README.md
1# nltools
2[![Build Status](https://travis-ci.org/huskyproject/nltools.svg?branch=master)](https://travis-ci.org/huskyproject/nltools)
3[![Build status](https://ci.appveyor.com/api/projects/status/w31hn6m0xwqbh8c1/branch/master?svg=true)](https://ci.appveyor.com/project/dukelsky/nltools/branch/master)
4[![Codacy Badge](https://api.codacy.com/project/badge/Grade/c1970dc5395945839afbae7c79b66a3a)](https://www.codacy.com/app/huskyproject/nltools?utm_source=github.com&utm_medium=referral&utm_content=huskyproject/nltools&utm_campaign=Badge_Grade)
5
6Husky Nodelist Utilities Documentation
7======================================
8
9Written 2000 by Tobias Ernst and released to the Public Domain.
10Updated by Husky Team.
11
12## I) Abstract
13
14This archive contains a set of nodelist management utilities, in source code
15and, if you got a full distribution, also with binaries for some common
16operating systems. The following functionality is provided:
17
18- Compiling raw nodelists into a FIDOUSER.LST file, that can be used for
19 nodelist lookup by sysop name, as supported by Msged and others.
20
21- Keeping your raw nodelists up to date, i.E. logic is provided to find the
22 nodediff files that apply to your current nodelist files, and a tool to
23 apply nodediffs to nodelists.
24
25This package does NOT presently include something like a V7 nodelist
26compiler. Therefore, this package is enough if you are running an IP node,
27but you need additional software if you are running Binkley or some other
28software that requires an V7 nodelist index.
29
30The tools can be compiled on Unix, as well as a lot of other platforms (OS/2,
31Win32, DOS/DJGPP), provided you have a working fidoconfig library and
32installation for that platform.
33
34A major advantage of these tools on Unix is that they have no case
35sensitivity problems. I.e. nodelist files and nodelist updates are found no
36matter if they are in lower, capital, or MiXeD case.
37
38## II) About Fidoconfig
39
40These tools are written as an addition to the Husky suite of Fidonet
41software. The tools do not read a separate configuration file, but rather
42they read a common, global "Fidoconfig" configuration file. This does not
43mean that you need to install the whole Husky suite - you can just use the
44programs of this package stand alone if you like, because the precompiled
45binaries have the fidoconfig linked in statically.
46
47But if you want to compile the tools, you do need the
48[fidoconfig](https://github.com/huskyproject/fidoconf) library.
49If you do not know what Fidoconfig is, visit
50[Husky project](https://github.com/huskyproject) or its
51[homepage](https://huskyproject.github.io/).
52This release of the nltools is designed to be built against fidoconfig.
53You will also require the [huskylib](https://github.com/huskyproject/huskylib)
54and [smapi](https://github.com/huskyproject/smapi) libraries because of some
55header files only found therein.
56
57If you only want to use the precompiled binaries, you must set the
58environment variable FIDOCONFIG to point to a fidoconfig-style configuration
59file, e.g. on OS/2, put SET FIDOCONFIG=e:\bbs\etc\fidoconf.cfg in your
60CONFIG.SYS file. On DOS, put the same command into autoexec.bat
61
62The keywords that you can use in this configuration file will be explained
63below, in section V, or of course in the fidoconfig manual.
64
65## III) General Considerations
66
67These tools expect the Nodelist and Nodediff files to be in the FTS-5000 (and
68obsoleted FTS-0005) defined format, that means: text lines that are finished
69with a CRLF sequence, and an EOF character after the last CRLF. This is
70normally true on any DOSish computer, and also on a Unix system if you just
71extract the distribution archives of Nodelist and Nodediff without giving some
72sort of auto conversion option. You might loose the CRs (^Ms) at the end when
73working with a text editor on these files, though. If the files do not have
74the ^M at the line end, the CRC checks will fail.
75
76## IV) About the tools
77
78### a) nlcrc
79
80 You normally do not need to call nlcrc manually, unless you are curious.
81
82 This simple tool checks the CRC checksum in a given Nodelist (not
83 Nodediff!) file. Simply invoke it with the filename as argument:
84
85 nlcrc NODELIST.260
86
87 If there is not any output, the check succeeded and the return code will
88 be zero. If the file does not contain a CRC checksum, a message will be
89 printed to stderr and the return code will be 4. If a file I/O error or
90 similar happens, a message will be printed to stderr and the return code
91 will be 8. If the file has a checksum, but the check fails, a message
92 will be printed to stderr and the return code will be 16.
93
94
95### b) nldiff
96
97 You normally do not need to call nldiff manually. nldiff is designed to be
98 called automatically by nlupdate.
99
100 This tool applies a Nodediff file to a given Nodelist. It has no
101 intelligence as to determining which of multiple Nodediff files is the
102 correct one (you have to use other tools for this), but it expects the
103 Nodelist filename and the Nodediff filename as explicit arguments with the
104 correct day file name extension, as for example in:
105
106 nldiff NODELIST.260 NODEDIFF.267
107
108 This will crate a new file NODELIST.267.
109
110 If you want the old file (NODELIST.260 in this case) to be deleted if the
111 process succeeds (this means that no I/O errors occured and that the new
112 nodelist file passes a CRC check), you can give the -n parameter:
113
114 nldiff -n NODELIST.260 NODEDIFF.267
115
116 If you also want the nodediff file (NODEDIFF.260) to be deleted, you can
117 specify the -d parameter:
118
119 nldiff -d -n NODELIST.260 NODEDIFF.267
120
121### c) ulc
122
123 ulc is the Husky Fido Userlist Compiler. ulc reads all nodelists that are
124 configured in Fidoconfig (via the "nodelist" keyword) and creates the
125 FIDOUSER.LST file (the name has to be configured with the "fidouserlist"
126 keyword). ulc does not take any command line options; it uses fidoconfig
127 to determine where to find the nodelist files. A log file named
128 "nltools.log" is placed in the fidoconfig log file directory.
129
130 The FIDOUSER.LST file format is defined as follows: The file consists of
131 text record of fixed length (65 characters including the terminal \r\n
132 sequence). The name of the sysop is left-aligned in the line in reverse
133 order (e.g. "Tobias_Ernst" would become "Ernst, Tobias"). Aligned to the
134 right of the record is the node number of the user. The records are sorted
135 alphabetically, so that a program can use a binary search algorithm to
136 find the corresponding node number for a given user name very fast.
137
138 The FIDOUSER.LST file format is supported by many mail readers, e.g. Timed
139 and Msged. For Msged, FIDOUSER.LST is currently the best method to
140 implement a node lookup at all, because Msged's V7 routines are flawed.
141
142### d) nlupdate
143
144 Nlupdate manages your nodelists and keeps them up to date. For each
145 nodelist that you have configured with a "Nodelist" statement (see
146 below), it will search the latest nodelist in the "NodelistDir", then
147 caluclate the day number of the difference file that is needed to
148 update this nodelist (the algorithm is Y2K safe and knows that 2000 is a
149 leap year), then searches the difference file, unpacks it if necessary,
150 and applies it to the nodelist using nldiff. nlupdate can also find full
151 replacement files if you configure this, and just unpack them and copy
152 them over the old nodelist. This is useful for othernets that do not have
153 nodediff files. - With only a few keywords, nlupdate manages the whole
154 process of updating your nodelist files for you.
155
156After you have set the configuration file up properly (see below), you just
157need to put the two commands
158
159 nlupdate
160 ulc
161 ;call your V7 index generator here if necessary
162
163into your weekly maintainance script and all the nodelist tasks are done for
164you.
165
166## V) CONFIGURATION SYNTAX
167
168The following text describes the configuration statements in the fidoconfig
169file that control the behaviour of ulc and nlupdate. Just put the
170appropriate configuration statements into a text file and point the
171FIDOCONFIG environment variable to this file, or add the statements to your
172existing fidoconfig file if you have one.
173```
174 LogFileDir
175 ----------
176 Syntax: logFileDir <path>
177 Example: logFileDir /var/spool/fido/log
178
179 Where the logfile goes. This is a *path* name, the filename inside this
180 path is hardcoded to be "nltools.log".
181
182 MsgBaseDir
183 ---------
184 Syntax: MsgBaseDir <patch>
185 Example: MsgBaseDir /var/spool/fido/msgbase
186
187 The nodelist tools do not use this, but you need to have it in the config
188 file. Just point it to any existing path. (Other Husky tools would place
189 files that store the message base in this directory).
190
191 NodelistDir
192 -----------
193 Syntax: nodelistDir <path>
194 Example: nodelistDir /var/spool/fido/nodelist
195
196 This command specifies the path where the actual nodelists are or should
197 be written to. This path contains the raw nodelist. Also, compiled
198 nodelists like the FIDOUSER.LST will be stored here.
199 This statement cannot be repeated.
200
201 Unpack
202 ------
203 Syntax: Unpack "<unpacker command>" <id pos> <id bytes>
204```
205 This configures the unpackers to use. It is crucial that you get these
206 lines right. For details, please consider the Fidoconfig manual. Below, I
207 simply give examples that work. I assume that you use the Freeware unrar
208 (available for OS/2, DOS, Windows, and in Source for Unix), the Freeware
209 Infozip unzip program (dito), and ARC 5.21. This last packer is required
210 for Fidonet node diffs especially; you can get it by f'requesting
211 ARC521_2.ZIP at 2:2476/418. This file contains a family mode executable
212 (DOS + OS/2, also runs on Windows 95 and NT), as well as source code that
213 can be compiled on Unix without major problems.
214
215 On Unix/Linux, use the following (adapt the path if necessary):
216```
217 Unpack "/usr/local/bin/unzip -joLqq $a -d $p" 0 504b0304
218 Unpack "/usr/local/bin/unrar e -o+ -y -c- -p- $a $p/ >/dev/null" 0 52617221
219 Unpack "/usr/local/bin/arc eno $a $p'/*.*'" 0 1a
220```
221 On DOS, OS/2 and Windows, use the following:
222```
223 Unpack "unzip -joLqq $a -d $p" 0 504b0304
224 Unpack "unrar e -o+ -y -c- -p- $a $p\ >/dev/null" 0 52617221
225 Unpack "arc eno $a $p\*.*" 0 1a
226```
227```
228 FidoUserList
229 -------------
230 Syntax: fidoUserList <filename>
231 Example: fidoUserList fidouser.lst
232```
233 If this keyword is present, the nodelist compiler (ulc) is instructed to
234 build a user list file with the given filename in the nodelist directory
235 (see nodelistdir). This is a simple text file with fixed line length that
236 contains user names (nodes, points) and their corresponding node or
237 pointnumbers. The file is sorted alphabetically by user name (case
238 insensitive), so that it can be bsearched to implement a quick node numer
239 lookup functinality. The fido user list file format is understood by
240 Msged, for example.
241```
242 NodeList
243 --------
244 Syntax: Nodelist <name>
245 Example: Nodelist points24
246```
247 This statement starts a new nodelist definition. All the following
248 nodelist-related stamtements change the configuration of this nodelist
249 until a new nodeelist statement is found.
250
251 The name that you specify must match the base name (without extension and
252 without pathname) of the raw, unpacked, nodelist file. The husky tools
253 ulc and nlupdate match the file name case-insensitively, but other tools
254 may need the exact spelling. The raw nodelist file is expected to reside
255 in the nodelist directory (see nodelistdir)-
256```
257 DiffUpdate
258 ----------
259 Syntax: DiffUpdate <path_and_basename>
260 Example: DiffUpdate /var/spool/filebase/nodediff/nodediff
261```
262 Here you can specify the base filename of nodelist difference files
263 (nodediffs) that are used to keep the corresponding nodelist up to date.
264 The argument to the DiffUpdate is the full file name with path of a
265 difference file, without the file extension. For example, if you have a
266 file area at /var/spool/filebase/24000, where your ticker places the
267 updates for the German Pointlist, and those update files are called
268 points24.a26, points24.a33, and so on, you would use
269```
270 DiffUpdate /var/spool/filebase/24000/points24
271```
272 The Diffupdate keyword is used by nlupdate, for example. The nodelist
273 updater will unpack the difference file (if it is archived, of course,
274 unpacked diffs are also supported), apply the diff to the corresponding
275 nodelist, and delete the temporary unpacked diff again.
276```
277 FullUpdate
278 ----------
279 Syntax: FullUpdate <path_and_basename>
280 Example: FullUpdate /var/spool/filebase/nodelist/nodelist
281```
282 This statement works like DiffUpdate. The difference is that here you
283 don't specify the location of a nodelist difference file, but the
284 locations where complete nodelist files/archives can be found. Some
285 othernets do not (regularly) distribute a nodediff file, but just hatch a
286 new nodelist every few weeks. In this case, you need the FullUpdate
287 statement.
288```
289 Defaultzone
290 -----------
291 Syntax: DefaultZone <zone>
292 Example: DefaultZone 2
293```
294 Some nodelist files do not start with a Zone entry. This is the case for
295 the German Points24 list, for example, but could also happen for othernets
296 that only have one zone. In this case, you can use the DefaultZone
297 keyword to specify the default zone number for all nodes listed in this
298 nodelist.
299```
300 Nodelistformat
301 --------------
302 Syntax: Nodelistformat <format>
303 Example: NodelistFormat Standard
304 Example: NodelistFormat Points24
305 Example: NodelistFormat Points4D
306```
307 Here you can specify the format of the unpacked nodelist. The default is
308 "standard": this is the normal Fidonet nodelist format. You can also
309 specify "points24", which is needed for the nodelist compiler to recognise
310 a point list in the German points24 format as such, so that it can see the
311 proper 4D point numbers instead of the fakenet numbers. Or you can specify
312 "points4d", which means a 4D point list with "Boss" entries.
313
314
315## VI) SAMPLE CONFIG
316
317The following lines show a sample configuration file. If you are only
318interested in the nodelist tools, you can just copy those commands into a
319text file, modify them according to your needs (OS/2, Win and DOS users may
320of course use backslashes instead of forward slashes, and drive letters), and
321point the FIDOCONFIG variable to this file. If, on the other hand, you
322already have a fidoconfig installtion, you can copy these commands into your
323existing fidoconfig file.
324```
325 NodelistDir /var/spool/fido/nodelist
326 FidoUserList fidouser.lst
327
328 Nodelist nodelist
329 DiffUpdate /var/spool/fido/filebase/nodediff/nodediff
330 NodelistFormat Standard
331
332 Nodelist points24
333 DiffUpdate /var/spool/fido/filebase/24000/pr24diff
334 DefaultZone 2
335 NodelistFormat Points24
336```
337## VII) COMPILING
338
339You can compile nltools as part of the Husky project using the top level
340Makefile and a proper huskymak.cfg. This works particularly well and easy on
341Linux and other Unixes. Instructions on how to do this can be found in the
342"huskybse" package, in the file INSTALL.
343
344The following instructions are for users that cannot use the huskymak.cfg
345way, but must use legacy makefiles.
346
347- Download and compile SMAPI and FIDOCONFIG.
348- Extract nltools on the same level as fidoconfig, e.g. you could have:
349 ~/fido/smapi
350 ~/fido/fidoconfig
351 ~/fido/nltools
352- Change to the src subdirectory.
353- Select the proper makefile and rename it to "makefile". Edit it. Set the
354 INSTDIR and LIBDIR variables according to your needs.
355- Type "make"
356- If you get an unresolved symbol error for "fexist", upgrade your
357 fidoconfig source code to the latest git level.
358- If it worked, type "make install" (Unix only)
359- If you like, type "make clean".
360
361
362## VIII) LICENCSE
363
364These tools are donated to the Public Domain, which means that you can do with
365with the SOURCE CODE whatever you want, but also that the author does not
366take any responsibilites whatsoever.
367
368In order to produce executables of the tools, you need the fidoconfig library
369(and I needed it as well). The fidoconfig library is not Public Domain, but
370it is covered by the GNU LIBRARY GENERAL PUBLIC LICENSE. A copy of this
371license can be found in the "legal" subdirectory. The binary executables of
372the nodelist tools therefore are also covered by the LGPL.
373
374In order to comply with the LGPL, I hereby invite you to download the source
375code of both the nodelist tools and fidoconfig from
376https://github.com/huskyproject.
377
378## IX) CONTACT
379
380Questions are appropriate in the LINUX.FIDO.GER, FIDOSOFT.HUSKY, and
381FIDO_UTIL Fidonet echo conferences.
382
383For more information see the [Husky project](https://github.com/huskyproject)
384and its [homepage](https://huskyproject.github.io/).
385