news/newsx/newsx-1.6

	     Newsx - An NNTP client for posting and fetching news
	     ====================================================

			       Version 1.6
			Written by Egil Kvaleberg

				 ------

Newsx is an NNTP client for Unix. It will connect to a remote NNTP server
and post outgoing articles batched by the news system, as well as fetch
incoming articles.

It provides the NNTP capabilities required for small local news spools on
installations with NNTP access only through limited ISP accounts. It works
well via a dialup SLIP/PPP connection.

Newsx is also well suited for large spools with normal feeds, being used for
pulling newsgroups from specific NNTP servers that are not distributed in
the usual manner. Since newsx obeys the normal news spool configuration file
and requires little or no specific configuration, the administrative burden
should be minimized.

- Compatible with C News and INN local news servers.

- Configuration relies on standard C News and INN mechanisms for setting
  up newsservers and newsgroups, making administration easy.

- Comprehensive error recovery for posting as well as for fetching. Failed
  postings will be retried at the next opportunity, while fetching will
  resume at the point where it stopped.

- Logging of errors that occurred, as well as article transfer statistics.
  Optional log file for actual articles posted, and collection of posted
  articles in folders.

- Has been designed to be nice, so as not to overload the remote news
  server.

- Uses only standard RFC-977 functions, so should be compatible with the
  vast majority of news servers.

- Refers to news history database to prevent fetching of articles already
  in the local spool. Will not fetch crossposted articles more than once.
  Does not rely on Xref, since many sites does not provide it.

- Interfaces to spam filters like cleanfeed.

Newsx was originally written in C under Linux, but due to the magic of
Gnu automake and autoconf it should now be very portable. Unassisted
system configuration is done by means of said Gnu autoconf.


PULLING NEWS
------------

News transfer via fetching (sucking, slurping, pulling) to local news
spools is sometimes claimed to be an inefficient way of transferring news
compared to connecting newsreaders directly to the remote NNTP server.

The fact of the matter is that if set up and used correctly, exactly the
opposite is the case.

A local news spool allows news transfer to occur at off-peak hours,
thereby decreasing the host server load in the critical period. This will
tie up less modems at peak hours, as well as decreasing the connect time
since the actual transfer will run much quicker.

A fetch-based system is also easier to administrate then the feeding kind,
since every site decides for itself what groups it will exchange with whom.
The major down-side is that article propagation times will be longer. For
sites that are leaf nodes in the news distribution tree, this is usually not
very important.

Much of the problems connected with news pulling is connected to the use
of the RFC-977 NEWNEWS command. Most news servers has implemented this
command in an inefficient manner, which will put severe loads on the
newsserver. Newsx does not use NEWNEWS.


ARCHIVE
-------

In this archive you should find the following files:

README            To you what `Drink Me' was to Alice.
INSTALL           Documentation re. configuration and installation
COPYING           The GNU General Public License
FAQ               Frequently asked questions
NEWS              Versions
configure         For making a `Makefile'
Makefile.in       Input to `configure'
README.in         README original - make changes here
autogen.sh        For rebuild everything maintained by autoconf and automake
configure.ac      For autoconf
config.guess      For autoconf
config.h.in       For autoconf, generated by autoheader
install-sh        For autoconf
mkinstalldirs     For autoconf
missing           For autoconf
acconfig.h        For autoconf
aclocal.m4        For autoconf
newsx.spec        For building a rpm: rpm -bb newsx.spec
doc               Directory for man-pages and misc. documentation
dbz               Directory for database source code
src               Directory for newsx source code
lib               Directory for misc. source code
test              Directory for test code


CONFIGURE, COMPILE, AND INSTALL
-------------------------------

1. Set up your local news server first.  Newsx auto-configuration depends
   on the news server files and directories it finds and where.

2. Glance at ./INSTALL to see if you need any special configuration
   options.  Run ./configure with the options you prefer.

3. Type `make', and hopefully you will get the newsx executable.

4. Type `su' to become root, and then `make install' to install the
   executables and the man pages.


QUICK START -- GENERAL
----------------------

Newsx moves articles between servers, so you need to have a working local
news server with at least one newsgroup in order to test newsx.  There are
two basic tests:

1.  Can an article locally posted to the newsgroup and queued by the news
    server for delivery to the remote server be successfully sent to the
    remote server, and

2.  Can articles from the remote server be fetched and placed in the
    local server's "in" queue.

The "QUICK START WITH INN/C-NEWS" instructions below walk you through these
tests.

By default, an in.hosts/$server file is built automatically and will fetch
all articles in all groups listed in active (except as limited by the sys
file or command line options).  It is sometimes worthwhile to set up this
file by hand rather than accept the default.

Special case #1:
   * you've been reading news DIRECTLY from one or more REMOTE servers,
   * you're now setting up a local server, AND
   * you'd like to avoid downloading articles you've already seen.

   You can copy your $HOME/.newsrc file and use it as the initial
   in.hosts/$server file!  The (remote server's) article numbers will be
   extracted, and if you use "--forget-inactive" the first time you call
   `newsx', groups that were in .newsrc that you've unsubscribed to will be
   eliminated from in.hosts/$server, and current and future downloading
   will be limited to only the groups you're subscribed to at the time.
   (This does not "track" newsgroup subscription changes you make later.)

Special case #2:
   * you've been running a local server for some time and have lots of
     newsgroups in 'active' that you don't want fetched.

   Use the 'active' file or a .newsrc file, remove the article numbers
   (assuming they're the local server's article numbers), trim the list
   down to the groups you want, and use that as the in.hosts/$server
   file.

WARNING:
   Newsgroups can have LOTS of articles.  Please look at the newsx man page
   descriptions of --maxnew, --syncnew, and --maxart before running newsx
   to pull articles from the remote server the first time.


QUICK START WITH INN
--------------------

We assume that you already have INN installed. What you have to do is:

SETUP:
 1. Do all maintenance of the news system as user `news'. Ensure
    that the `.profile' of this user defines a path that includes
    `/usr/lib/news/bin'. E.g.:

	export PATH="$PATH:/usr/lib/news/bin"

 2. Ensure that an outgoing batch has been configured in
    `/etc/news/newsfeeds'. If the NNTP host is named `news.acme.net',
    a minimum configuration might be:

	ME:::
	news.acme.net/newsx:*,!junk*,!control*:Tf,Wf:

    In many cases, it will be a good to add the last line shown above
    to an existing configuration file.

    The outgoing batch will be named `news.acme.net' - replace
    `news.acme.net' with what is appropriate for your configuration.

    The name of the batch does not HAVE the same as the name of the server,
    but it is often convenient.

    All groups that you do NOT want to exchange with the host in question
    should be listed with an ! before it, the `junk' group being a
    standard fitment.

    NOTE: Depending on how you use the spool, you may want to replace
    the special `newsx' in the example above with the particular exclude
    string for the newsserver in question.

 3. If you haven't done so already, create one or more newsgroups in the
    `/var/lib/news/active' file that you will be fetching from the remote
    newsserver:

	ctlinnd newgroup acme.test

FETCH TEST:
 4. Dry-run test fetching news articles for these groups by doing:

	newsx -ddddd -n --maxnew 100 news.acme.net

    If the output looks correct, try running it for real (using your
    preference of --maxnew, --maxart, and --syncnew options):

	newsx -dd --maxnew 100 news.acme.net

 5. The articles should now appear in the incoming batch
    `/var/spool/news/incoming'. If you invoke:

	rnews -U

    the incoming articles should appear in the news spool. In the
    example, you may look for them in `/var/spool/news/articles/acme/test'.

POST TEST:
 6. Using a newsreader, post an article to a suitable test group. Try to
    use a group with as small distribution as possible, preferably local
    to the external newsserver you are using. If you haven't done so
    already, the group must be in the `/var/lib/news/active' file.

    The next time that `rnews' is invoked, the articles will be output to
    the spool file `/var/spool/news/outgoing/acme' specified in the
    `/etc/news/newsfeeds' file.

 7. Run:

	newsx -ddddd -n acme news.acme.net

    to do a `dry test' to ensure that everything is set up correctly. No
    external connection is required.

 8. Now, fire up the connection and do it for real:

	newsx -dd acme news.acme.net

    Check that everything runs smoothly. The `-dd' may be removed when you
    are sure everything works as expected. To maintain a log of posted
    articles, as well as a folder containing all articles, you can do:

	newsx -l posted.log -f posted acme news.acme.net

 9. You may want to arrange for automatic news exchange by using `crontab'.
    It is usually good to invoke `newsrun' before and after the newsx
    session.

10. For more information about how to set up news and other subjects related
    to use of ordinary ISP accounts, you might want to consult:

	http://www.kvaleberg.com/ISP-Hookup-HOWTO.html
	ftp://ftp.sn.no/user/egilk/ISP-Hookup-HOWTO.txt


QUICK START WITH C News
-----------------------

We assume that you already have C News installed. What you have to do is:

SETUP:
 1. Do all maintenance of the news system as user `news'. Ensure
    that the `.profile' of this user defines a path that includes
    `/usr/lib/news/bin'. E.g.:

	export PATH="$PATH:/usr/lib/news/bin"

 2. Ensure that an outgoing batch has been configured in `/usr/lib/news/sys'.
    If the NNTP host is named `news.acme.net', a minimum configuration
    might be:

	ME:all/all::
	acme/newsx:all,!junk,!control/all:FL:

    The filename of the outgoing batch will be the server's name, `acme' in
    the example above.  Replace `acme' with what is appropriate for your
    configuration.

    All groups that you do NOT want to exchange with the host in question
    should be listed with an `!' before it, the `junk' group being a
    standard fitment.

    The L-flag is a double insurance that only actual local postings will
    be sent out.

    NOTE: Depending on how you use the spool, you may want to replace the
    special `newsx' in the example above with the particular exclude string
    for the newsserver in question.  This is particularly true if you are
    sending and receiving news with multiple servers or if the local system
    names you're using aren't the name the servers use for themselvers in
    "Path:".

 3. If you haven't done so already, create one or more newsgroups in the
    `/var/lib/news/active' file that you will be fetching from the remote
    newsserver:

	addgroup acme.test y

FETCH TEST:
 4. Dry-run test fetching news articles for these groups by doing:

	newsx -ddddd -n --maxnew 100 acme news.acme.net

    If the output looks correct, try running it for real (using your
    preference of --maxnew, --maxart, and --syncnew options):

	newsx -dd --maxnew 100 acme news.acme.net

 5. The articles should now appear in the incoming batch
    `/var/spool/news/incoming'.  If you invoke:

	newsrun

    the incoming articles should appear in the news spool.  For newsgroup
    acme.test, articles would be placed in `/var/spool/news/articles/acme/test/'.

POST TEST:
 6. Using a newsreader, post an article to a suitable test group. Try to
    use a group with as small distribution as possible, preferably local
    to the external newsserver you are using. If you haven't done so
    already, the group must be in the `/var/lib/news/active' file.

    The next time that `newsrun' is invoked, the articles will be output
    to the spool file `/var/spool/news/outgoing/acme/togo' specified in the
    `/usr/lib/news/sys' file.

 7. Continue with point 7 in the INN explanation above.


MAILING LISTS
-------------

If you want to be informed about future newsx versions and receive
possible bug alerts, you can send an email message to:

	majordomo@kvaleberg.no

with the following line in the message body:

	subscribe newsx-announce

If you want to join the newsx discussion list, you can send an
email message to the same address with the following line in the
message body:

	subscribe newsx

The Usenet newsgroup news.software.nntp is also a suitable forum for
discussing newsx.


LEGAL MATTER
------------

Copyright 2003 Egil Kvaleberg <egil@kvaleberg.no>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.


ACKNOWLEDGMENTS
---------------

The DBZ database manager is copyright 1988 Jon Zeeff, and has been
modified by a number of people.

The NNTP header file is copyright 1994 David Alden and The Ohio State
University.

The getopt library is copyright 1987, 88, 89, 90, 91, 92, 93, 94, 95
Free Software Foundation, Inc.

Autoconf is copyright 1992, 1993, 1994, 1996, 1997 Free Software
Foundation, Inc.

Automake is copyright 1994, 1995, 1996 1997 Free Software Foundation,
Inc.

The wildmat function is written by Rich $alz in 1986.

Ansi2knr is copyright 1989, 1991, 1993, 1994, 1995, 1996 Aladdin Enterprises.

The hashing function used for the INN 2.0 history database is derived
from the RSA Data Security, Inc. MD5 Message-Digest Algorithm, and is
copyright 1990 RSA Data Security, Inc.

INN 2.0 is Copyright (c) 1996 by Internet Software Consortium.

The setenv function is Copyright (c) 1987 Regents of the University of
California.

Please refer to the respective source files for a full description of the
terms that apply.

Thanks to the following individuals for constructive comments and
improvements:

	Greg Wooledge <wooledge@kellnet.com>
	Riku Saikkonen <rjs@isil.lloke.dna.fi>
	Simon J Mudd <sjmudd@redestb.es>
	Laurent Frigault <frigault@isicom.fr>
	Gerhard Zuber <zuber@berlin.snafu.de>
	Bruce Fisher <bruce@smtl.co.uk>
	Tim Tuck <tim_tuck@yes.optus.com.au>
	Stanislav Protassov <st@sw.org.sg>
	Brian Buhrow <buhrow@cats.ucsc.edu>
	Michael Faurot <mfaurot@phzzzt.atww.org>
	Rachel Polanskis <r.polanskis@nepean.uws.edu.au>
	Stoty <stoty@legba.tvnet.hu>
	Arne Georg Gleditsch <arnegl@ifi.uio.no>
	Randall Shutt <rshutt@extacy.ravenet.com>
	Rene Hoejbjerg Larsen <renehl@post1.tele.dk>
	J. Richard Sladkey <jrs@foliage.com>
	Filip Lingier <filip@filtronix.eunet.be>
	Joacim Persson <sp2joap1@ida.his.se>
	Andreas Jaeger <aj@arthur.pfalz.de>
	Janne Snabb <snabb@niksula.hut.fi>
	Frank Tarczynski <ftarz@mindspring.com>
	Jason Brown <jbrown@interalpha.co.uk>
	Knut Anders Hatlen <kahatlen@riksnett.no>
	Adrian Bridgett <adrian.bridgett@poboxes.com>
	Paul Tomblin <ptomblin@xcski.com>
	Helmut Heller <heller@altoetting-online.de>
	Michele Bini <mbini@dada.it>
	Per Hedeland <per@erix.ericsson.se>
	Uli Zappe <uli@zappe.de>
	Nikolay Grigoriev <shadow@aanet.ru>
	Peter Maydell <pmaydell@chiark.greenend.org.uk>
	Steinar Haug <sthaug@nethelp.no>
	Stefan Huelswitt <huels@iname.com>
	Winston Edmond <wbe@psr.com>
	Petter Gustad <petter@gustad.com>
	G. Paul Ziemba <paul@treehouse.napa.ca.us>
	Bernhard R. Erdmann <be@berdmann.de>
	Briggs, John H <John.H.Briggs@team.telstra.com>
	Carlo Fusco <messer_samvise@libero.it>
	Gorka Olaizola <gorka@escomposlinux.org>
	Niels Heinen <zillion@snosoft.com>
	Jochen Schmitt <Jochen@herr-schmitt-de>
	Andreas Metzler <ametzler@logic.univie.ac.at>


CONTACT INFORMATION
-------------------

All comments, changes you had to make for specific systems, bug
reports, bug fixes, etc. can be reported to:

	http://www.kvaleberg.com/bug/

There is also a mailing list available, as described before.

Although not a requirement of the license terms, you are according
to ancient news software tradition encouraged to send a postcard to
the following address if you find the program useful:

	Egil Kvaleberg
	Husebybakken 14A
	NO-0379 Oslo
	Norway


$Id: README.in,v 1.59.2.15 2003/01/22 13:15:25 egil Exp $
Name		Date	Size	#Lines	LOC
..		03-May-2022	-
dbz/	H	20-May-2003	-	5,823	3,735
doc/	H	20-May-2003	-	2,134	1,720
lib/	H	20-May-2003	-	1,997	1,084
more/	H	20-May-2003	-	1,163	794
src/	H	03-May-2022	-	12,419	7,176
test/	H	20-May-2003	-	943	620
AUTHORS	H A D	20-May-2003	320	7	5
COPYING	H A D	31-Oct-2002	17.6 KiB	341	281
ChangeLog	H A D	18-May-2003	10	3	1
FAQ	H A D	22-Jan-2003	18.2 KiB	534	346
INSTALL	H A D	22-Jan-2003	6.5 KiB	175	122
Makefile	H A D	20-May-2003	68	9	7
Makefile.am	H A D	22-Jan-2003	1.3 KiB	57	12
Makefile.in	H A D	18-May-2003	15.4 KiB	515	393
NEWS	H A D	18-May-2003	21.3 KiB	695	433
README	H A D	18-May-2003	17.3 KiB	494	353
README.in	H A D	22-Jan-2003	17.1 KiB	494	353
TODO	H A D	21-Jan-2003	2.2 KiB	108	57
aclocal.m4	H A D	18-May-2003	35.8 KiB	1,009	855
autogen.sh	H A D	22-Jan-2003	2.3 KiB	88	66
config.guess	H A D	31-Oct-2002	16.9 KiB	601	510
config.h.in	H A D	18-May-2003	6.9 KiB	299	199
configure	H A D	03-May-2022	242.6 KiB	8,608	7,162
configure.ac	H A D	18-May-2003	25.8 KiB	893	831
depcomp	H A D	12-Dec-2002	11.8 KiB	424	278
install-sh	H A D	31-Oct-2002	4.7 KiB	239	152
missing	H A D	21-Jan-2003	10 KiB	337	263
mkinstalldirs	H A D	31-Oct-2002	730	37	21
newsx.spec.in	H A D	21-Jan-2003	3.3 KiB	152	120