Automatically generated by Pod::Man 4.10 (Pod::Simple 3.35) Standard preamble: ========================================================================..
..
..
Set up some character translations and predefined strings. \*(-- will give an unbreakable dash, \*(PI will give pi, \*(L" will give a left double quote, and \*(R" will give a right double quote. \*(C+ will give a nicer C++. Capital omega is used to do unbreakable dashes and therefore won't be available. \*(C` and \*(C' expand to `' in nroff, nothing in troff, for use with C<>..tr \(*W-
. ds -- \(*W-
. ds PI pi
. if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
. if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch
. ds L" ""
. ds R" ""
. ds C` ""
. ds C' ""
'br\}
. ds -- \|\(em\|
. ds PI \(*p
. ds L" ``
. ds R" ''
. ds C`
. ds C'
'br\}
Escape single quotes in literal strings from groff's Unicode transform. If the F register is >0, we'll generate index entries on stderr for titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index entries marked with X<> in POD. Of course, you'll have to process the output yourself in some meaningful fashion. Avoid warning from groff about undefined register 'F'...
.nr rF 0
. if \nF \{\
. de IX
. tm Index:\\$1\t\\n%\t"\\$2"
..
. if !\nF==2 \{\
. nr % 0
. nr F 2
. \}
. \}
.\}
.rr rF
Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). Fear. Run. Save yourself. No user-serviceable parts.. \" fudge factors for nroff and troff
. ds #H 0
. ds #V .8m
. ds #F .3m
. ds #[ \f1
. ds #]
.\}
. ds #H ((1u-(\\\\n(.fu%2u))*.13m)
. ds #V .6m
. ds #F 0
. ds #[ \&
. ds #] \&
.\}
. \" simple accents for nroff and troff
. ds ' \&
. ds ` \&
. ds ^ \&
. ds , \&
. ds ~ ~
. ds /
.\}
. ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
. ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
. ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
. ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
. ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
. ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
. \" troff and (daisy-wheel) nroff accents
. \" corrections for vroff
. \" for low resolution devices (crt and lpr)
\{\
. ds : e
. ds 8 ss
. ds o a
. ds d- d\h'-1'\(ga
. ds D- D\h'-1'\(hy
. ds th \o'bp'
. ds Th \o'LP'
. ds ae ae
. ds Ae AE
.\}
======================================================================== Title "INNFEED 8" INNFEED 8 "2015-09-12" "INN 2.6.4" "InterNetNews Documentation"
For nroff, turn off justification. Always turn off hyphenation; it makes way too many mistakes in technical documents. "NAME"
innfeed, imapfeed - Multi-host, multi-connection, streaming NNTP feeder
"SYNOPSIS"
Header "SYNOPSIS" \fBinnfeed [
-ChmMvxyz] [
-a spool-dir] [
-b directory]
[
-c config-file] [
-d log-level] [
-e bytes]
[
-l logfile] [
-o bytes] [
-p pid-file] [
-s command]
[
-S status-file] [
file]
"DESCRIPTION"
Header "DESCRIPTION" \fBinnfeed implements the \s-1NNTP\s0 protocol for transferring news between
computers. It handles the standard \s-1IHAVE\s0 protocol as well as the
\
s-1CHECK/
TAKETHIS\s0 streaming extension.
innfeed can feed any number of
remote hosts at once and will open multiple connections to each host if
configured to do so. The only limitations are the process limits for open
file descriptors and memory.
As an alternative to using \s-1NNTP, INN\s0 may also be fed to an \s-1IMAP\s0 server.
This is done by using an executable called imapfeed, which is identical
to innfeed except for the delivery process. The new version has two
types of connections: an \s-1LMTP\s0 connection to deliver regular messages and
an \s-1IMAP\s0 connection to handle control messages.
"MODES"
Header "MODES" \fBinnfeed has three modes of operation: channel, funnel-file and batch.
Channel mode is used when no filename is given on the command line,
the input-file keyword is not given in the config file, and
the -x option is not given. In channel mode, innfeed runs with
stdin connected via a pipe to innd. Whenever innd closes this pipe
(and it has several reasons during normal processing to do so), innfeed
will exit. It first will try to finish sending all articles it was in
the middle of transmitting, before issuing a \s-1QUIT\s0 command. This means
\fBinnfeed may take a while to exit depending on how slow your peers are.
It never (well, almost never) just drops the connection. The recommended
way to restart innfeed when run in channel mode is therefore to tell
\fBinnd to close the pipe and spawn a new innfeed process. This can
be done with \*(C`ctlinnd flush \f(CIfeed\*(C' where feed is the name of
the innfeed channel feed in the newsfeeds file.
Funnel-file mode is used when a filename is given as an argument or the
\fIinput-file keyword is given in the config file. In funnel-file mode,
it reads the specified file for the same formatted information as innd
would give in channel mode. It is expected that innd is continually
writing to this file, so when innfeed reaches the end of the file, it
will check periodically for new information. To prevent the funnel file
from growing without bounds, you will need to periodically move the file
to the side (or simply remove it) and have innd flush the file. Then,
after the file is flushed by innd, you can send innfeed a \s-1SIGALRM,\s0
and it too will close the file and open the new file created by innd.
Something like:
.Vb 7
innfeed -p <pathrun in inn.conf>/innfeed.pid my-funnel-file &
while true; do
sleep 43200
rm -f my-funnel-file
ctlinnd flush funnel-file-site
kill -ALRM \`cat <pathrun>/innfeed.pid\`
done
.Ve
Batch mode is used when the -x flag is used. In batch mode, innfeed
will ignore stdin, and will simply process any backlog created by a
previously running innfeed. This mode is not normally needed as
\fBinnfeed will take care of backlog processing.
"CONFIGURATION"
Header "CONFIGURATION" \fBinnfeed expects a couple of things to be able to run correctly: a
directory where it can store backlog files and a configuration file to
describe which peers it should handle.
The configuration file is described in innfeed.conf\|(5). The -c option
can be used to specify a different file. For each peer (say, \*(C`foo\*(C'),
\fBinnfeed manages up to 4 files in the backlog directory:
"\(bu" 2
A
foo.lock file, which prevents other instances of
innfeed from
interfering with this one.
"\(bu" 2
A
foo.input file which has old article information
innfeed is reading
for re-processing.
"\(bu" 2
A
foo.output file where
innfeed is writing information on articles
that could not be processed (normally due to a slow or blocked peer).
"\(bu" 2
A
foo file that is never created by
innfeed, but if
innfeed
notices it, it will rename it to
foo.input at the next opportunity and
will start reading from it. This lets you create a batch file and put it
in a place where
innfeed will find it.
You should never alter the foo.input or foo.output files of a running
\fBinnfeed. The format of these last three files is one of the following:
.Vb 2
/path/to/article <message-id>
@token@ <message-id>
.Ve
This is the same as the first two fields of the lines innd feeds to
\fBinnfeed, and the same as the first two fields of the lines of the batch
file innd will write if innfeed is unavailable for some reason.
When innfeed processes its own batch files, it ignores everything after
the first two whitespace separated fields, so moving the innd-created
batch file to the appropriate spot will work, even though the lines have
extra fields.
The first field can also be a storage \s-1API\s0 token. The two types of lines
can be intermingled; innfeed will use the storage manager if appropriate,
and otherwise treat the first field as a filename to read directly.
\fBinnfeed writes its current status to the file innfeed.status (or
the file given by the -S option). This file contains details on the
process as a whole, and on each peer this instance of innfeed is managing.
If innfeed is told to send an article to a host it is not managing,
then the article information will be put into a file matching the pattern
\fIinnfeed-dropped.*, with part of the file name matching the pid of the
\fBinnfeed process that is writing to it. innfeed will not process this
file except to write to it. If nothing is written to the file, then it
will be removed if innfeed exits normally. Otherwise, the file remains,
and procbatch can be invoked to process it afterwards.
"SIGNALS"
Header "SIGNALS" Upon receipt of a \s-1SIGALRM,\s0
innfeed will close the funnel file specified
on the command line, and will reopen it (see funnel file description above).
\fBinnfeed with catch \s-1SIGINT\s0 and will write a large debugging snapshot of
the state of the running system.
\fBinnfeed will catch \s-1SIGHUP\s0 and will reload both the config and the
log files. See innfeed.conf\|(5) for more details.
\fBinnfeed will catch \s-1SIGCHLD\s0 and will close and reopen all backlog files.
\fBinnfeed will catch \s-1SIGTERM\s0 and will do an orderly shutdown.
Upon receipt of a \s-1SIGUSR1,\s0 innfeed will increment the debugging level
by one; receipt of a \s-1SIGUSR2\s0 will decrement it by one. The debugging
level starts at zero (unless the -d option it used), in which case no
debugging information is emitted. A larger value for the level means more
debugging information. Numbers up to 5 are currently useful.
"SYSLOG ENTRIES"
Header "SYSLOG ENTRIES" There are 3 different categories of syslog entries for statistics: host,
connection and global.
The host statistics are generated for a given peer at regular intervals
after the first connection is made (or, if the remote is unreachable, after
spooling starts). The host statistics give totals over all connections
that have been active during the given time frame. For example (broken
here to fit the page, with \*(C`vixie\*(C' being the peer):
.Vb 5
May 23 12:49:08 news innfeed[16015]: vixie checkpoint
seconds 1381 offered 2744 accepted 1286 refused 1021 rejected 437
missing 0 accsize 8506220 rejsize 142129 spooled 990
on_close 0 unspooled 240 deferred 10/15.3 requeued 25
queue 42.1/100:14,35,13,4,24,10
.Ve
The meanings of these fields are:
"seconds" 2
Item "seconds" The time since
innfeed connected to the host or since the statistics
were reset by a
\*(C`final\*(C' log entry.
"offered" 2
Item "offered" The number of \s-1IHAVE\s0 commands sent to the host if it is not in streaming
mode. The sum of the number of \s-1TAKETHIS\s0 commands sent when no-CHECK mode
is in effect plus the number of \s-1CHECK\s0 commands sent in streaming mode
(when no-CHECK mode is not in effect).
"accepted" 2
Item "accepted" The number of articles which were sent to the remote host and accepted by it.
"refused" 2
Item "refused" The number of articles offered to the host that it indicated it did not
want because it had already seen the message-ID. The remote host indicates
this by sending a 435 response to an \s-1IHAVE\s0 command or a 438 response to
a \s-1CHECK\s0 command.
"rejected" 2
Item "rejected" The number of articles transferred to the host that it did not accept because
it determined either that it already had the article or it did not want it
because of the article's Newsgroups: or Distribution: header fields, etc.
The remote host indicates that it is rejecting the article by sending a
437 or 439 response after
innfeed sent the entire article.
"missing" 2
Item "missing" The number of articles which
innfeed was told to offer to the host but
which were not present in the article spool. These articles were probably
cancelled or expired before
innfeed was able to offer them to the host.
"accsize" 2
Item "accsize" The number of bytes of all accepted articles transferred to the host.
"rejsize" 2
Item "rejsize" The number of bytes of all rejected articles transferred to the host.
"spooled" 2
Item "spooled" The number of article entries that were written to the
.output
backlog file because the articles either could not be sent to the host
or were refused by it. Articles are generally spooled either because new
articles are arriving more quickly than they can be offered to the host,
or because
innfeed closed all the connections to the host and pushed
all the articles currently in progress to the
.output backlog file.
"on_close" 2
Item "on_close" The number of articles that were spooled when
innfeed closed all the
connections to the host.
"unspooled" 2
Item "unspooled" The number of article entries that were read from the
.input backlog file.
"deferred" 2
Item "deferred" The first number is the number of articles that the host told
innfeed
to retry later by sending a 431 or 436 response.
innfeed immediately
puts these articles back on the tail of the queue.
.Sp
The second number is the average (mean) size of deferred articles during
the previous logging interval
"requeued" 2
Item "requeued" The number of articles that were in progress on connections when
innfeed
dropped those connections and put the articles back on the queue.
These connections may have been broken by a network problem or became
unresponsive causing
innfeed to time them out.
"queue" 2
Item "queue" The first number is the average (mean) queue size during the previous
logging interval. The second number is the maximum allowable queue size.
The third number is the percentage of the time that the queue was empty.
The fourth through seventh numbers are the percentages of the time that
the queue was >0% to 25% full, 25% to 50% full, 50% to 75% full, and 75%
to <100% full. The last number is the percentage of the time that the
queue was totally full.
If the -z option is used (see below), then when the peer stats are
generated, each connection will log its stats too. For example, for
connection number zero (from a set of five):
.Vb 3
May 23 12:49:08 news innfeed[16015]: vixie:0 checkpoint
seconds 1381 offered 596 accepted 274 refused 225
rejected 97 accsize 773623 rejsize 86591
.Ve
If you only open a maximum of one connection to a remote, then there will
be a close correlation between connection numbers and host numbers, but
in general you cannot tie the two sets of number together in any easy or
very meaningful way. When a connection closes, it will always log its stats.
If all connections for a host get closed together, then the host logs its
stats as \*(C`final\*(C' and resets its counters. If the feed is so busy that
there is always at least one connection open and running, then after some
amount of time (set via the config file), the host stats are logged as final
and reset. This is to make generating higher level stats from log files,
by other programs, easier.
There is one log entry that is emitted for a host just after its last
connection closes and innfeed is preparing to exit. This entry contains
counts over the entire life of the process. The \*(C`seconds\*(C' field is from the
first time a connection was successfully built, or the first time spooling
started. If a host has been completely idle, it will have no such log entry.
.Vb 3
May 23 12:49:08 news innfeed[16015]: decwrl global
seconds 1381 offered 34 accepted 22 refused 3 rejected 7
missing 0 accsize 81277 rejsize 12738 spooled 0 unspooled 0
.Ve
The final log entry is emitted immediately before exiting. It contains a
summary of the statistics over the entire life of the process.
.Vb 4
Feb 13 14:43:41 news innfeed[22344]: ME global
seconds 15742 offered 273441 accepted 45750 refused 222008
rejected 3334 missing 217 accsize 93647166 rejsize 7421839
spooled 10 unspooled 0
.Ve
"OPTIONS"
Header "OPTIONS" \fBinnfeed takes the following options.
"-a spool-dir" 4
Item "-a spool-dir" The
-a flag is used to specify the top of the article spool tree.
\fBinnfeed does a
chdir\|(2) to this directory, so it should probably be an
absolute path. The default is
patharticles as set in
inn.conf.
"-b directory" 4
Item "-b directory" The
-b flag may be used to specify a different directory for backlog
file storage and retrieval, as well as for lock files. If the path is
relative, then it is relative to
pathspool as set in
inn.conf.
The default is
\*(C`innfeed\*(C'.
"-c config-file" 4
Item "-c config-file" The
-c flag may be used to specify a different config file from the
default value. If the path is relative, then it is relative to
pathetc
as set in
inn.conf. The default is
innfeed.conf.
"-C" 4
Item "-C" The
-C flag is used to have
innfeed simply check the config file,
report on any errors and then exit.
"-d log-level" 4
Item "-d log-level" The
-d flag may be used to specify the initial logging level.
All debugging messages go to stderr (which may not be what you want,
see the
-l flag below).
"-e bytes" 4
Item "-e bytes" The
-e flag may be used to specify the size limit (in bytes) for the
\fI.output backlog files
innfeed creates. If the output file gets
bigger than 10% more than the given number,
innfeed will replace the
output file with the tail of the original version. The default value is
\f(CW0, which means there is no limit.
"-h" 4
Item "-h" Use the
-h flag to print the usage message.
"-l logfile" 4
Item "-l logfile" The
-l flag may be used to specify a different log file from stderr.
As
innd starts
innfeed with stderr attached to /
dev/
null, using
this option can be useful in catching any abnormal error messages, or any
debugging messages (all \*(L"normal\*(R" errors messages go to syslog).
"-m" 4
Item "-m" The
-m flag is used to turn on logging of all missing articles.
Normally, if an article is missing,
innfeed keeps a count, but logs no
further information. When this flag is used, details about message-IDs
and expected path names are logged.
"-M" 4
Item "-M" If
innfeed has been built with mmap support, then the
-M flag turns
\s-1OFF\s0 the use of
mmap(); otherwise, it has no effect.
"-o bytes" 4
Item "-o bytes" The
-o flag sets a value of the maximum number of bytes of article data
\fBinnfeed is supposed to keep in memory. This does not work properly yet.
"-p pid-file" 4
Item "-p pid-file" The
-p flag is used to specify the file name to write the pid of
the process into. A relative path is relative to
pathrun as set in
\
fIinn.conf. The default is
innfeed.pid.
"-s command" 4
Item "-s command" The
-s flag specifies the name of a command to run in a subprocess and
read article information from. This is similar to channel mode operation,
only that
command takes the place usually occupied by
innd.
"-S status-file" 4
Item "-S status-file" The
-S flag specifies the name of the file to write the periodic status
to. If the path is relative, it is considered relative to
pathlog
as set in
inn.conf. The default is
innfeed.status.
"-v" 4
Item "-v" When the
-v flag is given, version information is printed to stderr
and then
innfeed exits.
"-x" 4
Item "-x" The
-x flag is used to tell
innfeed not to expect any article
information from
innd but just to process any backlog files that exist
and then exit.
"-y" 4
Item "-y" The
-y flag is used to allow dynamic peer binding. If this flag is
used and article information is received from
innd that specifies an
unknown peer, then the peer name is taken to be the \s-1IP\s0 name too, and an
association with it is created. Using this, it is possible to only have
the global defaults in the
innfeed.conf file, provided the peer name
as used by
innd is the same as the \s-1IP\s0 name.
.Sp
Note that
innfeed with
-y and no peer in
innfeed.conf would cause
a problem that
innfeed drops the first article.
"-z" 4
Item "-z" The
-z flag is used to cause each connection, in a parallel feed
configuration, to report statistics when the controller for the connections
prints its statistics.
"BUGS"
Header "BUGS" When using the
-x option, the config file entry's
initial-connections
field will be the total number of connections created and used, no matter
how many big the batch file, and no matter how big the
max-connections
field specifies. Thus a value of
0 for
initial-connections means
nothing will happen in
-x mode.
\fBinnfeed does not automatically grab the file out of pathoutgoing.
This needs to be prepared for it by external means.
Probably too many other bugs to count.
"ALTERNATIVE"
Header "ALTERNATIVE" An alternative to
innfeed can be
\fBinnduct, maintained by Ian Jackson and available at
<http://
www.chiark.greenend.org.uk/
ucgi/~
ian/
git-
manpage/
innduct.git/
innduct.8>.
It is intended to solve a design issue in the way
innfeed works.
As a matter of fact, the program feed protocol spoken between
innd
and
innfeed is lossy: if
innfeed dies unexpectedly, articles
which
innd has written to the pipe to
innfeed will be skipped.
\fBinnd has no way of telling which articles those are, no useful
records, and no attempts to resend these articles.
"FILES"
Header "FILES" "pathbin/innfeed" 4
Item "pathbin/innfeed" The binary program itself.
Item "pathetc/innfeed.conf" The configuration file.
"pathspool/innfeed" 4
Item "pathspool/innfeed" The directory for backlog files.
"HISTORY"
Header "HISTORY" Written by James Brister <brister@vix.com> for InterNetNews. Converted to
\s-1POD\s0 by Julien Elie.
Earlier versions of innfeed (up to 0.10.1) were shipped separately;
\fBinnfeed is now part of \s-1INN\s0 and shares the same version number.
\f(CW$Id: innfeed.pod 9588 2013-12-19 17:46:41Z iulius $
"SEE ALSO"
Header "SEE ALSO" \fBctlinnd\|(8),
inn.conf\|(5),
innfeed.conf\|(5),
innd\|(8),
procbatch\|(8).