Files in this directory
~~~~~~~~~~~~~~~~~~~~~~~

README.contrib - the file you are now reading.

QDBM-transactions.patch - a patch to teach QDBM to use transactions,
    to show how QDBM transaction enhancements might look like.

bfproxy.pl - performs bogofilter functions via email

bogo.R - Script to check calculations performed by bogofilter.

bogofilter-milter.pl - Sendmail::Milter Perl script for filtering mail
   using individual users' bogofilter databases.

bogogrep.c - This file emulates GNU grep -ab with a plain text pattern
   anchored to the left.

bogominitrain.pl - Script for training to exhaustion from mbox, see FAQ.

dot-qmail-bogofilter-default - Sample .qmail or .qmail-default file to
    go along with ...
bogofilter-qfe.sh - Qmail specific bogofilter frontend script which allows
   the use of a centralized bogofilter running on an smtp mail server.

mailfilter.example - Sample maildrop configuration file.

mime.get.rfc822.pl - Perl script to extract message/rfc822 attachments into
   an mbox.

parmtest.sh - Script to test a bogofilter option.

printmaildir.pl - Converts a Maildir to mbox format.

procmailrc.example - Sample .procmailrc file.

randomtrain.sh - Script to train on error with random order from mbox,
    see FAQ. Comes with (and uses) ...
scramble.sh - Script needed by randomtrain.

spamitarium.pl - helps to remove unnecessary noise from email headers
    and to highlight just the portions which contribute positively to
    spam filtering using statistical methods.

stripsearch.pl - Stripsearch investigates the body of your emails for
    evidence of spamvertized URLs by looking them up in a Realtime
    BlockList (RBL) such as surbl.org or spamhaus.org.

trainbogo.sh - Script to train from qmail maildirs.

vm-bogofilter.el - An interface between the VM mail reader and the
    bogofilter spam filter.

It seems that training bogofilter on its errors _only_ is a very good
way to train, at least with the Robinson-Fisher or Bayes chain rule
calculation methods.  The way this works is: messages from the training
corpus are picked at random (without replacement, ie no message is used
more than once) and fed to bogofilter for evaluation.  If bogofilter
gets the classification right, nothing further is done.  If it's wrong,
or uncertain if ternary mode is in use, then the message is fed to
bogofilter again with the -s or -n option, as appropriate.

That's all very well, except that it's not an easy process to execute
with just a couple of shell commands.  I've now written a bash script
that does the job [Matthias Andree: this has been changed so it may now
work on a regular POSIX compliant sh, too, feedback is welcome]; you
give it the directory in which to build the bogofilter database, and a
list of files flagged with either -s or -n to indicate spam or nonspam,
and it performs training-on-error using all the messages in all the
files in random order.

My production version of bogofilter returns the following exit codes:
0 for spam
1 for nonspam
2 for uncertain
3 for error

Normal bogofilter returns (I think)
0 for spam
1 for nonspam
2 for error

This script will work with either.

You can use it to build from scratch; the first message evaluated will
return the error exit code, and randomtrain (as this script is called)
will train with that message, thus creating the databases.

The script needs rather a lot of auxiliary commands (they're listed in
the comments at the top of the file); in particular, perl is called for
the randomization function.  (The embedded perl script is "useful" in
its own right: it takes text on standard input and returns the lines in
random sequence.)  Known portability issue: on HP-UX (10.20 at least),
grep -b returns a block offset instead of a byte offset, so randomtrain
won't work unless gnu grep is substituted for the HP-UX one.

I rebuilt my training lists with randomtrain.  The training corpus
consists of 9878 spams and 7896 nonspams.  The message-counts from
bogoutil -w bogodir are 1475 and 408.  The database sizes from full
training were 10 and 4 Mb; randomtrain produced .db files of 7 and 1.2
Mb.  I don't yet have figures comparing discrimination by bogofilter
with these two training sets, but yesterday's smaller-scale test (which
motivated me to write this script) clearly indicated an improvement
could be expected.

Greg Louis <glouis@dynamicro.on.ca>