crm114/crm114-20100106-BlameMichelson.src/README

57 This is a large set of changes. Old .css files will not be compatible
74 alternating_example_neural.crm" you also get an amusing look at the
153 A PPC GCC compiler bug was found; PPC users need to use at least a
172 sized .ben files.  The .BEN structure has been cleaned up (uses less
173 disk space at no accuracy cost!), and both Hyperspace and bit-entropy
189 per class) but in long runs like TREC06-P at 90+Kmsgs it is about
198 Read the long comments at the top of crm_bit_entropy.c to see how the
206 with .css files or with any other classifier.
215 contemplating further at this point is putting "autotrain anything in
251 any but an intentional "tickler" program, and at worst simply forces
264 Mailreaver.crm takes all of the same flags and command lines (or at
275 you save your old mailfilter.cf file as mailfilter.cf.old, and use the
308 "css version" and reserved-bucket code, as well as improved the
347 scoring less than this amount (plus or minus) is tagged as UNSURE
355 fast accurate .css files.  After testing, we've changed the defaults
372 mailfilter.cf file, so you'll get .css files that are optimized for
386 It is safe to run mailtrainer.crm repeatedly on a .css fileset and
423                        .css files.  Use this only if your host machine
431                        new .css files.  The result will give you an
432                        idea of how well your .css files will work.
441 filename; instead, save those up and use them as a "test corpus" for
456 * If the .css statistics files don't exist, they will be created for you.
459 assumed that these are empty .css statistics files, and that ONE file
471 .css files after each run, otherwise you will not get valid results.
473 * N-fold validation does NOT run training at all on the validation set,
484 This is a bugfix/bugchase release; it should remedy or at least
532 statement, rather than a compile-time default value.  If a variable
573 accurate.  It still only uses 400K per .css file though,
589            head \/dev\/zero --bytes=4 > spam.css
590            head \/dev\/zero --bytes=4 > nonspam.css
664 			to edit the Makefile- look at the options for
688 	"make install_utils" -- will build the css utilities "cssutil",
692 			the state of a .css file, cssdiff lets you
693 			check the differences between two .css files,
694 			and cssmerge lets you merge two css files.
697 			"nonspamtext.txt", builds BRAND NEW spam.css
698                         and nonspam.css files.
712 *** Utilities for looking into .css files.
714 This release also contains the cssutil utility to take a look at
715 and manage .css spectral files used in the mailfilter.
761    used when the distribution .css files were built.  If you build
762    your own .css files from scratch, you can ignore this.
783    c o m m a n d, right at the start of the line.  You mail a message
808   spam.css and nonspam.css) are meant for example, mostly.
828 	- spam.css and nonspam.css:  These are large files and as of
829 		2003-09-20, are included only in the .css kits.  CRM
830 	        .css files are "Sparse Spectra" files and they
833 		currently configured at five words, so a little spam
843 		css file.  Matching a hash means a word or phrase under
855 the derivative .css files: it's because I don't own the copyright on
858 them.  But never fear, if you don't trust my .css files to be clean,
860 Your .css files will be slightly different than mine, but they will
891 think about saving your old .css files and switching over to the new
924 been improved.  Cached .css mapping activated.  Win32 mmap adapters
932 Fidelis Assis), CALL/RETURN now work less unintuitively, SYSCALL can
947 not interconvertable between .css files and OSBF .cfc files, and that
952 training all nonspams with pR's less than 10, and all spams with pR's
961 you aren't at least 10 pR units "sure", and _is_ upward compatible
962 with prior OSB and Markovian .css files, and with the same speed
1011 not interconvertable between .css files and OSBF .cfc files, and that
1016 training all nonspams with pR's less than 10, and all spams with pR's
1025 you aren't at least 10 pR units "sure", and _is_ upward compatible
1026 with prior OSB and Markovian .css files, and with the same speed
1070 (Dump and Restore) are available in cssutil for moving .css files between
1080 bug has been fixed (at least I can no longer coerce it to appear)-
1117 .css file.  Cssutil, cssdiff, and cssmerge think they can handle the
1158 been improved.  It now gives less counterintuitive pR results and
1160 still isn't _right_, but at least it's not totally _wrong_.  :) Using
1188    to read 16 bytes starting at byte 32 of an MP3 file (to grab one of
1247    found in.  Since #insert _cannot_ do this (it's compile time, not
1251    To use it, put all of the files (the .css's, the .mfp's etc) that are
1266 	/var/spool/mail/crm.conf/joe-spam.css
1267 	/var/spool/mail/crm.conf/joe-nonspam.css
1270    options; rather, the actual .css filenames are the concatenation of
1275    Version 1.00, at last!  The only fixes here are to make the Makefile
1285    better than a human at the same task.
1331 to it).  Mailfilter accuracy is up past 99.9% (less than 1 error per
1388   Yes, we're now at RC3.  Changes are that EVAL now works right, lots
1403 optimized, and it's ready.  Accuracy is steady at around 3 nines.
1415 sets; <microgroom> will keep your .css files nicely trimmed; ISOLATE
1421 to download 8 megs of .css if you are building your .css locally, and
1435 (and has anyone else done the cssutil --> cssmerge to build small .css files
1447 Good news: the new .css file format seems to be working very well;
1448 although we spend a little more time in .css evaluation, the accuracy
1453 Merging works well; you can now make your .css files as big (or small)
1458 a less-space-for-more-time tradeoff.
1468 As usual, prior news and updates are at the end of this file.
1484 totally incompatible but far better .css format has been implemented.
1486 want huge .css files, and for people who want _smaller_ .css files.
1505   1) The format of the .css files has changed drastically.  What used to
1510   both using the same bucket.  Bucket values were maxxed at 255 (they
1517   do other very useful things to .css files.  Right now, the only
1518   supported operation is to _merge_ one .css file onto another... but
1520   So, the VERY good news is that you can look at your .css files with
1521   cssutil, decide if (or where) you want to zero out less significant
1522   data, and then use dd to create a blank, new outfile.css file that will
1523   be about half to 2/3 full, then use cssmerge outfile.css infile.css to
1524   merge your infile.css into the outfile.css.
1527   small .css files.  :)
1529   You can create the blank .css file with the command 'dd' as in:
1531  dd bs=12 count=<number of feature buckets desired> if=/dev/zero of=mynew.css
1545   nest at least vaguely correctly; a nonfatal trap that is bounced does
1562   distribution .css files with.
1578   the standalone utility rewriteutil.crm, just look at the example/test
1585 This is a further major bugstomping release.  The .css files are
1591 encouraged to rebuild your .css files with a hundred Kbytees or so of
1593 spamtext.txt and nonspamtext.txt).  The included spam.css and
1594 nonspam.css files are already rebuilt for the corrected bug in LEARN
1595 and CLASSIFY.  These .css files are also completely fresh and new;
1606 preprocessor no longer inserts random carriage returns.  A '\' at the
1655 	yet more additions to the .css files
1699 to the main install directory, and do a "make experimental" to compile
1723 means match for the string "foobar" with at most one substitution,
1736 5) Further improvements to the classifier and the shipped .css files.
1743 The pR number is the log base 10 of the .css matchfile signal strength
1843        will learn that spam into your current spam.css database.  Old
1848        will learn that nonspam into your current nonspam.css database.  Old
1868 NEW Built-in Debugger - the "-d" flag at the end of the command line
1873 trained to your errors, you should see less than one spam per
1876 Bug fixes - the code base now should compile more cleanly on newer
1898 	   use my pre-supplied .css files should delete the supplied
1899 	   .css files and retrain from their own spamtext.txt and
1903 	   compatible with previous .css files.
1906 	   .css files (needed only if you have HUGE training sets.)
1907 	   Rule of thumb: the .css files should be at least 4x the
1910 	*) Multiple .css files will now combine correctly - that is,
1926 use my pre-supplied .css files should delete the supplied .css files
1929 with previous .css files.  -s will let you change the default size for
1930 creating new .css files (needed only if you have HUGE training sets.)
1931 Rule of thumb: the .css files should be at least 4x the size of the
1932 training set.  Multiple .css files will now combine correctly - that
1938   classify (flames.css rants.css spam.css | beer.css parties.css kernel.css)
1941 linux-kernel email.  (I don't supply .css files for anything but spam
1952     .css files, you must use learntest.crm to LEARN the new data into
1953     the new .css files as the old file used a less effective polynomial.)