libexttextcat is an N-Gram-Based Text Categorization library primarily intended
for language guessing.

Fundamentally this is an adaption of wiseguys libtextcat extended to be UTF-8
aware. See README.libtextcat for details on original libtextcat.

Building:

 * ./configure
 * make
 * make check

the tests can be run under valgrind's memcheck with export VALGRIND=memcheck,
e.g.

 * export VALGRIND=memcheck
 * make check

Quickstart: language guesser

 Assuming that you have successfully compiled the library, you need some
language models to start guessing languages. A collection of over 150 language
models, mostly derived from using the included "createfp" utility on UDHR
translations, is bundled, with a matching configuration file, in the langclass
directory:

  * cd langclass/LM
  * ../../src/testtextcat ../fpdb.conf

Paste some text onto the commandline, and watch it get classified.

Using the API:

Classifying the language of a textbuffer can be as easy as:

 #include "textcat.h"
 ...
 void *h = textcat_Init( "fpdb.conf" );
 ...
 printf( "Language: %s\n", textcat_Classify(h, buffer, 400);
 ...
 textcat_Done(h);

Creating your own fingerprints:

The createfp program allows you to easily create your own document
fingerprints. Just feed it an example document on standard input, and store the
standard output:

Put the names of your fingerprints in a configuration file, add some id's and
you're ready to classify.

Here's a worked example. The UN Declaration of Human Rights is available in a
massive pile of translations[4], and and unicode.org makes much of these
available as plain text[5], so...

% cd langclass/ShortTexts/
% wget http://unicode.org/udhr/d/udhr_abk.txt
% tail -n+7 udhr_abk.txt > ab.txt #skip english header, name is using BCP-47
% cd ../LM
% ../../src/createfp < ../ShortTexts/ab.txt > ab.lm
% echo "ab.lm       ab--utf8" >> ../fpdb.conf

Eventually we'll drop fpdb.conf and assume the name of the fingerprint .lm file
is the correct BCP-47 tag for the language it detects.

Performance tuning:

This library was made with efficiency in mind. There are couple of
parameters you may wish to tweak if you intend to use it for other
tasks than language guessing.

The most important thing is buffer size. For reliable language
guessing the classifier only needs a couple of hundreds of bytes max.
So don't feed it 100KB of text unless you are creating a fingerprint.

If you insist on feeding the classifier lots of text, try fiddling
with TABLEPOW, which determines the size of the hash table that is
used to store the n-grams. Making it too small will result in many
hashtable clashes, making it too large will cause wild memory
behaviour and both are bad for the performance.

Putting the most probable models at the top of the list in your config
file improves performance, because this will raise the threshold for
likely candidates more quickly.

Since the speed of the classifier is roughly linear with respect to
the number of models, you should consider how many models you really
need. In case of language guessing: do you really want to recognize
every language ever invented?

Acknowledgements

UTF-8 conversion and adaption for OpenOffice.org, Jocelyn Merand.
Original libTextCat, Frank Scheelen & Rob de Wit at wise-guys.nl.
Original language models, copyright Gertjan van Noord.

References:

[1] The document that started it all can be downloaded at John M.
Trenkle's site: N-Gram-Based Text Categorization

http://www.novodynamics.com/trenkle/papers/sdair-94-bc.ps.gz

[2] The Perl implementation by Gertjan van Noord (code + language
models): downloadable from his website

http://odur.let.rug.nl/~vannoord/TextCat/

[3] Original libtextcat implementation at

http://software.wise-guys.nl/libtextcat/

[4] http://www.ohchr.org/EN/UDHR/Pages/SearchByLang.aspx

[5] https://unicode.org/udhr/translations.html

Contact:

Questions or patches can be directed to libreoffice@lists.freedesktop.org.
Bugs can be directed to https://bugs.freedesktop.org

  :: libTextCat 2.2 ::

  What is it?

     Libtextcat is a library with functions that implement the
     classification technique described in Cavnar & Trenkle, "N-Gram-Based
     Text Categorization" [1]. It was primarily developed for language
     guessing, a task on which it is known to perform with near-perfect
     accuracy.

     The central idea of the Cavnar & Trenkle technique is to calculate a
     "fingerprint" of a document with an unknown category, and compare this
     with the fingerprints of a number of documents of which the categories
     are known. The categories of the closest matches are output as the
     classification. A fingerprint is a list of the most frequent n-grams
     occurring in a document, ordered by frequency. Fingerprints are
     compared with a simple out-of-place metric. See the article for more
     details.

     Considerable effort went into making this implementation fast and
     efficient. The language guesser processes over 100 documents/second on
     a simple PC, which makes it practical for many uses. It was developed
     for use in our webcrawler and search engine software, in which it it
     handles millions of documents a day.

  Download

     The library is released under the BSD License, which basicly states
     that you can do anything you like with it as long as you mention us
     and make it clear that this library is covered by the BSD License. It
     also exempts us from any liability, should this library eat your hard
     disc, kill your cat or classify your attorney's e-mails as spam.

     The current version is 2.1.

     It can be downloaded from our website:

       http://software.wise-guys.nl/libtextcat/

     As of yet there is no development version.

  Installation

     Do the familiar dance:

       tar xzf libtextcat-2.2.tar.gz
       cd libtextcat-2.2
       ./configure
       make
       make install

     This will install the library in /usr/local/lib/ and the createfp
     binary in /usr/local/bin.

     The library is known to compile flawlessly on GNU/Linux for x86, and
     IRIX64 (both 32 and 64 bits).

  Quickstart: language guesser

     Assuming that you have successfully compiled the library, you still
     need some language models to start guessing languages. If you don't
     feel like creating them yourself (cf. [2]Creating your own
     fingerprints below), you can use the excellent collection of over 70
     language models provided in Gertjan van Noord's "TextCat" package.
     You can find these models and a matching configuration file
     in the langclass directory:

       * cd libtextcat-2.2/langclass/
       * ../src/testtextcat conf.txt

     Paste some text onto the commandline, and watch it get classified.

  Using the API

     Classifying the language of a textbuffer can be as easy as:

       #include "textcat.h"
       ...
       void *h = textcat_Init( "conf.txt" );
       ...
       printf( "Language: %s\n", textcat_Classify(h, buffer, 400);
       ...
       textcat_Done(h);

  Creating your own fingerprints

     The createfp program allows you to easily create your own document
     fingerprints. Just feed it an example document on standard input, and
     store the standard output:

       % createfp < mydocument.txt > myfingerprint.txt

     Put the names of your fingerprints in a configuration file, add some
     id's and you're ready to classify.

  Performance tuning

     This library was made with efficiency in mind. There are couple of
     parameters you may wish to tweak if you intend to use it for other
     tasks than language guessing.

     The most important thing is buffer size. For reliable language
     guessing the classifier only needs a couple of hundreds of bytes max.
     So don't feed it 100KB of text unless you are creating a fingerprint.

     If you insist on feeding the classifier lots of text, try fiddling
     with TABLEPOW, which determines the size of the hash table that is
     used to store the n-grams. Making it too small will result in many
     hashtable clashes, making it too large will cause wild memory
     behaviour and both are bad for the performance.

     Putting the most probable models at the top of the list in your config
     file improves performance, because this will raise the threshold for
     likely candidates more quickly.

     Since the speed of the classifier is roughly linear with respect to
     the number of models, you should consider how many models you really
     need. In case of language guessing: do you really want to recognize
     every language ever invented?

  Acknowledgements

     The language models are copyright Gertjan van Noord.

  References

     [1] The document that started it all can be downloaded at John M.
     Trenkle's site: N-Gram-Based Text Categorization

       http://www.novodynamics.com/trenkle/papers/sdair-94-bc.ps.gz

     [2] The Perl implementation by Gertjan van Noord (code + language
     models): downloadable from his [7]website

       http://odur.let.rug.nl/~vannoord/TextCat/

  Contact

     Praise and flames may be directed at us through
     libtextcat@wise-guys.nl. If there is enough interest, we'll whip up
     a mailing list. The current project maintainer is Frank Scheelen.

  c. 2003 WiseGuys Internet B.V.