README.md
1FTS Xapian plugin for Dovecot
2=============================
3
4What is this?
5-------------
6
7This project intends to provide a straightforward, simple and maintenance free, way to configure FTS plugin for [Dovecot](https://github.com/dovecot/), leveraging the efforts by the [Xapian.org](https://xapian.org/) team.
8
9This effort came after Dovecot team decided to deprecate "fts_squat" included in the dovecot core, and due to the complexity of the Solr plugin capabilitles, un-needed for most users.
10
11
12Availability in major distributions
13-----------------------------------
14
15THis plugin is readly available in major distributions under the name "dovecot-fts-xapian"
16- Archlinux : https://archlinux.org/packages/?q=dovecot-fts-xapian
17- Debian : https://packages.debian.org/bullseye/dovecot-fts-xapian
18- Fedora : https://src.fedoraproject.org/rpms/dovecot-fts-xapian
19
20
21Configuration - dovecot.conf file
22---------------------------------
23
24Update your dovecot.conf file with something similar to:
25
26(Example in [conf.d/90-fts.conf](https://github.com/grosjo/fts-xapian/blob/master/contrib/conf.d/90-fts.conf) )
27
28```
29mail_plugins = (...) fts fts_xapian
30
31(...)
32
33plugin {
34 fts = xapian
35 fts_xapian = partial=3 full=20
36
37 fts_autoindex = yes
38 fts_enforced = yes
39
40 fts_autoindex_exclude = \Trash
41
42 # Index attachements
43 fts_decoder = decode2text
44}
45
46service indexer-worker {
47 # Increase vsz_limit to 2GB or above.
48 # Or 0 if you have rather large memory usable on your server, which is preferred for performance)
49 vsz_limit = 2G
50}
51
52service decode2text {
53 executable = script /usr/libexec/dovecot/decode2text.sh
54 user = dovecot
55 unix_listener decode2text {
56 mode = 0666
57 }
58}
59```
60
61
62Configuration - Indexing options
63--------------------------------
64
65| Option | Optional | Description | Possible values | Default value |
66|----------------|----------|---------------------------------|--------------------------------------|---------------|
67| partial & full | no | NGram values for header fields | between 3 and 20 characters | 3 & 20 |
68| verbose | yes | Logs verbosity | 0 (silent), 1 (verbose) or 2 (debug) | 0 |
69| lowmemory | yes | Memory limit before disk commit | 0 (default), or set value (in MB) | 0 |
70
71
72Configuration - NGrams details
73------------------------------
74
75The partial & full parameters are the NGram values for header fields, which means the keywords created for fields (To,
76Cc, ...) are between 3 and 20 chars long. Full words are also added by default (if not longer than 245 chars, which is
77the limit of Xapian capability).
78
79Example: "<john@doe>" will create joh, ohn, hn@, ..., john@d, ohn@do, ..., and finally john@doe as searchable keywords.
80
81Configuration - Index updating
82------------------------------
83
84Just restart Dovecot:
85
86```sh
87sudo service restart dovecot
88```
89
90If this is not a fresh install of dovecot, you need to re-index your mailboxes:
91
92```sh
93doveadm index -A -q \*
94```
95
96- With argument `-A`, it will re-index all mailboxes, therefore may take a while.
97- With argument `-q`, doveadm queues the indexing to be run by indexer process.
98 Remove `-q` if you want to index immediately.
99
100You shall put in a cron the following command (for daily run for instance) :
101
102```sh
103doveadm fts optimize -A
104```
105
106
107Building yourself - Prerequisites
108----------------------------------
109
110You are going to need the following things to get this going:
111
112```
113* Dovecot 2.2.x (or above)
114* Xapian 1.2.x (or above)
115* ICU 50.x (or above)
116```
117
118You will need to configure properly [Users Home Directories](https://wiki.dovecot.org/VirtualUsers/Home) in dovecot configuration
119
120
121Building yourself - Installing the Dovecot plugin
122-----------------------------
123
124First install the following packages, or equivalent for your operating system.
125
126```
127Ubuntu:
128apt-get build-dep dovecot-core
129apt-get install dovecot-dev git libxapian-dev libicu-dev
130
131Archlinux:
132pacman -S dovecot
133pacman -S xapian-core icu git
134
135FreeBSD:
136pkg install xapian-core
137pkg install xapian-bindings
138pkg install icu
139pkg install git
140```
141
142Clone this project:
143
144```
145git clone https://github.com/grosjo/fts-xapian
146cd fts-xapian
147```
148
149Compile and install the plugin.
150
151```
152autoupdate
153autoreconf -vi
154./configure --with-dovecot=/path/to/dovecot
155make
156sudo make install
157```
158
159Replace /path/to/dovecot by the actual path to 'dovecot-config'.
160Type 'locate dovecot-config' in a shell to figure this out. On ArchLinux , it is /usr/lib/dovecot.
161
162For specific configuration, you may have to 'export PKG_CONFIG_PATH=...'. To check that, type 'pkg-config --cflags-only-I icu-uc icu-io icu-i18n', it shall return no error.
163
164The module will be placed into the module directory of your dovecot configuration
165
166
167Debugging/Support
168-----------------
169
170Please submit requests/bugs via the [GitHub issue tracker](https://github.com/grosjo/fts-xapian/issues).
171
172A Matrix Room exists also at : #xapian-dovecot:grosjo.net
173
174Thanks to Aki Tuomi <aki.tuomi@open-xchange.com>, Stephan Bosch <stephan@rename-it.nl>, Paul Hecker <paul@iwascoding.com>
175