mailest - search mail messages with Hyper Estrairer
===================================================

"mailest" provides a search engine backend for
[Hyper Estraier](http://fallabs.com/hyperestraier/index.html)
to
[Mew](http://www.mew.org/)
and maintains its id index database.

- Makes `V`, `k/` or other related commands ready just after
  receiving/refiling mails into folders.

- No need to create index manually.
  - "mailest" daemon (mailestd) does it automatically by monitoring
    changes in the mail folders using kqueue or inotify.


Install
-------

On OpenBSD or FreeBSD, pre-compiled package is available.  The following
procedure is the way to compile it from source code.

Prerequisite:

- "hyperestraier" and "qdbm" installed
- "libevent" and "libiconv" are required


OpenBSD:

    % ftp https://github.com/yasuoka/mailest/archive/mailest-0.9.24.tar.gz
    % tar xzf mailest-0.9.24.tar.gz
    % cd mailest-mailest-0.9.24.
    % make
    % sudo make install

Others:

    % wget https://github.com/yasuoka/mailest/archive/mailest-0.9.24.tar.gz
    % tar xzf mailest-0.9.24.tar.gz
    % cd mailest-mailest-0.9.24.
    % ./configure
    % make
    % sudo make install

 On BSD, You may need to use "gmake" instead of "make" to use GNUmakefile


Upgrading from previous releases
--------------------------------

Terminate the running mailest daemons before upgrading because:

  - The mailest communication protocol between its server and its
    clients may be changed between the versions and the compatibility
    between the versions is not guaranteed.
  - Use `mailestctl stop` or send a termination signal (SIGTERM) to stop
    the mailest programs.


How to start
------------

+ For existing `mewest` users: Backup and/or delete the database
  (`~/Mail/casket`).

  - `V` (dialog) is not compatible with database generated by `mewest`
    while the other functionality of "mailestd" works with it.  Remove
    the database if you want to use `V`.

+ Add the following line to `.mew.el` to let Mew use "Hyper Estrairer" for
  search:

      (setq mew-search-method 'est)

+ Add the following lines to `~/.emacs` to use the "mailest" instead of
  "mewest".

      (defvar mew-prog-est        "mew-mailest")
      (defvar mew-prog-est-update "mew-mailest")
      (defvar mew-prog-cmew       "cmew-mailest")
      (defvar mew-prog-smew       "smew-mailest")
      (defvar mew-id-db-file      ".mailest.sock")

+ Try `km`, `kM`, `k/`, `k?`, `V` in Mew.  See
  http://mew.org/en/info/release/mew_6.html#dbsearch for search commands
  in Mew.


Usage
-----

- "mailest" daemon will be invoked by the following operations on Mew
  - Search or virutal folder operations (`V`, `k/` and so on)
  - Making index operations (`km` or `kM`)
- or you can use "@reboot" in your crontab like the following line if
  you want start it automatically when boot.

      @reboot mailestctl start

- After the daemon starts, it monitors the changes on the folders
  which have already been indexed.
- Upon `kM`, "mailest" starts monitoring for newly created folders
  as well.  Otherwise `km` is required for monitoring new folders.
- Others
  - to stop the daemon: `mailestctl stop`
  - to watch behaviour of the daemon: `tail -f ~/Mail/mailestd.log`
  - to suspend/resume indexing: `mailestctl suspend` or `mailestctl resume`
  - to stop monitoring the folders: add `monitor disable`
    to `~/Mail/mailestd.conf`.

See [man pages](#man-pages) also.


ChangeLog
---------

0.9.24

  - Fix a wrong ASSERT() introduced at 0.9.23 (e5076c8a)
  - Apply many fixes for parse.y on OpenBSD


0.9.23

  - Fix the creating draft tasks to be run not on the busiest database
    thread but on the main thread.  This had been desired by design, but
    it hadn't been implemented correctly.
  - Fix a bug which may refer the tree for all messages during its
    modification.  If this happens the daemon crashes with ASSERT if it
    is enabled.
  - Improve the man page.  Description for `trim-size` is added.
  - Mew had freezed by `csearch` command from `mailestctl` because the
    command doesn't get any response when the database is unusable.
  - Make guess-parid work even for a message which doesn't have a "Date"
    field.


0.9.22

  - Fix smew not to loop infinately


0.9.21

  - Fix mailestd not to mistakenly delete a message from the database,
    as a result, it stopps working since it presumes the database is
    broken.  (Reported by by Yoshiaki Kasahara)
  - Fix: When mailestd starts up, "DB Sync" splits the messages in some
    chunks.  The last message in chunk had been mistakenly treated it is
    a target of "guess-parid".
  - Fix: index for @title was not created
  - Add size check in log function.  Pointed out by Yuuich Someya for
    npppd.
  - Fix some LOG_DEBUG messages to use a canonical folder name.
  - Fix the way of referencing to libestdir.  Suggested by Hiroki Sato.
  - Fix BSD make.
  - Fix BSD make build on NetBSD.
  - Fix a bug in mailestd.conf(5).


0.9.20

  - Use the realpath always for folders not to treat the path with
    symbolic links and the realpath as different folders.  (Found
    by Yoshiaki Kasahara)
  - Improve log and comment a bit.


0.9.19

  - Fix some variables in BSD make not to have DESTDIR doublely.
  - Remove a debug log output in #ifdef MONITOR_INOTIFY.
  - Fix typos in log message.


0.9.18

  - Fix off-by-one in mailestctl.  It crashed by using "csearch"
    (mew-mailest).  (Found and debugged by Yoshiaki Kasahara)
  - Fix ./configure to select kqueue or inotify properly.
  - s/cascket/casket/ (Pointed out by Yoshiaki Kasahara)


0.9.17

  - Fix "update" related logs to show the entire path name.
    (diff from Hiroki Sato)


0.9.16

  - Fix "smew -p" to work.


0.9.15

  - Monitoring were not enabled on some systems (atleast on FreeBSD).
    Fixed autoconf to choice kqueue() or inotify() properly.  (Pointed
    out by Yoshiaki Kasahara)


0.9.14

  - Support "smew -p" and "smew -c" so that Mew '^' and '&' work.
    (pointed out by Yoshiaki Kasahara)
  - Fix and refine build on FreeBSD (diff from Hiroki Saito)
  - Fix build on the systems which need -liconv for iconv_open().
    (diff from Yoshiaki Kasahara)
  - Internal size for message-id may be too short.  Use 255 for it.
  - When "guess-parid" failed to find the parent message, add dummy
    x-mew-parid not to try to find the parent message ever.
  - Fixed: "guess-parid" is too slow since mailestd failed to create the
    index of "@title" on startup.
  - Fixed to create the index for "@title" properly.


0.9.13

  - Add "guess-parid" configration option to find the parent message
    for the messages which don't contain "In-Reply-To" or "Reference"
    header by guessing with "Subject" and "Date".  Add the following
    to `~/Mail/mailestd.conf`.

        guess-parid

0.9.12

  - Improve logging
  - Tweak log levels
  - Fix parsing "monitor" in mailestd.conf

0.9.11

  - Fix not to change current working directory.
  - Fix: invalid memory access in mailestd_db_smew().

0.9.10

  - Fix build on operating system which use kqueue for monitor.

0.9.9

  - Monitor the inodes the directories (by kevent and inotify) and start
    update automatically.
  - Fixed error in replace.h which is to redefine RB_FOREACH_SAFE.
  - Optimize database when many documents are put or deleted.  As the
    Hyper Estrairer's documet is recomended.
  - Unlimit the resource limit of the data size when process startup.

0.9.8

  - Add 'smew' functionality.  Add `smew` command to mailestctl(1).
    To use 'smew' of 'mailestd', add below lines:

        (defvar mew-prog-cmew       "cmew-mailest")
        (defvar mew-prog-smew       "smew-mailest")
        (defvar mew-id-db-file      ".mailest.sock")

    and delete the existing database and create the database again.
  - Made the 'message-id' attribute indexed.
  -  Create `x-mew-parid` indexed attribute for each message.
  - Add `message-id` command to mailestdctl(1).

0.9.7

  - Fix: When the database doesn't exists, the database thread stopps
    forever.
  - Fix: When the database is error, the first update causes a lot
    of errors.

0.9.6

  -  Fix: cannot search non ASCII/UTF-8 mails
  -  Fix: mailestd exits abnormally sometimes when it is stopped during
     syncing DB.


0.9.5

  -  First "update" to huge database taked long time.  Fix it not to wait
     database cache updating before update starts.
  -  Fix the results of "update" to be passed correctly.
  -  Support build on FreeBSD and NetBSD

Man pages
---------

- [mailestctl(1)](http://yasuoka.github.io/mailest/mailestctl.1.html)
- [mailestd(8)](http://yasuoka.github.io/mailest/mailestd.8.html)
- [mailestd.conf(5)](http://yasuoka.github.io/mailest/mailestd.conf.5.html)
- [mew-mailest(8)](http://yasuoka.github.io/mailest/mew-mailest.1.html)


Differences from `mewest`
-------------------------

Creation of backup copy of the database

  "mailest" does not create a backup copy of the database before updating it
  whereas `mewest` does.  Hyper Estrairer database breaks easily
  if a program (or a system) crashes during updating it.


TODO
----

- When indexing huge amount of mails, smew takes very long time.  Find
  a way to workaround this.
- Delete the index physically
- Automatically create a backup copy of the database when closing the
  writable DB connection.  Also recover the database automatically
  when it is broken.


Copyright
---------

Licenses other than "libestdraft" are BSD or OpenBSD license.  Only
"libestdraft" is derived from Hyper Estrairer's estcmd.c, it's LGPL.
See the source code to check the entire copyright.