Pinot
Copyright 2005-2021 Fabrice Colin <fabrice dot colin at gmail dot com>

Homepage - https://github.com/FabriceColin/pinot
 previously hosted at http://code.google.com/p/pinot-search/
 and http://pinot.berlios.de/
Translations - https://translations.launchpad.net/pinot/trunk/+pots/pinot


1. What is Pinot
2. Building Pinot
2. Available engines
3. Indexes
4. Indexing and monitoring
5. Searching
6. Viewing cached results
7. File formats
8. File patterns
9. Digging deeper
10. Saving results
11. D-Bus service & daemon
12. CJKV support
13. Environment variables and aliases
14. How to reset indexes
15. Compiling


1. What is Pinot


  Pinot combines desktop search and metasearch. It consists of :
    * a D-Bus service daemon that crawls, indexes, monitors your documents
      and that plugs into the GNOME Shell search system ("pinot-dbus-daemon")
    * a GTK3-based user interface that enables to query the index built by
      the service as well as Web engines, and which can display and analyze
      the results ("pinot")
    * other command-line tools

  It was developed and tested on GNU/Linux and should work on other Unix-like
  systems.


2. Available engines


  One of the main functionalities of Pinot is metasearch. This lets you query
  a variety of sources, including Web-based search engines. By default, the
  list of available engines is hidden and defaults to internal indexes (see
  section "3. Indexes"). To show the list of engines, click on the Show All
  Search Engines button, next to the Query field immediately below the menu
  bar. Click on the same button again to hide the list.

  Any number of engine or engine group may be selected at any one time.
  Multi-selection is done like in any other application. All queries are always
  run against the list of currently selected engines.

  Pinot supports both Sherlock and OpenSearch Description plugins. They are
  installed in $PREFIX/share/pinot/engines/, where PREFIX is usually /usr.
  Additional engines can be installed in that directory or in ~/.pinot/engines.
  Note this directory is not created automatically.

  Sherlock is what Firefox and the Mozilla Suite use. Chances are that somebody
  wrote a plugin for the engine you are interested in. Beware that a lot are
  out of date and will require some changes. Use pinot-search on the
  command-line to run a quick check on a plugin, eg
  $ pinot-search sherlock $PREFIX/share/pinot/engines/Bozo.src "clowns"

  Plugins are categorized by channels. For Sherlock plugins, the routeType
  element under SEARCH specifies the name of the channel the plugin belongs to.

  As for OpenSearch, Pinot should work with OpenSearch Description 1.0 and 1.1
  (draft 2) plugins. Keep in mind that the spec doesn't describe how to parse
  the results pages returned by search engines, therefore Pinot assumes that
  engines return results formatted according to the OpenSearch Response
  standard.
  In practice, this means that plugins that don't stick to the following rules
  will be ignored or won't show any result :
    * For Description 1.1 plugins, the type attribute on the Url field must be
      set to "application/atom+xml" or "application/rss+xml" (default).
      "text/html" will be rejected.
    * The search engine's results page content type must be some form of XML,
      otherwise Pinot won't attempt parsing it.
  Pinot differs from the Description spec in that it interprets the Tags field
  as a channel name. The standard defines Tags as a "space-delimited set of
  words that are used as keywords to identify and categorize this search
  content".

  The "Xapian Omega" plugin allows to query a locally installed instance of
  Xapian Omega at http://localhost/. If Omega is installed elsewhere, edit
  $PREFIX/share/pinot/engines/OmegaDescription.xml.


3. Indexes


  Pinot has two internal indexes. My Documents is populated by the D-Bus
  service and contains documents found on your computer. My Web Pages is
  populated by the UI whenever you :
    * import an external document, using the Index, Import URL menu
    * index results returned by Web engines, using the Results, Index menu
      or through a Stored Query
  Both index may have any of the file types listed in section "7. File formats".

  Indexes built by any other Xapian-based tools can be added to Pinot. To add
  an external index, click the + button at the bottom of the engines list.
  It can either be local, in which case you will have to select the directory
  where it is found, or served from a remote machine by xapian-tcpsrv. See
  the manual page for xapian-tcpsrv(1).

  All indexes are grouped together under the channel Current User in the
  engines list.


4. Indexing and monitoring


  Pinot can index any directory configured under the Indexing tab of the
  Preferences box. Monitoring is optional and should be disabled for the
  directories whose contents seldom change, eg $PREFIX/share/doc.
  Indexing and monitoring of directories is handled by the D-Bus service.
  The number of files and directories that can be monitored is capped by
  the value of /proc/sys/fs/inotify/max_user_watches - 1024.

  Symlinks are not followed but are still indexed, with the MIME type
  "inode/symlink".

  While Pinot is not currently able to get to and index application-specific
  data held in dot-directories, it can index common file formats as listed
  in section "7. File formats".

  All files and directories with a name that starts with a dot, eg
  ".thunderbird", are skipped and their content is not indexed. If you wish
  to include the contents of some dot-directory, create a symlink to a
  directory that is configured in Preferences. For instance, if "~/Documents"
  is configured for indexing, create a symlink from "~/.thunderbird" to
  "~/Documents/TMail". For this to work, the dot-directory must not be in a
  directory configured for indexing.

  If you want to exclude any specific files or directories from indexing, use
  patterns as described in section "8. File patterns".

  Pinot supports stopwords removal. While no such list is provided by default,
  they can be easily found on the Internet. Each language has its own stopword
  list, for instance a stopwords list for English should be copied to
  $PREFIX/share/pinot/stopwords/stopwords.en

  Language detection is done with libexttextcat. Ensure that the paths listed
  in /etc/pinot/textcat_conf.txt are correct.

  The pinot-index program allows indexing and peeking at documents' properties
  from the command-line. Using the -i/--index option with the My Documents or
  My Web Pages index is not recommended. For more details, see the manual page
  for pinot-index(1).


5. Searching


  Searches are run differently based on the type of engine being queried.

  When querying a Web engine, Pinot assumes this engine understands the query,
  which is sent as is. No pre-processing is performed on the text of the query,
  and the results list is more or less presented as retrieved from the Web
  engine.

  When querying an index, things are somewhat different. Queries can be
  expressed in a very natural way, using a combination of operators, filters
  and ranges. This query syntax is the syntax supported natively by Xapian's
  QueryParser and is documented at http://www.xapian.org/docs/queryparser.html
  For instance, the query "type:text/html AND lang:en AND (tcp NEAR ip)" will
  look for HTML files in English that mention TCP/IP. Note that all operators
  should be specified in capitals, eg "AND" not "and". The latter will be
  treated as a regular term.

  Pinot supports these query filters :
      "site" for host name, eg "site:github.com"
      "file" for file name, eg "file:index.html"
      "ext" for file extension, eg "ext:html"
      "title" for title, eg "title:pinot"
      "url" for URL, eg "url:https://github.com/"
      "dir" for directory, eg "dir:/home/fabrice"
      "inurl" for documents embedded in a URL, eg "inurl:file:///home/fabrice/Documents/backup.tar.gz"
      "lang" for ISO language code, eg "lang:en"
      "type" for MIME type, eg "type:text/html"
      "class" for MIME type classification, eg "class:text"
      "label" for label, eg "label:Important"

  The directory filter is recursive, ie it applies to sub-directories.
  Allowed language codes are "da", "nl", "en", "fi", "fr", "de", "hu", "it",
  "nn", "pt", "ro", "ru", "es", "sv" and "tr".

  Stemming is available to stored queries for which a stemming language is
  defined. If such a query doesn't return any exact match, the query terms are
  stemmed and the query is run again. Stopwords are also then removed if a
  stopwords list was found for the stemming language.

  The values of "file", "url", "dir" and "label" may be double-quoted. It's also
  worth pointing out that the query "dir:/X/Y" will return files and directories
  located in /X/Y, but not Y itself, which is what "dir:/X file:Y" would do.

  In addition, these ranges are supported :
      "YYYYMMDD..YYYYMMDD" for date ranges, eg "20070801..20070831"
      "HHMMSS..HHMMSS" for time ranges, eg "090000..180000"
      "size0..size1b" for size in bytes, eg "0..10240b"

  See the manual page for pinot-search(1) for examples.


6. Viewing cached results


  Results returned by search engines can be viewed "live" by selecting the View
  menuitem under Results. This opens whatever application defined for the
  result's MIME type and/or protocol scheme.
  In addition, Pinot allows to view the page as cached by Google and the Wayback
  Machine. Cache providers are actually configured in globalconfig.xml, located
  in /etc/pinot/. For instance :
  <cache>
    <name>Google</name>
    <location>http://www.google.com/search?q=cache:%url0</location>
    <protocols>http, https</protocols>
  </cache>

  This is self-explanatory :-) Here it configures a cache provider called
  "Google" that handles both http and https. The location field supports
  two parameters that are substituted to obtain the URL to open :
    * %url is the result's URL as displayed by the UI, eg
      https://github.com/FabriceColin/pinot
    * %url0 is the result's URL without the protocol, eg
      github.com/FabriceColin/pinot


7. File formats


  The following document types are supported internally :
    * plain text
    * HTML
    * XML
    * mbox, including attachments and embedded documents
    * MP3, Ogg Vorbis, FLAC
    * JPEG
    * common archive formats (tar, Z, gz, bzip2, deb)
    * ISO 9660 images

  The following document types are supported through external programs :
    * PDF (pdftotext required)
    * RTF (unrtf required)
    * ReStructured Text (rst2txt required)
    * OpenDocument/StarOffice files (unzip required)
    * MS Word (antiword required)
    * PowerPoint (catppt required)
    * Excel (xls2csv required)
    * DVI (catdvi required)
    * DjVu (djvutext required)
    * RPM (rpm required)

  For other document types, Pinot will only index metadata such as name,
  location etc... If you wish to add support for another document type, and
  know of a command-line program that can handle that type, add it to
  external-filters.xml, located in /etc/pinot/.


8. File patterns


  It is possible to skip indexing of files that match glob(3) patterns.
  These patterns are configured in the Indexing tab of the Preferences box,
  and can be used as a blacklist or a whitelist.

  Patterns apply to files and directories. For instance, blacklisting
  "*/Desktop*" will skip "~/Desktop" and not crawl nor monitor this directory's
  contents. Similarly, a blacklist entry for "*.avi" means that Pinot will not
  attempt indexing the content of AVI files, and will ignore all monitor events
  related to these files.

  If you have never run Pinot before, the list will be pre-configured to skip
  some picture, video and archive file types such as GIF, MPG and RAR.


9. Digging deeper


  Pinot offers two ways you can dig deeper in your documents : More Like This
  suggests terms specific to documents that may help in finding related
  documents, and Search This For allows to search in results.
  Both features are enabled if one or more of the results currently selected
  is indexed, and only operate on those.

  When activated, More Like This will create a new Stored Query prefixed with
  "More Like". For instance, if you run a Stored Query with name "Me", the
  expanded query's name will be "More Like Me".

  Search For This will search those results for the Stored Query selected in
  the sub-menu and will present results in a new tab. For instance, running
  the Stored Query "Me" on a set of results will open a "Me In Results" tab.

  In addition to these, Pinot may suggest alternative spellings for queries
  that don't return any result. If it does, a new Stored Query prefixed with
  "Corrected" will be created.


10. Saving results


  Lists of results can be saved to disk by selecting the Save As menuitem
  under Results. Two output formats are available to choose from in the file
  selector opened by Save As :
    * CSV, a text format
      The semi-colon character (';') is used to delimit fields.
    * OpenSearch response, a XML/RSS format
      See https://en.wikipedia.org/wiki/OpenSearch for details.


11. D-Bus service & daemon


  Unless Pinot was built without support for D-Bus, the daemon program
  "pinot-dbus-daemon" implements the D-Bus service and should be
  auto-started through the desktop file installed at
  /etc/xdg/autostart/pinot-dbus-daemon.desktop.

  D-Bus activation makes sure the service is running whenever one of its
  methods is invoked by any consumer application. For instance, clicking
  OK on the Preferences box will call the service's Reload method, which
  should start the service. This method also causes the service to reload
  the configuration file.

  A few things to keep in mind :
    * when starting, the service will first crawl all configured locations
      and (re)index new and modified files. The daemon's scheduling priority
      is set very low (15, can be adjusted with --priority) so that it
      hopefully doesn't prevent other activities. Crawling is suspended
      while the system is on battery.
    * when finished crawling, the service will monitor some locations for
      changes (as per preferences) and should consume little resources, unless
      a huge quantity of files needs its attention.
    * any change detected by the monitor is queued and acted upon as soon as
      possible, eg reindex a file that was modified.
    * operations that involve communicating with the service, such as editing
      documents metadata, may timeout if the system is under heavy load and/or
      the daemon is busy. In most cases, the message will have been received
      by the daemon, but the reply may take longer than expected. The Pinot
      UI may report that the operation failed, even though it was queued for
      processing and will be acted upon by the daemon.

  See section "13. Environment variables and aliases" for some tips on how to
  query the D-Bus interface. A list of available D-Bus methods can be found
  in the file pinot-dbus-daemon.xml.

  Pinot v1.20 implements the GNOME Shell search provider interface to allow
  searching the contents of files the daemon found at locations it crawled,
  basically the My Documents index. Go to the GNOME Settings' Search screen
  to enable Pinot as a provider. For this to work, the file
  com.github.fabricecolin.Pinot.search-provider.ini should be in the folder
  $PREFIX/share/gnome-shell/search-providers/


12. CJKV support


  Pinot supports indexing and searching CJKV text.

  At search time, queries that include CJKV characters are processed in a manner
  compatible with the CJKV indexing scheme. There is no need to format the query
  in a specific format, ie no need to separate characters with spaces.
  For example, the query :
      Fabrice 你好 title:身体好吗
  will be modified internally to :
      Fabrice  (你 你好 好) title:身 title:身体 title:体 title:体好 title:好 title:好吗 title:吗

  It is recommended that filters (eg "title") be used at the end of the query
  for it to be processed as expected.

  You can get a list of documents in which CJKV characters were detected
  by the indexer with the special filter "tokens:CJKV".


13. Environment variables and aliases


  Pinot tries to provide reasonable defaults for most systems, but there may be
  situations where you want to tweak these values through environment variables :
    * PINOT_SPELLING_DB
      By default, Pinot builds indexes with a spelling database. This spelling
      database may make up as much as a third of the size of the index.
      If your system is low on disk space, you can disable this with
      $ export PINOT_SPELLING_DB=NO
      Make sure this is set for your login session, ie whenever the daemon is
      auto-started. You will also have to reset indexes, as described in
      section "16. How to reset indexes".
    * PINOT_MINIMUM_DISK_SPACE
      The daemon will stop crawling and indexing files when the partition on
      which the index resides runs out of free space. By default, this means
      less than 50 Mb. To change this value to 100 Mb for instance, use
      $ export PINOT_MINIMUM_DISK_SPACE=100
    * PINOT_MAXIMUM_INDEX_THREADS
      This sets the maximum number of concurrent indexing threads used by the
      daemon. The default value is 1.
    * PINOT_MAXIMUM_NESTED_SIZE
      This limits the extraction of documents nested inside others, such as
      archives or mail messages, based on their size. By default, this is
      deactivated and set to 0.
    * PINOT_MAXIMUM_QUERY_RESULTS
      This overrides the number of results returned by queries run through
      the UI's Query field as well as the number of results initially set
      for new stored queries.

  Another environment variable that you may want to tweak comes from Xapian.
  XAPIAN_FLUSH_THRESHOLD can be set to the number of documents after which
  Xapian is to flush changes to the index. The default value is set to 10000
  at the time of writing this.
  Lowering this value should decrease the amount of memory used to cache
  changes to the index.

  Pinot provides a "tagged cd" script that enables to change a shell's
  current directory to the directory that matches the path elements passed
  as parameter. For instance, after setting :
  $ alias pcd='. $PREFIX/share/pinot/pinot-cd.sh'
  if ~/Documents is configured for indexing in Preferences, the following
  command would change the current directory to ~/Documents/Web/Stats :
  $ pcd Documents Stats
  If other directories match the given paths, pinot-cd.sh will display a list
  of matches. Future work will focus on disambiguation.

  If you have dbus-send installed, you may also want to set the following
  aliases :
  $ alias pinot-stats='dbus-send --session --print-reply --type=method_call \
    --dest=com.github.fabricecolin.Pinot /com/github/fabricecolin/Pinot com.github.fabricecolin.Pinot.GetStatistics'
  $ alias pinot-stop='dbus-send --session --print-reply --type=method_call \
    --dest=com.github.fabricecolin.Pinot /com/github/fabricecolin/Pinot com.github.fabricecolin.Pinot.Stop'
  The first will start the service daemon by calling its GetStatistics method,
  while the second alias will send it a request to stop and exit.


14. How to reset indexes


  You may wish to reset one of the index and start from scratch. There
  are several ways to do this, depending on which index it is.

  If you want to reset My Web Pages, you can either :
    * use Pinot to unindex every single document by selecting them all
      and choosing Unindex in the Index menu
    * or stop Pinot and delete ~/.pinot/index recursively

  If you want to reset My Documents, special considerations apply because
  of the historical data maintained by the daemon. There are two ways to
  proceed, and both require that the daemon be stopped.

  The manual way is to delete the index with
  $ rm -rf ~/.pinot/daemon
  and remove historical data with
  $ sqlite3 ~/.pinot/history-daemon "delete from CrawlHistory; delete from CrawlSources; delete from ActionQueue;"
  If you want to start from scratch and drop metadata (eg labels) that may
  exist on some documents, remove the history file altogether with
  $ rm -f ~/.pinot/history-daemon

  The automated way is to tell the daemon to reindex everything by launching
  it with the "--reindex" option, ie
  $ pinot-dbus-daemon --reindex
  It may be useful to take a look at the log file located at
  ~/.pinot/pinot-dbus-daemon.log.

15. Compiling


  Pinot's configure understands the following optional switches.

  --enable-debug enable debug [default=no]
  --enable-dbus enable DBus support [default=yes]
  --enable-libnotify enable libnotify support [default=no]
  --enable-mempool enable memory pool [default=no]
  --enable-libarchive [enable the libarchive filter [default=no]
  --enable-chmlib [enable the chmlib filter [default=no]

  Enable support for libarchive and chmlib if the necessary
  libraries are available. Enable libnotify support when building
  on BSD systems. Other switches should most likely stay unchanged.

  See the list below for dependencies. The version numbers indicate
  the minimum version Pinot has been tested with; older versions may
  or may not work.

---------------------------------------------------------------
Libraries and tools					Version
---------------------------------------------------------------
SQLite							3.3.1
http://www.sqlite.org/

xapian-core						1.4.10
http://www.xapian.org/

 zlib							1.2.0
 http://www.gzip.org/zlib/

curl (1)						7.13.1
http://curl.haxx.se/
- OR -
neon (1)						0.24.7
http://www.webdav.org/neon/

gdbus-codegen-glibmm (2)
https://github.com/Pelagicore/gdbus-codegen-glibmm

gtkmm							3.24
http://www.gtkmm.org/

libxml++						2.12.0
http://libxmlplusplus.sourceforge.net/

libexttextcat						3.2
http://cgit.freedesktop.org/libreoffice/libexttextcat/

gmime (3)						2.6.0
http://spruce.sourceforge.net/gmime

boost (4)						1.75
http://www.boost.org/

D-Bus with GLib bindings				0.61
http://www.freedesktop.org/wiki/Software/dbus

shared-mime-info					0.17
http://freedesktop.org/Software/shared-mime-info

desktop-file-utils					0.10
http://www.freedesktop.org/software/desktop-file-utils

TagLib							1.4
http://ktown.kde.org/~wheeler/taglib/

libarchive (5)						2.6.2
http://people.freebsd.org/~kientzle/libarchive/

exiv2							0.21
http://www.exiv2.org/

chmlib (6)						0.40
http://www.jedrea.com/chmlib/

openssh-askpass (7)					4.3
http://www.openssh.com/portable.html

---------------------------------------------------------------
External filter programs
---------------------------------------------------------------
unzip
http://www.info-zip.org/pub/infozip/UnZip.html

pdftotext
http://www.foolabs.com/xpdf/
http://poppler.freedesktop.org/

antiword
http://www.winfield.demon.nl/

unrtf
http://www.gnu.org/software/unrtf/unrtf.html

rst2txt
https://github.com/stephenfin/rst2txt

djvutxt
http://djvu.sourceforge.net/

catdvi
http://catdvi.sourceforge.net/

catppt
xls2csv
http://www.wagner.pp.ru/~vitus/software/catdoc/

---------------------------------------------------------------------
Notes :
(1) enabled with "./configure --with-http=neon|curl"
(2) only to regenerate DBus code, with "make dbus-code"
(3) for gmime 2.4.0 support, edit configure.in
(4) for building only
    with boost > 1.48 and < 1.54, turning off memory pooling with "./configure --enable-mempool=no" may be preferable
(5) optional - enabled with "./configure --enable-libarchive=yes"
(6) optional - enabled with "./configure --enable-chmlib=yes"
(7) experimental - required only if _SSH_TUNNEL is set
---------------------------------------------------------------------