1Remote Backend Protocol 2======================= 3 4This document describes *version 35.1* of the protocol used by Xapian's 5remote backend. The major protocol version increased to 35 in Xapian 61.1.5, and the minor protocol version to 1 in Xapian 1.2.4. Clients and 7servers must support matching major protocol versions and the client's 8minor protocol version must be the same or lower. This means that for a 9minor protocol version change, you can upgrade first servers and then 10clients and everything should work during the upgrades. 11 12The protocol assumes a reliable two-way connection across which 13arbitrary data can be sent - this could be provided by a TCP socket for 14example (as it is with xapian-tcpsrv), but any such connection could be 15used. For example, you could used xapian-progsrv across an ssh 16connection, or even a custom server across a suitable serial connection. 17 18All messages start with a single byte identifying code. A message from 19client to server has a ``MSG_XXX`` identifying code, while a message 20from server to client has a ``REPLY_XXX`` identifying code (but note 21that a reply might not actually be in response to a message - 22``REPLY_GREETING`` isn't - and some messages result in multiple replies). 23 24The identifying code is followed by the encoded length of the contents 25followed by the contents themselves. 26 27Inside the contents, strings are generally passed as an encoded length 28followed by the string data (this is indicated below by ``L<...>``) 29except when the string is the last or only thing in the contents in 30which case we know the length because we know the length of the contents 31so we don't need to explicitly specify it. 32 33Integers are encoded using the same encoding used for string lengths 34(indicated by ``I<...>`` below). 35 36Floating pointing values are passed using a bit packed encoding of the 37sign and exponent and a base-256 encoding of the mantissa which avoids 38any rounding issues (assuming that both machines have ``FLT_RADIX`` equal 39to some power of 2). This is indicated by ``F<...>`` below. 40 41Boolean values are passed as a single byte which is the ASCII character 42value for ``0`` or ``1``. This is indicated by ``B<...>`` below. 43 44Server greeting and statistics 45------------------------------ 46 47- ``REPLY_GREETING <protocol major version> <protocol minor version> I<db doc count> I<last docid> B<has positions?> I<db total length> <UUID>`` 48 49The protocol major and minor versions are passed as a single byte each 50(e.g. ``'\x1e\x01'`` for version 30.1). The server and client must 51understand the same protocol major version, and the server protocol 52minor version must be greater than or equal to that of the client (this 53means that the server understands newer MSG\_\ *XXX*, but will only send 54newer REPLY\_\ *YYY* in response to an appropriate client message. 55 56Exception 57--------- 58 59- ``REPLY_EXCEPTION <serialised Xapian::Error object>`` 60 61If an unknown exception is caught by the server, this message is sent 62but with empty contents. 63 64This message can be sent at any point - the serialised exception is 65unserialised by the client and thrown. The server and client both abort 66any current sequence of messages. 67 68Write Access 69------------ 70 71- ``MSG_WRITEACCESS`` 72- ``REPLY_UPDATE I<db doc count> I<last docid> B<has positions?> I<db total length> <UUID>`` 73 74The reply is the same as for ``MSG_UPDATE``. If write access isn't 75supported or the database is locked by another writer, then an exception 76is thrown. 77 78By default the server is also read-only, even if writing is supported. 79If the client wants to be able to write, it needs to request this 80explicitly. We do this so that the same server can support multiple 81read-only clients and one writing client at once, without the protocol 82for read-only clients requiring an extra message. The overhead of an 83extra message exchange for a writer is unlikely to matter as indexing is 84rarely so real-time critical as searching. 85 86All Terms 87--------- 88 89- ``MSG_ALLTERMS`` 90- ``REPLY_ALLTERMS I<term freq> L<term name>`` 91- ``...`` 92- ``REPLY_DONE`` 93 94Term Exists 95----------- 96 97- ``MSG_TERMEXISTS <term name>`` 98- ``REPLY_TERMEXISTS`` or ``REPLY_TERMDOESNTEXIST`` 99 100Term Frequency 101-------------- 102 103- ``MSG_TERMFREQ <term name>`` 104- ``REPLY_TERMFREQ I<term freq>`` 105 106Collection Frequency 107-------------------- 108 109- ``MSG_COLLFREQ <term name>`` 110- ``REPLY_COLLFREQ I<collection freq>`` 111 112Document 113-------- 114 115- ``MSG_DOCUMENT I<document id>`` 116- ``REPLY_DOCDATA <document data>`` 117- ``REPLY_VALUE I<value no> <value>`` 118- ``...`` 119- ``REPLY_DONE`` 120 121Document Length 122--------------- 123 124- ``MSG_DOCLENGTH I<document id>`` 125- ``REPLY_DOCLENGTH I<document length>`` 126 127Keep Alive 128---------- 129 130- ``MSG_KEEPALIVE`` 131- ``REPLY_DONE`` 132 133Reopen 134------ 135 136- ``MSG_REOPEN`` 137- ``REPLY_UPDATE I<db doc count> I<last docid> B<has positions?> I<db total length> <UUID>`` 138 139The reply is the same as for ``MSG_UPDATE``. 140 141Query 142----- 143 144- ``MSG_QUERY L<serialised Xapian::Query object> I<query length> I<collapse max> [I<collapse key number> (if collapse_max non-zero)] <docid order> I<sort key number> <sort by> B<sort value forward> <percent cutoff> F<weight cutoff> <serialised Xapian::Weight object> <serialised Xapian::RSet object> [L<serialised Xapian::MatchSpy object>...]`` 145- ``REPLY_STATS <serialised Stats object>`` 146- ``MSG_GETMSET I<first> I<max items> I<check at least> <serialised global Stats object>`` 147- ``REPLY_RESULTS L<the result of calling serialise_results() on each Xapian::MatchSpy> <serialised Xapian::MSet object>`` 148 149docid order is ``'0'``, ``'1'`` or ``'2'``. 150 151sort by is ``'0'``, ``'1'``, ``'2'`` or ``'3'``. 152 153Termlist 154-------- 155 156- ``MSG_TERMLIST I<document id>`` 157- ``REPLY_DOCLENGTH I<document length>`` 158- ``REPLY_TERMLIST I<wdf> I<term freq> L<term name>`` 159- ``...`` 160- ``REPLY_DONE`` 161 162Positionlist 163------------ 164 165- ``MSG_POSITIONLIST I<document id> <term name>`` 166- ``REPLY_POSITIONLIST I<termpos delta - 1>`` 167- ``...`` 168- ``REPLY_DONE`` 169 170Since positions must be strictly monotonically increasing, we encode 171``(pos - lastpos - 1)`` so that small differences between large position 172values can still be encoded compactly. The first position is encoded as 173its true value. 174 175Postlist 176-------- 177 178- ``MSG_POSTLIST <term name>`` 179- ``REPLY_POSTLISTSTART I<termfreq> I<collfreq>`` 180- ``REPLY_POSTLISTITEM I<docid delta - 1> I<wdf> F<document length>`` 181- ``...`` 182- ``REPLY_DONE`` 183 184Since document IDs in postlists must be strictly monotonically 185increasing, we encode ``(docid - lastdocid - 1)`` so that small 186differences between large document IDs can still be encoded compactly. 187The first document ID is encoded as its true value - 1 (since document 188IDs are always > 0). 189 190Shut Down 191--------- 192 193- ``MSG_SHUTDOWN`` 194 195No reply is sent - this message signals that the client has ended the 196session. 197 198Update 199------ 200 201- ``MSG_UPDATE`` 202- ``REPLY_UPDATE I<db doc count> I<last docid> B<has positions?> I<db total length> <UUID>`` 203 204Only useful for a ``WritableDatabase`` (since the same statistics are 205sent when the connection is initiated in the ``REPLY_GREETING`` and they 206don't change if the database can't change). 207 208Add document 209------------ 210 211- ``MSG_ADDDOCUMENT <serialised Xapian::Document object>`` 212- ``REPLY_ADDDOCUMENT I<document id>`` 213 214Delete document 215--------------- 216 217- ``MSG_DELETEDOCUMENT I<document id>`` 218- ``REPLY_DONE`` 219 220Delete document by term 221----------------------- 222 223- ``MSG_DELETEDOCUMENTTERM <term name>`` 224 225Replace document 226---------------- 227 228- ``MSG_REPLACEDOCUMENT I<document id> <serialised Xapian::Document object>`` 229 230Replace document by term 231------------------------ 232 233- ``MSG_REPLACEDOCUMENTTERM L<term name> <serialised Xapian::Document object>`` 234 235Cancel 236------ 237 238- ``MSG_CANCEL`` 239 240Commit 241------ 242 243- ``MSG_COMMIT`` 244- ``REPLY_DONE`` 245 246Set metadata 247------------ 248 249- ``MSG_SETMETADATA L<key> <value>`` 250 251Get metadata 252------------ 253 254- ``MSG_GETMETADATA <key>`` 255- ``REPLY_METADATA <value>`` 256 257Metadata keys 258------------- 259 260- ``MSG_METADATAKEYLIST <prefix>`` 261- ``REPLY_METADATAKEYLIST <key>`` 262- ``...`` 263- ``REPLY_DONE`` 264 265Add spelling 266------------ 267 268- ``MSG_ADDSPELLING I<freqinc> <word>`` 269 270Remove spelling 271--------------- 272 273- ``MSG_REMOVESPELLING I<freqdec> <word>`` 274 275