1.. Copyright (C) 2008 Lemur Consulting Ltd
2.. Copyright (C) 2010,2014,2019 Olly Betts
3
4====================================
5Xapian Database Replication Protocol
6====================================
7
8.. contents:: Table of contents
9
10This document contains details of the implementation of the replication
11protocol, version 1.  For details of how and why to use the replication
12protocol, see the separate `Replication Users Guide <replication.html>`_
13document.
14
15Protocol description
16====================
17
18The protocol used to transfer the updates is based on the RemoteConnection
19system (also used for remote Xapian databases).  This provides a "message"
20layer abstraction for the connections; so the communication is based on a set
21of messages, each with a type, and some associated data.
22
23Where the following description refers to "packed" strings or integers, this
24means packed according to the same methods for packing these into databases.
25
26Client messages
27---------------
28
29The client sends two message types to the server: a message of type 'R' which
30specifies the revision string for that database (or an empty string to force
31a copy) followed by a message of type 'D' which specifies the name of the
32database to be replicated.  These messages are sent whenever the client wants
33to receive updates for a database.
34
35Server messages
36---------------
37
38The server can send a variety of messages.  The message types are currently
39defined in an enum in common/replicationprotocol.h (in which each type is
40preceded by ``REPL_REPLY_``):
41
42 - END_OF_CHANGES: this indicates that no further changes are needed, and ends
43   the response to the original request.  It contains no data.
44
45 - FAIL: this indicates that a consistent set of changes couldn't be sent.  It
46   may occur because the database is being changed too quickly at the senders
47   end, or for other reasons.  It ends the response to the original request,
48   and may occur when any other messages are expected.
49
50 - DB_HEADER: this indicates that an entire database copy is about to be sent.
51   It contains a string representing the UUID of the database which is about to
52   be sent, followed by a (packed) unsigned integer, representing the revision
53   number of the copy which is about to be sent.
54
55 - DB_FILENAME: this contains the name of the next file to be sent in a DB copy
56   operation.
57
58 - DB_FILEDATA: this contains the contents of a file in a DB copy operation.
59   The contents of the message are the details of the file.
60
61 - DB_FOOTER: this indicates the end of a DB copy operation.  The contents of
62   this message are a single (packed) unsigned integer, which represents a
63   revision number.  The newly copied database is not safe to make live until
64   changesets up to the specified revision have been applied.
65
66 - CHANGESET: this indicates that a changeset file (see below) is being sent.
67
68Changeset files
69===============
70
71Changes are represented by changeset files.  When changeset logging is enabled
72for a database, just before each commit a changeset file is created in
73the database directory.  This file contains a record of the changes made,
74currently in the following format (but note that this format may change in
75the future):
76
77 - 12 bytes holding the string "FlintChanges", "ChertChanges" or "GlassChanges"
78   (used to check that a file is a changeset file).
79
80 - The format of the changeset (as a variable length unsigned integer).
81
82 - The revision number of the database before the changes were applied (as a
83   variable length unsigned integer).
84
85 - The revision number of the database after the changes were applied (as a
86   variable length unsigned integer).
87
88 - A byte:
89
90   - 0 if the changes can safely be applied to a live database
91
92   - 1 if the changes cannot be applied while searching is in progress.  (This
93     will be set if the database was modified in "DANGEROUS" mode).
94
95 - A series of items:
96
97   - A byte: 0 if there are no more items in the changeset, 1 if the next item
98     is a base file, 2 if the next item is a list of blocks.
99
100   - A (packed) string, holding a table name.
101
102   - If a base file:
103
104     - The letter of the base file (currently 'A' or 'B').
105
106     - The length of the file (as a variable length unsigned integer).
107
108     - The contents of the base file.
109
110   - If a list of blocks:
111
112     - The blocksize in use (for glass, divided by 2048).
113
114     - A list of items:
115
116       - A variable length unsigned integer holding 0 if the list is at an end,
117	 or holding (block number + 1) otherwise.
118
119       - The contents of the block.
120
121 - A revision number that the database must be upgraded to, with more
122   changesets, before it is safe to be made live.  This will normally be the
123   revision number of the database after the changes were applied, but if the
124   changeset was created by copying parts of the database without a read lock,
125   and modifications were made to the database while the copy was in progress,
126   parts of the copy may contain later revisions than intended - in this
127   situation further changesets will be needed to ensure that these parts of
128   the database are fully integrated.
129