1.. Copyright (C) 2008 Lemur Consulting Ltd 2.. Copyright (C) 2010,2014,2019 Olly Betts 3 4==================================== 5Xapian Database Replication Protocol 6==================================== 7 8.. contents:: Table of contents 9 10This document contains details of the implementation of the replication 11protocol, version 1. For details of how and why to use the replication 12protocol, see the separate `Replication Users Guide <replication.html>`_ 13document. 14 15Protocol description 16==================== 17 18The protocol used to transfer the updates is based on the RemoteConnection 19system (also used for remote Xapian databases). This provides a "message" 20layer abstraction for the connections; so the communication is based on a set 21of messages, each with a type, and some associated data. 22 23Where the following description refers to "packed" strings or integers, this 24means packed according to the same methods for packing these into databases. 25 26Client messages 27--------------- 28 29The client sends two message types to the server: a message of type 'R' which 30specifies the revision string for that database (or an empty string to force 31a copy) followed by a message of type 'D' which specifies the name of the 32database to be replicated. These messages are sent whenever the client wants 33to receive updates for a database. 34 35Server messages 36--------------- 37 38The server can send a variety of messages. The message types are currently 39defined in an enum in common/replicationprotocol.h (in which each type is 40preceded by ``REPL_REPLY_``): 41 42 - END_OF_CHANGES: this indicates that no further changes are needed, and ends 43 the response to the original request. It contains no data. 44 45 - FAIL: this indicates that a consistent set of changes couldn't be sent. It 46 may occur because the database is being changed too quickly at the senders 47 end, or for other reasons. It ends the response to the original request, 48 and may occur when any other messages are expected. 49 50 - DB_HEADER: this indicates that an entire database copy is about to be sent. 51 It contains a string representing the UUID of the database which is about to 52 be sent, followed by a (packed) unsigned integer, representing the revision 53 number of the copy which is about to be sent. 54 55 - DB_FILENAME: this contains the name of the next file to be sent in a DB copy 56 operation. 57 58 - DB_FILEDATA: this contains the contents of a file in a DB copy operation. 59 The contents of the message are the details of the file. 60 61 - DB_FOOTER: this indicates the end of a DB copy operation. The contents of 62 this message are a single (packed) unsigned integer, which represents a 63 revision number. The newly copied database is not safe to make live until 64 changesets up to the specified revision have been applied. 65 66 - CHANGESET: this indicates that a changeset file (see below) is being sent. 67 68Changeset files 69=============== 70 71Changes are represented by changeset files. When changeset logging is enabled 72for a database, just before each commit a changeset file is created in 73the database directory. This file contains a record of the changes made, 74currently in the following format (but note that this format may change in 75the future): 76 77 - 12 bytes holding the string "FlintChanges", "ChertChanges" or "GlassChanges" 78 (used to check that a file is a changeset file). 79 80 - The format of the changeset (as a variable length unsigned integer). 81 82 - The revision number of the database before the changes were applied (as a 83 variable length unsigned integer). 84 85 - The revision number of the database after the changes were applied (as a 86 variable length unsigned integer). 87 88 - A byte: 89 90 - 0 if the changes can safely be applied to a live database 91 92 - 1 if the changes cannot be applied while searching is in progress. (This 93 will be set if the database was modified in "DANGEROUS" mode). 94 95 - A series of items: 96 97 - A byte: 0 if there are no more items in the changeset, 1 if the next item 98 is a base file, 2 if the next item is a list of blocks. 99 100 - A (packed) string, holding a table name. 101 102 - If a base file: 103 104 - The letter of the base file (currently 'A' or 'B'). 105 106 - The length of the file (as a variable length unsigned integer). 107 108 - The contents of the base file. 109 110 - If a list of blocks: 111 112 - The blocksize in use (for glass, divided by 2048). 113 114 - A list of items: 115 116 - A variable length unsigned integer holding 0 if the list is at an end, 117 or holding (block number + 1) otherwise. 118 119 - The contents of the block. 120 121 - A revision number that the database must be upgraded to, with more 122 changesets, before it is safe to be made live. This will normally be the 123 revision number of the database after the changes were applied, but if the 124 changeset was created by copying parts of the database without a read lock, 125 and modifications were made to the database while the copy was in progress, 126 parts of the copy may contain later revisions than intended - in this 127 situation further changesets will be needed to ensure that these parts of 128 the database are fully integrated. 129