README.md
1[![Build Status](https://travis-ci.org/xaionaro/clsync.png?branch=master)](https://travis-ci.org/xaionaro/clsync)
2[![Coverage Status](https://coveralls.io/repos/xaionaro/clsync/badge.png)](https://coveralls.io/r/xaionaro/clsync)
3
4clsync
5======
6Contents
7--------
8
91. Name
102. Motivation
113. inotify vs fanotify
124. Installing
135. How to use
146. Example of usage
157. Other uses
168. Clustering
179. Known building issues
1810. FreeBSD support
1911. Support
2012. Developing
2113. Articles
22
23
241. Name
25-------
26
27Why "clsync"? The first name of the utility was "insync" (due to inotify), but
28then I suggested to use "fanotify" instead of "inotify" and utility was been
29renamed to "fasync". After that I started to intensively write the program.
30However I faced with some problems in "fanotify", so I was have to temporary
31fallback to "inotify", then I decided that the best name is "Runtime Sync" or
32"Live Sync", but "rtsync" is a name of some corporation and "lsync" is busy
33by "[lsyncd](https://github.com/axkibe/lsyncd)". So I called it
34"clsync", that should be interpreted as "lsync, but on c" due to "lsyncd" that
35written on "LUA" and may be used for the same purposes.
36
37UPD: Also I was have to add somekind of clustering support. It's multicast
38notifing subsystem to prevent loops on bidirection syncing. So "clsync" also
39can be interpreted as "cluster live sync". ;)
40
412. Motivation
42-------------
43
44This utility was been writted for two purposes:
45- for making high availability clusters
46- for making backups of them
47
48To do HA cluster I've tried a lot of different solutions, like "simple
49rsync by cron", "glusterfs", "ocfs2 over drbd", "common mirrorable external
50storage", "incron + perl + rsync", "inosync", "lsyncd" and so on. When I
51started to write the utility we was using "lsyncd", "ceph" and
52"ocfs2 over drbd". However all of this solutions doesn't arrange me, so I
53was have to write own utility for this purpose.
54
55To do backups we also tried a lot of different solution, and again I was have
56to write own utility for this purpose.
57
58The best known (for me) replacement for this utility is "lsyncd", however:
59- It's code is `>½` on LUA. There a lot of problems connected with it,
60for example:
61 - It's more difficult to maintain the code with ordinary sysadmin.
62 - It really eats 100% CPU sometimes.
63 - It requires LUA libs, that cannot be easily installed to few
64of our systems.
65- It's a little buggy. That may be easily fixed for our cases,
66but LUA. :(
67- It doesn't support pthread or something like that. It's necessary
68to serve huge directories with a lot of containers right.
69- It cannot run rsync for a pack of files. It runs rsync for every
70event. :(
71- Sometimes, it's too complex in configuration for our situation.
72- It can't set another event-collecting delay for big files. We don't
73want to sync big files (`>1GiB`) so often as ordinary files.
74- Shared object (.so file) cannot be used as rsync-wrapper.
75- It doesn't support kqueue/bsm
76
77Sorry, if I'm wrong. Let me know if it is, please :). "lsyncd" - is really
78interesting and useful utility, just it's not appropriate for us.
79
80UPD.: Also clsync was used to replace incron/csync2/etc in HPC-clusters for
81syncing /etc/{passwd,shadow,group,shells} files.
82
833. inotify vs fanotify:
84-----------------------
85
86It's said, that fanotify is much better, than inotify. So I started to write
87this program with using of fanotify. However I encountered the problem, that
88fanotify was unable to catch some important events at the moment of writing
89the program, like "directory creation" or "file deletion". So I switched to
90"inotify", leaving the code for "fanotify" in the safety... So, don't use
91"fanotify" in this utility ;).
92
93
944. Installing
95-------------
96
97Debian/ubuntu-users can try to install it directly with apt-get:
98
99 apt-get install clsync
100
101If it's required to install clsync from the source, first of all, you should
102install dependencies to compile it. On debian-like systems you should
103execute something like:
104
105 apt-get install libglib2.0-dev autoreconf gcc
106
107Next step is generating Makefile. To do that usually it's enought to execute:
108
109 autoreconf -i && ./configure
110
111Next step is compiling. To compile usually it's enough to execute:
112
113 make
114
115Next step is installing. To install usually it's enough to execute:
116
117 su -c 'make install'
118
119
1205. How to use
121-------------
122
123How to use is described in "man" ;). What is not described, you can ask me
124personally (see "Support").
125
126
1276. Example of usage
128-------------------
129
130Example of usage, that works on my PC is in directory "examples". Just run
131"clsync-start-rsyncdirect.sh" and try to create/modify/delete files/dirs in
132"example/testdir/from". All modifications should appear (with some delay) in
133directory "example/testdir/to" ;)
134
135For dummies:
136
137 pushd /tmp
138 git clone https://github.com/xaionaro/clsync
139 cd clsync
140 autoreconf -fi
141 ./configure
142 make
143 export PATH_OLD="$PATH"
144 export PATH="$(pwd):$PATH"
145 cd examples
146 ./clsync-start-rsyncdirect.sh
147 export PATH="$PATH_OLD"
148
149Now you can try to make changes in directory
150"/tmp/clsync/examples/testdir/from" (in another terminal).
151Wait about 7 seconds after the changes and check directory
152"/tmp/clsync/examples/testdir/to". To finish the experiment press ^C
153(control+c) in clsync's terminal.
154
155 cd ../..
156 rm -rf clsync
157 popd
158
159Note: There's no need to change PATH's value if clsync is installed
160system-wide, e.g. with
161
162 make install
163
164For dummies, again (with "make install"):
165
166 pushd /tmp
167 git clone https://github.com/xaionaro/clsync
168 cd clsync
169 autoreconf -fi
170 ./configure
171 make
172 sudo make install
173 cd examples
174 ./clsync-start-rsyncdirect.sh
175
176Directory "/tmp/clsync/examples/testdir/from" is now synced to
177"/tmp/clsync/examples/testdir/to" with 7 seconds delay. To terminate
178the clsync press ^C (control+c) in clsync's terminal.
179
180 cd ..
181 sudo make uninstall
182 cd ..
183 rm -rf clsync
184 popd
185
186For really dummies or/and lazy users, there's a video demonstration:
187[http://ut.mephi.ru/oss/clsync](http://ut.mephi.ru/oss/clsync)
188
189
1907. Other uses
191-------------
192
193Also, clsync may be used to do nearly atomic directory recursive copy.
194
195For example, command
196
197 ionice -c 3 clsync -L /dev/shm/clsync --exit-on-no-events -x 23 -x 24 -M rsyncdirect -S $(which rsync) -W /path/from -D /path/to -d1
198
199may be used to copy "/path/from" into "/path/to" with sync up of changes made (in "/path/from") while the copying. It will copy new changes over and over until there will be no changes, and then clsync will exit.
200
201
2028. Clustering
203-------------
204
205I've started to implement support of bi-directional syncing with using
206multicast notifing of other nodes. However it became a long task, so it was
207suspended for next releases.
208
209However let's solve next hypothetical problem. For example, you're using
210LXC and trying to replicate containers between two servers (to make failover
211and load balancing).
212
213In this case you have to sync containers in both directions. However, if you
214just run clsync to sync containers to neighboring node on both of them, you'll
215get sync-loop [file-update on A causes file-update on B causes file-update
216on A causes ...].
217
218Well, in this case I with my colleagues were using separate directories for
219every node of cluster (e.g. "`/srv/nodes/<NODE NAME>/containers/<CONTAINERS>`")
220and syncing every directory only in one direction. That was failover with
221load-balancing, but very unconvenient. So I've started to write code for
222bi-directional syncing, however it's no time to complete it :(. So
223Andrew Savchenko proposed to run one clsync-instance per container. And this's
224really good solution. It's just need to start clsync-process when container
225starts and stop the process when containers stops. The only problem is
226split-brain, that can be solved two ways:
227- by human every time;
228- by scripts that chooses which variant of container to save.
229
230Example of the script is just a script that calls "find" on both sides to
231determine which side has the latest changes :)
232
2339. Known building issues
234------------------------
235
236May be problems with "configuring" or compilation. In this case just try
237next command:
238 echo '#define REVISION "-custom"' > revision.h; gcc -std=gnu99 -D\_FORTIFY\_SOURCE=2 -DPARANOID -pipe -Wall -ggdb3 --param ssp-buffer-size=4 -fstack-check -fstack-protector-all -Xlinker -zrelro -pthread $(pkg-config --cflags glib-2.0) $(pkg-config --libs glib-2.0) -ldl \*.c -o /tmp/clsync
239
240
24110. FreeBSD support
242-------------------
243
244clsync was been ported to FreeBSD.
245
246FreeBSD doesn't support inotify, so there're 3 ways to use clsync on it:
247* using [libinotify](https://github.com/dmatveev/libinotify-kqueue);
248* using BSM API;
249* using kqueue/kevent directly.
250
251However:
252* kqueue/kevent doesn't allow to catch file creation events. However it allows to catch an event of directory content change (without details). So clsync waits for such events and rescans (non-recursively) the whole dir on each such event. This algorithm is not tested and may be buggy. Moreover kqueue/kevent requires to open a file descriptor for every watched file. So this way may eat a lot of CPU and file descriptors.
253* libinotify is not production ready. There may be problems with it. Moreover libinotify backends to kqueue API anyway. On the other hand inotify support is well tested in clsync, so this way should be stable (if libinotify is stable) in contrast to kqueue direct use.
254* Using of BSM API requires auditd reconfiguration. It may hopple to real audit. Moreover this's a global OS setting. And using of this way forces clsync to catch all FS events of the whole system.
255
256I recommend to use the BSM API at the moment. However when the libinotify will be production ready you should try that way.
257
258I hope you will send me bugreports to make me able to improve the FreeBSD support :)
259
260
26111. Support
262-----------
263
264To get support, you can contact with me this ways:
265- Official IRC channel of "clsync": irc.freenode.net#clsync
266- Where else can you find me: IRC:SSL+UTF-8 irc.campus.mephi.ru:6695#mephi,xaionaro,xai
267- And e-mail: <dyokunev@ut.mephi.ru>, <xaionaro@gmail.com>; PGP pubkey: 0x8E30679C
268
26912. Developing
270--------------
271
272I started to write "DEVELOPING" and "PROTOCOL" files.
273You can look there if you wish. ;)
274
275I'll be glad to receive code contribution :)
276
27713. Articles
278------------
279
280Russian:
281- [HA clustering](https://gitlab.ut.mephi.ru/ut/articles/blob/master/clsync/ha)
282- [syncing to many nodes](https://gitlab.ut.mephi.ru/ut/articles/blob/master/clsync/inotify-to-many-nodes)
283- [atomic sync](https://gitlab.ut.mephi.ru/ut/articles/blob/master/clsync/atomicsync)
284
285
286
287 -- Dmitry Yu Okunev <dyokunev@ut.mephi.ru> 0x8E30679C
288
289