1.\" Copyright (c) 2008 Poul-Henning Kamp
2.\" All rights reserved.
3.\"
4.\" Redistribution and use in source and binary forms, with or without
5.\" modification, are permitted provided that the following conditions
6.\" are met:
7.\" 1. Redistributions of source code must retain the above copyright
8.\"    notice, this list of conditions and the following disclaimer.
9.\" 2. Redistributions in binary form must reproduce the above copyright
10.\"    notice, this list of conditions and the following disclaimer in the
11.\"    documentation and/or other materials provided with the distribution.
12.\"
13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
23.\" SUCH DAMAGE.
24.\"
25.Dd February 9, 2008
26.Dt FIFOLOG 1
27.Os
28.Sh NAME
29.Nm fifolog_create , fifolog_writer , fifolog_reader
30.Nd "initialize, write, seek and extract data from a fifolog"
31.Sh SYNOPSIS
32.Nm fifolog_create
33.Op Fl l Ar record-size
34.Op Fl r Ar record-count
35.Op Fl s Ar size
36.Ar file
37.Nm fifolog_reader
38.Op Fl t
39.Op Fl b Ar tstart
40.Op Fl B Ar Tstart
41.Op Fl e Ar tend
42.Op Fl E Ar Tend
43.Op Fl o Ar ofile
44.Op Fl R Ar regexp
45.Op Fl T Ar timefmt
46.Ar file
47.Nm fifolog_writer
48.Op Fl w Ar write-rate
49.Op Fl s Ar sync-rate
50.Op Fl z Ar compression
51.Ar file
52.Sh DESCRIPTION
53Fifologs provide a compact round-robin circular storage for
54recording text and binary information to permanent storage in a bounded
55and predictable fashion, time and space wise.
56.Pp
57A fifolog can be stored either directly on a disk partition or in a
58regular file.
59.Pp
60The input data stream is encoded, compressed and marked up with
61timestamps before it is written to storage, such that it is possible
62to seek out a particular time interval in the stored data, without
63having to decompress the entire logfile.
64.Pp
65The
66.Nm fifolog_create
67utility
68is used to initialize the first sector of a disk device
69or file system file to make it a fifolog and should be called only
70once.
71.Pp
72Running
73.Nm fifolog_create
74on an existing fifolog will reset it so that
75.Nm fifolog_reader
76and
77.Nm fifolog_writer
78will not see the previous contents.
79(The previous contents are not physically erased, and with a bit
80of hand-work all but the first record can be easily recovered.)
81.Pp
82If the
83.Ar file
84does not already exist,
85.Nm fifolog_create
86will attempt to create and
87.Xr ftruncate 2
88it to the specified size, defaulting to 86400 records of 512 bytes
89if the
90.Fl r , l
91or
92.Fl s
93options do not specify otherwise.
94.Pp
95The
96.Nm fifolog_writer
97utility
98will read standard input and write it to the end of the fifolog
99according to the parameters given.
100.Pp
101Writes happen whenever the output buffer is filled with compressed
102data or when either of two timers expire, forcing a partially filled
103buffer to be written.
104.Pp
105The first and faster timer,
106.Fl w Ar write-rate ,
107forces available data to be written
108but does not flush and reset the compression dictionary.
109This timer is intended to minimize the amount of logdata lost in RAM
110in case of a crash and by default it fires 10 seconds after
111the previous write.
112.Pp
113The second and slower timer,
114.Fl s Ar sync-rate ,
115forces a full flush and reset of the compression
116engine and causes the next record written to be a synchronization
117point with an uncompressed timestamp, making it possible to start
118reading the logfile from that record.
119By default this timer fires a minute after the previous sync.
120.Pp
121The
122.Fl z Ar compression
123option controls the
124.Xr zlib 3
125compression level; legal values are zero to nine which is the default.
126.Pp
127The
128.Nm fifolog_reader
129utility
130will retrieve records from the fifolog according to the specified
131parameters and write them either to standard output or the file specified
132with
133.Fl o .
134.Pp
135It is possible to specify a start and end time to limit the amount
136of data
137.Nm fifolog_reader
138will report.
139The lower-case variants
140.Fl b
141and
142.Fl e
143take a
144.Vt time_t
145value, whereas the upper-case variants
146.Fl B
147and
148.Fl E
149take human-readable specifications such as
150.Dq Li "1 hour ago" .
151.Pp
152The
153.Fl t
154option forces timestamps to be formatted as
155.Dq Li "YYYYMMDDhhmmss"
156instead of as
157.Vt time_t ,
158and
159.Fl T
160allows the specification of an
161.Xr strftime 3
162formatting string.
163.Pp
164Finally, records can be filtered such that only records matching the
165.Pq Dv REG_BASIC
166regular expression specified with
167.Fl R
168are output.
169.Sh IMPLEMENTATION NOTES
170The data stored in the fifolog consists of three layers, an outer
171layer that allows searches to synchronization points based on timestamps
172without having to decompress and decode the actual contents, a
173compression layer implemented with
174.Xr zlib 3 ,
175and an inner serialization and timestamping layer.
176.Pp
177The exact encoding is described in the
178.Pa fifolog.h
179file.
180.Pp
181Fifolog is particularly well suited for use on Flash based media, where
182it results in much lower write-wear, than a file system with regular
183log files rotated with
184.Xr newsyslog 8
185etc.
186.Sh EXAMPLES
187Create a fifolog with 1024*1024 records of 512 bytes:
188.Pp
189.Dl "fifolog_create -r 10m /tmp/fifolog"
190.Pp
191Write a single record to this file:
192.Pp
193.Dl "date | fifolog_writer /tmp/fifolog"
194.Pp
195Read it back with human readable timestamps:
196.Pp
197.Dl "fifolog_reader -t /tmp/fifolog"
198.Pp
199One particular useful use of
200.Nm fifolog_writer
201is with
202.Xr syslogd 8
203using a line such as this in
204.Xr syslog.conf 5 :
205.Pp
206.Dl "*.* |fifolog_writer /var/log/syslog_fifolog"
207.Sh HISTORY
208The fifolog tools have been liberated from an open source
209.Tn SCADA
210applications called
211.Dq measured ,
212which monitors and controls remote radio navigation
213transmitters for the Danish Air Traffic Control system.
214.Sh AUTHORS
215The fifolog tools were written by
216.An Poul-Henning Kamp .
217