1.\" Copyright (c) 2008 Poul-Henning Kamp
2.\" All rights reserved.
3.\"
4.\" Redistribution and use in source and binary forms, with or without
5.\" modification, are permitted provided that the following conditions
6.\" are met:
7.\" 1. Redistributions of source code must retain the above copyright
8.\"    notice, this list of conditions and the following disclaimer.
9.\" 2. Redistributions in binary form must reproduce the above copyright
10.\"    notice, this list of conditions and the following disclaimer in the
11.\"    documentation and/or other materials provided with the distribution.
12.\"
13.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
23.\" SUCH DAMAGE.
24.\"
25.\" $FreeBSD$
26.\"
27.Dd February 9, 2008
28.Dt FIFOLOG 1
29.Os
30.Sh NAME
31.Nm fifolog_create , fifolog_writer , fifolog_reader
32.Nd "initialize, write, seek and extract data from a fifolog"
33.Sh SYNOPSIS
34.Nm fifolog_create
35.Op Fl l Ar record-size
36.Op Fl r Ar record-count
37.Op Fl s Ar size
38.Ar file
39.Nm fifolog_reader
40.Op Fl t
41.Op Fl b Ar tstart
42.Op Fl B Ar Tstart
43.Op Fl e Ar tend
44.Op Fl E Ar Tend
45.Op Fl o Ar ofile
46.Op Fl R Ar regexp
47.Op Fl T Ar timefmt
48.Ar file
49.Nm fifolog_writer
50.Op Fl w Ar write-rate
51.Op Fl s Ar sync-rate
52.Op Fl z Ar compression
53.Ar file
54.Sh DESCRIPTION
55Fifologs provide a compact round-robin circular storage for
56recording text and binary information to permanent storage in a bounded
57and predictable fashion, time and space wise.
58.Pp
59A fifolog can be stored either directly on a disk partition or in a
60regular file.
61.Pp
62The input data stream is encoded, compressed and marked up with
63timestamps before it is written to storage, such that it is possible
64to seek out a particular time interval in the stored data, without
65having to decompress the entire logfile.
66.Pp
67The
68.Nm fifolog_create
69utility
70is used to initialize the first sector of a disk device
71or file system file to make it a fifolog and should be called only
72once.
73.Pp
74Running
75.Nm fifolog_create
76on an existing fifolog will reset it so that
77.Nm fifolog_reader
78and
79.Nm fifolog_writer
80will not see the previous contents.
81(The previous contents are not physically erased, and with a bit
82of hand-work all but the first record can be easily recovered.)
83.Pp
84If the
85.Ar file
86does not already exist,
87.Nm fifolog_create
88will attempt to create and
89.Xr ftruncate 2
90it to the specified size, defaulting to 86400 records of 512 bytes
91if the
92.Fl r , l
93or
94.Fl s
95options do not specify otherwise.
96.Pp
97The
98.Nm fifolog_writer
99utility
100will read standard input and write it to the end of the fifolog
101according to the parameters given.
102.Pp
103Writes happen whenever the output buffer is filled with compressed
104data or when either of two timers expire, forcing a partially filled
105buffer to be written.
106.Pp
107The first and faster timer,
108.Fl w Ar write-rate ,
109forces available data to be written
110but does not flush and reset the compression dictionary.
111This timer is intended to minimize the amount of logdata lost in RAM
112in case of a crash and by default it fires 10 seconds after
113the previous write.
114.Pp
115The second and slower timer,
116.Fl s Ar sync-rate ,
117forces a full flush and reset of the compression
118engine and causes the next record written to be a synchronization
119point with an uncompressed timestamp, making it possible to start
120reading the logfile from that record.
121By default this timer fires a minute after the previous sync.
122.Pp
123The
124.Fl z Ar compression
125option controls the
126.Xr zlib 3
127compression level; legal values are zero to nine which is the default.
128.Pp
129The
130.Nm fifolog_reader
131utility
132will retrieve records from the fifolog according to the specified
133parameters and write them either to standard output or the file specified
134with
135.Fl o .
136.Pp
137It is possible to specify a start and end time to limit the amount
138of data
139.Nm fifolog_reader
140will report.
141The lower-case variants
142.Fl b
143and
144.Fl e
145take a
146.Vt time_t
147value, whereas the upper-case variants
148.Fl B
149and
150.Fl E
151take human-readable specifications such as
152.Dq Li "1 hour ago" .
153.Pp
154The
155.Fl t
156option forces timestamps to be formatted as
157.Dq Li "YYYYMMDDhhmmss"
158instead of as
159.Vt time_t ,
160and
161.Fl T
162allows the specification of an
163.Xr strftime 3
164formatting string.
165.Pp
166Finally, records can be filtered such that only records matching the
167.Pq Dv REG_BASIC
168regular expression specified with
169.Fl R
170are output.
171.Sh IMPLEMENTATION NOTES
172The data stored in the fifolog consists of three layers, an outer
173layer that allows searches to synchronization points based on timestamps
174without having to decompress and decode the actual contents, a
175compression layer implemented with
176.Xr zlib 3 ,
177and an inner serialization and timestamping layer.
178.Pp
179The exact encoding is described in the
180.Pa fifolog.h
181file.
182.Pp
183Fifolog is particularly well suited for use on Flash based media, where
184it results in much lower write-wear, than a file system with regular
185log files rotated with
186.Xr newsyslog 8
187etc.
188.Sh EXAMPLES
189Create a fifolog with 1024*1024 records of 512 bytes:
190.Pp
191.Dl "fifolog_create -r 10m /tmp/fifolog"
192.Pp
193Write a single record to this file:
194.Pp
195.Dl "date | fifolog_writer /tmp/fifolog"
196.Pp
197Read it back with human readable timestamps:
198.Pp
199.Dl "fifolog_reader -t /tmp/fifolog"
200.Pp
201One particular useful use of
202.Nm fifolog_writer
203is with
204.Xr syslogd 8
205using a line such as this in
206.Xr syslog.conf 5 :
207.Pp
208.Dl "*.* |fifolog_writer /var/log/syslog_fifolog"
209.Sh HISTORY
210The fifolog tools have been liberated from an open source
211.Tn SCADA
212applications called
213.Dq measured ,
214which monitors and controls remote radio navigation
215transmitters for the Danish Air Traffic Control system.
216.Sh AUTHORS
217The fifolog tools were written by
218.An Poul-Henning Kamp .
219