README.md
1[![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=for-the-badge)](LICENSE)
2
3**A parser for Syslog messages and transports**.
4
5> [Blazing fast](#Performances) Syslog parsers
6
7_By [@leodido](https://github.com/leodido)_.
8
9To wrap up, this package provides:
10
11- a [RFC5424-compliant parser and builder](/rfc5424)
12- a [RFC3164-compliant parser](/rfc3164) - ie., BSD-syslog messages
13- a parser which works on streams for syslog with [octet counting](https://tools.ietf.org/html/rfc5425#section-4.3) framing technique, see [octetcounting](/octetcounting)
14- a parser which works on streams for syslog with [non-transparent](https://tools.ietf.org/html/rfc6587#section-3.4.2) framing technique, see [nontransparent](/nontransparent)
15
16This library provides the pieces to parse Syslog messages transported following various RFCs.
17
18For example:
19
20- TLS with octet count ([RFC5425](https://tools.ietf.org/html/rfc5425))
21- TCP with non-transparent framing or with octet count ([RFC 6587](https://tools.ietf.org/html/rfc6587))
22- UDP carrying one message per packet ([RFC5426](https://tools.ietf.org/html/rfc5426))
23
24## Installation
25
26```
27go get github.com/influxdata/go-syslog/v3
28```
29
30## Docs
31
32[![Documentation](https://img.shields.io/badge/godoc-reference-blue.svg?style=for-the-badge)](http://godoc.org/github.com/influxdata/go-syslog)
33
34The [docs](docs/) directory contains `.dot` files representing the finite-state machines (FSMs) implementing the syslog parsers and transports.
35
36## Usage
37
38Suppose you want to parse a given sequence of bytes as a RFC5424 message.
39
40_Notice that the same interface applies for RFC3164. But you can always take a look at the [examples file](./rfc3164/example_test.go)._
41
42```go
43i := []byte(`<165>4 2018-10-11T22:14:15.003Z mymach.it e - 1 [ex@32473 iut="3"] An application event log entry...`)
44p := rfc5424.NewParser()
45m, e := p.Parse(i)
46```
47
48This results in `m` being equal to:
49
50```go
51// (*rfc5424.SyslogMessage)({
52// Base: (syslog.Base) {
53// Facility: (*uint8)(20),
54// Severity: (*uint8)(5),
55// Priority: (*uint8)(165),
56// Timestamp: (*time.Time)(2018-10-11 22:14:15.003 +0000 UTC),
57// Hostname: (*string)((len=9) "mymach.it"),
58// Appname: (*string)((len=1) "e"),
59// ProcID: (*string)(<nil>),
60// MsgID: (*string)((len=1) "1"),
61// Message: (*string)((len=33) "An application event log entry...")
62// },
63// Version: (uint16) 4,
64// StructuredData: (*map[string]map[string]string)((len=1) {
65// (string) (len=8) "ex@32473": (map[string]string) (len=1) {
66// (string) (len=3) "iut": (string) (len=1) "3"
67// }
68// })
69// })
70```
71
72And `e` being equal to `nil` since the `i` byte slice contains a perfectly valid RFC5424 message.
73
74### Best effort mode
75
76RFC5424 parser has the ability to perform partial matches (until it can).
77
78With this mode enabled, when the parsing process errors out it returns the message collected until that position, and the error that caused the parser to stop.
79
80Notice that in this modality the output is returned _iff_ it represents a minimally valid message - ie., a message containing almost a priority field in `[1,191]` within angular brackets, followed by a version in `]0,999]` (in the case of RFC5424).
81
82Let's look at an example.
83
84```go
85i := []byte("<1>1 A - - - - - -")
86p := NewParser(WithBestEffort())
87m, e := p.Parse(i)
88```
89
90This results in `m` being equal to the following `SyslogMessage` instance.
91
92```go
93// (*rfc5424.SyslogMessage)({
94// Base: (syslog.Base) {
95// Facility: (*uint8)(0),
96// Severity: (*uint8)(1),
97// Priority: (*uint8)(1),
98// Timestamp: (*time.Time)(<nil>),
99// Hostname: (*string)(<nil>),
100// Appname: (*string)(<nil>),
101// ProcID: (*string)(<nil>),
102// MsgID: (*string)(<nil>),
103// Message: (*string)(<nil>)
104// },
105// Version: (uint16) 1,
106// StructuredData: (*map[string]map[string]string)(<nil>)
107// })
108```
109
110And, at the same time, in `e` reporting the error that actually stopped the parser.
111
112```go
113// expecting a RFC3339MICRO timestamp or a nil value [col 5]
114```
115
116Both `m` and `e` have a value since at the column the parser stopped it already was able to construct a minimally valid RFC5424 `SyslogMessage`.
117
118### Builder
119
120This library also provides a builder to construct valid syslog messages.
121
122Notice that its API ignores input values that does not match the grammar.
123
124Let's have a look to an example.
125
126```go
127msg := &rfc5424.SyslogMessage{}
128msg.SetTimestamp("not a RFC3339MICRO timestamp")
129msg.Valid() // Not yet a valid message (try msg.Valid())
130msg.SetPriority(191)
131msg.SetVersion(1)
132msg.Valid() // Now it is minimally valid
133```
134
135Printing `msg` you will verify it contains a `nil` timestamp (since an invalid one has been given).
136
137```go
138// (*rfc5424.SyslogMessage)({
139// Base: (syslog.Base) {
140// Facility: (*uint8)(23),
141// Severity: (*uint8)(7),
142// Priority: (*uint8)(191),
143// Timestamp: (*time.Time)(<nil>),
144// Hostname: (*string)(<nil>),
145// Appname: (*string)(<nil>),
146// ProcID: (*string)(<nil>),
147// MsgID: (*string)(<nil>),
148// Message: (*string)(<nil>)
149// },
150// Version: (uint16) 1,
151// StructuredData: (*map[string]map[string]string)(<nil>)
152// })
153```
154
155Finally you can serialize the message into a string.
156
157```go
158str, _ := msg.String()
159// <191>1 - - - - - -
160```
161
162## Message transfer
163
164Excluding encapsulating one message for packet in packet protocols there are two ways to transfer syslog messages over streams.
165
166The older - ie., the **non-transparent** framing - and the newer one - ie., the **octet counting** framing - which is reliable and has not been seen to cause problems noted with the non-transparent one.
167
168This library provide stream parsers for both.
169
170### Octet counting
171
172In short, [RFC5425](https://tools.ietf.org/html/rfc5425#section-4.3) and [RFC6587](https://tools.ietf.org/html/rfc6587), aside from the protocol considerations, describe a **transparent framing** technique for Syslog messages that uses the **octect counting** technique - ie., the message length of the incoming message.
173
174Each Syslog message is sent with a prefix representing the number of bytes it is made of.
175
176The [octecounting package](./octetcounting) parses messages stream following such rule.
177
178To quickly understand how to use it please have a look at the [example file](./octetcounting/example_test.go).
179
180### Non transparent
181
182The [RFC6587](https://tools.ietf.org/html/rfc6587#section-3.4.2) also describes the **non-transparent framing** transport of syslog messages.
183
184In such case the messages are separated by a trailer, usually a line feed.
185
186The [nontransparent package](./nontransparent) parses message stream following such [technique](https://tools.ietf.org/html/rfc6587#section-3.4.2).
187
188To quickly understand how to use it please have a look at the [example file](./nontransparent/example_test.go).
189
190Things we do not support:
191
192- trailers other than `LF` or `NUL`
193- trailers which length is greater than 1 byte
194- trailer change on a frame-by-frame basis
195
196## Performances
197
198To run the benchmark execute the following command.
199
200```bash
201make bench
202```
203
204On my machine<sup>[1](#mymachine)</sup> these are the results obtained paring RFC5424 syslog messages with best effort mode on.
205
206```
207[no]_empty_input__________________________________ 4524100 274 ns/op 272 B/op 4 allocs/op
208[no]_multiple_syslog_messages_on_multiple_lines___ 3039513 361 ns/op 288 B/op 8 allocs/op
209[no]_impossible_timestamp_________________________ 1244562 951 ns/op 512 B/op 11 allocs/op
210[no]_malformed_structured_data____________________ 2389249 512 ns/op 512 B/op 9 allocs/op
211[no]_with_duplicated_structured_data_id___________ 1000000 1183 ns/op 712 B/op 17 allocs/op
212[ok]_minimal______________________________________ 6876235 178 ns/op 227 B/op 5 allocs/op
213[ok]_average_message______________________________ 730473 1653 ns/op 1520 B/op 24 allocs/op
214[ok]_complicated_message__________________________ 908776 1344 ns/op 1264 B/op 24 allocs/op
215[ok]_very_long_message____________________________ 392737 3114 ns/op 2448 B/op 25 allocs/op
216[ok]_all_max_length_and_complete__________________ 510740 2431 ns/op 1872 B/op 28 allocs/op
217[ok]_all_max_length_except_structured_data_and_mes 755124 1593 ns/op 867 B/op 13 allocs/op
218[ok]_minimal_with_message_containing_newline______ 6142984 199 ns/op 230 B/op 6 allocs/op
219[ok]_w/o_procid,_w/o_structured_data,_with_message 1670286 732 ns/op 348 B/op 10 allocs/op
220[ok]_minimal_with_UTF-8_message___________________ 3013480 407 ns/op 339 B/op 6 allocs/op
221[ok]_minimal_with_UTF-8_message_starting_with_BOM_ 2926410 423 ns/op 355 B/op 6 allocs/op
222[ok]_with_structured_data_id,_w/o_structured_data_ 1558971 814 ns/op 570 B/op 11 allocs/op
223[ok]_with_multiple_structured_data________________ 1000000 1243 ns/op 1205 B/op 16 allocs/op
224[ok]_with_escaped_backslash_within_structured_data 1000000 1025 ns/op 896 B/op 17 allocs/op
225[ok]_with_UTF-8_structured_data_param_value,_with_ 1000000 1241 ns/op 1034 B/op 19 allocs/op
226```
227
228As you can see it takes:
229
230* ~250ns to parse the smallest legal message
231
232* less than 2µs to parse an average legal message
233
234* ~3µs to parse a very long legal message
235
236Other RFC5424 implementations, like this [one](https://github.com/roguelazer/rust-syslog-rfc5424) in Rust, spend 8µs to parse an average legal message.
237
238_TBD: comparison against other Go parsers_.
239
240---
241
242* <a name="mymachine">[1]</a>: Intel Core i7-8850H CPU @ 2.60GHz
243