1.. _eve-json-output: 2 3Eve JSON Output 4=============== 5 6The EVE output facility outputs alerts, anomalies, metadata, file info and protocol 7specific records through JSON. 8 9The most common way to use this is through 'EVE', which is a firehose approach 10where all these logs go into a single file. 11 12.. literalinclude:: ../../partials/eve-log.yaml 13 14Each alert, http log, etc will go into this one file: 'eve.json'. This file 15can then be processed by 3rd party tools like Logstash (ELK) or jq. 16 17If ``ethernet`` is set to yes, then ethernet headers will be added to events 18if available. 19 20Output types 21~~~~~~~~~~~~ 22 23EVE can output to multiple methods. ``regular`` is a normal file. Other 24options are ``syslog``, ``unix_dgram``, ``unix_stream`` and ``redis``. 25 26Output types:: 27 28 filetype: regular #regular|syslog|unix_dgram|unix_stream|redis 29 filename: eve.json 30 # Enable for multi-threaded eve.json output; output files are amended 31 # with an identifier, e.g., eve.9.json. Default: off 32 #threaded: off 33 #prefix: "@cee: " # prefix to prepend to each log entry 34 # the following are valid when type: syslog above 35 #identity: "suricata" 36 #facility: local5 37 #level: Info ## possible levels: Emergency, Alert, Critical, 38 ## Error, Warning, Notice, Info, Debug 39 #ethernet: no # log ethernet header in events when available 40 #redis: 41 # server: 127.0.0.1 42 # port: 6379 43 # async: true ## if redis replies are read asynchronously 44 # mode: list ## possible values: list|lpush (default), rpush, channel|publish 45 # ## lpush and rpush are using a Redis list. "list" is an alias for lpush 46 # ## publish is using a Redis channel. "channel" is an alias for publish 47 # key: suricata ## key or channel to use (default to suricata) 48 # Redis pipelining set up. This will enable to only do a query every 49 # 'batch-size' events. This should lower the latency induced by network 50 # connection at the cost of some memory. There is no flushing implemented 51 # so this setting as to be reserved to high traffic suricata. 52 # pipelining: 53 # enabled: yes ## set enable to yes to enable query pipelining 54 # batch-size: 10 ## number of entry to keep in buffer 55 56Alerts 57~~~~~~ 58 59Alerts are event records for rule matches. They can be amended with 60metadata, such as the application layer record (HTTP, DNS, etc) an 61alert was generated for, and elements of the rule. 62 63Metadata:: 64 65 - alert: 66 #payload: yes # enable dumping payload in Base64 67 #payload-buffer-size: 4kb # max size of payload buffer to output in eve-log 68 #payload-printable: yes # enable dumping payload in printable (lossy) format 69 #packet: yes # enable dumping of packet (without stream segments) 70 #http-body: yes # Requires metadata; enable dumping of http body in Base64 71 #http-body-printable: yes # Requires metadata; enable dumping of http body in printable format 72 73 # metadata: 74 75 # Include the decoded application layer (ie. http, dns) 76 #app-layer: true 77 78 # Log the the current state of the flow record. 79 #flow: true 80 81 #rule: 82 # Log the metadata field from the rule in a structured 83 # format. 84 #metadata: true 85 86 # Log the raw rule text. 87 #raw: false 88 89Anomaly 90~~~~~~~ 91 92Anomalies are event records created when packets with unexpected or anomalous 93values are handled. These events include conditions such as incorrect protocol 94values, incorrect protocol length values, and other conditions which render the 95packet suspect. Other conditions may occur during the normal progression of a stream; 96these are termed ``stream`` events are include control sequences with incorrect 97values or that occur out of expected sequence. 98 99Anomalies are reported by and configured by type: 100 101- Decode 102- Stream 103- Application layer 104 105Metadata:: 106 107 - anomaly: 108 # Anomaly log records describe unexpected conditions such as truncated packets, 109 # packets with invalid IP/UDP/TCP length values, and other events that render 110 # the packet invalid for further processing or describe unexpected behavior on 111 # an established stream. Networks which experience high occurrences of 112 # anomalies may experience packet processing degradation. 113 # 114 # Anomalies are reported for the following: 115 # 1. Decode: Values and conditions that are detected while decoding individual 116 # packets. This includes invalid or unexpected values for low-level protocol 117 # lengths as well. 118 # 2. Stream: This includes stream related events (TCP 3-way handshake issues, 119 # unexpected sequence number, etc). 120 # 3. Application layer: These denote application layer specific conditions that 121 # are unexpected, invalid or are unexpected given the application monitoring 122 # state. 123 # 124 # By default, anomaly logging is disabled. When anomaly logging is enabled, 125 # application-layer anomaly reporting is enabled. 126 # 127 # Choose one or both types of anomaly logging and whether to enable 128 # logging of the packet header for packet anomalies. 129 types: 130 #decode: no 131 #stream: no 132 #applayer: yes 133 #packethdr: no 134 135HTTP 136~~~~ 137 138HTTP transaction logging. 139 140Config:: 141 142 - http: 143 extended: yes # enable this for extended logging information 144 # custom allows additional http fields to be included in eve-log 145 # the example below adds three additional fields when uncommented 146 #custom: [Accept-Encoding, Accept-Language, Authorization] 147 # set this value to one among {both, request, response} to dump all 148 # http headers for every http request and/or response 149 # dump-all-headers: [both, request, response] 150 151List of custom fields: 152 153====================== ====================== 154Yaml Option HTTP Header 155====================== ====================== 156accept accept 157accept_charset accept-charset 158accept_encoding accept-encoding 159accept_language accept-language 160accept_datetime accept-datetime 161authorization authorization 162cache_control cache-control 163cookie cookie 164from from 165max_forwards max-forwards 166origin origin 167pragma pragma 168proxy_authorization proxy-authorization 169range range 170te te 171via via 172x_requested_with x-requested-with 173dnt dnt 174x_forwarded_proto x-forwarded-proto 175x_authenticated_user x-authenticated-user 176x_flash_version x-flash-version 177accept_range accept-range 178age age 179allow allow 180connection connection 181content_encoding content-encoding 182content_language content-language 183content_length content-length 184content_location content-location 185content_md5 content-md5 186content_range content-range 187content_type content-type 188date date 189etag etags 190expires expires 191last_modified last-modified 192link link 193location location 194proxy_authenticate proxy-authenticate 195referrer referrer 196refresh refresh 197retry_after retry-after 198server server 199set_cookie set-cookie 200trailer trailer 201transfer_encoding transfer-encoding 202upgrade upgrade 203vary vary 204warning warning 205www_authenticate www-authenticate 206true_client_ip true-client-ip 207org_src_ip org-src-ip 208x_bluecoat_via x-bluecoat-via 209====================== ====================== 210 211In the ``custom`` option values from both columns can be used. The 212``HTTP Header`` column is case insensitive. 213 214DNS 215~~~ 216 217.. note:: As of Suricata 5.0, the version 2 format of the EVE DNS log 218 is the default. 219 220DNS records are logged as one entry for the request, and one entry for 221the response. 222 223YAML:: 224 225 - dns: 226 # As of Suricata 5.0, version 2 of the eve dns output 227 # format is the default. 228 #version: 2 229 230 # Enable/disable this logger. Default: enabled. 231 #enabled: yes 232 233 # Control logging of requests and responses: 234 # - requests: enable logging of DNS queries 235 # - responses: enable logging of DNS answers 236 # By default both requests and responses are logged. 237 #requests: no 238 #responses: no 239 240 # Format of answer logging: 241 # - detailed: array item per answer 242 # - grouped: answers aggregated by type 243 # Default: all 244 #formats: [detailed, grouped] 245 246 # Types to log, based on the query type. 247 # Default: all. 248 #types: [a, aaaa, cname, mx, ns, ptr, txt] 249 250DNS v1 Format 251~~~~~~~~~~~~~ 252 253The version 1 DNS output has been obsoleted by the version 2 output 254above. The v1 format logs a record per answer in the response possibly 255resulting in much more than 2 log records per request and response. 256 257YAML:: 258 259 - dns: 260 # Must set the version to 1 to get the old style format. 261 version: 1 262 # control logging of queries and answers 263 # default yes, no to disable 264 query: yes # enable logging of DNS queries 265 answer: yes # enable logging of DNS answers 266 # control which RR types are logged 267 # all enabled if custom not specified 268 #custom: [a, aaaa, cname, mx, ns, ptr, txt] 269 270To reduce verbosity the output can be filtered by supplying the record types 271to be logged under ``custom``. 272 273TLS 274~~~ 275 276TLS records are logged one record per session. 277 278YAML:: 279 280 - tls: 281 extended: yes # enable this for extended logging information 282 # custom allows to control which tls fields that are included 283 # in eve-log 284 #custom: [subject, issuer, serial, fingerprint, sni, version, not_before, not_after, certificate, chain, ja3, ja3s] 285 286The default is to log certificate subject and issuer. If ``extended`` is 287enabled, then the log gets more verbose. 288 289By using ``custom`` it is possible to select which TLS fields to log. 290 291Date modifiers in filename 292~~~~~~~~~~~~~~~~~~~~~~~~~~ 293 294It is possible to use date modifiers in the eve-log filename. 295 296:: 297 298 outputs: 299 - eve-log: 300 filename: eve-%s.json 301 302The example above adds epoch time to the filename. All the date modifiers from the 303C library should be supported. See the man page for ``strftime`` for all supported 304modifiers. 305 306.. _output_eve_rotate: 307 308Threaded file output 309~~~~~~~~~~~~~~~~~~~~ 310 311By default, all output is written to the named filename in the outputs section. The ``threaded`` option enables 312each output thread to write to individual files. In this case, the ``filename`` will include a unique identifier. 313 314With ``threaded`` enabled, the output will be split among many files -- and 315the aggregate of each file's contents must be treated together. 316 317:: 318 319 outputs: 320 - eve-log: 321 filename: eve.json 322 threaded: on 323 324This example will cause each Suricata thread to write to its own "eve.json" file. Filenames are constructed 325by adding a unique identifier to the filename. For example, ``eve.7.json``. 326 327 328Rotate log file 329~~~~~~~~~~~~~~~ 330 331Eve-log can be configured to rotate based on time. 332 333:: 334 335 outputs: 336 - eve-log: 337 filename: eve-%Y-%m-%d-%H:%M.json 338 rotate-interval: minute 339 340The example above creates a new log file each minute, where the filename contains 341a timestamp. Other supported ``rotate-interval`` values are ``hour`` and ``day``. 342 343In addition to this, it is also possible to specify the ``rotate-interval`` as a 344relative value. One example is to rotate the log file each X seconds. 345 346:: 347 348 outputs: 349 - eve-log: 350 filename: eve-%Y-%m-%d-%H:%M:%S.json 351 rotate-interval: 30s 352 353The example above rotates eve-log each 30 seconds. This could be replaced with 354``30m`` to rotate every 30 minutes, ``30h`` to rotate every 30 hours, ``30d`` 355to rotate every 30 days, or ``30w`` to rotate every 30 weeks. 356 357.. _multiple-eve-instances: 358 359Multiple Logger Instances 360~~~~~~~~~~~~~~~~~~~~~~~~~ 361 362It is possible to have multiple 'EVE' instances, for example the following is valid: 363 364:: 365 366 outputs: 367 - eve-log: 368 enabled: yes 369 type: file 370 filename: eve-ips.json 371 types: 372 - alert 373 - drop 374 375 - eve-log: 376 enabled: yes 377 type: file 378 filename: eve-nsm.json 379 types: 380 - http 381 - dns 382 - tls 383 384So here the alerts and drops go into 'eve-ips.json', while http, dns and tls go into 'eve-nsm.json'. 385 386With the exception of ``drop``, you can specify multiples of the same 387logger type, however, ``drop`` can only be used once. 388 389.. note:: The use of independent json loggers such as alert-json-log, 390 dns-json-log, etc. has been deprecated and will be removed 391 by June 2020. Please use multiple eve-log instances as 392 documented above instead. Please see the `deprecation 393 policy`_ for more information. 394 395File permissions 396~~~~~~~~~~~~~~~~ 397 398Log file permissions can be set individually for each logger. ``filemode`` can be used to 399control the permissions of a log file, e.g.: 400 401:: 402 403 outputs: 404 - eve-log: 405 enabled: yes 406 filename: eve.json 407 filemode: 600 408 409The example above sets the file permissions on ``eve.json`` to 600, which means that it is 410only readable and writable by the owner of the file. 411 412JSON flags 413~~~~~~~~~~ 414 415Several flags can be specified to control the JSON output in EVE: 416 417:: 418 419 outputs: 420 - eve-log: 421 json: 422 # Sort object keys in the same order as they were inserted 423 preserve-order: yes 424 425 # Make the output more compact 426 compact: yes 427 428 # Escape all unicode characters outside the ASCII range 429 ensure-ascii: yes 430 431 # Escape the '/' characters in string with '\/' 432 escape-slash: yes 433 434All these flags are enabled by default, and can be modified per EVE instance. 435 436Community Flow ID 437~~~~~~~~~~~~~~~~~ 438 439Often Suricata is used in combination with other tools like Bro/Zeek. Enabling 440the community-id option in the eve-log section adds a new ``community_id`` 441field to each output. 442 443Example:: 444 445 { 446 "timestamp": "2003-12-16T13:21:44.891921+0000", 447 "flow_id": 1332028388187153, 448 "pcap_cnt": 1, 449 "event_type": "alert", 450 ... 451 "community_id": "1:LQU9qZlK+B5F3KDmev6m5PMibrg=", 452 "alert": { 453 "action": "allowed", 454 "gid": 1, 455 "signature_id": 1, 456 }, 457 } 458 { 459 "timestamp": "2003-12-16T13:21:45.037333+0000", 460 "flow_id": 1332028388187153, 461 "event_type": "flow", 462 "flow": { 463 "pkts_toserver": 5, 464 "pkts_toclient": 4, 465 "bytes_toserver": 338, 466 "bytes_toclient": 272, 467 "start": "2003-12-16T13:21:44.891921+0000", 468 "end": "2003-12-16T13:21:45.346457+0000", 469 "age": 1, 470 "state": "closed", 471 "reason": "shutdown", 472 "alerted": true 473 }, 474 "community_id": "1:LQU9qZlK+B5F3KDmev6m5PMibrg=", 475 } 476 477Options 478""""""" 479 480The output can be enabled per instance of the EVE logger. 481 482The ``community-id`` option is boolean. If set to ``true`` it is enabled. 483The ``community-id-seed`` option specifies a unsigned 16 bit value that 484is used a seed to the hash that is calculated for the ``community-id`` 485output. This must be set to the same value on all tools that output this 486record. 487 488YAML:: 489 490 - eve-log: 491 # Community Flow ID 492 # Adds a 'community_id' field to EVE records. These are meant to give 493 # a records a predictable flow id that can be used to match records to 494 # output of other tools such as Bro. 495 # 496 # Takes a 'seed' that needs to be same across sensors and tools 497 # to make the id less predictable. 498 499 # enable/disable the community id feature. 500 community-id: false 501 # Seed value for the ID output. Valid values are 0-65535. 502 community-id-seed: 0 503 504 505.. _deprecation policy: https://suricata-ids.org/about/deprecation-policy/ 506