1=========== 2Napalm-logs 3=========== 4 5Python library to parse syslog messages from network devices and produce JSON 6serializable Python objects, in a vendor agnostic shape. The output objects are 7structured following the `OpenConfig <http://www.openconfig.net/>`_ or 8`IETF <https://github.com/YangModels/yang/tree/master/standard/ietf>`_ YANG models. 9 10For example, the following syslog message from a Juniper device: 11 12.. code-block:: text 13 14 <149>Jun 21 14:03:12 vmx01 rpd[2902]: BGP_PREFIX_THRESH_EXCEEDED: 192.168.140.254 (External AS 4230): Configured maximum prefix-limit threshold(140) exceeded for inet4-unicast nlri: 141 (instance master) 15 16Will produce the following object: 17 18.. code-block:: json 19 20 { 21 "yang_message": { 22 "bgp": { 23 "neighbors": { 24 "neighbor": { 25 "192.168.140.254": { 26 "state": { 27 "peer_as": "4230" 28 }, 29 "afi_safis": { 30 "afi_safi": { 31 "inet4": { 32 "state": { 33 "prefixes": { 34 "received": 141 35 } 36 }, 37 "ipv4_unicast": { 38 "prefix_limit": { 39 "state": { 40 "max_prefixes": 140 41 } 42 } 43 } 44 } 45 } 46 } 47 } 48 } 49 } 50 } 51 }, 52 "message_details": { 53 "processId": "2902", 54 "severity": 5, 55 "facility": 18, 56 "hostPrefix": null, 57 "pri": "149", 58 "processName": "rpd", 59 "host": "vmx01", 60 "tag": "BGP_PREFIX_THRESH_EXCEEDED", 61 "time": "14:03:12", 62 "date": "Jun 21", 63 "message": "192.168.140.254 (External AS 4230): Configured maximum prefix-limit threshold(140) exceeded for inet4-unicast nlri: 141 (instance master)" 64 }, 65 "timestamp": 1498053792, 66 "facility": 18, 67 "ip": "127.0.0.1", 68 "host": "vmx01", 69 "yang_model": "openconfig-bgp", 70 "error": "BGP_PREFIX_THRESH_EXCEEDED", 71 "os": "junos", 72 "severity": 5 73 } 74 75The library is provided with a command line program which acts as a daemon, 76running in background and listening to syslog messages continuously, then 77publishing them over secured channels, where multiple clients can subscribe. 78 79It is flexible to listen to the syslog messages via UDP or TCP, but also from 80brokers such as Apache Kafka. Similarly, the output objects can be published via 81various channels such as ZeroMQ, Kafka, or remote server logging. It is also 82pluggable enough to extend these capabilities and listen or publish to other 83services, depending on the needs. 84 85The messages are published over a secured channel, encrypted and signed. 86Although the security can be disabled, this is highly discouraged. 87 88Output data 89----------- 90 91The objects published by napalm-logs are structured data, with the hierarchy 92standardized in the OpenConfig and IETF models. To check what models are used for 93each message type, together with examples of raw syslog messages and sample 94output objects, please check the :ref:`messages` section. 95 96Install 97------- 98 99napalm-logs is available on PyPi and can easily be installed using the following 100command: 101 102.. code-block:: bash 103 104 $ pip install napalm-logs 105 106For advanced installation notes, see :ref:`installation`. 107 108How to use napalm-logs 109---------------------- 110 111Basic Configuration 112+++++++++++++++++++ 113 114Firstly you need to decide if you would like all messages between *napalm-logs* 115and the clients to be encrypted. If you do want them to be encrypted you will 116require a certificate and key, which you can generate using the following 117command: 118 119.. code-block:: bash 120 121 openssl req -nodes -x509 -newkey rsa:4096 -keyout /var/cache/napalm-logs.key -out /var/cache/napalm-logs.crt -days 365 122 123This will provide a self-signed certificate ``napalm-logs.crt`` and key 124``napalm-logs.key`` under the ``/var/cache`` directory. 125 126If you do not require the messages to be encrypted you can ignore the above 127step and just use the command line argument ``--disable-security`` when starting 128*napalm-logs*. 129 130Each of the other config options come with defaults, so you can now start 131*napalm-logs* with default options and your chosen security options. 132 133Starting napalm-logs 134++++++++++++++++++++ 135 136*Napalm-logs* will need to be run with root privileges if you want it to be able 137to listen on ``udp`` port ``514`` - the standard syslog port. If you need to 138run it via sudo and it has been installed in a virtual env, you will need to 139include the full path. In these examples I will run as root. 140 141To start napalm-logs using the crt and key generated above you should run the 142following command: 143 144.. code-block:: bash 145 146 napalm-logs --certificate /var/cache/napalm-logs.crt --keyfile /var/cache/napalm-logs.key 147 148This will start napalm-logs listening for incoming syslog messages on 149``0.0.0.0`` port ``514``. It will also start to listen for incoming client 150requests on ``0.0.0.0`` port ``49017``, and incoming authentication requests on 151``0.0.0.0`` port ``49018``. 152For more information on authentication please see the :ref:`authentication` 153section. 154 155Further Configuration 156+++++++++++++++++++++ 157 158It is possible to change the address and ports that napalm-logs will use, let's 159take a look at these options: 160 161.. code-block:: bash 162 163 -a ADDRESS, --address=ADDRESS 164 Listener address. Default: 0.0.0.0 165 -p PORT, --port=PORT Listener bind port. Default: 514 166 --publish-address=PUBLISH_ADDRESS 167 Publisher bind address. Default: 0.0.0.0 168 --publish-port=PUBLISH_PORT 169 Publisher bind port. Default: 49017 170 --auth-address=AUTH_ADDRESS 171 Authenticator bind address. Default: 0.0.0.0 172 --auth-port=AUTH_PORT 173 Authenticator bind port. Default: 49018 174 175There are several ``plugable`` parts to napalm-logs, two of which are the 176``listener`` and the ``publisher``. The listener is the part that ingests the 177incoming syslog messages, and the publisher is the part that outputs them to the 178client. 179 180You can chose which listener to use, and which publisher to use by using the 181following arguments: 182 183.. code-block:: bash 184 185 --listener=LISTENER Listener type. Default: udp 186 -t TRANSPORT, --transport=TRANSPORT 187 Publish transport. Default: zmq 188 189There are more configuration options, please see :ref:`configuration-options` 190for more details. 191 192Configuration file example 193++++++++++++++++++++++++++ 194 195The napalm-logs server can be started without any CLI aguments, as long as they 196are correctly specified under the configuration file. The default path of the 197configuration file is under ``/etc/napalm/logs``. To select a different 198filepath, we can use the ``-c`` option: 199 200.. code-block:: bash 201 202 napalm-logs -c /home/admin/napalm/logs 203 204The configuration file is formatted as YAML, which makes it more human readable. 205In general, any configuration option available on the CLI can be specified in 206the configuration file, with the mention that hyphen is replaced by underscore, 207e.g.: the CLI option ``auth-address`` becomes ``auth_address`` in the 208*napalm-logs* configuration file. 209 210.. code-block:: yaml 211 212 address: 172.17.17.1 213 port: 5514 214 publish_address: 172.17.17.2 215 publish_port: 49017 216 transport: zmq 217 listener: 218 kafka: 219 bootstrap_servers: 220 - 10.10.10.1 221 - 10.10.10.2 222 - 10.10.10.3 223 224The configuration above listens to the syslog messages from the Kafka bootstrap 225servers ``10.10.10.1``, ``10.10.10.2`` and ``10.10.10.3`` then publishes the 226structured objects encrypted and serialized via ZeroMQ, serving them at the 227address ``172.17.17.2``, port ``49017``. 228 229Check the complete list of configuration options under 230:ref:`configuration-options`. 231 232Starting a Client 233+++++++++++++++++ 234 235The client structure depends on how you start the napalm-logs daemon. If the 236security is disabled (via the CLI option ``--disable-security`` or through the 237configuration file, where the ``disable_security`` field is set as ``false``), 238the client script is as simple as: 239 240.. code-block:: python 241 242 #!/usr/bin/env python 243 244 import zmq 245 import napalm_logs.utils 246 247 server_address = '127.0.0.1' 248 server_port = 49017 249 250 context = zmq.Context() 251 socket = context.socket(zmq.SUB) 252 socket.connect('tcp://{address}:{port}'.format(address=server_address, 253 port=server_port)) 254 socket.setsockopt(zmq.SUBSCRIBE, '') 255 256 while True: 257 raw_object = socket.recv() 258 print(napalm_logs.utils.unserialize(raw_object)) 259 260Which subscribes to the ZeroMQ bus and deserializes messages using the 261``napalm_logs.utils.unserialise`` helper. The ``server_address`` and the 262``server_port`` of the client represent the ``--publish-address`` and the 263``--publish-port`` of the napalm-logs daemon. 264 265When the program is started with security enabled (**recommended**), the 266clients can use the ``napalm_logs.utils.ClientAuth`` class, which executes the 267handshake to retrieve the encryption key and hex of the verification key. This 268class requires the certificate (the same certificate specified when starting 269the napalm-logs daemon), as well as the authentication address and port 270(corresponding to the ``--auth-address`` and ``--auth-port`` CLI arguments or 271``auth_address`` and ``auth_port`` configuration fields sent to the napalm-logs 272daemon): 273 274.. code-block:: python 275 276 #!/usr/bin/env python 277 278 import napalm_logs.utils 279 import zmq 280 281 server_address = '127.0.0.1' 282 server_port = 49017 283 auth_address = '127.0.0.1' 284 auth_port = 49018 285 286 certificate = '/var/cache/napalm-logs.crt' # This is the server crt generated earlier 287 288 context = zmq.Context() 289 socket = context.socket(zmq.SUB) 290 socket.connect('tcp://{address}:{port}'.format(address=server_address, 291 port=server_port)) 292 socket.setsockopt(zmq.SUBSCRIBE, '') 293 294 auth = napalm_logs.utils.ClientAuth(certificate, 295 address=auth_address, 296 port=auth_port) 297 298 while True: 299 raw_object = socket.recv() 300 decrypted = auth.decrypt(raw_object) 301 print(decrypted) 302 303 304.. toctree:: 305 :maxdepth: 1 306 307 installation/index 308 docker/index 309 device_config/index 310 options/index 311 clients/index 312 messages/index 313 metrics/index 314 authentication/index 315 listener/index 316 publisher/index 317 serializer/index 318 logger/index 319 buffer/index 320 syslog/index 321 developers/index 322 releases/index 323