1# Distributed Monitoring with Master, Satellites and Agents <a id="distributed-monitoring"></a> 2 3This chapter will guide you through the setup of a distributed monitoring 4environment, including high-availability clustering and setup details 5for Icinga masters, satellites and agents. 6 7## Roles: Master, Satellites and Agents <a id="distributed-monitoring-roles"></a> 8 9Icinga 2 nodes can be given names for easier understanding: 10 11* A `master` node which is on top of the hierarchy. 12* A `satellite` node which is a child of a `satellite` or `master` node. 13* An `agent` node which is connected to `master` and/or `satellite` nodes. 14 15![Icinga 2 Distributed Roles](images/distributed-monitoring/icinga2_distributed_monitoring_roles.png) 16 17Rephrasing this picture into more details: 18 19* A `master` node has no parent node. 20 * A `master`node is where you usually install Icinga Web 2. 21 * A `master` node can combine executed checks from child nodes into backends and notifications. 22* A `satellite` node has a parent and a child node. 23 * A `satellite` node may execute checks on its own or delegate check execution to child nodes. 24 * A `satellite` node can receive configuration for hosts/services, etc. from the parent node. 25 * A `satellite` node continues to run even if the master node is temporarily unavailable. 26* An `agent` node only has a parent node. 27 * An `agent` node will either run its own configured checks or receive command execution events from the parent node. 28 29A client can be a secondary master, a satellite or an agent. It 30typically requests something from the primary master or parent node. 31 32The following sections will refer to these roles and explain the 33differences and the possibilities this kind of setup offers. 34 35> **Note** 36> 37> Previous versions of this documentation used the term `Icinga client`. 38> This has been refined into `Icinga agent` and is visible in the docs, 39> backends and web interfaces. 40 41**Tip**: If you just want to install a single master node that monitors several hosts 42(i.e. Icinga agents), continue reading -- we'll start with 43simple examples. 44In case you are planning a huge cluster setup with multiple levels and 45lots of satellites and agents, read on -- we'll deal with these cases later on. 46 47The installation on each system is the same: You need to install the 48[Icinga 2 package](02-installation.md#setting-up-icinga2) and the required [plugins](02-installation.md#setting-up-check-plugins). 49 50The required configuration steps are mostly happening 51on the command line. You can also [automate the setup](06-distributed-monitoring.md#distributed-monitoring-automation). 52 53The first thing you need learn about a distributed setup is the hierarchy of the single components. 54 55## Zones <a id="distributed-monitoring-zones"></a> 56 57The Icinga 2 hierarchy consists of so-called [zone](09-object-types.md#objecttype-zone) objects. 58Zones depend on a parent-child relationship in order to trust each other. 59 60![Icinga 2 Distributed Zones](images/distributed-monitoring/icinga2_distributed_monitoring_zones.png) 61 62Have a look at this example for the `satellite` zones which have the `master` zone as a parent zone: 63 64``` 65object Zone "master" { 66 //... 67} 68 69object Zone "satellite region 1" { 70 parent = "master" 71 //... 72} 73 74object Zone "satellite region 2" { 75 parent = "master" 76 //... 77} 78``` 79 80There are certain limitations for child zones, e.g. their members are not allowed 81to send configuration commands to the parent zone members. Vice versa, the 82trust hierarchy allows for example the `master` zone to send 83configuration files to the `satellite` zone. Read more about this 84in the [security section](06-distributed-monitoring.md#distributed-monitoring-security). 85 86`agent` nodes also have their own unique zone. By convention you 87must use the FQDN for the zone name. 88 89## Endpoints <a id="distributed-monitoring-endpoints"></a> 90 91Nodes which are a member of a zone are so-called [Endpoint](09-object-types.md#objecttype-endpoint) objects. 92 93![Icinga 2 Distributed Endpoints](images/distributed-monitoring/icinga2_distributed_monitoring_endpoints.png) 94 95Here is an example configuration for two endpoints in different zones: 96 97``` 98object Endpoint "icinga2-master1.localdomain" { 99 host = "192.168.56.101" 100} 101 102object Endpoint "icinga2-satellite1.localdomain" { 103 host = "192.168.56.105" 104} 105 106object Zone "master" { 107 endpoints = [ "icinga2-master1.localdomain" ] 108} 109 110object Zone "satellite" { 111 endpoints = [ "icinga2-satellite1.localdomain" ] 112 parent = "master" 113} 114``` 115 116All endpoints in the same zone work as high-availability setup. For 117example, if you have two nodes in the `master` zone, they will load-balance the check execution. 118 119Endpoint objects are important for specifying the connection 120information, e.g. if the master should actively try to connect to an agent. 121 122The zone membership is defined inside the `Zone` object definition using 123the `endpoints` attribute with an array of `Endpoint` names. 124 125> **Note** 126> 127> There is a known [problem](https://github.com/Icinga/icinga2/issues/3533) 128> with >2 endpoints in a zone and a message routing loop. 129> The config validation will log a warning to let you know about this too. 130 131If you want to check the availability (e.g. ping checks) of the node 132you still need a [Host](09-object-types.md#objecttype-host) object. 133 134## ApiListener <a id="distributed-monitoring-apilistener"></a> 135 136In case you are using the CLI commands later, you don't have to write 137this configuration from scratch in a text editor. 138The [ApiListener](09-object-types.md#objecttype-apilistener) object is 139used to load the TLS certificates and specify restrictions, e.g. 140for accepting configuration commands. 141 142It is also used for the [Icinga 2 REST API](12-icinga2-api.md#icinga2-api) which shares 143the same host and port with the Icinga 2 Cluster protocol. 144 145The object configuration is stored in the `/etc/icinga2/features-enabled/api.conf` 146file. Depending on the configuration mode the attributes `accept_commands` 147and `accept_config` can be configured here. 148 149In order to use the `api` feature you need to enable it and restart Icinga 2. 150 151```bash 152icinga2 feature enable api 153``` 154 155## Conventions <a id="distributed-monitoring-conventions"></a> 156 157By convention all nodes should be configured using their FQDN. 158 159Furthermore, you must ensure that the following names 160are exactly the same in all configuration files: 161 162* Host certificate common name (CN). 163* Endpoint configuration object for the host. 164* NodeName constant for the local host. 165 166Setting this up on the command line will help you to minimize the effort. 167Just keep in mind that you need to use the FQDN for endpoints and for 168common names when asked. 169 170## Security <a id="distributed-monitoring-security"></a> 171 172While there are certain mechanisms to ensure a secure communication between all 173nodes (firewalls, policies, software hardening, etc.), Icinga 2 also provides 174additional security: 175 176* TLS v1.2+ is required. 177* TLS cipher lists are hardened [by default](09-object-types.md#objecttype-apilistener). 178* TLS certificates are mandatory for communication between nodes. The CLI command wizards 179help you create these certificates. 180* Child zones only receive updates (check results, commands, etc.) for their configured objects. 181* Child zones are not allowed to push configuration updates to parent zones. 182* Zones cannot interfere with other zones and influence each other. Each checkable host or service object is assigned to **one zone** only. 183* All nodes in a zone trust each other. 184* [Config sync](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync) and [remote command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) is disabled by default. 185 186The underlying protocol uses JSON-RPC event notifications exchanged by nodes. 187The connection is secured by TLS. The message protocol uses an internal API, 188and as such message types and names may change internally and are not documented. 189 190Zones build the trust relationship in a distributed environment. If you do not specify 191a zone for an agent/satellite and specify the parent zone, its zone members e.g. the master instance 192won't trust the agent/satellite. 193 194Building this trust is key in your distributed environment. That way the parent node 195knows that it is able to send messages to the child zone, e.g. configuration objects, 196configuration in global zones, commands to be executed in this zone/for this endpoint. 197It also receives check results from the child zone for checkable objects (host/service). 198 199Vice versa, the agent/satellite trusts the master and accepts configuration and commands if enabled 200in the api feature. If the agent/satellite would send configuration to the parent zone, the parent nodes 201will deny it. The parent zone is the configuration entity, and does not trust agents/satellites in this matter. 202An agent/satellite could attempt to modify a different agent/satellite for example, or inject a check command 203with malicious code. 204 205While it may sound complicated for agent/satellite setups, it removes the problem with different roles 206and configurations for a master and child nodes. Both of them work the same way, are configured 207in the same way (Zone, Endpoint, ApiListener), and you can troubleshoot and debug them in just one go. 208 209## Versions and Upgrade <a id="distributed-monitoring-versions-upgrade"></a> 210 211It generally is advised to use the newest releases with the same version on all instances. 212Prior to upgrading, make sure to plan a maintenance window. 213 214The Icinga project aims to allow the following compatibility: 215 216``` 217master (2.11) >= satellite (2.10) >= agent (2.9) 218``` 219 220Older agent versions may work, but there's no guarantee. Always keep in mind that 221older versions are out of support and can contain bugs. 222 223In terms of an upgrade, ensure that the master is upgraded first, then 224involved satellites, and last the Icinga agents. If you are on v2.10 225currently, first upgrade the master instance(s) to 2.11, and then proceed 226with the satellites. Things are getting easier with any sort of automation 227tool (Puppet, Ansible, etc.). 228 229Releases and new features may require you to upgrade master/satellite instances at once, 230this is highlighted in the [upgrading docs](16-upgrading-icinga-2.md#upgrading-icinga-2) if needed. 231One example is the CA Proxy and on-demand signing feature 232available since v2.8 where all involved instances need this version 233to function properly. 234 235## Master Setup <a id="distributed-monitoring-setup-master"></a> 236 237This section explains how to install a central single master node using 238the `node wizard` command. If you prefer to do an automated installation, please 239refer to the [automated setup](06-distributed-monitoring.md#distributed-monitoring-automation) section. 240 241Install the [Icinga 2 package](02-installation.md#setting-up-icinga2) and setup 242the required [plugins](02-installation.md#setting-up-check-plugins) if you haven't done 243so already. 244 245**Note**: Windows is not supported for a master node setup. 246 247The next step is to run the `node wizard` CLI command. Prior to that 248ensure to collect the required information: 249 250 Parameter | Description 251 --------------------|-------------------- 252 Common name (CN) | **Required.** By convention this should be the host's FQDN. Defaults to the FQDN. 253 Master zone name | **Optional.** Allows to specify the master zone name. Defaults to `master`. 254 Global zones | **Optional.** Allows to specify more global zones in addition to `global-templates` and `director-global`. Defaults to `n`. 255 API bind host | **Optional.** Allows to specify the address the ApiListener is bound to. For advanced usage only. 256 API bind port | **Optional.** Allows to specify the port the ApiListener is bound to. For advanced usage only (requires changing the default port 5665 everywhere). 257 Disable conf.d | **Optional.** Allows to disable the `include_recursive "conf.d"` directive except for the `api-users.conf` file in the `icinga2.conf` file. Defaults to `y`. Configuration on the master is discussed below. 258 259The setup wizard will ensure that the following steps are taken: 260 261* Enable the `api` feature. 262* Generate a new certificate authority (CA) in `/var/lib/icinga2/ca` if it doesn't exist. 263* Create a certificate for this node signed by the CA key. 264* Update the [zones.conf](04-configuration.md#zones-conf) file with the new zone hierarchy. 265* Update the [ApiListener](06-distributed-monitoring.md#distributed-monitoring-apilistener) and [constants](04-configuration.md#constants-conf) configuration. 266* Update the [icinga2.conf](04-configuration.md#icinga2-conf) to disable the `conf.d` inclusion, and add the `api-users.conf` file inclusion. 267 268Here is an example of a master setup for the `icinga2-master1.localdomain` node on CentOS 7: 269 270``` 271[root@icinga2-master1.localdomain /]# icinga2 node wizard 272 273Welcome to the Icinga 2 Setup Wizard! 274 275We will guide you through all required configuration details. 276 277Please specify if this is a satellite/agent setup ('n' installs a master setup) [Y/n]: n 278 279Starting the Master setup routine... 280 281Please specify the common name (CN) [icinga2-master1.localdomain]: icinga2-master1.localdomain 282Reconfiguring Icinga... 283Checking for existing certificates for common name 'icinga2-master1.localdomain'... 284Certificates not yet generated. Running 'api setup' now. 285Generating master configuration for Icinga 2. 286Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect. 287 288Master zone name [master]: 289 290Default global zones: global-templates director-global 291Do you want to specify additional global zones? [y/N]: N 292 293Please specify the API bind host/port (optional): 294Bind Host []: 295Bind Port []: 296 297Do you want to disable the inclusion of the conf.d directory [Y/n]: 298Disabling the inclusion of the conf.d directory... 299Checking if the api-users.conf file exists... 300 301Done. 302 303Now restart your Icinga 2 daemon to finish the installation! 304``` 305 306You can verify that the CA public and private keys are stored in the `/var/lib/icinga2/ca` directory. 307Keep this path secure and include it in your [backups](02-installation.md#install-backup). 308 309In case you lose the CA private key you have to generate a new CA for signing new agent/satellite 310certificate requests. You then have to also re-create new signed certificates for all 311existing nodes. 312 313Once the master setup is complete, you can also use this node as primary [CSR auto-signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing) 314master. The following section will explain how to use the CLI commands in order to fetch their 315signed certificate from this master node. 316 317## Signing Certificates on the Master <a id="distributed-monitoring-setup-sign-certificates-master"></a> 318 319All certificates must be signed by the same certificate authority (CA). This ensures 320that all nodes trust each other in a distributed monitoring environment. 321 322This CA is generated during the [master setup](06-distributed-monitoring.md#distributed-monitoring-setup-master) 323and should be the same on all master instances. 324 325You can avoid signing and deploying certificates [manually](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-certificates-manual) 326by using built-in methods for auto-signing certificate signing requests (CSR): 327 328* [CSR Auto-Signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing) which uses a client (an agent or a satellite) ticket generated on the master as trust identifier. 329* [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) which allows to sign pending certificate requests on the master. 330 331Both methods are described in detail below. 332 333> **Note** 334> 335> [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) is available in Icinga 2 v2.8+. 336 337### CSR Auto-Signing <a id="distributed-monitoring-setup-csr-auto-signing"></a> 338 339A client can be a secondary master, a satellite or an agent. It sends a certificate signing request (CSR) 340and must authenticate itself in a trusted way. The master generates a client ticket which is included in this request. 341That way the master can verify that the request matches the previously trusted ticket 342and sign the request. 343 344> **Note** 345> 346> Icinga 2 v2.8 added the possibility to forward signing requests on a satellite 347> to the master node. This is called `CA Proxy` in blog posts and design drafts. 348> This functionality helps with the setup of [three level clusters](06-distributed-monitoring.md#distributed-monitoring-scenarios-master-satellite-agents) 349> and more. 350 351Advantages: 352 353* Nodes (secondary master, satellites, agents) can be installed by different users who have received the client ticket. 354* No manual interaction necessary on the master node. 355* Automation tools like Puppet, Ansible, etc. can retrieve the pre-generated ticket in their client catalog 356and run the node setup directly. 357 358Disadvantages: 359 360* Tickets need to be generated on the master and copied to client setup wizards. 361* No central signing management. 362 363#### CSR Auto-Signing: Preparation <a id="distributed-monitoring-setup-csr-auto-signing-preparation"></a> 364 365Prior to using this mode, ensure that the following steps are taken on 366the signing master: 367 368* The [master setup](06-distributed-monitoring.md#distributed-monitoring-setup-master) was run successfully. This includes: 369 * Generated a CA key pair 370 * Generated a private ticket salt stored in the `TicketSalt` constant, set as `ticket_salt` attribute inside the [api](09-object-types.md#objecttype-apilistener) feature. 371* Restart of the master instance. 372 373#### CSR Auto-Signing: On the master <a id="distributed-monitoring-setup-csr-auto-signing-master"></a> 374 375Setup wizards for agent/satellite nodes will ask you for this specific client ticket. 376 377There are two possible ways to retrieve the ticket: 378 379* [CLI command](11-cli-commands.md#cli-command-pki) executed on the master node. 380* [REST API](12-icinga2-api.md#icinga2-api) request against the master node. 381 382 383Required information: 384 385 Parameter | Description 386 --------------------|-------------------- 387 Common name (CN) | **Required.** The common name for the agent/satellite. By convention this should be the FQDN. 388 389The following example shows how to generate a ticket on the master node `icinga2-master1.localdomain` for the agent `icinga2-agent1.localdomain`: 390 391``` 392[root@icinga2-master1.localdomain /]# icinga2 pki ticket --cn icinga2-agent1.localdomain 393``` 394 395Querying the [Icinga 2 API](12-icinga2-api.md#icinga2-api) on the master requires an [ApiUser](12-icinga2-api.md#icinga2-api-authentication) 396object with at least the `actions/generate-ticket` permission. 397 398``` 399[root@icinga2-master1.localdomain /]# vim /etc/icinga2/conf.d/api-users.conf 400 401object ApiUser "client-pki-ticket" { 402 password = "bea11beb7b810ea9ce6ea" //change this 403 permissions = [ "actions/generate-ticket" ] 404} 405 406[root@icinga2-master1.localdomain /]# systemctl restart icinga2 407 408Retrieve the ticket on the master node `icinga2-master1.localdomain` with `curl`, for example: 409 410 [root@icinga2-master1.localdomain /]# curl -k -s -u client-pki-ticket:bea11beb7b810ea9ce6ea -H 'Accept: application/json' \ 411 -X POST 'https://localhost:5665/v1/actions/generate-ticket' -d '{ "cn": "icinga2-agent1.localdomain" }' 412``` 413 414Store that ticket number for the [agent/satellite setup](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite) below. 415 416> **Note** 417> 418> Never expose the ticket salt and/or ApiUser credentials to your client nodes. 419> Example: Retrieve the ticket on the Puppet master node and send the compiled catalog 420> to the authorized Puppet agent node which will invoke the 421> [automated setup steps](06-distributed-monitoring.md#distributed-monitoring-automation-cli-node-setup). 422 423 424### On-Demand CSR Signing <a id="distributed-monitoring-setup-on-demand-csr-signing"></a> 425 426The client can be a secondary master, satellite or agent. 427It sends a certificate signing request to specified parent node without any 428ticket. The admin on the primary master is responsible for reviewing and signing the requests 429with the private CA key. 430 431This could either be directly the master, or a satellite which forwards the request 432to the signing master. 433 434Advantages: 435 436* Central certificate request signing management. 437* No pre-generated ticket is required for client setups. 438 439Disadvantages: 440 441* Asynchronous step for automated deployments. 442* Needs client verification on the master. 443 444#### On-Demand CSR Signing: Preparation <a id="distributed-monitoring-setup-on-demand-csr-signing-preparation"></a> 445 446Prior to using this mode, ensure that the following steps are taken on 447the signing master: 448 449* The [master setup](06-distributed-monitoring.md#distributed-monitoring-setup-master) was run successfully. This includes: 450 * Generated a CA key pair 451* Restart of the master instance. 452 453#### On-Demand CSR Signing: On the master <a id="distributed-monitoring-setup-on-demand-csr-signing-master"></a> 454 455You can list pending certificate signing requests with the `ca list` CLI command. 456 457``` 458[root@icinga2-master1.localdomain /]# icinga2 ca list 459Fingerprint | Timestamp | Signed | Subject 460-----------------------------------------------------------------|---------------------|--------|-------- 46171700c28445109416dd7102038962ac3fd421fbb349a6e7303b6033ec1772850 | 2017/09/06 17:20:02 | | CN = icinga2-agent2.localdomain 462``` 463 464In order to show all requests, use the `--all` parameter. 465 466``` 467[root@icinga2-master1.localdomain /]# icinga2 ca list --all 468Fingerprint | Timestamp | Signed | Subject 469-----------------------------------------------------------------|---------------------|--------|-------- 470403da5b228df384f07f980f45ba50202529cded7c8182abf96740660caa09727 | 2017/09/06 17:02:40 | * | CN = icinga2-agent1.localdomain 47171700c28445109416dd7102038962ac3fd421fbb349a6e7303b6033ec1772850 | 2017/09/06 17:20:02 | | CN = icinga2-agent2.localdomain 472``` 473 474**Tip**: Add `--json` to the CLI command to retrieve the details in JSON format. 475 476If you want to sign a specific request, you need to use the `ca sign` CLI command 477and pass its fingerprint as argument. 478 479``` 480[root@icinga2-master1.localdomain /]# icinga2 ca sign 71700c28445109416dd7102038962ac3fd421fbb349a6e7303b6033ec1772850 481information/cli: Signed certificate for 'CN = icinga2-agent2.localdomain'. 482``` 483 484> **Note** 485> 486> `ca list` cannot be used as historical inventory. Certificate 487> signing requests older than 1 week are automatically deleted. 488 489You can also remove an undesired CSR using the `ca remove` command using the 490syntax as the `ca sign` command. 491 492``` 493[root@pym ~]# icinga2 ca remove 5c31ca0e2269c10363a97e40e3f2b2cd56493f9194d5b1852541b835970da46e 494information/cli: Certificate 5c31ca0e2269c10363a97e40e3f2b2cd56493f9194d5b1852541b835970da46e removed. 495``` 496If you want to restore a certificate you have removed, you can use `ca restore`. 497 498<!-- Keep this for compatibility --> 499<a id="distributed-monitoring-setup-satellite-client"></a> 500 501## Agent/Satellite Setup <a id="distributed-monitoring-setup-agent-satellite"></a> 502 503This section describes the setup of an agent or satellite connected to an 504existing master node setup. If you haven't done so already, please [run the master setup](06-distributed-monitoring.md#distributed-monitoring-setup-master). 505 506Icinga 2 on the master node must be running and accepting connections on port `5665`. 507 508<!-- Keep this for compatibility --> 509<a id="distributed-monitoring-setup-client-linux"></a> 510 511### Agent/Satellite Setup on Linux <a id="distributed-monitoring-setup-agent-satellite-linux"></a> 512 513Please ensure that you've run all the steps mentioned in the [agent/satellite section](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite). 514 515Install the [Icinga 2 package](02-installation.md#setting-up-icinga2) and setup 516the required [plugins](02-installation.md#setting-up-check-plugins) if you haven't done 517so already. 518 519The next step is to run the `node wizard` CLI command. 520 521In this example we're generating a ticket on the master node `icinga2-master1.localdomain` for the agent `icinga2-agent1.localdomain`: 522 523``` 524[root@icinga2-master1.localdomain /]# icinga2 pki ticket --cn icinga2-agent1.localdomain 5254f75d2ecd253575fe9180938ebff7cbca262f96e 526``` 527 528Note: You don't need this step if you have chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing). 529 530Start the wizard on the agent `icinga2-agent1.localdomain`: 531 532``` 533[root@icinga2-agent1.localdomain /]# icinga2 node wizard 534 535Welcome to the Icinga 2 Setup Wizard! 536 537We will guide you through all required configuration details. 538``` 539 540Press `Enter` or add `y` to start a satellite or agent setup. 541 542``` 543Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]: 544``` 545 546Press `Enter` to use the proposed name in brackets, or add a specific common name (CN). By convention 547this should be the FQDN. 548 549``` 550Starting the Agent/Satellite setup routine... 551 552Please specify the common name (CN) [icinga2-agent1.localdomain]: icinga2-agent1.localdomain 553``` 554 555Specify the direct parent for this node. This could be your primary master `icinga2-master1.localdomain` 556or a satellite node in a multi level cluster scenario. 557 558``` 559Please specify the parent endpoint(s) (master or satellite) where this node should connect to: 560Master/Satellite Common Name (CN from your master/satellite node): icinga2-master1.localdomain 561``` 562 563Press `Enter` or choose `y` to establish a connection to the parent node. 564 565``` 566Do you want to establish a connection to the parent node from this node? [Y/n]: 567``` 568 569> **Note:** 570> 571> If this node cannot connect to the parent node, choose `n`. The setup 572> wizard will provide instructions for this scenario -- signing questions are disabled then. 573 574Add the connection details for `icinga2-master1.localdomain`. 575 576``` 577Please specify the master/satellite connection information: 578Master/Satellite endpoint host (IP address or FQDN): 192.168.56.101 579Master/Satellite endpoint port [5665]: 5665 580``` 581 582You can add more parent nodes if necessary. Press `Enter` or choose `n` 583if you don't want to add any. This comes in handy if you have more than one 584parent node, e.g. two masters or two satellites. 585 586``` 587Add more master/satellite endpoints? [y/N]: 588``` 589 590Verify the parent node's certificate: 591 592``` 593Parent certificate information: 594 595 Subject: CN = icinga2-master1.localdomain 596 Issuer: CN = Icinga CA 597 Valid From: Sep 7 13:41:24 2017 GMT 598 Valid Until: Sep 3 13:41:24 2032 GMT 599 Fingerprint: AC 99 8B 2B 3D B0 01 00 E5 21 FA 05 2E EC D5 A9 EF 9E AA E3 600 601Is this information correct? [y/N]: y 602``` 603 604The setup wizard fetches the parent node's certificate and ask 605you to verify this information. This is to prevent MITM attacks or 606any kind of untrusted parent relationship. 607 608You can verify the fingerprint by running the following command on the node to connect to: 609 610```bash 611openssl x509 -noout -fingerprint -sha256 -in \ 612 "/var/lib/icinga2/certs/$(hostname --fqdn).crt" 613``` 614 615Note: The certificate is not fetched if you have chosen not to connect 616to the parent node. 617 618Proceed with adding the optional client ticket for [CSR auto-signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing): 619 620``` 621Please specify the request ticket generated on your Icinga 2 master (optional). 622 (Hint: # icinga2 pki ticket --cn 'icinga2-agent1.localdomain'): 6234f75d2ecd253575fe9180938ebff7cbca262f96e 624``` 625 626In case you've chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) 627you can leave the ticket question blank. 628 629Instead, Icinga 2 tells you to approve the request later on the master node. 630 631``` 632No ticket was specified. Please approve the certificate signing request manually 633on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details). 634``` 635 636You can optionally specify a different bind host and/or port. 637 638``` 639Please specify the API bind host/port (optional): 640Bind Host []: 641Bind Port []: 642``` 643 644The next step asks you to accept configuration (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)) 645and commands (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). 646 647``` 648Accept config from parent node? [y/N]: y 649Accept commands from parent node? [y/N]: y 650``` 651 652Next you can optionally specify the local and parent zone names. This will be reflected 653in the generated zone configuration file. 654 655Set the local zone name to something else, if you are installing a satellite or secondary master instance. 656 657``` 658Local zone name [icinga2-agent1.localdomain]: 659``` 660 661Set the parent zone name to something else than `master` if this agents connects to a satellite instance instead of the master. 662 663``` 664Parent zone name [master]: 665``` 666 667You can add more global zones in addition to `global-templates` and `director-global` if necessary. 668Press `Enter` or choose `n`, if you don't want to add any additional. 669 670``` 671Reconfiguring Icinga... 672 673Default global zones: global-templates director-global 674Do you want to specify additional global zones? [y/N]: N 675``` 676 677Last but not least the wizard asks you whether you want to disable the inclusion of the local configuration 678directory in `conf.d`, or not. Defaults to disabled, as agents either are checked via command endpoint, or 679they receive configuration synced from the parent zone. 680 681``` 682Do you want to disable the inclusion of the conf.d directory [Y/n]: Y 683Disabling the inclusion of the conf.d directory... 684``` 685 686 687The wizard proceeds and you are good to go. 688 689``` 690Done. 691 692Now restart your Icinga 2 daemon to finish the installation! 693``` 694 695> **Note** 696> 697> If you have chosen not to connect to the parent node, you cannot start 698> Icinga 2 yet. The wizard asked you to manually copy the master's public 699> CA certificate file into `/var/lib/icinga2/certs/ca.crt`. 700> 701> You need to [manually sign the CSR on the master node](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing-master). 702 703Restart Icinga 2 as requested. 704 705``` 706[root@icinga2-agent1.localdomain /]# systemctl restart icinga2 707``` 708 709Here is an overview of all parameters in detail: 710 711 Parameter | Description 712 --------------------|-------------------- 713 Common name (CN) | **Required.** By convention this should be the host's FQDN. Defaults to the FQDN. 714 Master common name | **Required.** Use the common name you've specified for your master node before. 715 Establish connection to the parent node | **Optional.** Whether the node should attempt to connect to the parent node or not. Defaults to `y`. 716 Master/Satellite endpoint host | **Required if the the agent needs to connect to the master/satellite.** The parent endpoint's IP address or FQDN. This information is included in the `Endpoint` object configuration in the `zones.conf` file. 717 Master/Satellite endpoint port | **Optional if the the agent needs to connect to the master/satellite.** The parent endpoints's listening port. This information is included in the `Endpoint` object configuration. 718 Add more master/satellite endpoints | **Optional.** If you have multiple master/satellite nodes configured, add them here. 719 Parent Certificate information | **Required.** Verify that the connecting host really is the requested master node. 720 Request ticket | **Optional.** Add the [ticket](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing) generated on the master. 721 API bind host | **Optional.** Allows to specify the address the ApiListener is bound to. For advanced usage only. 722 API bind port | **Optional.** Allows to specify the port the ApiListener is bound to. For advanced usage only (requires changing the default port 5665 everywhere). 723 Accept config | **Optional.** Whether this node accepts configuration sync from the master node (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this defaults to `n`. 724 Accept commands | **Optional.** Whether this node accepts command execution messages from the master node (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this defaults to `n`. 725 Local zone name | **Optional.** Allows to specify the name for the local zone. This comes in handy when this instance is a satellite, not an agent. Defaults to the FQDN. 726 Parent zone name | **Optional.** Allows to specify the name for the parent zone. This is important if the agent has a satellite instance as parent, not the master. Defaults to `master`. 727 Global zones | **Optional.** Allows to specify more global zones in addition to `global-templates` and `director-global`. Defaults to `n`. 728 Disable conf.d | **Optional.** Allows to disable the inclusion of the `conf.d` directory which holds local example configuration. Clients should retrieve their configuration from the parent node, or act as command endpoint execution bridge. Defaults to `y`. 729 730The setup wizard will ensure that the following steps are taken: 731 732* Enable the `api` feature. 733* Create a certificate signing request (CSR) for the local node. 734* Request a signed certificate (optional with the provided ticket number) on the master node. 735* Allow to verify the parent node's certificate. 736* Store the signed agent/satellite certificate and ca.crt in `/var/lib/icinga2/certs`. 737* Update the `zones.conf` file with the new zone hierarchy. 738* Update `/etc/icinga2/features-enabled/api.conf` (`accept_config`, `accept_commands`) and `constants.conf`. 739* Update `/etc/icinga2/icinga2.conf` and comment out `include_recursive "conf.d"`. 740 741You can verify that the certificate files are stored in the `/var/lib/icinga2/certs` directory. 742 743> **Note** 744> 745> If the agent is not directly connected to the certificate signing master, 746> signing requests and responses might need some minutes to fully update the agent certificates. 747> 748> If you have chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) 749> certificates need to be signed on the master first. Ticket-less setups require at least Icinga 2 v2.8+ on all involved instances. 750 751Now that you've successfully installed a Linux/Unix agent/satellite instance, please proceed to 752the [configuration modes](06-distributed-monitoring.md#distributed-monitoring-configuration-modes). 753 754 755<!-- Keep this for compatibility --> 756<a id="distributed-monitoring-setup-client-windows"></a> 757 758### Agent Setup on Windows <a id="distributed-monitoring-setup-agent-windows"></a> 759 760The supported Windows agent versions are listed [here](https://icinga.com/subscription/support-details/). 761 762Requirements: 763 764* [Microsoft .NET Framework 4.6](https://www.microsoft.com/en-US/download/details.aspx?id=53344) or higher. This is the default on Windows Server 2016 or later. 765* [Universal C Runtime for Windows](https://support.microsoft.com/en-us/help/2999226/update-for-universal-c-runtime-in-windows) for Windows Server 2012 and older. 766 767#### Agent Setup on Windows: Installer <a id="distributed-monitoring-setup-agent-windows-installer"></a> 768 769Download the MSI-Installer package from [https://packages.icinga.com/windows/](https://packages.icinga.com/windows/). 770The preferred flavor is `x86_64` for modern Windows systems. 771 772The Windows package provides native [monitoring plugin binaries](06-distributed-monitoring.md#distributed-monitoring-windows-plugins) 773to get you started more easily. 774The installer package also includes the [NSClient++](https://www.nsclient.org/) package 775to allow using its built-in plugins. You can find more details in 776[this chapter](06-distributed-monitoring.md#distributed-monitoring-windows-nscp). 777 778> **Note** 779> 780> Please note that Icinga 2 was designed to run as light-weight agent on Windows. 781> There is no support for satellite instances. 782 783Run the MSI-Installer package and follow the instructions shown in the screenshots. 784 785![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_01.png) 786![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_02.png) 787![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_03.png) 788![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_04.png) 789![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_installer_05.png) 790 791The graphical installer offers to run the [Icinga Agent setup wizard](06-distributed-monitoring.md#distributed-monitoring-setup-agent-windows-configuration-wizard) 792after the installation. Select the check box to proceed. 793 794> **Tip** 795> 796> You can also run the Icinga agent setup wizard from the Start menu later. 797 798#### Agent Setup on Windows: Configuration Wizard <a id="distributed-monitoring-setup-agent-windows-configuration-wizard"></a> 799 800On a fresh installation the setup wizard guides you through the initial configuration. 801It also provides a mechanism to send a certificate request to the [CSR signing master](06-distributed-monitoring.md#distributed-monitoring-setup-sign-certificates-master). 802 803The following configuration details are required: 804 805 Parameter | Description 806 --------------------|-------------------- 807 Instance name | **Required.** By convention this should be the host's FQDN. Defaults to the FQDN. 808 Setup ticket | **Optional.** Paste the previously generated [ticket number](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing). If left blank, the certificate request must be [signed on the master node](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing). 809 810Fill in the required information and click `Add` to add a new master connection. 811 812![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_01.png) 813 814Add the following details: 815 816 Parameter | Description 817 -------------------------------|------------------------------- 818 Instance name | **Required.** The master/satellite endpoint name where this agent is a direct child of. 819 Master/Satellite endpoint host | **Required.** The master or satellite's IP address or FQDN. This information is included in the `Endpoint` object configuration in the `zones.conf` file. 820 Master/Satellite endpoint port | **Optional.** The master or satellite's listening port. This information is included in the `Endpoint` object configuration. 821 822![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_02.png) 823 824When needed you can add an additional global zone (the zones `global-templates` and `director-global` are added by default): 825 826![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_02_global_zone.png) 827 828Optionally enable the following settings: 829 830 Parameter | Description 831 --------------------------------------------------------|---------------------------------- 832 Accept commands from master/satellite instance(s) | **Optional.** Whether this node accepts command execution messages from the master node (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this is disabled by default. 833 Accept config updates from master/satellite instance(s) | **Optional.** Whether this node accepts configuration sync from the master node (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)). For [security reasons](06-distributed-monitoring.md#distributed-monitoring-security) this is disabled by default. 834 Run Icinga 2 service as this user | **Optional.** Specify a different Windows user. This defaults to `NT AUTHORITY\Network Service` and is required for more privileged service checks. 835 Install/Update bundled NSClient++ | **Optional.** The Windows installer bundles the NSClient++ installer for additional [plugin checks](06-distributed-monitoring.md#distributed-monitoring-windows-nscp). 836 Disable including local 'conf.d' directory | **Optional.** Allows to disable the `include_recursive "conf.d"` directive except for the `api-users.conf` file in the `icinga2.conf` file. Defaults to `true`. 837 838![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_03.png) 839 840Verify the certificate from the master/satellite instance where this node should connect to. 841 842![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_04.png) 843 844 845#### Bundled NSClient++ Setup <a id="distributed-monitoring-setup-agent-windows-nsclient"></a> 846 847If you have chosen to install/update the NSClient++ package, the Icinga 2 setup wizard asks 848you to do so. 849 850![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_01.png) 851 852Choose the `Generic` setup. 853 854![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_02.png) 855 856Choose the `Custom` setup type. 857 858![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_03.png) 859 860NSClient++ does not install a sample configuration by default. Change this as shown in the screenshot. 861 862![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_04.png) 863 864Generate a secure password and enable the web server module. **Note**: The webserver module is 865available starting with NSClient++ 0.5.0. Icinga 2 v2.6+ is required which includes this version. 866 867![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_05.png) 868 869Finish the installation. 870 871![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_06.png) 872 873Open a web browser and navigate to `https://localhost:8443`. Enter the password you've configured 874during the setup. In case you lost it, look into the `C:\Program Files\NSClient++\nsclient.ini` 875configuration file. 876 877![Icinga 2 Windows Setup NSClient++](images/distributed-monitoring/icinga2_windows_setup_wizard_05_nsclient_07.png) 878 879The NSClient++ REST API can be used to query metrics. [check_nscp_api](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api) 880uses this transport method. 881 882 883#### Finish Windows Agent Setup <a id="distributed-monitoring-setup-agent-windows-finish"></a> 884 885Finish the Windows setup wizard. 886 887![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_06_finish_with_ticket.png) 888 889If you did not provide a setup ticket, you need to sign the certificate request on the master. 890The setup wizards tells you to do so. The Icinga 2 service is running at this point already 891and will automatically receive and update a signed client certificate. 892 893![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_06_finish_no_ticket.png) 894 895Icinga 2 is automatically started as a Windows service. 896 897![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_running_service.png) 898 899The Icinga 2 configuration is stored inside the `C:\ProgramData\icinga2` directory. 900Click `Examine Config` in the setup wizard to open a new Explorer window. 901 902![Icinga 2 Windows Setup](images/distributed-monitoring/icinga2_windows_setup_wizard_examine_config.png) 903 904The configuration files can be modified with your favorite editor e.g. Notepad++ or vim in Powershell (via chocolatey). 905 906In order to use the [top down](06-distributed-monitoring.md#distributed-monitoring-top-down) agent 907configuration prepare the following steps. 908 909You don't need any local configuration on the agent except for 910CheckCommand definitions which can be synced using the global zone 911above. Therefore disable the inclusion of the `conf.d` directory 912in the `icinga2.conf` file. 913 914Navigate to `C:\ProgramData\icinga2\etc\icinga2` and open 915the `icinga2.conf` file in your preferred editor. Remove or comment (`//`) 916the following line: 917 918``` 919// Commented out, not required on an agent with top down mode 920//include_recursive "conf.d" 921``` 922 923> **Note** 924> 925> Packages >= 2.9 provide an option in the setup wizard to disable this. 926> Defaults to disabled. 927 928Validate the configuration on Windows open an administrative Powershell 929and run the following command: 930 931``` 932C:\> cd C:\Program Files\ICINGA2\sbin 933 934C:\Program Files\ICINGA2\sbin> .\icinga2.exe daemon -C 935``` 936 937**Note**: You have to run this command in a shell with `administrator` privileges. 938 939Now you need to restart the Icinga 2 service. Run `services.msc` from the start menu and restart the `icinga2` service. 940Alternatively open an administrative Powershell and run the following commands: 941 942``` 943C:\> Restart-Service icinga2 944 945C:\> Get-Service icinga2 946``` 947 948 949Now that you've successfully installed a Windows agent, please proceed to 950the [detailed configuration modes](06-distributed-monitoring.md#distributed-monitoring-configuration-modes). 951 952 953## Configuration Modes <a id="distributed-monitoring-configuration-modes"></a> 954 955There are different ways to ensure that the Icinga 2 cluster nodes execute 956checks, send notifications, etc. 957 958The preferred method is to configure monitoring objects on the master 959and distribute the configuration to satellites and agents. 960 961The following chapters explain this in detail with hands-on manual configuration 962examples. You should test and implement this once to fully understand how it works. 963 964Once you are familiar with Icinga 2 and distributed monitoring, you 965can start with additional integrations to manage and deploy your 966configuration: 967 968* [Icinga Director](https://icinga.com/docs/director/latest/) provides a web interface to manage configuration and also allows to sync imported resources (CMDB, PuppetDB, etc.) 969* [Ansible Roles](https://icinga.com/products/integrations/) 970* [Puppet Module](https://icinga.com/products/integrations/puppet/) 971* [Chef Cookbook](https://icinga.com/products/integrations/chef/) 972 973More details can be found [here](13-addons.md#configuration-tools). 974 975### Top Down <a id="distributed-monitoring-top-down"></a> 976 977There are two different behaviors with check execution: 978 979* Send a command execution event remotely: The scheduler still runs on the parent node. 980* Sync the host/service objects directly to the child node: Checks are executed locally. 981 982Again, technically it does not matter whether this is an `agent` or a `satellite` 983which is receiving configuration or command execution events. 984 985### Top Down Command Endpoint <a id="distributed-monitoring-top-down-command-endpoint"></a> 986 987This mode forces the Icinga 2 node to execute commands remotely on a specified endpoint. 988The host/service object configuration is located on the master/satellite and the agent only 989needs the CheckCommand object definitions available. 990 991Every endpoint has its own remote check queue. The amount of checks executed simultaneously 992can be limited on the endpoint with the `MaxConcurrentChecks` constant defined in [constants.conf](04-configuration.md#constants-conf). Icinga 2 may discard check requests, 993if the remote check queue is full. 994 995![Icinga 2 Distributed Top Down Command Endpoint](images/distributed-monitoring/icinga2_distributed_monitoring_agent_checks_command_endpoint.png) 996 997Advantages: 998 999* No local checks need to be defined on the child node (agent). 1000* Light-weight remote check execution (asynchronous events). 1001* No [replay log](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-command-endpoint-log-duration) is necessary for the child node. 1002* Pin checks to specific endpoints (if the child zone consists of 2 endpoints). 1003 1004Disadvantages: 1005 1006* If the child node is not connected, no more checks are executed. 1007* Requires additional configuration attribute specified in host/service objects. 1008* Requires local `CheckCommand` object configuration. Best practice is to use a [global config zone](06-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync). 1009 1010To make sure that all nodes involved will accept configuration and/or 1011commands, you need to configure the `Zone` and `Endpoint` hierarchy 1012on all nodes. 1013 1014* `icinga2-master1.localdomain` is the configuration master in this scenario. 1015* `icinga2-agent1.localdomain` acts as agent which receives command execution messages via command endpoint from the master. In addition, it receives the global check command configuration from the master. 1016 1017Include the endpoint and zone configuration on **both** nodes in the file `/etc/icinga2/zones.conf`. 1018 1019The endpoint configuration could look like this, for example: 1020 1021``` 1022[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 1023 1024object Endpoint "icinga2-master1.localdomain" { 1025 host = "192.168.56.101" 1026} 1027 1028object Endpoint "icinga2-agent1.localdomain" { 1029 host = "192.168.56.111" 1030 log_duration = 0 // Disable the replay log for command endpoint agents 1031} 1032``` 1033 1034Next, you need to define two zones. There is no naming convention, best practice is to either use `master`, `satellite`/`agent-fqdn` or to choose region names for example `Europe`, `USA` and `Asia`, though. 1035 1036**Note**: Each agent requires its own zone and endpoint configuration. Best practice 1037is to use the agent's FQDN for all object names. 1038 1039The `master` zone is a parent of the `icinga2-agent1.localdomain` zone: 1040 1041``` 1042[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 1043 1044object Zone "master" { 1045 endpoints = [ "icinga2-master1.localdomain" ] //array with endpoint names 1046} 1047 1048object Zone "icinga2-agent1.localdomain" { 1049 endpoints = [ "icinga2-agent1.localdomain" ] 1050 1051 parent = "master" //establish zone hierarchy 1052} 1053``` 1054 1055You don't need any local configuration on the agent except for 1056CheckCommand definitions which can be synced using the global zone 1057above. Therefore disable the inclusion of the `conf.d` directory 1058in `/etc/icinga2/icinga2.conf`. 1059 1060``` 1061[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/icinga2.conf 1062 1063// Commented out, not required on an agent as command endpoint 1064//include_recursive "conf.d" 1065``` 1066 1067> **Note** 1068> 1069> Packages >= 2.9 provide an option in the setup wizard to disable this. 1070> Defaults to disabled. 1071 1072Now it is time to validate the configuration and to restart the Icinga 2 daemon 1073on both nodes. 1074 1075Example on CentOS 7: 1076 1077``` 1078[root@icinga2-agent1.localdomain /]# icinga2 daemon -C 1079[root@icinga2-agent1.localdomain /]# systemctl restart icinga2 1080 1081[root@icinga2-master1.localdomain /]# icinga2 daemon -C 1082[root@icinga2-master1.localdomain /]# systemctl restart icinga2 1083``` 1084 1085Once the agents have successfully connected, you are ready for the next step: **execute 1086a remote check on the agent using the command endpoint**. 1087 1088Include the host and service object configuration in the `master` zone 1089-- this will help adding a secondary master for high-availability later. 1090 1091``` 1092[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master 1093``` 1094 1095Add the host and service objects you want to monitor. There is 1096no limitation for files and directories -- best practice is to 1097sort things by type. 1098 1099By convention a master/satellite/agent host object should use the same name as the endpoint object. 1100You can also add multiple hosts which execute checks against remote services/agents. 1101 1102The following example adds the `agent_endpoint` custom variable to the 1103host and stores its name (FQDN). _Versions older than 2.11 1104used the `client_endpoint` custom variable._ 1105 1106This custom variable serves two purposes: 1) Service apply rules can match against it. 11072) Apply rules can retrieve its value and assign it to the `command_endpoint` attribute. 1108 1109``` 1110[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 1111[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 1112 1113object Host "icinga2-agent1.localdomain" { 1114 check_command = "hostalive" //check is executed on the master 1115 address = "192.168.56.111" 1116 1117 vars.agent_endpoint = name //follows the convention that host name == endpoint name 1118} 1119``` 1120 1121Given that you are monitoring a Linux agent, add a remote [disk](10-icinga-template-library.md#plugin-check-command-disk) 1122check. 1123 1124``` 1125[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 1126 1127apply Service "disk" { 1128 check_command = "disk" 1129 1130 // Specify the remote agent as command execution endpoint, fetch the host custom variable 1131 command_endpoint = host.vars.agent_endpoint 1132 1133 // Only assign where a host is marked as agent endpoint 1134 assign where host.vars.agent_endpoint 1135} 1136``` 1137 1138If you have your own custom `CheckCommand` definition, add it to the global zone: 1139 1140``` 1141[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/global-templates 1142[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-templates/commands.conf 1143 1144object CheckCommand "my-cmd" { 1145 //... 1146} 1147``` 1148 1149Save the changes and validate the configuration on the master node: 1150 1151``` 1152[root@icinga2-master1.localdomain /]# icinga2 daemon -C 1153``` 1154Restart the Icinga 2 daemon (example for CentOS 7): 1155 1156``` 1157[root@icinga2-master1.localdomain /]# systemctl restart icinga2 1158``` 1159 1160The following steps will happen: 1161 1162* Icinga 2 validates the configuration on `icinga2-master1.localdomain` and restarts. 1163* The `icinga2-master1.localdomain` node schedules and executes the checks. 1164* The `icinga2-agent1.localdomain` node receives the execute command event with additional command parameters. 1165* The `icinga2-agent1.localdomain` node maps the command parameters to the local check command, executes the check locally, and sends back the check result message. 1166 1167As you can see, no interaction from your side is required on the agent itself, and it's not necessary to reload the Icinga 2 service on the agent. 1168 1169You have learned the basics about command endpoint checks. Proceed with 1170the [scenarios](06-distributed-monitoring.md#distributed-monitoring-scenarios) 1171section where you can find detailed information on extending the setup. 1172 1173 1174### Top Down Config Sync <a id="distributed-monitoring-top-down-config-sync"></a> 1175 1176This mode syncs the object configuration files within specified zones. 1177It comes in handy if you want to configure everything on the master node 1178and sync the satellite checks (disk, memory, etc.). The satellites run their 1179own local scheduler and will send the check result messages back to the master. 1180 1181![Icinga 2 Distributed Top Down Config Sync](images/distributed-monitoring/icinga2_distributed_monitoring_satellite_config_sync.png) 1182 1183Advantages: 1184 1185* Sync the configuration files from the parent zone to the child zones. 1186* No manual restart is required on the child nodes, as syncing, validation, and restarts happen automatically. 1187* Execute checks directly on the child node's scheduler. 1188* Replay log if the connection drops (important for keeping the check history in sync, e.g. for SLA reports). 1189* Use a global zone for syncing templates, groups, etc. 1190 1191Disadvantages: 1192 1193* Requires a config directory on the master node with the zone name underneath `/etc/icinga2/zones.d`. 1194* Additional zone and endpoint configuration needed. 1195* Replay log is replicated on reconnect after connection loss. This might increase the data transfer and create an overload on the connection. 1196 1197> **Note** 1198> 1199> This mode only supports **configuration text files** for Icinga. Do not abuse 1200> this for syncing binaries, this is not supported and may harm your production 1201> environment. The config sync uses checksums to detect changes, binaries may 1202> trigger reload loops. 1203> 1204> This is a fair warning. If you want to deploy plugin binaries, create 1205> packages for dependency management and use infrastructure lifecycle tools 1206> such as Foreman, Puppet, Ansible, etc. 1207 1208To make sure that all involved nodes accept configuration and/or 1209commands, you need to configure the `Zone` and `Endpoint` hierarchy 1210on all nodes. 1211 1212* `icinga2-master1.localdomain` is the configuration master in this scenario. 1213* `icinga2-satellite1.localdomain` acts as satellite which receives configuration from the master. Checks are scheduled locally. 1214 1215Include the endpoint and zone configuration on **both** nodes in the file `/etc/icinga2/zones.conf`. 1216 1217The endpoint configuration could look like this: 1218 1219``` 1220[root@icinga2-satellite1.localdomain /]# vim /etc/icinga2/zones.conf 1221 1222object Endpoint "icinga2-master1.localdomain" { 1223 host = "192.168.56.101" 1224} 1225 1226object Endpoint "icinga2-satellite1.localdomain" { 1227 host = "192.168.56.105" 1228} 1229``` 1230 1231Next, you need to define two zones. There is no naming convention, best practice is to either use `master`, `satellite`/`agent-fqdn` or to choose region names for example `Europe`, `USA` and `Asia`, though. 1232 1233The `master` zone is a parent of the `satellite` zone: 1234 1235``` 1236[root@icinga2-agent2.localdomain /]# vim /etc/icinga2/zones.conf 1237 1238object Zone "master" { 1239 endpoints = [ "icinga2-master1.localdomain" ] //array with endpoint names 1240} 1241 1242object Zone "satellite" { 1243 endpoints = [ "icinga2-satellite1.localdomain" ] 1244 1245 parent = "master" //establish zone hierarchy 1246} 1247``` 1248 1249Edit the `api` feature on the satellite `icinga2-satellite1.localdomain` in 1250the `/etc/icinga2/features-enabled/api.conf` file and set 1251`accept_config` to `true`. 1252 1253``` 1254[root@icinga2-satellite1.localdomain /]# vim /etc/icinga2/features-enabled/api.conf 1255 1256object ApiListener "api" { 1257 //... 1258 accept_config = true 1259} 1260``` 1261 1262Now it is time to validate the configuration and to restart the Icinga 2 daemon 1263on both nodes. 1264 1265Example on CentOS 7: 1266 1267``` 1268[root@icinga2-satellite1.localdomain /]# icinga2 daemon -C 1269[root@icinga2-satellite1.localdomain /]# systemctl restart icinga2 1270 1271[root@icinga2-master1.localdomain /]# icinga2 daemon -C 1272[root@icinga2-master1.localdomain /]# systemctl restart icinga2 1273``` 1274 1275**Tip**: Best practice is to use a [global zone](06-distributed-monitoring.md#distributed-monitoring-global-zone-config-sync) 1276for common configuration items (check commands, templates, groups, etc.). 1277 1278Once the satellite(s) have connected successfully, it's time for the next step: **execute 1279a local check on the satellite using the configuration sync**. 1280 1281Navigate to `/etc/icinga2/zones.d` on your master node 1282`icinga2-master1.localdomain` and create a new directory with the same 1283name as your satellite/agent zone name: 1284 1285``` 1286[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/satellite 1287``` 1288 1289Add the host and service objects you want to monitor. There is 1290no limitation for files and directories -- best practice is to 1291sort things by type. 1292 1293By convention a master/satellite/agent host object should use the same name as the endpoint object. 1294You can also add multiple hosts which execute checks against remote services/agents via [command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) 1295checks. 1296 1297``` 1298[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite 1299[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim hosts.conf 1300 1301object Host "icinga2-satellite1.localdomain" { 1302 check_command = "hostalive" 1303 address = "192.168.56.112" 1304 zone = "master" //optional trick: sync the required host object to the satellite, but enforce the "master" zone to execute the check 1305} 1306``` 1307 1308Given that you are monitoring a Linux satellite add a local [disk](10-icinga-template-library.md#plugin-check-command-disk) 1309check. 1310 1311``` 1312[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim services.conf 1313 1314object Service "disk" { 1315 host_name = "icinga2-satellite1.localdomain" 1316 1317 check_command = "disk" 1318} 1319``` 1320 1321Save the changes and validate the configuration on the master node: 1322 1323``` 1324[root@icinga2-master1.localdomain /]# icinga2 daemon -C 1325``` 1326 1327Restart the Icinga 2 daemon (example for CentOS 7): 1328 1329``` 1330[root@icinga2-master1.localdomain /]# systemctl restart icinga2 1331``` 1332 1333The following steps will happen: 1334 1335* Icinga 2 validates the configuration on `icinga2-master1.localdomain`. 1336* Icinga 2 copies the configuration into its zone config store in `/var/lib/icinga2/api/zones`. 1337* The `icinga2-master1.localdomain` node sends a config update event to all endpoints in the same or direct child zones. 1338* The `icinga2-satellite1.localdomain` node accepts config and populates the local zone config store with the received config files. 1339* The `icinga2-satellite1.localdomain` node validates the configuration and automatically restarts. 1340 1341Again, there is no interaction required on the satellite itself. 1342 1343You can also use the config sync inside a high-availability zone to 1344ensure that all config objects are synced among zone members. 1345 1346**Note**: You can only have one so-called "config master" in a zone which stores 1347the configuration in the `zones.d` directory. 1348Multiple nodes with configuration files in the `zones.d` directory are 1349**not supported**. 1350 1351Now that you've learned the basics about the configuration sync, proceed with 1352the [scenarios](06-distributed-monitoring.md#distributed-monitoring-scenarios) 1353section where you can find detailed information on extending the setup. 1354 1355 1356 1357If you are eager to start fresh instead you might take a look into the 1358[Icinga Director](https://icinga.com/docs/director/latest/). 1359 1360## Scenarios <a id="distributed-monitoring-scenarios"></a> 1361 1362The following examples should give you an idea on how to build your own 1363distributed monitoring environment. We've seen them all in production 1364environments and received feedback from our [community](https://community.icinga.com/) 1365and [partner support](https://icinga.com/support/) channels: 1366 1367* [Single master with agents](06-distributed-monitoring.md#distributed-monitoring-master-agents). 1368* [HA master with agents as command endpoint](06-distributed-monitoring.md#distributed-monitoring-scenarios-ha-master-agents) 1369* [Three level cluster](06-distributed-monitoring.md#distributed-monitoring-scenarios-master-satellite-agents) with config HA masters, satellites receiving config sync, and agents checked using command endpoint. 1370 1371You can also extend the cluster tree depth to four levels e.g. with 2 satellite levels. 1372Just keep in mind that multiple levels become harder to debug in case of errors. 1373 1374You can also start with a single master setup, and later add a secondary 1375master endpoint. This requires an extra step with the [initial sync](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-initial-sync) 1376for cloning the runtime state. This is described in detail [here](06-distributed-monitoring.md#distributed-monitoring-scenarios-ha-master-agents). 1377 1378<!-- Keep this for compatiblity --> 1379<a id="distributed-monitoring-master-clients"></a> 1380 1381### Master with Agents <a id="distributed-monitoring-master-agents"></a> 1382 1383In this scenario, a single master node runs the check scheduler, notifications 1384and IDO database backend and uses the [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) 1385to execute checks on the remote agents. 1386 1387![Icinga 2 Distributed Master with Agents](images/distributed-monitoring/icinga2_distributed_monitoring_scenarios_master_with_agents.png) 1388 1389* `icinga2-master1.localdomain` is the primary master node. 1390* `icinga2-agent1.localdomain` and `icinga2-agent2.localdomain` are two child nodes as agents. 1391 1392Setup requirements: 1393 1394* Set up `icinga2-master1.localdomain` as [master](06-distributed-monitoring.md#distributed-monitoring-setup-master). 1395* Set up `icinga2-agent1.localdomain` and `icinga2-agent2.localdomain` as [agent](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite). 1396 1397Edit the `zones.conf` configuration file on the master: 1398 1399``` 1400[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf 1401 1402object Endpoint "icinga2-master1.localdomain" { 1403 // That's us 1404} 1405 1406object Endpoint "icinga2-agent1.localdomain" { 1407 host = "192.168.56.111" // The master actively tries to connect to the agent 1408 log_duration = 0 // Disable the replay log for command endpoint agents 1409} 1410 1411object Endpoint "icinga2-agent2.localdomain" { 1412 host = "192.168.56.112" // The master actively tries to connect to the agent 1413 log_duration = 0 // Disable the replay log for command endpoint agents 1414} 1415 1416object Zone "master" { 1417 endpoints = [ "icinga2-master1.localdomain" ] 1418} 1419 1420object Zone "icinga2-agent1.localdomain" { 1421 endpoints = [ "icinga2-agent1.localdomain" ] 1422 1423 parent = "master" 1424} 1425 1426object Zone "icinga2-agent2.localdomain" { 1427 endpoints = [ "icinga2-agent2.localdomain" ] 1428 1429 parent = "master" 1430} 1431 1432/* sync global commands */ 1433object Zone "global-templates" { 1434 global = true 1435} 1436object Zone "director-global" { 1437 global = true 1438} 1439``` 1440 1441The two agent nodes do not need to know about each other. The only important thing 1442is that they know about the parent zone and their endpoint members (and optionally the global zone). 1443 1444If you specify the `host` attribute in the `icinga2-master1.localdomain` endpoint object, 1445the agent will actively try to connect to the master node. Since you've specified the agent 1446endpoint's attribute on the master node already, you don't want the agents to connect to the 1447master. **Choose one [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction).** 1448 1449``` 1450[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 1451 1452object Endpoint "icinga2-master1.localdomain" { 1453 // Do not actively connect to the master by leaving out the 'host' attribute 1454} 1455 1456object Endpoint "icinga2-agent1.localdomain" { 1457 // That's us 1458} 1459 1460object Zone "master" { 1461 endpoints = [ "icinga2-master1.localdomain" ] 1462} 1463 1464object Zone "icinga2-agent1.localdomain" { 1465 endpoints = [ "icinga2-agent1.localdomain" ] 1466 1467 parent = "master" 1468} 1469 1470/* sync global commands */ 1471object Zone "global-templates" { 1472 global = true 1473} 1474object Zone "director-global" { 1475 global = true 1476} 1477``` 1478``` 1479[root@icinga2-agent2.localdomain /]# vim /etc/icinga2/zones.conf 1480 1481object Endpoint "icinga2-master1.localdomain" { 1482 // Do not actively connect to the master by leaving out the 'host' attribute 1483} 1484 1485object Endpoint "icinga2-agent2.localdomain" { 1486 // That's us 1487} 1488 1489object Zone "master" { 1490 endpoints = [ "icinga2-master1.localdomain" ] 1491} 1492 1493object Zone "icinga2-agent2.localdomain" { 1494 endpoints = [ "icinga2-agent2.localdomain" ] 1495 1496 parent = "master" 1497} 1498 1499/* sync global commands */ 1500object Zone "global-templates" { 1501 global = true 1502} 1503object Zone "director-global" { 1504 global = true 1505} 1506``` 1507 1508Now it is time to define the two agent hosts and apply service checks using 1509the command endpoint execution method on them. Note: You can also use the 1510config sync mode here. 1511 1512Create a new configuration directory on the master node: 1513 1514``` 1515[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master 1516``` 1517 1518Add the two agent nodes as host objects: 1519 1520``` 1521[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 1522[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 1523 1524object Host "icinga2-agent1.localdomain" { 1525 check_command = "hostalive" 1526 address = "192.168.56.111" 1527 1528 vars.agent_endpoint = name //follows the convention that host name == endpoint name 1529} 1530 1531object Host "icinga2-agent2.localdomain" { 1532 check_command = "hostalive" 1533 address = "192.168.56.112" 1534 1535 vars.agent_endpoint = name //follows the convention that host name == endpoint name 1536} 1537``` 1538 1539Add services using command endpoint checks: 1540 1541``` 1542[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 1543 1544apply Service "ping4" { 1545 check_command = "ping4" 1546 1547 //check is executed on the master node 1548 assign where host.address 1549} 1550 1551apply Service "disk" { 1552 check_command = "disk" 1553 1554 // Execute the check on the remote command endpoint 1555 command_endpoint = host.vars.agent_endpoint 1556 1557 // Assign the service onto an agent 1558 assign where host.vars.agent_endpoint 1559} 1560``` 1561 1562Validate the configuration and restart Icinga 2 on the master node `icinga2-master1.localdomain`. 1563 1564``` 1565[root@icinga2-master1.localdomain /]# icinga2 daemon -C 1566[root@icinga2-master1.localdomain /]# systemctl restart icinga2 1567``` 1568 1569Open Icinga Web 2 and check the two newly created agent hosts with two new services 1570-- one executed locally (`ping4`) and one using command endpoint (`disk`). 1571 1572> **Note** 1573> 1574> You don't necessarily need to add the agent endpoint/zone configuration objects 1575> into the master's zones.conf file. Instead, you can put them into `/etc/icinga2/zones.d/master` 1576> either in `hosts.conf` shown above, or in a new file called `agents.conf`. 1577 1578> **Tip**: 1579> 1580> It's a good idea to add [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks) 1581to make sure that your cluster notifies you in case of failure. 1582 1583In terms of health checks, consider adding the following for this scenario: 1584 1585- Master node(s) check the connection to the agents 1586- Optional: Add dependencies for the agent host to prevent unwanted notifications when agents are unreachable 1587 1588Proceed in [this chapter](06-distributed-monitoring.md#distributed-monitoring-health-checks-master-agents). 1589 1590<!-- Keep this for compatibility --> 1591<a id="distributed-monitoring-scenarios-ha-master-clients"></a> 1592 1593### High-Availability Master with Agents <a id="distributed-monitoring-scenarios-ha-master-agents"></a> 1594 1595This scenario is similar to the one in the [previous section](06-distributed-monitoring.md#distributed-monitoring-master-agents). The only difference is that we will now set up two master nodes in a high-availability setup. 1596These nodes must be configured as zone and endpoints objects. 1597 1598![Icinga 2 Distributed High Availability Master with Agents](images/distributed-monitoring/icinga2_distributed_monitoring_scenario_ha_masters_with_agents.png) 1599 1600The setup uses the capabilities of the Icinga 2 cluster. All zone members 1601replicate cluster events between each other. In addition to that, several Icinga 2 1602features can enable [HA functionality](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). 1603 1604Best practice is to run the database backend on a dedicated server/cluster and 1605only expose a virtual IP address to Icinga and the IDO feature. By default, only one 1606endpoint will actively write to the backend then. Typical setups for MySQL clusters 1607involve Master-Master-Replication (Master-Slave-Replication in both directions) or Galera, 1608more tips can be found on our [community forums](https://community.icinga.com/). 1609The IDO object must have the same `instance_name` on all master nodes. 1610 1611**Note**: All nodes in the same zone require that you enable the same features for high-availability (HA). 1612 1613Overview: 1614 1615* `icinga2-master1.localdomain` is the config master master node. 1616* `icinga2-master2.localdomain` is the secondary master master node without config in `zones.d`. 1617* `icinga2-agent1.localdomain` and `icinga2-agent2.localdomain` are two child nodes as agents. 1618 1619Setup requirements: 1620 1621* Set up `icinga2-master1.localdomain` as [master](06-distributed-monitoring.md#distributed-monitoring-setup-master). 1622* Set up `icinga2-master2.localdomain` as [satellite](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite) (**we will modify the generated configuration**). 1623* Set up `icinga2-agent1.localdomain` and `icinga2-agent2.localdomain` as [agents](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite) (when asked for adding multiple masters, set to `y` and add the secondary master `icinga2-master2.localdomain`). 1624 1625In case you don't want to use the CLI commands, you can also manually create and sync the 1626required TLS certificates. We will modify and discuss all the details of the automatically generated configuration here. 1627 1628Since there are now two nodes in the same zone, we must consider the 1629[high-availability features](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). 1630 1631* Checks and notifications are balanced between the two master nodes. That's fine, but it requires check plugins and notification scripts to exist on both nodes. 1632* The IDO feature will only be active on one node by default. Since all events are replicated between both nodes, it is easier to just have one central database. 1633 1634One possibility is to use a dedicated MySQL cluster VIP (external application cluster) 1635and leave the IDO feature with enabled HA capabilities. Alternatively, 1636you can disable the HA feature and write to a local database on each node. 1637Both methods require that you configure Icinga Web 2 accordingly (monitoring 1638backend, IDO database, used transports, etc.). 1639 1640> **Note** 1641> 1642> You can also start with a single master shown [here](06-distributed-monitoring.md#distributed-monitoring-master-agents) and later add 1643> the second master. This requires an extra step with the [initial sync](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-initial-sync) 1644> for cloning the runtime state after done. Once done, proceed here. 1645 1646In this scenario, we are not adding the agent configuration immediately 1647to the `zones.conf` file but will establish the hierarchy later. 1648 1649The first master looks like this: 1650 1651``` 1652[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf 1653 1654object Endpoint "icinga2-master1.localdomain" { 1655 // That's us 1656} 1657 1658object Endpoint "icinga2-master2.localdomain" { 1659 host = "192.168.56.102" // Actively connect to the secondary master 1660} 1661 1662object Zone "master" { 1663 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ] 1664} 1665 1666/* sync global commands */ 1667object Zone "global-templates" { 1668 global = true 1669} 1670object Zone "director-global" { 1671 global = true 1672} 1673``` 1674 1675The secondary master waits for connection attempts from the first master, 1676and therefore does not try to connect to it again. 1677 1678``` 1679[root@icinga2-master2.localdomain /]# vim /etc/icinga2/zones.conf 1680 1681object Endpoint "icinga2-master1.localdomain" { 1682 // The first master already connects to us 1683} 1684 1685object Endpoint "icinga2-master2.localdomain" { 1686 // That's us 1687} 1688 1689object Zone "master" { 1690 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ] 1691} 1692 1693/* sync global commands */ 1694object Zone "global-templates" { 1695 global = true 1696} 1697object Zone "director-global" { 1698 global = true 1699} 1700``` 1701 1702Restart both masters and ensure the initial connection and TLS handshake works. 1703 1704The two agent nodes do not need to know about each other. The only important thing 1705is that they know about the parent zone and their endpoint members (and optionally about the global zone). 1706 1707If you specify the `host` attribute in the `icinga2-master1.localdomain` and `icinga2-master2.localdomain` 1708endpoint objects, the agent will actively try to connect to the master node. Since we've specified the agent 1709endpoint's attribute on the master node already, we don't want the agent to connect to the 1710master nodes. **Choose one [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction).** 1711 1712``` 1713[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 1714 1715object Endpoint "icinga2-master1.localdomain" { 1716 // Do not actively connect to the master by leaving out the 'host' attribute 1717} 1718 1719object Endpoint "icinga2-master2.localdomain" { 1720 // Do not actively connect to the master by leaving out the 'host' attribute 1721} 1722 1723object Endpoint "icinga2-agent1.localdomain" { 1724 // That's us 1725} 1726 1727object Zone "master" { 1728 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ] 1729} 1730 1731object Zone "icinga2-agent1.localdomain" { 1732 endpoints = [ "icinga2-agent1.localdomain" ] 1733 1734 parent = "master" 1735} 1736 1737/* sync global commands */ 1738object Zone "global-templates" { 1739 global = true 1740} 1741object Zone "director-global" { 1742 global = true 1743} 1744 1745``` 1746 1747``` 1748[root@icinga2-agent2.localdomain /]# vim /etc/icinga2/zones.conf 1749 1750object Endpoint "icinga2-master1.localdomain" { 1751 // Do not actively connect to the master by leaving out the 'host' attribute 1752} 1753 1754object Endpoint "icinga2-master2.localdomain" { 1755 // Do not actively connect to the master by leaving out the 'host' attribute 1756} 1757 1758object Endpoint "icinga2-agent2.localdomain" { 1759 //That's us 1760} 1761 1762object Zone "master" { 1763 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ] 1764} 1765 1766object Zone "icinga2-agent2.localdomain" { 1767 endpoints = [ "icinga2-agent2.localdomain" ] 1768 1769 parent = "master" 1770} 1771 1772/* sync global commands */ 1773object Zone "global-templates" { 1774 global = true 1775} 1776object Zone "director-global" { 1777 global = true 1778} 1779``` 1780 1781Now it is time to define the two agent hosts and apply service checks using 1782the command endpoint execution method. 1783 1784Create a new configuration directory on the master node `icinga2-master1.localdomain`. 1785**Note**: The secondary master node `icinga2-master2.localdomain` receives the 1786configuration using the [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync). 1787 1788``` 1789[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master 1790``` 1791 1792Add the two agent nodes with their zone/endpoint and host object configuration. 1793 1794> **Note** 1795> 1796> In order to keep things in sync between the two HA masters, 1797> keep the `zones.conf` file as small as possible. 1798> 1799> You can create the agent zone and endpoint objects inside the 1800> master zone and have them synced to the secondary master. 1801> The cluster config sync enforces a reload allowing the secondary 1802> master to connect to the agents as well. 1803 1804Edit the `zones.conf` file and ensure that the agent zone/endpoint objects 1805are **not** specified in there. 1806 1807Then navigate into `/etc/icinga2/zones.d/master` and create a new file `agents.conf`. 1808 1809``` 1810[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 1811[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim agents.conf 1812 1813//----------------------------------------------- 1814// Endpoints 1815 1816object Endpoint "icinga2-agent1.localdomain" { 1817 host = "192.168.56.111" // The master actively tries to connect to the agent 1818 log_duration = 0 // Disable the replay log for command endpoint agents 1819} 1820 1821object Endpoint "icinga2-agent2.localdomain" { 1822 host = "192.168.56.112" // The master actively tries to connect to the agent 1823 log_duration = 0 // Disable the replay log for command endpoint agents 1824} 1825 1826//----------------------------------------------- 1827// Zones 1828 1829object Zone "icinga2-agent1.localdomain" { 1830 endpoints = [ "icinga2-agent1.localdomain" ] 1831 1832 parent = "master" 1833} 1834 1835object Zone "icinga2-agent2.localdomain" { 1836 endpoints = [ "icinga2-agent2.localdomain" ] 1837 1838 parent = "master" 1839} 1840``` 1841 1842Whenever you need to add an agent again, edit the mentioned files. 1843 1844Next, create the corresponding host objects for the agents. Use the same names 1845for host and endpoint objects. 1846 1847``` 1848[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 1849 1850object Host "icinga2-agent1.localdomain" { 1851 check_command = "hostalive" 1852 address = "192.168.56.111" 1853 vars.agent_endpoint = name //follows the convention that host name == endpoint name 1854} 1855 1856object Host "icinga2-agent2.localdomain" { 1857 check_command = "hostalive" 1858 address = "192.168.56.112" 1859 vars.agent_endpoint = name //follows the convention that host name == endpoint name 1860} 1861``` 1862 1863Add services using command endpoint checks: 1864 1865``` 1866[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 1867 1868apply Service "ping4" { 1869 check_command = "ping4" 1870 1871 // Check is executed on the master node 1872 assign where host.address 1873} 1874 1875apply Service "disk" { 1876 check_command = "disk" 1877 1878 // Check is executed on the remote command endpoint 1879 command_endpoint = host.vars.agent_endpoint 1880 1881 assign where host.vars.agent_endpoint 1882} 1883``` 1884 1885Validate the configuration and restart Icinga 2 on the master node `icinga2-master1.localdomain`. 1886 1887``` 1888[root@icinga2-master1.localdomain /]# icinga2 daemon -C 1889[root@icinga2-master1.localdomain /]# systemctl restart icinga2 1890``` 1891 1892Open Icinga Web 2 and check the two newly created agent hosts with two new services 1893-- one executed locally (`ping4`) and one using command endpoint (`disk`). 1894 1895> **Tip**: 1896> 1897> It's a good idea to add [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks) 1898to make sure that your cluster notifies you in case of failure. 1899 1900In terms of health checks, consider adding the following for this scenario: 1901 1902- Master node(s) check the connection to the agents 1903- Optional: Add dependencies for the agent host to prevent unwanted notifications when agents are unreachable 1904 1905Proceed in [this chapter](06-distributed-monitoring.md#distributed-monitoring-health-checks-master-agents). 1906 1907<!-- Keep this for compatibility --> 1908<a id="distributed-monitoring-scenarios-master-satellite-client"></a> 1909 1910### Three Levels with Masters, Satellites and Agents <a id="distributed-monitoring-scenarios-master-satellite-agents"></a> 1911 1912This scenario combines everything you've learned so far: High-availability masters, 1913satellites receiving their configuration from the master zone, and agents checked via command 1914endpoint from the satellite zones. 1915 1916![Icinga 2 Distributed Master and Satellites with Agents](images/distributed-monitoring/icinga2_distributed_monitoring_scenarios_master_satellites_agents.png) 1917 1918> **Tip**: 1919> 1920> It can get complicated, so grab a pen and paper and bring your thoughts to life. 1921> Play around with a test setup before using it in a production environment! 1922 1923There are various reasons why you might want to have satellites in your environment. The following list explains the more common ones. 1924 1925* Monitor remote locations. Besides reducing connections and traffic between different locations this setup also helps when the network connection to the remote network is lost. Satellites will keep checking and collecting data on their own and will send their check results when the connection is restored. 1926* Reduce connections between security zones. Satellites in a different zone (e.g. DMZ) than your masters will help reduce connections through firewalls. 1927* Offload resource hungry checks to other hosts. In very big setups running lots of plugins on your masters or satellites might have a significant impact on the performance during times of high load. You can introduce another level of satellites just to run these plugins and send their results to the upstream hosts. 1928 1929Best practice is to run the database backend on a dedicated server/cluster and 1930only expose a virtual IP address to Icinga and the IDO feature. By default, only one 1931endpoint will actively write to the backend then. Typical setups for MySQL clusters 1932involve Master-Master-Replication (Master-Slave-Replication in both directions) or Galera, 1933more tips can be found on our [community forums](https://community.icinga.com/). 1934 1935Overview: 1936 1937* `icinga2-master1.localdomain` is the configuration master master node. 1938* `icinga2-master2.localdomain` is the secondary master master node without configuration in `zones.d`. 1939* `icinga2-satellite1.localdomain` and `icinga2-satellite2.localdomain` are satellite nodes in a `master` child zone. They forward CSR signing requests to the master zone. 1940* `icinga2-agent1.localdomain` and `icinga2-agent2.localdomain` are two child nodes as agents. 1941 1942Setup requirements: 1943 1944* Set up `icinga2-master1.localdomain` as [master](06-distributed-monitoring.md#distributed-monitoring-setup-master). 1945* Set up `icinga2-master2.localdomain`, `icinga2-satellite1.localdomain` and `icinga2-satellite2.localdomain` as [agents](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite) (we will modify the generated configuration). 1946* Set up `icinga2-agent1.localdomain` and `icinga2-agent2.localdomain` as [agents](06-distributed-monitoring.md#distributed-monitoring-setup-agent-satellite). 1947 1948When being asked for the parent endpoint providing CSR auto-signing capabilities, 1949please add one of the satellite nodes. **Note**: This requires Icinga 2 v2.8+ 1950and the `CA Proxy` on all master, satellite and agent nodes. 1951 1952Example for `icinga2-agent1.localdomain`: 1953 1954``` 1955Please specify the parent endpoint(s) (master or satellite) where this node should connect to: 1956``` 1957 1958Parent endpoint is the first satellite `icinga2-satellite1.localdomain`: 1959 1960``` 1961Master/Satellite Common Name (CN from your master/satellite node): icinga2-satellite1.localdomain 1962Do you want to establish a connection to the parent node from this node? [Y/n]: y 1963 1964Please specify the master/satellite connection information: 1965Master/Satellite endpoint host (IP address or FQDN): 192.168.56.105 1966Master/Satellite endpoint port [5665]: 5665 1967``` 1968 1969Add the second satellite `icinga2-satellite2.localdomain` as parent: 1970 1971``` 1972Add more master/satellite endpoints? [y/N]: y 1973 1974Master/Satellite Common Name (CN from your master/satellite node): icinga2-satellite2.localdomain 1975Do you want to establish a connection to the parent node from this node? [Y/n]: y 1976 1977Please specify the master/satellite connection information: 1978Master/Satellite endpoint host (IP address or FQDN): 192.168.56.106 1979Master/Satellite endpoint port [5665]: 5665 1980 1981Add more master/satellite endpoints? [y/N]: n 1982``` 1983 1984The specified parent nodes will forward the CSR signing request to the master instances. 1985 1986Proceed with adding the optional client ticket for [CSR auto-signing](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing): 1987 1988``` 1989Please specify the request ticket generated on your Icinga 2 master (optional). 1990 (Hint: # icinga2 pki ticket --cn 'icinga2-agent1.localdomain'): 19914f75d2ecd253575fe9180938ebff7cbca262f96e 1992``` 1993 1994In case you've chosen to use [On-Demand CSR Signing](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing) 1995you can leave the ticket question blank. 1996 1997Instead, Icinga 2 tells you to approve the request later on the master node. 1998 1999``` 2000No ticket was specified. Please approve the certificate signing request manually 2001on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details). 2002``` 2003 2004You can optionally specify a different bind host and/or port. 2005 2006``` 2007Please specify the API bind host/port (optional): 2008Bind Host []: 2009Bind Port []: 2010``` 2011 2012The next step asks you to accept configuration (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)) 2013and commands (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). 2014 2015``` 2016Accept config from parent node? [y/N]: y 2017Accept commands from parent node? [y/N]: y 2018``` 2019 2020Next you can optionally specify the local and parent zone names. This will be reflected 2021in the generated zone configuration file. 2022 2023``` 2024Local zone name [icinga2-agent1.localdomain]: icinga2-agent1.localdomain 2025``` 2026 2027Set the parent zone name to `satellite` for this agent. 2028 2029``` 2030Parent zone name [master]: satellite 2031``` 2032 2033You can add more global zones in addition to `global-templates` and `director-global` if necessary. 2034Press `Enter` or choose `n`, if you don't want to add any additional. 2035 2036``` 2037Reconfiguring Icinga... 2038 2039Default global zones: global-templates director-global 2040Do you want to specify additional global zones? [y/N]: N 2041``` 2042 2043Last but not least the wizard asks you whether you want to disable the inclusion of the local configuration 2044directory in `conf.d`, or not. Defaults to disabled, since agents are checked via command endpoint and the example 2045configuration would collide with this mode. 2046 2047``` 2048Do you want to disable the inclusion of the conf.d directory [Y/n]: Y 2049Disabling the inclusion of the conf.d directory... 2050``` 2051 2052 2053**We'll discuss the details of the required configuration below. Most of this 2054configuration can be rendered by the setup wizards.** 2055 2056The zone hierarchy can look like this. We'll define only the directly connected zones here. 2057 2058The master instances should actively connect to the satellite instances, therefore 2059the configuration on `icinga2-master1.localdomain` and `icinga2-master2.localdomain` 2060must include the `host` attribute for the satellite endpoints: 2061 2062``` 2063[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf 2064 2065object Endpoint "icinga2-master1.localdomain" { 2066 // That's us 2067} 2068 2069object Endpoint "icinga2-master2.localdomain" { 2070 host = "192.168.56.102" // Actively connect to the second master. 2071} 2072 2073object Endpoint "icinga2-satellite1.localdomain" { 2074 host = "192.168.56.105" // Actively connect to the satellites. 2075} 2076 2077object Endpoint "icinga2-satellite2.localdomain" { 2078 host = "192.168.56.106" // Actively connect to the satellites. 2079} 2080 2081object Zone "master" { 2082 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ] 2083} 2084``` 2085 2086The endpoint configuration on the secondary master looks similar, 2087but changes the connection attributes - the first master already 2088tries to connect, there is no need for a secondary attempt. 2089 2090``` 2091[root@icinga2-master2.localdomain /]# vim /etc/icinga2/zones.conf 2092 2093object Endpoint "icinga2-master1.localdomain" { 2094 // First master already connects to us 2095} 2096 2097object Endpoint "icinga2-master2.localdomain" { 2098 // That's us 2099} 2100 2101object Endpoint "icinga2-satellite1.localdomain" { 2102 host = "192.168.56.105" // Actively connect to the satellites. 2103} 2104 2105object Endpoint "icinga2-satellite2.localdomain" { 2106 host = "192.168.56.106" // Actively connect to the satellites. 2107} 2108``` 2109 2110The zone configuration on both masters looks the same. Add this 2111to the corresponding `zones.conf` entries for the endpoints. 2112 2113``` 2114object Zone "satellite" { 2115 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ] 2116 2117 parent = "master" 2118} 2119 2120/* sync global commands */ 2121object Zone "global-templates" { 2122 global = true 2123} 2124 2125object Zone "director-global" { 2126 global = true 2127} 2128 2129``` 2130 2131In contrast to that, the satellite instances `icinga2-satellite1.localdomain` 2132and `icinga2-satellite2.localdomain` should not actively connect to the master 2133instances. 2134 2135``` 2136[root@icinga2-satellite1.localdomain /]# vim /etc/icinga2/zones.conf 2137 2138object Endpoint "icinga2-master1.localdomain" { 2139 // This endpoint will connect to us 2140} 2141 2142object Endpoint "icinga2-master2.localdomain" { 2143 // This endpoint will connect to us 2144} 2145 2146object Endpoint "icinga2-satellite1.localdomain" { 2147 // That's us 2148} 2149 2150object Endpoint "icinga2-satellite2.localdomain" { 2151 host = "192.168.56.106" // Actively connect to the secondary satellite 2152} 2153``` 2154 2155Again, only one side is required to establish the connection inside the HA zone. 2156Since satellite1 already connects to satellite2, leave out the `host` attribute 2157for `icinga2-satellite1.localdomain` on satellite2. 2158 2159``` 2160[root@icinga2-satellite2.localdomain /]# vim /etc/icinga2/zones.conf 2161 2162object Endpoint "icinga2-master1.localdomain" { 2163 // This endpoint will connect to us 2164} 2165 2166object Endpoint "icinga2-master2.localdomain" { 2167 // This endpoint will connect to us 2168} 2169 2170object Endpoint "icinga2-satellite1.localdomain" { 2171 // First satellite already connects to us 2172} 2173 2174object Endpoint "icinga2-satellite2.localdomain" { 2175 // That's us 2176} 2177``` 2178 2179The zone configuration on both satellites looks the same. Add this 2180to the corresponding `zones.conf` entries for the endpoints. 2181 2182``` 2183object Zone "master" { 2184 endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ] 2185} 2186 2187object Zone "satellite" { 2188 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ] 2189 2190 parent = "master" 2191} 2192 2193/* sync global commands */ 2194object Zone "global-templates" { 2195 global = true 2196} 2197 2198object Zone "director-global" { 2199 global = true 2200} 2201``` 2202 2203Keep in mind to control the endpoint [connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction) 2204using the `host` attribute, also for other endpoints in the same zone. 2205 2206Since we want to use [top down command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) checks, 2207we must configure the agent endpoint and zone objects. 2208 2209In order to minimize the effort, we'll sync the agent zone and endpoint configuration to the 2210satellites where the connection information is needed as well. Note: This only works with satellite 2211and agents, since there already is a trust relationship between the master and the satellite zone. 2212The cluster config sync to the satellite invokes an automated reload causing the agent connection attempts. 2213 2214`icinga2-master1.localdomain` is the configuration master where everything is stored: 2215 2216``` 2217[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/{master,satellite,global-templates} 2218[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite 2219 2220[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-agent1.localdomain.conf 2221 2222object Endpoint "icinga2-agent1.localdomain" { 2223 host = "192.168.56.111" // The satellite actively tries to connect to the agent 2224 log_duration = 0 // Disable the replay log for command endpoint agents 2225} 2226 2227object Zone "icinga2-agent1.localdomain" { 2228 endpoints = [ "icinga2-agent1.localdomain" ] 2229 2230 parent = "satellite" 2231} 2232 2233[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-agent2.localdomain.conf 2234 2235object Endpoint "icinga2-agent2.localdomain" { 2236 host = "192.168.56.112" // The satellite actively tries to connect to the agent 2237 log_duration = 0 // Disable the replay log for command endpoint agents 2238} 2239 2240object Zone "icinga2-agent2.localdomain" { 2241 endpoints = [ "icinga2-agent2.localdomain" ] 2242 2243 parent = "satellite" 2244} 2245``` 2246 2247The two agent nodes do not need to know about each other. The only important thing 2248is that they know about the parent zone (the satellite) and their endpoint members (and optionally the global zone). 2249 2250> **Tipp** 2251> 2252> In the example above we've specified the `host` attribute in the agent endpoint configuration. In this mode, 2253> the satellites actively connect to the agents. This costs some resources on the satellite -- if you prefer to 2254> offload the connection attempts to the agent, or your DMZ requires this, you can also change the **[connection direction](06-distributed-monitoring.md#distributed-monitoring-advanced-hints-connection-direction).** 2255> 2256> 1) Don't set the `host` attribute for the agent endpoints put into `zones.d/satellite`. 2257> 2) Modify each agent's zones.conf file and add the `host` attribute to all parent satellites. You can automate this with using the `node wizard/setup` CLI commands. 2258 2259The agents are waiting for the satellites to connect, therefore they don't specify 2260the `host` attribute in the endpoint objects locally. 2261 2262Example for `icinga2-agent1.localdomain`: 2263 2264``` 2265[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 2266 2267object Endpoint "icinga2-satellite1.localdomain" { 2268 // Do not actively connect to the satellite by leaving out the 'host' attribute 2269} 2270 2271object Endpoint "icinga2-satellite2.localdomain" { 2272 // Do not actively connect to the satellite by leaving out the 'host' attribute 2273} 2274 2275object Endpoint "icinga2-agent1.localdomain" { 2276 // That's us 2277} 2278 2279object Zone "satellite" { 2280 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ] 2281} 2282 2283object Zone "icinga2-agent1.localdomain" { 2284 endpoints = [ "icinga2-agent1.localdomain" ] 2285 2286 parent = "satellite" 2287} 2288 2289/* sync global commands */ 2290object Zone "global-templates" { 2291 global = true 2292} 2293 2294object Zone "director-global" { 2295 global = true 2296} 2297``` 2298 2299Example for `icinga2-agent2.localdomain`: 2300 2301``` 2302[root@icinga2-agent2.localdomain /]# vim /etc/icinga2/zones.conf 2303 2304object Endpoint "icinga2-satellite1.localdomain" { 2305 // Do not actively connect to the satellite by leaving out the 'host' attribute 2306} 2307 2308object Endpoint "icinga2-satellite2.localdomain" { 2309 // Do not actively connect to the satellite by leaving out the 'host' attribute 2310} 2311 2312object Endpoint "icinga2-agent2.localdomain" { 2313 // That's us 2314} 2315 2316object Zone "satellite" { 2317 endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ] 2318} 2319 2320object Zone "icinga2-agent2.localdomain" { 2321 endpoints = [ "icinga2-agent2.localdomain" ] 2322 2323 parent = "satellite" 2324} 2325 2326/* sync global commands */ 2327object Zone "global-templates" { 2328 global = true 2329} 2330 2331object Zone "director-global" { 2332 global = true 2333} 2334``` 2335 2336Now it is time to define the two agents hosts on the master, sync them to the satellites 2337and apply service checks using the command endpoint execution method to them. 2338Add the two agent nodes as host objects to the `satellite` zone. 2339 2340We've already created the directories in `/etc/icinga2/zones.d` including the files for the 2341zone and endpoint configuration for the agents. 2342 2343``` 2344[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite 2345``` 2346 2347Add the host object configuration for the `icinga2-agent1.localdomain` agent. You should 2348have created the configuration file in the previous steps and it should contain the endpoint 2349and zone object configuration already. 2350 2351``` 2352[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-agent1.localdomain.conf 2353 2354object Host "icinga2-agent1.localdomain" { 2355 check_command = "hostalive" 2356 address = "192.168.56.111" 2357 2358 vars.agent_endpoint = name // Follows the convention that host name == endpoint name 2359} 2360``` 2361 2362Add the host object configuration for the `icinga2-agent2.localdomain` agent configuration file: 2363 2364``` 2365[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-agent2.localdomain.conf 2366 2367object Host "icinga2-agent2.localdomain" { 2368 check_command = "hostalive" 2369 address = "192.168.56.112" 2370 2371 vars.agent_endpoint = name // Follows the convention that host name == endpoint name 2372} 2373``` 2374 2375Add a service object which is executed on the satellite nodes (e.g. `ping4`). Pin the apply rule to the `satellite` zone only. 2376 2377``` 2378[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim services.conf 2379 2380apply Service "ping4" { 2381 check_command = "ping4" 2382 2383 // Check is executed on the satellite node 2384 assign where host.zone == "satellite" && host.address 2385} 2386``` 2387 2388Add services using command endpoint checks. Pin the apply rules to the `satellite` zone only. 2389 2390``` 2391[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim services.conf 2392 2393apply Service "disk" { 2394 check_command = "disk" 2395 2396 // Execute the check on the remote command endpoint 2397 command_endpoint = host.vars.agent_endpoint 2398 2399 assign where host.zone == "satellite" && host.vars.agent_endpoint 2400} 2401``` 2402 2403Validate the configuration and restart Icinga 2 on the master node `icinga2-master1.localdomain`. 2404 2405``` 2406[root@icinga2-master1.localdomain /]# icinga2 daemon -C 2407[root@icinga2-master1.localdomain /]# systemctl restart icinga2 2408``` 2409 2410Open Icinga Web 2 and check the two newly created agent hosts with two new services 2411-- one executed locally (`ping4`) and one using command endpoint (`disk`). 2412 2413> **Tip**: 2414> 2415> It's a good idea to add [health checks](06-distributed-monitoring.md#distributed-monitoring-health-checks) 2416to make sure that your cluster notifies you in case of failure. 2417 2418In terms of health checks, consider adding the following for this scenario: 2419 2420- Master nodes check whether the satellite zone is connected 2421- Satellite nodes check the connection to the agents 2422- Optional: Add dependencies for the agent host to prevent unwanted notifications when agents are unreachable 2423 2424Proceed in [this chapter](06-distributed-monitoring.md#distributed-monitoring-health-checks-master-satellite-agent). 2425 2426 2427## Best Practice <a id="distributed-monitoring-best-practice"></a> 2428 2429We've put together a collection of configuration examples from community feedback. 2430If you like to share your tips and tricks with us, please join the [community channels](https://icinga.com/community/)! 2431 2432### Global Zone for Config Sync <a id="distributed-monitoring-global-zone-config-sync"></a> 2433 2434Global zones can be used to sync generic configuration objects 2435to all nodes depending on them. Common examples are: 2436 2437* Templates which are imported into zone specific objects. 2438* Command objects referenced by Host, Service, Notification objects. 2439* Apply rules for services, notifications and dependencies. 2440* User objects referenced in notifications. 2441* Group objects. 2442* TimePeriod objects. 2443 2444Plugin scripts and binaries must not be synced, this is for Icinga 2 2445configuration files only. Use your preferred package repository 2446and/or configuration management tool (Puppet, Ansible, Chef, etc.) 2447for keeping packages and scripts uptodate. 2448 2449**Note**: Checkable objects (hosts and services) cannot be put into a global 2450zone. The configuration validation will terminate with an error. Apply rules 2451work as they are evaluated locally on each endpoint. 2452 2453The zone object configuration must be deployed on all nodes which should receive 2454the global configuration files: 2455 2456``` 2457[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf 2458 2459object Zone "global-commands" { 2460 global = true 2461} 2462``` 2463 2464The default global zones generated by the setup wizards are called `global-templates` and `director-global`. 2465 2466While you can and should use `global-templates` for your global configuration, `director-global` is reserved for use 2467by [Icinga Director](https://icinga.com/docs/director/latest/). Please don't 2468place any configuration in it manually. 2469 2470Similar to the zone configuration sync you'll need to create a new directory in 2471`/etc/icinga2/zones.d`: 2472 2473``` 2474[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/global-commands 2475``` 2476 2477Next, add a new check command, for example: 2478 2479``` 2480[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/global-commands/web.conf 2481 2482object CheckCommand "webinject" { 2483 //... 2484} 2485``` 2486 2487Restart the endpoints(s) which should receive the global zone before 2488before restarting the parent master/satellite nodes. 2489 2490Then validate the configuration on the master node and restart Icinga 2. 2491 2492**Tip**: You can copy the example configuration files located in `/etc/icinga2/conf.d` 2493into the default global zone `global-templates`. 2494 2495Example: 2496 2497``` 2498[root@icinga2-master1.localdomain /]# cd /etc/icinga2/conf.d 2499[root@icinga2-master1.localdomain /etc/icinga2/conf.d]# cp {commands,groups,notifications,services,templates,timeperiods,users}.conf /etc/icinga2/zones.d/global-templates 2500``` 2501 2502### Health Checks <a id="distributed-monitoring-health-checks"></a> 2503 2504In case of network failures or other problems, your monitoring might 2505either have late check results or just send out mass alarms for unknown 2506checks. 2507 2508In order to minimize the problems caused by this, you should configure 2509additional health checks. 2510 2511#### cluster-zone with Masters and Agents <a id="distributed-monitoring-health-checks-master-agents"></a> 2512 2513The `cluster-zone` check will test whether the configured target zone is currently 2514connected or not. This example adds a health check for the [ha master with agents scenario](06-distributed-monitoring.md#distributed-monitoring-scenarios-ha-master-agents). 2515 2516``` 2517[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/services.conf 2518 2519apply Service "agent-health" { 2520 check_command = "cluster-zone" 2521 2522 display_name = "cluster-health-" + host.name 2523 2524 /* This follows the convention that the agent zone name is the FQDN which is the same as the host object name. */ 2525 vars.cluster_zone = host.name 2526 2527 assign where host.vars.agent_endpoint 2528} 2529``` 2530 2531In order to prevent unwanted notifications, add a service dependency which gets applied to 2532all services using the command endpoint mode. 2533 2534``` 2535[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/dependencies.conf 2536 2537apply Dependency "agent-health-check" to Service { 2538 parent_service_name = "agent-health" 2539 2540 states = [ OK ] // Fail if the parent service state switches to NOT-OK 2541 disable_notifications = true 2542 2543 assign where host.vars.agent_endpoint // Automatically assigns all agent endpoint checks as child services on the matched host 2544 ignore where service.name == "agent-health" // Avoid a self reference from child to parent 2545} 2546``` 2547 2548#### cluster-zone with Masters, Satellites and Agents <a id="distributed-monitoring-health-checks-master-satellite-agent"></a> 2549 2550This example adds health checks for the [master, satellites and agents scenario](06-distributed-monitoring.md#distributed-monitoring-scenarios-master-satellite-agents). 2551 2552Whenever the connection between the master and satellite zone breaks, 2553you may encounter late check results in Icinga Web. In order to view 2554this failure and also send notifications, add the following configuration: 2555 2556First, add the two masters as host objects to the master zone, if not already 2557existing. 2558 2559``` 2560[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/hosts.conf 2561 2562object Host "icinga2-master1.localdomain" { 2563 check_command = "hostalive" 2564 2565 address = "192.168.56.101" 2566} 2567 2568object Host "icinga2-master2.localdomain" { 2569 check_command = "hostalive" 2570 2571 address = "192.168.56.102" 2572} 2573``` 2574 2575Add service health checks against the satellite zone. 2576 2577``` 2578[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/health.conf 2579 2580apply Service "satellite-zone-health" { 2581 check_command = "cluster-zone" 2582 check_interval = 30s 2583 retry_interval = 10s 2584 2585 vars.cluster_zone = "satellite" 2586 2587 assign where match("icinga2-master*.localdomain", host.name) 2588} 2589``` 2590 2591**Don't forget to create notification apply rules for these services.** 2592 2593Next are health checks for agents connected to the satellite zone. 2594Navigate into the satellite directory in `zones.d`: 2595 2596``` 2597[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite 2598``` 2599 2600You should already have configured agent host objects following [the master, satellite, agents scenario](06-distributed-monitoring.md#distributed-monitoring-scenarios-master-satellite-agents). 2601Add a new configuration file where all the health checks are defined. 2602 2603``` 2604[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim health.conf 2605 2606apply Service "agent-health" { 2607 check_command = "cluster-zone" 2608 2609 display_name = "agent-health-" + host.name 2610 2611 // This follows the convention that the agent zone name is the FQDN which is the same as the host object name. 2612 vars.cluster_zone = host.name 2613 2614 // Create this health check for agent hosts in the satellite zone 2615 assign where host.zone == "satellite" && host.vars.agent_endpoint 2616} 2617``` 2618 2619In order to prevent unwanted notifications, add a service dependency which gets applied to 2620all services using the command endpoint mode. 2621 2622``` 2623[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim health.conf 2624 2625apply Dependency "agent-health-check" to Service { 2626 parent_service_name = "agent-health" 2627 2628 states = [ OK ] // Fail if the parent service state switches to NOT-OK 2629 disable_notifications = true 2630 2631 assign where host.zone == "satellite" && host.vars.agent_endpoint // Automatically assigns all agent endpoint checks as child services on the matched host 2632 ignore where service.name == "agent-health" // Avoid a self reference from child to parent 2633} 2634``` 2635 2636This is all done on the configuration master, and requires the scenario to be fully up and running. 2637 2638#### Cluster Check 2639 2640The `cluster` check will check if all endpoints in the current zone and the directly 2641connected zones are working properly. The disadvantage of using this check is that 2642you cannot monitor 3 or more cluster levels with it. 2643 2644``` 2645[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master 2646[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-master1.localdomain.conf 2647 2648object Host "icinga2-master1.localdomain" { 2649 check_command = "hostalive" 2650 address = "192.168.56.101" 2651} 2652 2653[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/cluster.conf 2654 2655object Service "cluster" { 2656 check_command = "cluster" 2657 check_interval = 5s 2658 retry_interval = 1s 2659 2660 host_name = "icinga2-master1.localdomain" 2661} 2662``` 2663 2664### Pin Checks in a Zone <a id="distributed-monitoring-pin-checks-zone"></a> 2665 2666In case you want to pin specific checks to their endpoints in a given zone you'll need to use 2667the `command_endpoint` attribute. This is reasonable if you want to 2668execute a local disk check in the `master` Zone on a specific endpoint then. 2669 2670``` 2671[root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master 2672[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/icinga2-master1.localdomain.conf 2673 2674object Host "icinga2-master1.localdomain" { 2675 check_command = "hostalive" 2676 address = "192.168.56.101" 2677} 2678 2679[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.d/master/services.conf 2680 2681apply Service "disk" { 2682 check_command = "disk" 2683 2684 command_endpoint = host.name //requires a host object matching the endpoint object name e.g. icinga2-master1.localdomain 2685 2686 assign where host.zone == "master" && match("icinga2-master*", host.name) 2687} 2688``` 2689 2690The `host.zone` attribute check inside the expression ensures that 2691the service object is only created for host objects inside the `master` 2692zone. In addition to that the [match](18-library-reference.md#global-functions-match) 2693function ensures to only create services for the master nodes. 2694 2695### Windows Firewall <a id="distributed-monitoring-windows-firewall"></a> 2696 2697#### ICMP Requests <a id="distributed-monitoring-windows-firewall-icmp"></a> 2698 2699By default ICMP requests are disabled in the Windows firewall. You can 2700change that by [adding a new rule](https://support.microsoft.com/en-us/kb/947709). 2701 2702``` 2703C:\> netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow 2704``` 2705 2706#### Icinga 2 <a id="distributed-monitoring-windows-firewall-icinga2"></a> 2707 2708If your master/satellite nodes should actively connect to the Windows agent 2709you'll also need to ensure that port `5665` is enabled. 2710 2711``` 2712C:\> netsh advfirewall firewall add rule name="Open port 5665 (Icinga 2)" dir=in action=allow protocol=TCP localport=5665 2713``` 2714 2715#### NSClient++ API <a id="distributed-monitoring-windows-firewall-nsclient-api"></a> 2716 2717If the [check_nscp_api](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api) 2718plugin is used to query NSClient++, you need to ensure that its port is enabled. 2719 2720``` 2721C:\> netsh advfirewall firewall add rule name="Open port 8443 (NSClient++ API)" dir=in action=allow protocol=TCP localport=8443 2722``` 2723 2724For security reasons, it is advised to enable the NSClient++ HTTP API for local 2725connection from the Icinga agent only. Remote connections to the HTTP API 2726are not recommended with using the legacy HTTP API. 2727 2728### Windows Agent and Plugins <a id="distributed-monitoring-windows-plugins"></a> 2729 2730The Icinga 2 package on Windows already provides several plugins. 2731Detailed [documentation](10-icinga-template-library.md#windows-plugins) is available for all check command definitions. 2732 2733Based on the [master with agents](06-distributed-monitoring.md#distributed-monitoring-master-agents) 2734scenario we'll now add a local disk check. 2735 2736First, add the agent node as host object: 2737 2738``` 2739[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 2740[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 2741 2742object Host "icinga2-agent2.localdomain" { 2743 check_command = "hostalive" 2744 address = "192.168.56.112" 2745 vars.agent_endpoint = name //follows the convention that host name == endpoint name 2746 vars.os_type = "windows" 2747} 2748``` 2749 2750Next, add the disk check using command endpoint checks (details in the 2751[disk-windows](10-icinga-template-library.md#windows-plugins-disk-windows) documentation): 2752 2753``` 2754[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 2755 2756apply Service "disk C:" { 2757 check_command = "disk-windows" 2758 2759 vars.disk_win_path = "C:" 2760 2761 //specify where the check is executed 2762 command_endpoint = host.vars.agent_endpoint 2763 2764 assign where host.vars.os_type == "windows" && host.vars.agent_endpoint 2765} 2766``` 2767 2768Validate the configuration and restart Icinga 2. 2769 2770``` 2771[root@icinga2-master1.localdomain /]# icinga2 daemon -C 2772[root@icinga2-master1.localdomain /]# systemctl restart icinga2 2773``` 2774 2775Open Icinga Web 2 and check your newly added Windows disk check :) 2776 2777![Icinga Windows Agent](images/distributed-monitoring/icinga2_distributed_windows_client_disk_icingaweb2.png) 2778 2779If you want to add your own plugins please check [this chapter](05-service-monitoring.md#service-monitoring-requirements) 2780for the requirements. 2781 2782### Windows Agent and NSClient++ <a id="distributed-monitoring-windows-nscp"></a> 2783 2784There are two methods available for querying NSClient++: 2785 2786* Query the [HTTP API](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api) locally from an Icinga agent (requires a running NSClient++ service) 2787* Run a [local CLI check](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-local) (does not require NSClient++ as a service) 2788 2789Both methods have their advantages and disadvantages. One thing to 2790note: If you rely on performance counter delta calculations such as 2791CPU utilization, please use the HTTP API instead of the CLI sample call. 2792 2793#### NSCLient++ with check_nscp_api <a id="distributed-monitoring-windows-nscp-check-api"></a> 2794 2795The [Windows setup](06-distributed-monitoring.md#distributed-monitoring-setup-agent-windows) already allows 2796you to install the NSClient++ package. In addition to the Windows plugins you can 2797use the [nscp_api command](10-icinga-template-library.md#nscp-check-api) provided by the Icinga Template Library (ITL). 2798 2799The initial setup for the NSClient++ API and the required arguments 2800is the described in the ITL chapter for the [nscp_api](10-icinga-template-library.md#nscp-check-api) CheckCommand. 2801 2802Based on the [master with agents](06-distributed-monitoring.md#distributed-monitoring-master-agents) 2803scenario we'll now add a local nscp check which queries the NSClient++ API to check the free disk space. 2804 2805Define a host object called `icinga2-agent2.localdomain` on the master. Add the `nscp_api_password` 2806custom variable and specify the drives to check. 2807 2808``` 2809[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 2810[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 2811 2812object Host "icinga2-agent1.localdomain" { 2813 check_command = "hostalive" 2814 address = "192.168.56.111" 2815 2816 vars.agent_endpoint = name //follows the convention that host name == endpoint name 2817 vars.os_type = "Windows" 2818 vars.nscp_api_password = "icinga" 2819 vars.drives = [ "C:", "D:" ] 2820} 2821``` 2822 2823The service checks are generated using an [apply for](03-monitoring-basics.md#using-apply-for) 2824rule based on `host.vars.drives`: 2825 2826``` 2827[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 2828 2829apply Service "nscp-api-" for (drive in host.vars.drives) { 2830 import "generic-service" 2831 2832 check_command = "nscp_api" 2833 command_endpoint = host.vars.agent_endpoint 2834 2835 //display_name = "nscp-drive-" + drive 2836 2837 vars.nscp_api_host = "localhost" 2838 vars.nscp_api_query = "check_drivesize" 2839 vars.nscp_api_password = host.vars.nscp_api_password 2840 vars.nscp_api_arguments = [ "drive=" + drive ] 2841 2842 ignore where host.vars.os_type != "Windows" 2843} 2844``` 2845 2846Validate the configuration and restart Icinga 2. 2847 2848``` 2849[root@icinga2-master1.localdomain /]# icinga2 daemon -C 2850[root@icinga2-master1.localdomain /]# systemctl restart icinga2 2851``` 2852 2853Two new services ("nscp-drive-D:" and "nscp-drive-C:") will be visible in Icinga Web 2. 2854 2855![Icinga 2 Distributed Monitoring Windows Agent with NSClient++ nscp-api](images/distributed-monitoring/icinga2_distributed_windows_nscp_api_drivesize_icingaweb2.png) 2856 2857Note: You can also omit the `command_endpoint` configuration to execute 2858the command on the master. This also requires a different value for `nscp_api_host` 2859which defaults to `host.address`. 2860 2861``` 2862 //command_endpoint = host.vars.agent_endpoint 2863 2864 //vars.nscp_api_host = "localhost" 2865``` 2866 2867You can verify the check execution by looking at the `Check Source` attribute 2868in Icinga Web 2 or the REST API. 2869 2870If you want to monitor specific Windows services, you could use the following example: 2871 2872``` 2873[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 2874[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 2875 2876object Host "icinga2-agent1.localdomain" { 2877 check_command = "hostalive" 2878 address = "192.168.56.111" 2879 2880 vars.agent_endpoint = name //follows the convention that host name == endpoint name 2881 vars.os_type = "Windows" 2882 vars.nscp_api_password = "icinga" 2883 vars.services = [ "Windows Update", "wscsvc" ] 2884} 2885 2886[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 2887 2888apply Service "nscp-api-" for (svc in host.vars.services) { 2889 import "generic-service" 2890 2891 check_command = "nscp_api" 2892 command_endpoint = host.vars.agent_endpoint 2893 2894 //display_name = "nscp-service-" + svc 2895 2896 vars.nscp_api_host = "localhost" 2897 vars.nscp_api_query = "check_service" 2898 vars.nscp_api_password = host.vars.nscp_api_password 2899 vars.nscp_api_arguments = [ "service=" + svc ] 2900 2901 ignore where host.vars.os_type != "Windows" 2902} 2903``` 2904 2905#### NSCLient++ with nscp-local <a id="distributed-monitoring-windows-nscp-check-local"></a> 2906 2907The [Windows setup](06-distributed-monitoring.md#distributed-monitoring-setup-agent-windows) allows 2908you to install the bundled NSClient++ package. In addition to the Windows plugins you can 2909use the [nscp-local commands](10-icinga-template-library.md#nscp-plugin-check-commands) 2910provided by the Icinga Template Library (ITL). 2911 2912Add the following `include` statement on all your nodes (master, satellite, agent): 2913 2914``` 2915vim /etc/icinga2/icinga2.conf 2916 2917include <nscp> 2918``` 2919 2920The CheckCommand definitions will automatically determine the installed path 2921to the `nscp.exe` binary. 2922 2923Based on the [master with agents](06-distributed-monitoring.md#distributed-monitoring-master-agents) 2924scenario we'll now add a local nscp check querying a given performance counter. 2925 2926First, add the agent node as host object: 2927 2928``` 2929[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/master 2930[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim hosts.conf 2931 2932object Host "icinga2-agent1.localdomain" { 2933 check_command = "hostalive" 2934 address = "192.168.56.111" 2935 2936 vars.agent_endpoint = name //follows the convention that host name == endpoint name 2937 vars.os_type = "windows" 2938} 2939``` 2940 2941Next, add a performance counter check using command endpoint checks (details in the 2942[nscp-local-counter](10-icinga-template-library.md#nscp-check-local-counter) documentation): 2943 2944``` 2945[root@icinga2-master1.localdomain /etc/icinga2/zones.d/master]# vim services.conf 2946 2947apply Service "nscp-local-counter-cpu" { 2948 check_command = "nscp-local-counter" 2949 command_endpoint = host.vars.agent_endpoint 2950 2951 vars.nscp_counter_name = "\\Processor(_total)\\% Processor Time" 2952 vars.nscp_counter_perfsyntax = "Total Processor Time" 2953 vars.nscp_counter_warning = 1 2954 vars.nscp_counter_critical = 5 2955 2956 vars.nscp_counter_showall = true 2957 2958 assign where host.vars.os_type == "windows" && host.vars.agent_endpoint 2959} 2960``` 2961 2962Validate the configuration and restart Icinga 2. 2963 2964``` 2965[root@icinga2-master1.localdomain /]# icinga2 daemon -C 2966[root@icinga2-master1.localdomain /]# systemctl restart icinga2 2967``` 2968 2969Open Icinga Web 2 and check your newly added Windows NSClient++ check :) 2970 2971![Icinga 2 Distributed Monitoring Windows Agent with NSClient++ nscp-local](images/distributed-monitoring/icinga2_distributed_windows_nscp_counter_icingaweb2.png) 2972 2973> **Tip** 2974> 2975> In order to measure CPU load, you'll need a running NSClient++ service. 2976> Therefore it is advised to use a local [nscp-api](06-distributed-monitoring.md#distributed-monitoring-windows-nscp-check-api) 2977> check against its REST API. 2978 2979## Advanced Hints <a id="distributed-monitoring-advanced-hints"></a> 2980 2981You can find additional hints in this section if you prefer to go your own route 2982with automating setups (setup, certificates, configuration). 2983 2984### Certificate Auto-Renewal <a id="distributed-monitoring-certificate-auto-renewal"></a> 2985 2986Icinga 2 v2.8+ added the possibility that nodes request certificate updates 2987on their own. If their expiration date is soon enough, they automatically 2988renew their already signed certificate by sending a signing request to the 2989parent node. You'll also see a message in the logs if certificate renewal 2990isn't necessary. 2991 2992### High-Availability for Icinga 2 Features <a id="distributed-monitoring-high-availability-features"></a> 2993 2994All nodes in the same zone require that you enable the same features for high-availability (HA). 2995 2996By default, the following features provide advanced HA functionality: 2997 2998* [Checks](06-distributed-monitoring.md#distributed-monitoring-high-availability-checks) (load balanced, automated failover). 2999* [Notifications](06-distributed-monitoring.md#distributed-monitoring-high-availability-notifications) (load balanced, automated failover). 3000* [DB IDO](06-distributed-monitoring.md#distributed-monitoring-high-availability-db-ido) (Run-Once, automated failover). 3001* [Elasticsearch](09-object-types.md#objecttype-elasticsearchwriter) 3002* [Gelf](09-object-types.md#objecttype-gelfwriter) 3003* [Graphite](09-object-types.md#objecttype-graphitewriter) 3004* [InfluxDB](09-object-types.md#objecttype-influxdb2writer) (v1 and v2) 3005* [OpenTsdb](09-object-types.md#objecttype-opentsdbwriter) 3006* [Perfdata](09-object-types.md#objecttype-perfdatawriter) (for PNP) 3007 3008#### High-Availability with Checks <a id="distributed-monitoring-high-availability-checks"></a> 3009 3010All instances within the same zone (e.g. the `master` zone as HA cluster) must 3011have the `checker` feature enabled. 3012 3013Example: 3014 3015```bash 3016icinga2 feature enable checker 3017``` 3018 3019All nodes in the same zone load-balance the check execution. If one instance shuts down, 3020the other nodes will automatically take over the remaining checks. 3021 3022#### High-Availability with Notifications <a id="distributed-monitoring-high-availability-notifications"></a> 3023 3024All instances within the same zone (e.g. the `master` zone as HA cluster) must 3025have the `notification` feature enabled. 3026 3027Example: 3028 3029```bash 3030icinga2 feature enable notification 3031``` 3032 3033Notifications are load-balanced amongst all nodes in a zone. By default this functionality 3034is enabled. 3035If your nodes should send out notifications independently from any other nodes (this will cause 3036duplicated notifications if not properly handled!), you can set `enable_ha = false` 3037in the [NotificationComponent](09-object-types.md#objecttype-notificationcomponent) feature. 3038 3039#### High-Availability with DB IDO <a id="distributed-monitoring-high-availability-db-ido"></a> 3040 3041All instances within the same zone (e.g. the `master` zone as HA cluster) must 3042have the DB IDO feature enabled. 3043 3044Example DB IDO MySQL: 3045 3046```bash 3047icinga2 feature enable ido-mysql 3048``` 3049 3050By default the DB IDO feature only runs on one node. All other nodes in the same zone disable 3051the active IDO database connection at runtime. The node with the active DB IDO connection is 3052not necessarily the zone master. 3053 3054**Note**: The DB IDO HA feature can be disabled by setting the `enable_ha` attribute to `false` 3055for the [IdoMysqlConnection](09-object-types.md#objecttype-idomysqlconnection) or 3056[IdoPgsqlConnection](09-object-types.md#objecttype-idopgsqlconnection) object on **all** nodes in the 3057**same** zone. 3058 3059All endpoints will enable the DB IDO feature and connect to the configured 3060database and dump configuration, status and historical data on their own. 3061 3062If the instance with the active DB IDO connection dies, the HA functionality will 3063automatically elect a new DB IDO master. 3064 3065The DB IDO feature will try to determine which cluster endpoint is currently writing 3066to the database and bail out if another endpoint is active. You can manually verify that 3067by running the following query command: 3068 3069``` 3070icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus; 3071 status_update_time | endpoint_name 3072------------------------+--------------- 3073 2016-08-15 15:52:26+02 | icinga2-master1.localdomain 3074(1 Zeile) 3075``` 3076 3077This is useful when the cluster connection between endpoints breaks, and prevents 3078data duplication in split-brain-scenarios. The failover timeout can be set for the 3079`failover_timeout` attribute, but not lower than 60 seconds. 3080 3081### Endpoint Connection Direction <a id="distributed-monitoring-advanced-hints-connection-direction"></a> 3082 3083Endpoints attempt to connect to another endpoint when its local [Endpoint](09-object-types.md#objecttype-endpoint) object 3084configuration specifies a valid `host` attribute (FQDN or IP address). 3085 3086Example for the master node `icinga2-master1.localdomain` actively connecting 3087to the agent node `icinga2-agent1.localdomain`: 3088 3089``` 3090[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf 3091 3092//... 3093 3094object Endpoint "icinga2-agent1.localdomain" { 3095 host = "192.168.56.111" // The master actively tries to connect to the agent 3096 log_duration = 0 // Disable the replay log for command endpoint agents 3097} 3098``` 3099 3100Example for the agent node `icinga2-agent1.localdomain` not actively 3101connecting to the master node `icinga2-master1.localdomain`: 3102 3103``` 3104[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 3105 3106//... 3107 3108object Endpoint "icinga2-master1.localdomain" { 3109 // Do not actively connect to the master by leaving out the 'host' attribute 3110 log_duration = 0 // Disable the replay log for command endpoint agents 3111} 3112``` 3113 3114It is not necessary that both the master and the agent node establish 3115two connections to each other. Icinga 2 will only use one connection 3116and close the second connection if established. This generates useless 3117CPU cycles and leads to blocking resources when the connection times out. 3118 3119**Tip**: Choose either to let master/satellite nodes connect to agent nodes 3120or vice versa. 3121 3122 3123### Disable Log Duration for Command Endpoints <a id="distributed-monitoring-advanced-hints-command-endpoint-log-duration"></a> 3124 3125The replay log is a built-in mechanism to ensure that nodes in a distributed setup 3126keep the same history (check results, notifications, etc.) when nodes are temporarily 3127disconnected and then reconnect. 3128 3129This functionality is not needed when a master/satellite node is sending check 3130execution events to an agent which is configured as [command endpoint](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) 3131for check execution. 3132 3133The [Endpoint](09-object-types.md#objecttype-endpoint) object attribute `log_duration` can 3134be lower or set to 0 to fully disable any log replay updates when the 3135agent is not connected. 3136 3137Configuration on the master node `icinga2-master1.localdomain`: 3138 3139``` 3140[root@icinga2-master1.localdomain /]# vim /etc/icinga2/zones.conf 3141 3142//... 3143 3144object Endpoint "icinga2-agent1.localdomain" { 3145 host = "192.168.56.111" // The master actively tries to connect to the agent 3146 log_duration = 0 3147} 3148 3149object Endpoint "icinga2-agent2.localdomain" { 3150 host = "192.168.56.112" // The master actively tries to connect to the agent 3151 log_duration = 0 3152} 3153``` 3154 3155Configuration on the agent `icinga2-agent1.localdomain`: 3156 3157``` 3158[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf 3159 3160//... 3161 3162object Endpoint "icinga2-master1.localdomain" { 3163 // Do not actively connect to the master by leaving out the 'host' attribute 3164 log_duration = 0 3165} 3166 3167object Endpoint "icinga2-master2.localdomain" { 3168 // Do not actively connect to the master by leaving out the 'host' attribute 3169 log_duration = 0 3170} 3171``` 3172 3173### Initial Sync for new Endpoints in a Zone <a id="distributed-monitoring-advanced-hints-initial-sync"></a> 3174 3175> **Note** 3176> 3177> This is required if you decide to change an already running single endpoint production 3178> environment into a HA-enabled cluster zone with two endpoints. 3179> The [initial setup](06-distributed-monitoring.md#distributed-monitoring-scenarios-ha-master-clients) 3180> with 2 HA masters doesn't require this step. 3181 3182In order to make sure that all of your zone endpoints have the same state you need 3183to pick the authoritative running one and copy the following content: 3184 3185* State file from `/var/lib/icinga2/icinga2.state` 3186* Internal config package for runtime created objects (downtimes, comments, hosts, etc.) at `/var/lib/icinga2/api/packages/_api` 3187 3188If you need already deployed config packages from the Director, or synced cluster zones, 3189you can also sync the entire `/var/lib/icinga2/api/packages` directory. This directory should also be 3190included in your [backup strategy](02-installation.md#install-backup). 3191 3192Do **not** sync `/var/lib/icinga2/api/zones*` manually - this is an internal directory 3193and handled by the Icinga cluster config sync itself. 3194 3195> **Note** 3196> 3197> Ensure that all endpoints are shut down during this procedure. Once you have 3198> synced the cached files, proceed with configuring the remaining endpoints 3199> to let them know about the new master/satellite node (zones.conf). 3200 3201### Manual Certificate Creation <a id="distributed-monitoring-advanced-hints-certificates-manual"></a> 3202 3203#### Create CA on the Master <a id="distributed-monitoring-advanced-hints-certificates-manual-ca"></a> 3204 3205Choose the host which should store the certificate authority (one of the master nodes). 3206 3207The first step is the creation of the certificate authority (CA) by running the following command 3208as root user: 3209 3210``` 3211[root@icinga2-master1.localdomain /root]# icinga2 pki new-ca 3212``` 3213 3214#### Create CSR and Certificate <a id="distributed-monitoring-advanced-hints-certificates-manual-create"></a> 3215 3216Create a certificate signing request (CSR) for the local instance: 3217 3218``` 3219[root@icinga2-master1.localdomain /root]# icinga2 pki new-cert --cn icinga2-master1.localdomain \ 3220 --key icinga2-master1.localdomain.key \ 3221 --csr icinga2-master1.localdomain.csr 3222``` 3223 3224Sign the CSR with the previously created CA: 3225 3226``` 3227[root@icinga2-master1.localdomain /root]# icinga2 pki sign-csr --csr icinga2-master1.localdomain.csr --cert icinga2-master1.localdomain 3228``` 3229 3230Repeat the steps for all instances in your setup. 3231 3232#### Copy Certificates <a id="distributed-monitoring-advanced-hints-certificates-manual-copy"></a> 3233 3234Copy the host's certificate files and the public CA certificate to `/var/lib/icinga2/certs`: 3235 3236``` 3237[root@icinga2-master1.localdomain /root]# mkdir -p /var/lib/icinga2/certs 3238[root@icinga2-master1.localdomain /root]# cp icinga2-master1.localdomain.{crt,key} /var/lib/icinga2/certs 3239[root@icinga2-master1.localdomain /root]# cp /var/lib/icinga2/ca/ca.crt /var/lib/icinga2/certs 3240``` 3241 3242Ensure that proper permissions are set (replace `icinga` with the Icinga 2 daemon user): 3243 3244``` 3245[root@icinga2-master1.localdomain /root]# chown -R icinga:icinga /var/lib/icinga2/certs 3246[root@icinga2-master1.localdomain /root]# chmod 600 /var/lib/icinga2/certs/*.key 3247[root@icinga2-master1.localdomain /root]# chmod 644 /var/lib/icinga2/certs/*.crt 3248``` 3249 3250The CA public and private key are stored in the `/var/lib/icinga2/ca` directory. Keep this path secure and include 3251it in your backups. 3252 3253#### Create Multiple Certificates <a id="distributed-monitoring-advanced-hints-certificates-manual-multiple"></a> 3254 3255Use your preferred method to automate the certificate generation process. 3256 3257``` 3258[root@icinga2-master1.localdomain /var/lib/icinga2/certs]# for node in icinga2-master1.localdomain icinga2-master2.localdomain icinga2-satellite1.localdomain; do icinga2 pki new-cert --cn $node --csr $node.csr --key $node.key; done 3259information/base: Writing private key to 'icinga2-master1.localdomain.key'. 3260information/base: Writing certificate signing request to 'icinga2-master1.localdomain.csr'. 3261information/base: Writing private key to 'icinga2-master2.localdomain.key'. 3262information/base: Writing certificate signing request to 'icinga2-master2.localdomain.csr'. 3263information/base: Writing private key to 'icinga2-satellite1.localdomain.key'. 3264information/base: Writing certificate signing request to 'icinga2-satellite1.localdomain.csr'. 3265 3266[root@icinga2-master1.localdomain /var/lib/icinga2/certs]# for node in icinga2-master1.localdomain icinga2-master2.localdomain icinga2-satellite1.localdomain; do sudo icinga2 pki sign-csr --csr $node.csr --cert $node.crt; done 3267information/pki: Writing certificate to file 'icinga2-master1.localdomain.crt'. 3268information/pki: Writing certificate to file 'icinga2-master2.localdomain.crt'. 3269information/pki: Writing certificate to file 'icinga2-satellite1.localdomain.crt'. 3270``` 3271 3272Copy and move these certificates to the respective instances e.g. with SSH/SCP. 3273 3274## Automation <a id="distributed-monitoring-automation"></a> 3275 3276These hints should get you started with your own automation tools (Puppet, Ansible, Chef, Salt, etc.) 3277or custom scripts for automated setup. 3278 3279These are collected best practices from various community channels. 3280 3281* [Silent Windows setup](06-distributed-monitoring.md#distributed-monitoring-automation-windows-silent) 3282* [Node Setup CLI command](06-distributed-monitoring.md#distributed-monitoring-automation-cli-node-setup) with parameters 3283 3284If you prefer an alternate method, we still recommend leaving all the Icinga 2 features intact (e.g. `icinga2 feature enable api`). 3285You should also use well known and documented default configuration file locations (e.g. `zones.conf`). 3286This will tremendously help when someone is trying to help in the [community channels](https://icinga.com/community/). 3287 3288 3289### Silent Windows Setup <a id="distributed-monitoring-automation-windows-silent"></a> 3290 3291If you want to install the agent silently/unattended, use the `/qn` modifier. The 3292installation should not trigger a restart, but if you want to be completely sure, you can use the `/norestart` modifier. 3293 3294``` 3295C:> msiexec /i C:\Icinga2-v2.5.0-x86.msi /qn /norestart 3296``` 3297 3298Once the setup is completed you can use the `node setup` cli command too. 3299 3300### Node Setup using CLI Parameters <a id="distributed-monitoring-automation-cli-node-setup"></a> 3301 3302Instead of using the `node wizard` CLI command, there is an alternative `node setup` 3303command available which has some prerequisites. 3304 3305**Note**: The CLI command can be used on Linux/Unix and Windows operating systems. 3306The graphical Windows setup wizard actively uses these CLI commands. 3307 3308#### Node Setup on the Master Node <a id="distributed-monitoring-automation-cli-node-setup-master"></a> 3309 3310In case you want to setup a master node you must add the `--master` parameter 3311to the `node setup` CLI command. In addition to that the `--cn` can optionally 3312be passed (defaults to the FQDN). 3313 3314 Parameter | Description 3315 --------------------|-------------------- 3316 `--cn` | **Optional.** Common name (CN). By convention this should be the host's FQDN. Defaults to the FQDN. 3317 `--zone` | **Optional.** Zone name. Defaults to `master`. 3318 `--listen` | **Optional.** Address to listen on. Syntax is `host,port`. 3319 `--disable-confd` | **Optional.** If provided, this disables the `include_recursive "conf.d"` directive and adds the `api-users.conf` file inclusion to `icinga2.conf`. Available since v2.9+. Not set by default for compatibility reasons with Puppet, Ansible, Chef, etc. 3320 3321Example: 3322 3323``` 3324[root@icinga2-master1.localdomain /]# icinga2 node setup --master 3325``` 3326 3327In case you want to bind the `ApiListener` object to a specific 3328host/port you can specify it like this: 3329 3330``` 3331--listen 192.68.56.101,5665 3332``` 3333 3334In case you don't need anything in `conf.d`, use the following command line: 3335 3336``` 3337[root@icinga2-master1.localdomain /]# icinga2 node setup --master --disable-confd 3338``` 3339 3340<!-- Keep this for compatibility --> 3341<a id="distributed-monitoring-automation-cli-node-setup-satellite-client"></a> 3342 3343#### Node Setup with Agents/Satellites <a id="distributed-monitoring-automation-cli-node-setup-agent-satellite"></a> 3344 3345##### Preparations 3346 3347Make sure that the `/var/lib/icinga2/certs` directory exists and is owned by the `icinga` 3348user (or the user Icinga 2 is running as). 3349 3350``` 3351[root@icinga2-agent1.localdomain /]# mkdir -p /var/lib/icinga2/certs 3352[root@icinga2-agent1.localdomain /]# chown -R icinga:icinga /var/lib/icinga2/certs 3353``` 3354 3355First you'll need to generate a new local self-signed certificate. 3356Pass the following details to the `pki new-cert` CLI command: 3357 3358 Parameter | Description 3359 --------------------|-------------------- 3360 `--cn` | **Required.** Common name (CN). By convention this should be the host's FQDN. 3361 `--key`, `--file` | **Required.** Client certificate files. These generated files will be put into the specified location. By convention this should be using `/var/lib/icinga2/certs` as directory. 3362 3363Example: 3364 3365``` 3366[root@icinga2-agent1.localdomain /]# icinga2 pki new-cert --cn icinga2-agent1.localdomain \ 3367--key /var/lib/icinga2/certs/icinga2-agent1.localdomain.key \ 3368--cert /var/lib/icinga2/certs/icinga2-agent1.localdomain.crt 3369``` 3370 3371##### Verify Parent Connection 3372 3373In order to verify the parent connection and avoid man-in-the-middle attacks, 3374fetch the parent instance's certificate and verify that it matches the connection. 3375The `trusted-parent.crt` file is a temporary file passed to `node setup` in the 3376next step and does not need to be stored for later usage. 3377 3378Pass the following details to the `pki save-cert` CLI command: 3379 3380 Parameter | Description 3381 --------------------|-------------------- 3382 `--trustedcert` | **Required.** Store the parent's certificate file. Manually verify that you're trusting it. 3383 `--host` | **Required.** FQDN or IP address of the parent host. 3384 3385Request the master certificate from the master host (`icinga2-master1.localdomain`) 3386and store it as `trusted-parent.crt`. Review it and continue. 3387 3388``` 3389[root@icinga2-agent1.localdomain /]# icinga2 pki save-cert \ 3390--trustedcert /var/lib/icinga2/certs/trusted-parent.crt \ 3391--host icinga2-master1.localdomain 3392 3393information/cli: Retrieving TLS certificate for 'icinga2-master1.localdomain:5665'. 3394 3395 Subject: CN = icinga2-master1.localdomain 3396 Issuer: CN = icinga2-master1.localdomain 3397 Valid From: Feb 4 08:59:05 2020 GMT 3398 Valid Until: Jan 31 08:59:05 2035 GMT 3399 Fingerprint: B4 90 DE 46 81 DD 2E BF EE 9D D5 47 61 43 EF C6 6D 86 A6 CC 3400 3401*** 3402*** You have to ensure that this certificate actually matches the parent 3403*** instance's certificate in order to avoid man-in-the-middle attacks. 3404*** 3405 3406information/pki: Writing certificate to file '/var/lib/icinga2/certs/trusted-parent.crt'. 3407``` 3408 3409##### Node Setup 3410 3411Continue with the additional `node setup` step. Specify a local endpoint and zone name (`icinga2-agent1.localdomain`) 3412and set the master host (`icinga2-master1.localdomain`) as parent zone configuration. Specify the path to 3413the previously stored trusted parent certificate (`trusted-parent.crt`). 3414 3415Pass the following details to the `node setup` CLI command: 3416 3417 Parameter | Description 3418 --------------------|-------------------- 3419 `--cn` | **Optional.** Common name (CN). By convention this should be the host's FQDN. 3420 `--ticket` | **Required.** Request ticket. Add the previously generated [ticket number](06-distributed-monitoring.md#distributed-monitoring-setup-csr-auto-signing). 3421 `--trustedcert` | **Required.** Trusted parent certificate file as connection verification (received via 'pki save-cert'). 3422 `--parent_host` | **Optional.** FQDN or IP address of the parent host. This is where the command connects for CSR signing. If not specified, you need to manually copy the parent's public CA certificate file into `/var/lib/icinga2/certs/ca.crt` in order to start Icinga 2. 3423 `--endpoint` | **Required.** Specifies the parent's endpoint name. 3424 `--zone` | **Required.** Specifies the agent/satellite zone name. 3425 `--parent_zone` | **Optional.** Specifies the parent's zone name. 3426 `--accept-config` | **Optional.** Whether this node accepts configuration sync from the master node (required for [config sync mode](06-distributed-monitoring.md#distributed-monitoring-top-down-config-sync)). 3427 `--accept-commands` | **Optional.** Whether this node accepts command execution messages from the master node (required for [command endpoint mode](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint)). 3428 `--global_zones` | **Optional.** Allows to specify more global zones in addition to `global-templates` and `director-global`. 3429 `--disable-confd` | **Optional.** If provided, this disables the `include_recursive "conf.d"` directive in `icinga2.conf`. Available since v2.9+. Not set by default for compatibility reasons with Puppet, Ansible, Chef, etc. 3430 3431> **Note** 3432> 3433> The `master_host` parameter is deprecated and will be removed. Please use `--parent_host` instead. 3434 3435Example: 3436 3437``` 3438[root@icinga2-agent1.localdomain /]# icinga2 node setup --ticket ead2d570e18c78abf285d6b85524970a0f69c22d \ 3439--cn icinga2-agent1.localdomain \ 3440--endpoint icinga2-master1.localdomain \ 3441--zone icinga2-agent1.localdomain \ 3442--parent_zone master \ 3443--parent_host icinga2-master1.localdomain \ 3444--trustedcert /var/lib/icinga2/certs/trusted-parent.crt \ 3445--accept-commands --accept-config \ 3446--disable-confd 3447``` 3448 3449In case the agent/satellite should connect to the master node, you'll 3450need to modify the `--endpoint` parameter using the format `cn,host,port`: 3451 3452``` 3453--endpoint icinga2-master1.localdomain,192.168.56.101,5665 3454``` 3455 3456Specify the parent zone using the `--parent_zone` parameter. This is useful 3457if the agent connects to a satellite, not the master instance. 3458 3459``` 3460--parent_zone satellite 3461``` 3462 3463In case the agent should know the additional global zone `linux-templates`, you'll 3464need to set the `--global-zones` parameter. 3465 3466``` 3467--global_zones linux-templates 3468``` 3469 3470The `--parent-host` parameter is optional since v2.9 and allows you to perform a connection-less setup. 3471You cannot restart Icinga 2 yet, the CLI command asked to to manually copy the parent's public CA 3472certificate file in `/var/lib/icinga2/certs/ca.crt`. Once Icinga 2 is started, it sends 3473a ticket signing request to the parent node. If you have provided a ticket, the master node 3474signs the request and sends it back to the agent/satellite which performs a certificate update in-memory. 3475 3476In case you did not provide a ticket, you need to [manually sign the CSR on the master node](06-distributed-monitoring.md#distributed-monitoring-setup-on-demand-csr-signing-master) 3477which holds the CA's key pair. 3478 3479 3480**You can find additional best practices below.** 3481 3482If this agent node is configured as [remote command endpoint execution](06-distributed-monitoring.md#distributed-monitoring-top-down-command-endpoint) 3483you can safely disable the `checker` feature. The `node setup` CLI command already disabled the `notification` feature. 3484 3485``` 3486[root@icinga2-agent1.localdomain /]# icinga2 feature disable checker 3487``` 3488 3489**Optional**: Add an ApiUser object configuration for remote troubleshooting. 3490 3491``` 3492[root@icinga2-agent1.localdomain /]# cat <<EOF >/etc/icinga2/conf.d/api-users.conf 3493object ApiUser "root" { 3494 password = "agentsupersecretpassword" 3495 permissions = ["*"] 3496} 3497EOF 3498``` 3499 3500Finally restart Icinga 2. 3501 3502``` 3503[root@icinga2-agent1.localdomain /]# systemctl restart icinga2 3504``` 3505 3506Your automation tool must then configure master node in the meantime. 3507 3508``` 3509# cat <<EOF >>/etc/icinga2/zones.conf 3510object Endpoint "icinga2-agent1.localdomain" { 3511 // Agent connects itself 3512} 3513 3514object Zone "icinga2-agent1.localdomain" { 3515 endpoints = [ "icinga2-agent1.localdomain" ] 3516 parent = "master" 3517} 3518 3519EOF 3520``` 3521 3522## Using Multiple Environments <a id="distributed-monitoring-environments"></a> 3523 3524> **Note** 3525> 3526> This documentation only covers the basics. Full functionality requires a not yet released addon. 3527 3528In some cases it can be desired to run multiple Icinga instances on the same host. 3529Two potential scenarios include: 3530 3531* Different versions of the same monitoring configuration (e.g. production and testing) 3532* Disparate sets of checks for entirely unrelated monitoring environments (e.g. infrastructure and applications) 3533 3534The configuration is done with the global constants `ApiBindHost` and `ApiBindPort` 3535or the `bind_host` and `bind_port` attributes of the 3536[ApiListener](09-object-types.md#objecttype-apilistener) object. 3537 3538The environment must be set with the global constant `Environment` or as object attribute 3539of the [IcingaApplication](09-object-types.md#objecttype-icingaapplication) object. 3540 3541In any case the constant is default value for the attribute and the direct configuration in the objects 3542have more precedence. The constants have been added to allow the values being set from the CLI on startup. 3543 3544When Icinga establishes a TLS connection to another cluster instance it automatically uses the [SNI extension](https://en.wikipedia.org/wiki/Server_Name_Indication) 3545to signal which endpoint it is attempting to connect to. On its own this can already be used to position multiple 3546Icinga instances behind a load balancer. 3547 3548SNI example: `icinga2-agent1.localdomain` 3549 3550However, if the environment is configured to `production`, Icinga appends the environment name to the SNI hostname like this: 3551 3552SNI example with environment: `icinga2-agent1.localdomain:production` 3553 3554Middleware like loadbalancers or TLS proxies can read the SNI header and route the connection to the appropriate target. 3555I.e., it uses a single externally-visible TCP port (usually 5665) and forwards connections to one or more Icinga 3556instances which are bound to a local TCP port. It does so by inspecting the environment name that is sent as part of the 3557SNI extension. 3558