1# Upgrading Icinga 2 <a id="upgrading-icinga-2"></a> 2 3Upgrading Icinga 2 is usually quite straightforward. 4Ordinarily the only manual steps involved 5are scheme updates for the IDO database. 6 7Specific version upgrades are described below. Please note that version 8updates are incremental. An upgrade from v2.6 to v2.8 requires to 9follow the instructions for v2.7 too. 10 11## Upgrading to v2.13 <a id="upgrading-to-2-13"></a> 12 13### DB IDO Schema Update <a id="upgrading-to-2-13-db-ido"></a> 14 15There is an optional schema update on MySQL which increases the max length of object names from 128 to 255 characters. 16 17Please proceed here for the [MySQL upgrading docs](16-upgrading-icinga-2.md#upgrading-mysql-db). 18 19### Behavior changes <a id="upgrading-to-2-13-behavior-changes"></a> 20 21#### Deletion of child downtimes on services 22 23Service downtimes created while using the `all_services` flag on the [schedule-downtime](12-icinga2-api.md#schedule-downtime) API action 24will now automatically be deleted when deleting the hosts downtime. 25 26#### Windows Event Log 27 28Icinga 2.13 now supports logging to the Windows Event Log. Icinga will now also log messages from the early 29startup phase to the Windows Event Log. These were previously missing from the log file and you could only 30see them by manually starting Icinga in the foreground. 31 32This feature is now enabled and replaces the existing mainlog feature logging to a file. When upgrading, the installer 33will enable the windowseventlog feature and disable the mainlog feature. Logging to a file is still possible. 34If you don't want this configuration migration on upgrade, you can opt-out by installing 35the `%ProgramData%\icinga2\etc\icinga2\features-available\windowseventlog.conf` file before upgrading to Icinga 2.13. 36 37#### Broken API package name validation 38 39This version has replaced a broken regex in the API package validation code which results in package names 40now being validated correctly. Package names should now only consist of alphanumeric characters, dashes and underscores. 41 42This change only applies to newly created packages to support already existing ones. 43 44## Upgrading to v2.12 <a id="upgrading-to-2-12"></a> 45 46* CLI 47 * New `pki verify` CLI command for better [TLS certificate troubleshooting](15-troubleshooting.md#troubleshooting-certificate-verification) 48 49### Behavior changes <a id="upgrading-to-2-12-behavior-changes"></a> 50 51The behavior of multi parent [dependencies](03-monitoring-basics.md#dependencies) was fixed to e.g. render hosts unreachable when both router uplinks are down. 52 53Previous behaviour: 54 551) parentHost1 DOWN, parentHost2 UP => childHost **not reachable** 562) parentHost1 DOWN, parentHost2 DOWN => childHost **not reachable** 57 58New behavior: 59 601) parentHost1 DOWN, parentHost2 UP => childHost **reachable** 612) parentHost1 DOWN, parentHost2 DOWN => childHost **not reachable** 62 63Please review your [Dependency](09-object-types.md#objecttype-dependency) configuration as 1) may lead to 64different results for 65 66- `last_reachable` via REST API query 67- Notifications not suppressed by faulty reachability calculation anymore 68 69### Breaking changes <a id="upgrading-to-2-12-breaking-changes"></a> 70 71As of v2.12 our [API](12-icinga2-api.md) URL endpoint [`/v1/actions/acknowledge-problem`](12-icinga2-api.md#icinga2-api-actions-acknowledge-problem) refuses acknowledging an already acknowledged checkable by overwriting the acknowledgement. 72To replace an acknowledgement you have to remove the old one before adding the new one. 73 74The deprecated parameters `--cert` and `--key` for the `pki save-cert` CLI command 75have been removed from the command and documentation. 76 77## Upgrading to v2.11 <a id="upgrading-to-2-11"></a> 78 79### Bugfixes for 2.11 <a id="upgrading-to-2-11-bugfixes"></a> 80 812.11.1 on agents/satellites fixes a problem where 2.10.x as config master would send out an unwanted config marker file, 82thus rendering the agent to think it is autoritative for the config, and never accepting any new 83config files for the zone(s). **If your config master is 2.11.x already, you are not affected by this problem.** 84 85In order to fix this, upgrade to at least 2.11.1, and purge away the local config sync storage once, then restart. 86 87```bash 88yum install icinga2 89 90rm -rf /var/lib/icinga2/api/zones/* 91rm -rf /var/lib/icinga2/api/zones-stage/* 92 93systemctl restart icinga2 94``` 95 962.11.2 fixes a problem where the newly introduced config sync "check-change-then-reload" functionality 97could cause endless reload loops with agents. The most visible parts are failing command endpoint checks 98with "not connected" UNKNOWN state. **Only applies to HA enabled zones with 2 masters and/or 2 satellites.** 99 100In order to fix this, upgrade all agents/satellites to at least 2.11.2 and restart them. 101 102### Packages <a id="upgrading-to-2-11-packages"></a> 103 104EOL distributions where no packages are available with this release: 105 106* SLES 11 107* Ubuntu 14 LTS 108* RHEL/CentOS 6 x86 109 110Raspbian Packages are available inside the `icinga-buster` repository 111on [https://packages.icinga.com](https://packages.icinga.com/raspbian/). 112Please note that Stretch is not supported suffering from compiler 113regressions. Upgrade to Raspbian Buster is highly recommended. 114 115#### Added: Boost 1.66+ 116 117The rewrite of our core network stack for cluster and REST API 118requires newer Boost versions, specifically >= 1.66. For technical 119details, please continue reading in [this issue](https://github.com/Icinga/icinga2/issues/7041). 120 121Distribution | Repository providing Boost Dependencies 122---------------------|------------------------------------- 123RHEL/CentOS 7 | [EPEL repository](02-installation.md#package-repositories-rhel-epel) 124RHEL/CentOS 6 x64 | [packages.icinga.com](https://packages.icinga.com) 125Fedora | Fedora Upstream 126Debian 10 Buster | Debian Upstream 127Debian 9 Stretch | [Backports repository](02-installation.md#package-repositories-debian-backports) **New since 2.11** 128Debian 8 Jessie | [packages.icinga.com](https://packages.icinga.com) 129Ubuntu 18 Bionic | [packages.icinga.com](https://packages.icinga.com) 130Ubuntu 16 Xenial | [packages.icinga.com](https://packages.icinga.com) 131SLES 15 | SUSE Upstream 132SLES 12 | [packages.icinga.com](https://packages.icinga.com) (replaces the SDK repository requirement) 133 134On platforms where EPEL or Backports cannot satisfy this dependency, 135we provide Boost as package on our [package repository](https://packages.icinga.com) 136for your convenience. 137 138After upgrade, you may remove the old Boost packages (1.53 or anything above) 139if you don't need them anymore. 140 141#### Added: .NET Framework 4.6 142 143We modernized the graphical Windows wizard to use the more recent .NET Framework 4.6. This requires that Windows versions 144older than Windows 10/Windows Server 2016 installs at least [.NET Framework 4.6](https://www.microsoft.com/en-US/download/details.aspx?id=53344). Starting with Windows 10/Windows Server 2016 a .NET Framework 4.6 or higher is installed by default. 145 146The MSI-Installer package checks if the .NET Framework 4.6 or higher is present, if not the installation wizard will abort with an error message telling you to install at least .NET Framework 4.6. 147 148#### Removed: YAJL 149 150Our JSON library, namely [YAJL](https://github.com/lloyd/yajl), isn't maintained anymore 151and may cause [crashes](https://github.com/Icinga/icinga2/issues/6684). 152 153It is replaced by [JSON for Modern C++](https://github.com/nlohmann/json) by Niels Lohmann 154and compiled into the binary as header only include. It helps our way to C++11 and allows 155to fix additional UTF8 issues more easily. Read more about its [design goals](https://github.com/nlohmann/json#design-goals) 156and [benchmarks](https://github.com/miloyip/nativejson-benchmark#parsing-time). 157 158### Core <a id="upgrading-to-2-11-core"></a> 159 160#### Reload Handling <a id="upgrading-to-2-11-core-reload-handling"></a> 161 1622.11 provides fixes for unwanted notifications during restarts. 163The updated systemd service file now uses the `KillMode=mixed` setting. 164 165The reload handling was improved with an umbrella process, which means 166that normal runtime operations include **3 processes**. You may need to 167adjust the local instance monitoring of the [procs](08-advanced-topics.md#monitoring-icinga) check. 168 169More details can be found in the [technical concepts](19-technical-concepts.md#technical-concepts-core-reload) chapter. 170 171#### Downtime Notifications <a id="upgrading-to-2-11-core-downtime-notifications"></a> 172 173Imagine that a host/service changes to a HARD NOT-OK state, 174and its check interval is set to a high interval e.g. 1 hour. 175 176A maintenance downtime prevents the notification being sent, 177but once it ends and the host/service is still in a downtime, 178no immediate notification is re-sent but you'll have to wait 179for the next check. 180 181Another scenario is with one-shot notifications (interval=0) 182which would never notify again after the downtime ends and 183the problem state being intact. The state change logic requires 184to recover and become HARD NOT-OK to notify again. 185 186In order to solve these problems with filtered/suppressed notifications 187in downtimes, v2.11 changes the behaviour like this: 188 189- If there was a notification suppressed in a downtime, the core stores that information 190- Once the downtime ends and the problem state is still intact, Icinga checks whether a re-notification should be sent immediately 191 192A new cluster message was added to keep this in sync amongst HA masters. 193 194> **Important** 195> 196> In order to properly use this new feature, all involved endpoints 197> must be upgraded to v2.11. 198 199### Network Stack <a id="upgrading-to-2-11-network-stack"></a> 200 201The core network stack has been rewritten in 2.11 (some say this could be Icinga 3). 202 203You can read the full story [here](https://github.com/Icinga/icinga2/issues/7041). 204 205The only visible changes for users are: 206 207- No more dead-locks with hanging TLS connections (Cluster, REST API) 208- Better log messages in error cases 209- More robust and stable with using external libraries instead of self-written socket I/O 210 211Coming with this release, we've also updated TLS specific requirements 212explained below. 213 214#### TLS 1.2 <a id="upgrading-to-2-11-network-stack-tls-1-2"></a> 215 216v2.11 raises the minimum required TLS version to 1.2. 217This is available since OpenSSL 1.0.1 (EL6 & Debian Jessie). 218 219Older Icinga satellites/agents need to support TLS 1.2 during the TLS 220handshake. 221 222The `api` feature attribute `tls_protocolmin` now only supports the 223value `TLSv1.2` being the default. 224 225#### Hardened Cipher List <a id="upgrading-to-2-11-network-stack-cipher-list"></a> 226 227The previous default cipher list allowed weak ciphers. There's no sane way 228other than explicitly setting the allowed ciphers. 229 230The new default sets this to: 231 232``` 233ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384:AES128-GCM-SHA256 234``` 235 236You can override this setting in the [api](09-object-types.md#objecttype-apilistener) 237feature with the `cipher_list` attribute. 238 239In case that one of these ciphers is marked as insecure in the future, 240please let us know with an issue on GitHub. 241 242### Cluster <a id="upgrading-to-2-11-cluster"></a> 243 244#### Agent Hosts with Command Endpoint require a Zone <a id="upgrading-to-2-11-cluster-agent-hosts-command-endpoint-zone"></a> 245 2462.11 fixes bugs where agent host checks would never be scheduled on 247the master. One definite requirement is that the checkable host/service 248is put into a zone. 249 250By default, the Director puts the agent host in `zones.d/master` 251and you're good to go. If you manually manage the configuration, 252the config compiler now throws an error with `command_endpoint` 253being set but no `zone` defined. 254 255The most convenient way with e.g. managing the objects in `conf.d` 256is to move them into the `master` zone. Please continue in the 257[troubleshooting docs](15-troubleshooting.md#troubleshooting-cluster-command-endpoint-errors-agent-hosts-command-endpoint-zone) 258for further instructions. 259 260#### Config Sync <a id="upgrading-to-2-11-cluster-config-sync"></a> 261 2622.11 overhauls the cluster config sync in many ways. This includes the 263following under the hood: 264 265- Synced configuration files are not immediately put into production, but left inside a stage. 266- Unsuccessful config validation never puts the config into production, additional logging and API states are available. 267- Zone directories which are not configured in zones.conf, are not included anymore on secondary master/satellites/clients. 268- Synced config change calculation use checksums instead of timestamps to trigger validation/reload. This is more safe, and the usage of timestamps is now deprecated. 269- Don't allow parallel cluster syncs to avoid race conditions with overridden files. 270- Deleted directories and files are now purged, previous versions had a bug. 271 272Whenever a newer child endpoint receives a configuration update without 273checksums, it will log a warning. 274 275``` 276Received configuration update without checksums from parent endpoint satellite1. This behaviour is deprecated. Please upgrade the parent endpoint to 2.11+ 277``` 278 279This is a gentle reminder to upgrade the master and satellites first, 280prior to installing new clients/agents. 281 282Technical details are available in the [technical concepts](19-technical-concepts.md#technical-concepts-cluster-config-sync) chapter. 283 284Since the config sync change detection now uses checksums, this may fail 285with anything else than syncing configuration text files. Syncing binary 286files were never supported, but rumors say that some users do so. 287 288This is now prohibited and logged. 289 290``` 291[2019-08-02 16:03:19 +0200] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/global-templates/forbidden.exe' for cluster config sync: Does not contain valid UTF8. Binary files are not supported. 292Context: 293 (0) Creating config update for file '/etc/icinga2/zones.d/global-templates/forbidden.exe' 294 (1) Activating object 'api' of type 'ApiListener' 295``` 296 297Such binaries wrapped into JSON-RPC cluster messages may always cause changes 298and trigger reload loops. In order to prevent such harm in production, 299use infrastructure tools such as Foreman, Puppet, Ansible, etc. to install 300plugins on the masters, satellites and agents. 301 302##### Config Sync: Zones in Zones <a id="upgrading-to-2-11-cluster-config-sync-zones-in-zones"></a> 303 304The cluster config sync works in the way that configuration 305put into `/etc/icinga2/zones.d` only is included when configured 306outside in `/etc/icinga2/zones.conf`. 307 308If you for example create a "Zone Inception" with defining the 309`satellite` zone in `zones.d/master`, the config compiler does not 310re-run and include this zone config recursively from `zones.d/satellite`. 311 312Since v2.11, the config compiler is only including directories where a 313zone has been configured. Otherwise it would include renamed old zones, 314broken zones, etc. and those long-lasting bugs have been now fixed. 315 316Please consult the [troubleshoot docs](15-troubleshooting.md#troubleshooting-cluster-config-zones-in-zones) 317for concrete examples and solutions. 318 319#### HA-aware Features <a id="upgrading-to-2-11-cluster-ha-aware-features"></a> 320 321v2.11 introduces additional HA functionality similar to the DB IDO feature. 322This enables the feature being active only on one endpoint while the other 323endpoint is paused. When one endpoint is shut down, automatic failover happens. 324 325This feature is turned off by default keeping the current behaviour. If you need 326it active on just one endpoint, set `enable_ha = true` on both endpoints in the 327feature configuration. 328 329This affects the following features: 330 331* [Elasticsearch](09-object-types.md#objecttype-elasticsearchwriter) 332* [Gelf](09-object-types.md#objecttype-gelfwriter) 333* [Graphite](09-object-types.md#objecttype-graphitewriter) 334* [InfluxDB](09-object-types.md#objecttype-influxdbwriter) 335* [OpenTsdb](09-object-types.md#objecttype-opentsdbwriter) 336* [Perfdata](09-object-types.md#objecttype-perfdatawriter) (for PNP) 337 338### HA Failover <a id="upgrading-to-2-11-ha-failover"></a> 339 340The reconnect failover has been improved, and the default `failover_timeout` 341for the DB IDO features has been lowered from 60 to 30 seconds. 342Object authority updates (required for balancing in the cluster) happen 343more frequenty (was 30, is 10 seconds). 344Also the cold startup without object authority updates has been reduced 345from 60 to 30 seconds. This is to allow cluster reconnects (lowered from 60s to 10s in 2.10) 346before actually considering a failover/split brain scenario. 347 348The [IdoMysqlConnection](09-object-types.md#objecttype-idomysqlconnection) and [IdoPgsqlConnection](09-object-types.md#objecttype-idopgsqlconnection) 349objects provide a new attribute named `last_failover` which shows the last failover timestamp. 350This value also is available in the [ido](10-icinga-template-library.md#itl-icinga-ido) CheckCommand output. 351 352 353### CLI Commands <a id="upgrading-to-2-11-cli-commands"></a> 354 355The `troubleshoot` CLI command has been removed. It was never completed, 356and turned out not to provide required details for GitHub issues anyways. 357 358We didn't ask nor endorse users on GitHub/Discourse in the past 2 years, so 359we're removing it without deprecation. 360 361Issue templates, the troubleshooting docs and support knowledge has 362proven to be better. 363 364#### Permissions <a id="upgrading-to-2-11-cli-commands-permissions"></a> 365 366CLI commands such as `api setup`, `node wizard/setup`, `feature enable/disable/list` 367required root permissions previously. Since the file permissions allow 368the Icinga user to change things already, and users kept asking to 369run Icinga on their own webspace without root permissions, this is now possible 370with 2.11. 371 372If you are running the commands with a different user than the 373compiled `ICINGA_USER` and `ICINGA_GROUP` CMake settings (`icinga` everywhere, 374except Debian with `nagios` for historical reasons), ensure that this 375user has the capabilities to change to a different user. 376 377If you still encounter problems, run the aforementioned CLI commands as root, 378or with sudo. 379 380#### CA List Behaviour Change <a id="upgrading-to-2-11-cli-commands-ca-list"></a> 381 382`ca list` only shows the pending certificate signing requests by default. 383 384You can use the new `--all` parameter to show all signing requests. 385Note that Icinga automatically purges signed requests older than 1 week. 386 387#### New: CA Remove/Restore <a id="upgrading-to-2-11-cli-commands-ca-remove-restore"></a> 388 389`ca remove` allows you to remove pending signing requests. Once the 390master receives a CSR, and it is marked as removed, the request is 391denied. 392 393`ca restore` allows you to restore a removed signing request. You 394can list removed signing requests with the new `--removed` parameter 395for `ca list`. 396 397### Configuration <a id="upgrading-to-2-11-configuration"></a> 398 399The deprecated `concurrent_checks` attribute in the [checker feature](09-object-types.md#objecttype-checkercomponent) 400has no effect anymore if set. Please use the [MaxConcurrentChecks](17-language-reference.md#icinga-constants-global-config) 401constant in [constants.conf](04-configuration.md#constants-conf) instead. 402 403### REST API <a id="upgrading-to-2-11-api"></a> 404 405#### Actions <a id="upgrading-to-2-11-api-actions"></a> 406 407The [schedule-downtime](12-icinga2-api.md#icinga2-api-actions-schedule-downtime-host-all-services) 408action supports the `all_services` parameter for Host types. Defaults to false. 409 410#### Config Packages <a id="upgrading-to-2-11-api-config-packages"></a> 411 412Deployed configuration packages require an active stage, with many previous 413allowed. This mechanism is used by the Icinga Director as external consumer, 414and Icinga itself for storing runtime created objects inside the `_api` 415package. 416 417This includes downtimes and comments, which where sometimes stored in the wrong 418directory path, because the active-stage file was empty/truncated/unreadable at 419this point. 420 4212.11 makes this mechanism more stable and detects broken config packages. 422It will also attempt to fix them, the following log entry is perfectly fine. 423 424``` 425[2019-05-10 12:12:09 +0200] information/ConfigObjectUtility: Repairing config package '_api' with stage 'dbe0bef8-c72c-4cc9-9779-da7c4527c5b2'. 426``` 427 428If you still encounter problems, please follow [this troubleshooting entry](15-troubleshooting.md#troubleshooting-api-missing-runtime-objects). 429 430### DB IDO MySQL Schema <a id="upgrading-to-2-11-db-ido"></a> 431 432The schema for MySQL contains an optional update which 433drops unneeded indexes. You don't necessarily need to apply 434this update. 435 436### Documentation <a id="upgrading-to-2-11-documentation"></a> 437 438* `Custom attributes` have been renamed to `Custom variables` following the name `vars` and their usage in backends and web interfaces. 439The term `custom attribute` still applies, but referring from the web to the core docs is easier. 440* The distributed environment term `client` has been refined into `agent`. Wordings and images have been adjusted, and a `client` only is used as 441general term when requesting something from a parent server role. 442* The images for basics, modes and scenarios in the distributed monitoring chapter have been re-created from scratch. 443* `02-getting-started.md` was renamed to `02-installation.md`, `04-configuring-icinga-2.md` into `04-configuration.md`. Apache redirects will be in place. 444 445## Upgrading to v2.10 <a id="upgrading-to-2-10"></a> 446 447### Path Constant Changes <a id="upgrading-to-2-10-path-constant-changes"></a> 448 449During package upgrades you may see a notice that the configuration 450content of features has changed. This is due to a more general approach 451with path constants in v2.10. 452 453The known constants `SysconfDir` and `LocalStateDir` stay intact and won't 454break on upgrade. 455If you are using these constants in your own custom command definitions 456or other objects, you are advised to revise them and update them according 457to the [documentation](17-language-reference.md#icinga-constants). 458 459Example diff: 460 461``` 462object NotificationCommand "mail-service-notification" { 463- command = [ SysconfDir + "/icinga2/scripts/mail-service-notification.sh" ] 464+ command = [ ConfigDir + "/scripts/mail-service-notification.sh" ] 465``` 466 467If you have the `ICINGA2_RUN_DIR` environment variable configured in the 468sysconfig file, you need to rename it to `ICINGA2_INIT_RUN_DIR`. `ICINGA2_STATE_DIR` 469has been removed and this setting has no effect. 470 471> **Note** 472> 473> This is important if you rely on the sysconfig configuration in your own scripts. 474 475### New Constants <a id="upgrading-to-2-10-path-new-constants"></a> 476 477New [Icinga constants](17-language-reference.md#icinga-constants) have been added in this release. 478 479* `Environment` for specifying the Icinga environment. Defaults to not set. 480* `ApiBindHost` and `ApiBindPort` to allow overriding the default ApiListener values. This will be used for an Icinga addon only. 481 482### Configuration: Namespaces <a id="upgrading-to-2-10-configuration-namespaces"></a> 483 484The keywords `namespace` and `using` are now [reserved](17-language-reference.md#reserved-keywords) for the namespace functionality provided 485with v2.10. Read more about how it works [here](17-language-reference.md#namespaces). 486 487### Configuration: ApiListener <a id="upgrading-to-2-10-configuration-apilistener"></a> 488 489Anonymous JSON-RPC connections in the cluster can now be configured with `max_anonymous_clients` 490attribute. 491The corresponding REST API results from `/v1/status/ApiListener` in `json_rpc` have been renamed 492from `clients` to `anonymous_clients` to better reflect their purpose. Authenticated clients 493are counted as connected endpoints. A similar change is there for the performance data metrics. 494 495The TLS handshake timeout defaults to 10 seconds since v2.8.2. This can now be configured 496with the configuration attribute `tls_handshake_timeout`. Beware of performance issues 497with setting a too high value. 498 499### API: schedule-downtime Action <a id="upgrading-to-2-10-api-schedule-downtime-action"></a> 500 501The attribute `child_options` was previously accepting 0,1,2 for specific child downtime settings. 502This behaviour stays intact, but the new proposed way are specific constants as values (`DowntimeNoChildren`, `DowntimeTriggeredChildren`, `DowntimeNonTriggeredChildren`). 503 504### Notifications: Recovery and Acknowledgement <a id="upgrading-to-2-10-notifications"></a> 505 506When a user should be notified on `Problem` and `Acknowledgement`, v2.10 now checks during 507the `Acknowledgement` notification event whether this user has been notified about a problem before. 508 509``` 510 types = [ Problem, Acknowledgement, Recovery ] 511``` 512 513If **no** `Problem` notification was sent, and the types filter includes problems for this user, 514the `Acknowledgement` notification is **not** sent. 515 516In contrast to that, the following configuration always sends `Acknowledgement` notifications. 517 518``` 519 types = [ Acknowledgement, Recovery ] 520``` 521 522This change also restores the old behaviour for `Recovery` notifications. The above configuration 523leaving out the `Problem` type can be used to only receive recovery notifications. If `Problem` 524is added to the types again, Icinga 2 checks whether it has notified a user of a problem when 525sending the recovery notification. 526 527More details can be found in [this PR](https://github.com/Icinga/icinga2/pull/6527). 528 529### Stricter configuration validation 530 531Some config errors are now fatal. While it never worked before, icinga2 refuses to start now! 532 533For example the following started to give a fatal error in 2.10: 534 535``` 536 object Zone "XXX" { 537 endpoints = [ "master-server" ] 538 parent = "global-templates" 539 } 540``` 541 542```critical/config: Error: Zone 'XXX' can not have a global zone as parent.``` 543 544### Package Changes <a id="upgrading-to-2-10-package-changes"></a> 545 546Debian/Ubuntu drops the `libicinga2` package. `apt-get upgrade icinga2` 547won't remove such packages leaving the upgrade in an unsatisfied state. 548 549Please use `apt-get full-upgrade` or `apt-get dist-upgrade` instead, as explained [here](https://github.com/Icinga/icinga2/issues/6695#issuecomment-430585915). 550 551On RHEL/CentOS/Fedora, `icinga2-libs` has been obsoleted. Unfortunately yum's dependency 552resolver doesn't allow to install older versions than 2.10 then. Please 553read [here](https://github.com/Icinga/icinga-packaging/issues/114#issuecomment-429264827) 554for details. 555 556RPM packages dropped the [Classic UI](16-upgrading-icinga-2.md#upgrading-to-2-8-removed-classicui-config-package) 557package in v2.8, Debian/Ubuntu packages were forgotten. This is now the case with this 558release. Icinga 1.x is EOL by the end of 2018, plan your migration to [Icinga Web 2](https://icinga.com/docs/icingaweb2/latest/). 559 560## Upgrading to v2.9 <a id="upgrading-to-2-9"></a> 561 562### Deprecation and Removal Notes <a id="upgrading-to-2-9-deprecation-removal-notes"></a> 563 564- Deprecation of 1.x compatibility features: `StatusDataWriter`, `CompatLogger`, `CheckResultReader`. Their removal is scheduled for 2.11. 565Icinga 1.x is EOL and will be out of support by the end of 2018. 566- Removal of Icinga Studio. It always has been experimental and did not satisfy our high quality standards. We've therefore removed it. 567 568### Sysconfig Changes <a id="upgrading-to-2-9-sysconfig-changes"></a> 569 570The security fixes in v2.8.2 required moving specific runtime settings 571into the Sysconfig file and environment. This included that Icinga 2 572would itself parse this file and read the required variables. This has generated 573numerous false-positive log messages and led to many support questions. v2.9.0 574changes this in the standard way to read these variables from the environment, and use 575sane compile-time defaults. 576 577> **Note** 578> 579> In order to upgrade, remove everything in the sysconfig file and re-apply 580> your changes. 581 582There is a bug with existing sysconfig files where path variables are not expanded 583because systemd [does not support](https://github.com/systemd/systemd/issues/2123) 584shell variable expansion. This worked with SysVInit though. 585 586Edit the sysconfig file and either remove everything, or edit this line 587on RHEL 7. Modify the path for other distributions. 588 589``` 590vim /etc/sysconfig/icinga2 591 592-ICINGA2_PID_FILE=$ICINGA2_RUN_DIR/icinga2/icinga2.pid 593+ICINGA2_PID_FILE=/run/icinga2/icinga2.pid 594``` 595 596If you want to adjust the number of open files for the Icinga application 597for example, you would just add this setting like this on RHEL 7: 598 599``` 600vim /etc/sysconfig/icinga2 601 602ICINGA2_RLIMIT_FILES=50000 603``` 604 605Restart Icinga 2 afterwards, the systemd service file automatically puts the 606value into the application's environment where this is read on startup. 607 608### Setup Wizard Changes <a id="upgrading-to-2-9-setup-wizard-changes"></a> 609 610Client and satellite setups previously had the example configuration in `conf.d` included 611by default. This caused trouble on config sync, or with locally executed checks generating 612wrong check results for command endpoint clients. 613 614In v2.9.0 `node wizard`, `node setup` and the graphical Windows wizard will disable 615the inclusion by default. You can opt-out and explicitly enable it again if needed. 616 617In addition to the default global zones `global-templates` and `director-global`, 618the setup wizards also offer to specify your own custom global zones and generate 619the required configuration automatically. 620 621The setup wizards also use full qualified names for Zone and Endpoint object generation, 622either the default values (FQDN for clients) or the user supplied input. This removes 623the dependency on the `NodeName` and `ZoneName` constant and helps to immediately see 624the parent-child relationship. Those doing support will also see the benefit in production. 625 626### CLI Command Changes <a id="upgrading-to-2-9-cli-changes"></a> 627 628The [node setup](06-distributed-monitoring.md#distributed-monitoring-automation-cli-node-setup) 629parameter `--master_host` was deprecated and replaced with `--parent_host`. 630This parameter is now optional to allow connection-less client setups similar to the `node wizard` 631CLI command. The `parent_zone` parameter has been added to modify the parent zone name e.g. 632for client-to-satellite setups. 633 634The `api user` command which was released in v2.8.2 turned out to cause huge problems with 635configuration validation, windows restarts and OpenSSL versions. It is therefore removed in 2.9, 636the `password_hash` attribute for the ApiUser object stays intact but has no effect. This is to ensure 637that clients don't break on upgrade. We will revise this feature in future development iterations. 638 639### Configuration Changes <a id="upgrading-to-2-9-config-changes"></a> 640 641The CORS attributes `access_control_allow_credentials`, `access_control_allow_headers` and 642`access_control_allow_methods` are now controlled by Icinga 2 and cannot be changed anymore. 643 644### Unique Generated Names <a id="upgrading-to-2-9-unique-name-changes"></a> 645 646With the removal of RHEL 5 as supported platform, we can finally use real unique IDs. 647This is reflected in generating names for e.g. API stage names. Previously it was a handcrafted 648mix of local FQDN, timestamps and random numbers. 649 650### Custom Vars not updating <a id="upgrading-to-2-9-custom-vars-not-updating"></a> 651 652A rare issue preventing the custom vars of objects created prior to 2.9.0 being updated when changed may occur. To 653remedy this, truncate the customvar tables and restart Icinga 2. The following is an example of how to do this with mysql: 654 655``` 656$ mysql -uroot -picinga icinga 657MariaDB [icinga]> truncate icinga_customvariables; 658Query OK, 0 rows affected (0.05 sec) 659MariaDB [icinga]> truncate icinga_customvariablestatus; 660Query OK, 0 rows affected (0.03 sec) 661MariaDB [icinga]> exit 662Bye 663$ sudo systemctl restart icinga2 664``` 665 666Custom vars should now stay up to date. 667 668 669## Upgrading to v2.8.2 <a id="upgrading-to-2-8-2"></a> 670 671With version 2.8.2 the location of settings formerly found in `/etc/icinga2/init.conf` has changed. They are now 672located in the sysconfig, `/etc/sysconfig/icinga2` (RPM) or `/etc/default/icinga2` (DPKG) on most systems. The 673`init.conf` file has been removed and its settings will be ignored. These changes are only relevant if you edited the 674`init.conf`. Below is a table displaying the new names for the affected settings. 675 676 Old `init.conf` | New `sysconfig/icinga2` 677 ----------------|------------------------ 678 RunAsUser | ICINGA2\_USER 679 RunAsGroup | ICINGA2\_GROUP 680 RLimitFiles | ICINGA2\_RLIMIT\_FILES 681 RLimitProcesses | ICINGA2\_RLIMIT\_PROCESSES 682 RLimitStack | ICINGA2\_RLIMIT\_STACK 683 684## Upgrading to v2.8 <a id="upgrading-to-2-8"></a> 685 686### DB IDO Schema Update to 2.8.0 <a id="upgrading-to-2-8-db-ido"></a> 687 688There are additional indexes and schema fixes which require an update. 689 690Please proceed here for [MySQL](16-upgrading-icinga-2.md#upgrading-mysql-db) or [PostgreSQL](16-upgrading-icinga-2.md#upgrading-postgresql-db). 691 692> **Note** 693> 694> `2.8.1.sql` fixes a unique constraint problem with fresh 2.8.0 installations. 695> You don't need this update if you are upgrading from an older version. 696 697### Changed Certificate Paths <a id="upgrading-to-2-8-certificate-paths"></a> 698 699The default certificate path was changed from `/etc/icinga2/pki` to 700`/var/lib/icinga2/certs`. 701 702 Old Path | New Path 703 ---------------------------------------------------|--------------------------------------------------- 704 `/etc/icinga2/pki/icinga2-agent1.localdomain.crt` | `/var/lib/icinga2/certs/icinga2-agent1.localdomain.crt` 705 `/etc/icinga2/pki/icinga2-agent1.localdomain.key` | `/var/lib/icinga2/certs/icinga2-agent1.localdomain.key` 706 `/etc/icinga2/pki/ca.crt` | `/var/lib/icinga2/certs/ca.crt` 707 708This applies to Windows clients in the same way: `%ProgramData%\etc\icinga2\pki` 709was moved to `%ProgramData%\var\lib\icinga2\certs`. 710 711 Old Path | New Path 712 ----------------------------------------------------------------|---------------------------------------------------------------- 713 `%ProgramData%\etc\icinga2\pki\icinga2-agent1.localdomain.crt` | `%ProgramData%\var\lib\icinga2\certs\icinga2-agent1.localdomain.crt` 714 `%ProgramData%\etc\icinga2\pki\icinga2-agent1.localdomain.key` | `%ProgramData%\var\lib\icinga2\certs\icinga2-agent1.localdomain.key` 715 `%ProgramData%\etc\icinga2\pki\ca.crt` | `%ProgramData%\var\lib\icinga2\certs\ca.crt` 716 717 718> **Note** 719> 720> The default expected path for client certificates is `/var/lib/icinga2/certs/ + NodeName + {.crt,.key}`. 721> The `NodeName` constant is usually the FQDN and certificate common name (CN). Check the [conventions](06-distributed-monitoring.md#distributed-monitoring-conventions) 722> section inside the Distributed Monitoring chapter. 723 724The [setup CLI commands](06-distributed-monitoring.md#distributed-monitoring-setup-master) and the 725default [ApiListener configuration](06-distributed-monitoring.md#distributed-monitoring-apilistener) 726have been adjusted to these paths too. 727 728The [ApiListener](09-object-types.md#objecttype-apilistener) object attributes `cert_path`, `key_path` 729and `ca_path` have been deprecated and removed from the example configuration. 730 731#### Migration Path <a id="upgrading-to-2-8-certificate-paths-migration-path"></a> 732 733> **Note** 734> 735> Icinga 2 automatically migrates the certificates to the new default location if they 736> are configured and detected in `/etc/icinga2/pki`. 737 738During startup, the migration kicks in and ensures to copy the certificates to the new 739location. This will also happen if someone updates the certificate files in `/etc/icinga2/pki` 740to ensure that the new certificate location always has the latest files. 741 742This has been implemented in the Icinga 2 binary to ensure it works on both Linux/Unix 743and the Windows platform. 744 745If you are not using the built-in CLI commands and setup wizards to deploy the client certificates, 746please ensure to update your deployment tools/scripts. This mainly affects 747 748* Puppet modules 749* Ansible playbooks 750* Chef cookbooks 751* Salt recipes 752* Custom scripts, e.g. Windows Powershell or self-made implementations 753 754In order to support a smooth migration between versions older than 2.8 and future releases, 755the built-in certificate migration path is planned to exist as long as the deprecated 756`ApiListener` object attributes exist. 757 758You are safe to use the existing configuration paths inside the `api` feature. 759 760**Example** 761 762Look at the following example taken from the Director Linux deployment script for clients. 763 764* Ensure that the default certificate path is changed from `/etc/icinga2/pki` to `/var/lib/icinga2/certs`. 765 766``` 767-ICINGA2_SSL_DIR="${ICINGA2_CONF_DIR}/pki" 768+ICINGA2_SSL_DIR="${ICINGA2_STATE_DIR}/lib/icinga2/certs" 769``` 770 771* Remove the ApiListener configuration attributes. 772 773``` 774object ApiListener "api" { 775- cert_path = SysconfDir + "/icinga2/pki/${ICINGA2_NODENAME}.crt" 776- key_path = SysconfDir + "/icinga2/pki/${ICINGA2_NODENAME}.key" 777- ca_path = SysconfDir + "/icinga2/pki/ca.crt" 778 accept_commands = true 779 accept_config = true 780} 781``` 782 783Test the script with a fresh client installation before putting it into production. 784 785> **Tip** 786> 787> Please support module and script developers in their migration. If you find 788> any project which would require these changes, create an issue or a patchset in a PR 789> and help them out. Thanks in advance! 790 791### On-Demand Signing and CA Proxy <a id="upgrading-to-2-8-on-demand-signing-ca-proxy"></a> 792 793Icinga 2 v2.8 supports the following features inside the cluster: 794 795* Forward signing requests from clients through a satellite instance to a signing master ("CA Proxy"). 796* Signing requests without a ticket. The master instance allows to list and sign CSRs ("On-Demand Signing"). 797 798In order to use these features, **all instances must be upgraded to v2.8**. 799 800More details in [this chapter](06-distributed-monitoring.md#distributed-monitoring-setup-sign-certificates-master). 801 802### Windows Client <a id="upgrading-to-2-8-windows-client"></a> 803 804Windows versions older than Windows 10/Server 2016 require the [Universal C Runtime for Windows](https://support.microsoft.com/en-us/help/2999226/update-for-universal-c-runtime-in-windows). 805 806### Removed Bottom Up Client Mode <a id="upgrading-to-2-8-removed-bottom-up-client-mode"></a> 807 808This client mode was deprecated in 2.6 and was removed in 2.8. 809 810The node CLI command does not provide `list` or `update-config` anymore. 811 812> **Note** 813> 814> The old migration guide can be found on [GitHub](https://github.com/Icinga/icinga2/blob/v2.7.0/doc/06-distributed-monitoring.md#bottom-up-migration-to-top-down-). 815 816The clients don't need to have a local `conf.d` directory included. 817 818Icinga 2 continues to run with the generated and imported configuration. 819You are advised to [migrate](https://github.com/Icinga/icinga2/issues/4798) 820any existing configuration to the "top down" mode with the help of the 821Icinga Director or config management tools such as Puppet, Ansible, etc. 822 823 824### Removed Classic UI Config Package <a id="upgrading-to-2-8-removed-classicui-config-package"></a> 825 826The config meta package `classicui-config` and the configuration files 827have been removed. You need to manually configure 828this legacy interface. Create a backup of the configuration 829before upgrading and re-configure it afterwards. 830 831 832### Flapping Configuration <a id="upgrading-to-2-8-flapping-configuration"></a> 833 834Icinga 2 v2.8 implements a new flapping detection algorithm which splits the 835threshold configuration into low and high settings. 836 837`flapping_threshold` is deprecated and does not have any effect when flapping 838is enabled. Please remove `flapping_threshold` from your configuration. This 839attribute will be removed in v2.9. 840 841Instead you need to use the `flapping_threshold_low` and `flapping_threshold_high` 842attributes. More details can be found [here](08-advanced-topics.md#check-flapping). 843 844### Deprecated Configuration Attributes <a id="upgrading-to-2-8-deprecated-configuration"></a> 845 846 Object | Attribute 847 --------------|------------------ 848 ApiListener | cert\_path (migration happens) 849 ApiListener | key\_path (migration happens) 850 ApiListener | ca\_path (migration happens) 851 Host, Service | flapping\_threshold (has no effect) 852 853## Upgrading to v2.7 <a id="upgrading-to-2-7"></a> 854 855v2.7.0 provided new notification scripts and commands. Please ensure to 856update your configuration accordingly. An advisory has been published [here](https://icinga.com/2017/08/23/advisory-for-icinga-2-v2-7-update-and-mail-notification-scripts/). 857 858In case are having troubles with OpenSSL 1.1.0 and the 859public CA certificates, please read [this advisory](https://icinga.com/2017/08/30/advisory-for-ssl-problems-with-leading-zeros-on-openssl-1-1-0/) 860and check the [troubleshooting chapter](15-troubleshooting.md#troubleshooting). 861 862If Icinga 2 fails to start with an empty reference to `$ICINGA2_CACHE_DIR` 863ensure to set it inside `/etc/sysconfig/icinga2` (RHEL) or `/etc/default/icinga2` (Debian). 864 865RPM packages will put a file called `/etc/sysconfig/icinga2.rpmnew` 866if you have modified the original file. 867 868Example on CentOS 7: 869 870``` 871vim /etc/sysconfig/icinga2 872 873ICINGA2_CACHE_DIR=/var/cache/icinga2 874 875systemctl restart icinga2 876``` 877 878## Upgrading the MySQL database <a id="upgrading-mysql-db"></a> 879 880If you want to upgrade an existing Icinga 2 instance, check the 881`/usr/share/icinga2-ido-mysql/schema/upgrade` directory for incremental schema upgrade file(s). 882 883> **Note** 884> 885> If there isn't an upgrade file for your current version available, there's nothing to do. 886 887Apply all database schema upgrade files incrementally. 888 889``` 890# mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/upgrade/<version>.sql 891``` 892 893The Icinga 2 DB IDO feature checks the required database schema version on startup 894and generates an log message if not satisfied. 895 896 897**Example:** You are upgrading Icinga 2 from version `2.4.0` to `2.8.0`. Look into 898the `upgrade` directory: 899 900``` 901$ ls /usr/share/icinga2-ido-mysql/schema/upgrade/ 9022.0.2.sql 2.1.0.sql 2.2.0.sql 2.3.0.sql 2.4.0.sql 2.5.0.sql 2.6.0.sql 2.8.0.sql 903``` 904 905There are two new upgrade files called `2.5.0.sql`, `2.6.0.sql` and `2.8.0.sql` 906which must be applied incrementally to your IDO database. 907 908```bash 909mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/upgrade/2.5.0.sql 910mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/upgrade/2.6.0.sql 911mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/upgrade/2.8.0.sql 912``` 913 914## Upgrading the PostgreSQL database <a id="upgrading-postgresql-db"></a> 915 916If you want to upgrade an existing Icinga 2 instance, check the 917`/usr/share/icinga2-ido-pgsql/schema/upgrade` directory for incremental schema upgrade file(s). 918 919> **Note** 920> 921> If there isn't an upgrade file for your current version available, there's nothing to do. 922 923Apply all database schema upgrade files incrementally. 924 925``` 926# export PGPASSWORD=icinga 927# psql -U icinga -d icinga < /usr/share/icinga2-ido-pgsql/schema/upgrade/<version>.sql 928``` 929 930The Icinga 2 DB IDO feature checks the required database schema version on startup 931and generates an log message if not satisfied. 932 933**Example:** You are upgrading Icinga 2 from version `2.4.0` to `2.8.0`. Look into 934the `upgrade` directory: 935 936``` 937$ ls /usr/share/icinga2-ido-pgsql/schema/upgrade/ 9382.0.2.sql 2.1.0.sql 2.2.0.sql 2.3.0.sql 2.4.0.sql 2.5.0.sql 2.6.0.sql 2.8.0.sql 939``` 940 941There are two new upgrade files called `2.5.0.sql`, `2.6.0.sql` and `2.8.0.sql` 942which must be applied incrementally to your IDO database. 943 944```bash 945export PGPASSWORD=icinga 946psql -U icinga -d icinga < /usr/share/icinga2-ido-pgsql/schema/upgrade/2.5.0.sql 947psql -U icinga -d icinga < /usr/share/icinga2-ido-pgsql/schema/upgrade/2.6.0.sql 948psql -U icinga -d icinga < /usr/share/icinga2-ido-pgsql/schema/upgrade/2.8.0.sql 949``` 950