1--- 2title: How to monitor your backups 3eleventyNavigation: 4 key: Monitor your backups 5 parent: How-to guides 6 order: 5 7--- 8 9## Monitoring and alerting 10 11Having backups is great, but they won't do you a lot of good unless you have 12confidence that they're running on a regular basis. That's where monitoring 13and alerting comes in. 14 15There are several different ways you can monitor your backups and find out 16whether they're succeeding. Which of these you choose to do is up to you and 17your particular infrastructure. 18 19### Job runner alerts 20 21The easiest place to start is with failure alerts from the [scheduled job 22runner](https://torsion.org/borgmatic/docs/how-to/set-up-backups/#autopilot) 23(cron, systemd, etc.) that's running borgmatic. But note that if the job 24doesn't even get scheduled (e.g. due to the job runner not running), you 25probably won't get an alert at all! Still, this is a decent first line of 26defense, especially when combined with some of the other approaches below. 27 28### Commands run on error 29 30The `on_error` hook allows you to run an arbitrary command or script when 31borgmatic itself encounters an error running your backups. So for instance, 32you can run a script to send yourself a text message alert. But note that if 33borgmatic doesn't actually run, this alert won't fire. See [error 34hooks](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#error-hooks) 35below for how to configure this. 36 37### Third-party monitoring services 38 39borgmatic integrates with monitoring services like 40[Healthchecks](https://healthchecks.io/), [Cronitor](https://cronitor.io), 41[Cronhub](https://cronhub.io), and [PagerDuty](https://www.pagerduty.com/) and 42pings these services whenever borgmatic runs. That way, you'll receive an 43alert when something goes wrong or (for certain hooks) the service doesn't 44hear from borgmatic for a configured interval. See [Healthchecks 45hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#healthchecks-hook), 46[Cronitor 47hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#cronitor-hook), 48[Cronhub 49hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#cronhub-hook), 50and [PagerDuty 51hook](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#pagerduty-hook) 52below for how to configure this. 53 54While these services offer different features, you probably only need to use 55one of them at most. 56 57### Third-party monitoring software 58 59You can use traditional monitoring software to consume borgmatic JSON output 60and track when the last successful backup occurred. See [scripting 61borgmatic](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#scripting-borgmatic) 62and [related 63software](https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/#related-software) 64below for how to configure this. 65 66### Borg hosting providers 67 68Most [Borg hosting 69providers](https://torsion.org/borgmatic/#hosting-providers) include 70monitoring and alerting as part of their offering. This gives you a dashboard 71to check on all of your backups, and can alert you if the service doesn't hear 72from borgmatic for a configured interval. 73 74### Consistency checks 75 76While not strictly part of monitoring, if you really want confidence that your 77backups are not only running but are restorable as well, you can configure 78particular [consistency 79checks](https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/#consistency-check-configuration) 80or even script full [extract 81tests](https://torsion.org/borgmatic/docs/how-to/extract-a-backup/). 82 83 84## Error hooks 85 86When an error occurs during a `prune`, `create`, or `check` action, borgmatic 87can run configurable shell commands to fire off custom error notifications or 88take other actions, so you can get alerted as soon as something goes wrong. 89Here's a not-so-useful example: 90 91```yaml 92hooks: 93 on_error: 94 - echo "Error while creating a backup or running a backup hook." 95``` 96 97The `on_error` hook supports interpolating particular runtime variables into 98the hook command. Here's an example that assumes you provide a separate shell 99script to handle the alerting: 100 101```yaml 102hooks: 103 on_error: 104 - send-text-message.sh "{configuration_filename}" "{repository}" 105``` 106 107In this example, when the error occurs, borgmatic interpolates a few runtime 108values into the hook command: the borgmatic configuration filename, and the 109path of the repository. Here's the full set of supported variables you can use 110here: 111 112 * `configuration_filename`: borgmatic configuration filename in which the 113 error occurred 114 * `repository`: path of the repository in which the error occurred (may be 115 blank if the error occurs in a hook) 116 * `error`: the error message itself 117 * `output`: output of the command that failed (may be blank if an error 118 occurred without running a command) 119 120Note that borgmatic runs the `on_error` hooks only for `prune`, `create`, or 121`check` actions or hooks in which an error occurs, and not other actions. 122borgmatic does not run `on_error` hooks if an error occurs within a 123`before_everything` or `after_everything` hook. For more about hooks, see the 124[borgmatic hooks 125documentation](https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/), 126especially the security information. 127 128 129## Healthchecks hook 130 131[Healthchecks](https://healthchecks.io/) is a service that provides "instant 132alerts when your cron jobs fail silently", and borgmatic has built-in 133integration with it. Once you create a Healthchecks account and project on 134their site, all you need to do is configure borgmatic with the unique "Ping 135URL" for your project. Here's an example: 136 137 138```yaml 139hooks: 140 healthchecks: https://hc-ping.com/addffa72-da17-40ae-be9c-ff591afb942a 141``` 142 143With this hook in place, borgmatic pings your Healthchecks project when a 144backup begins, ends, or errors. Specifically, after the <a 145href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup` 146hooks</a> run, borgmatic lets Healthchecks know that it has started if any of 147the `prune`, `create`, or `check` actions are run. 148 149Then, if the actions complete successfully, borgmatic notifies Healthchecks of 150the success after the `after_backup` hooks run, and includes borgmatic logs in 151the payload data sent to Healthchecks. This means that borgmatic logs show up 152in the Healthchecks UI, although be aware that Healthchecks currently has a 15310-kilobyte limit for the logs in each ping. 154 155If an error occurs during any action or hook, borgmatic notifies Healthchecks 156after the `on_error` hooks run, also tacking on logs including the error 157itself. But the logs are only included for errors that occur when a `prune`, 158`create`, or `check` action is run. 159 160You can customize the verbosity of the logs that are sent to Healthchecks with 161borgmatic's `--monitoring-verbosity` flag. The `--files` and `--stats` flags 162may also be of use. See `borgmatic --help` for more information. 163 164You can configure Healthchecks to notify you by a [variety of 165mechanisms](https://healthchecks.io/#welcome-integrations) when backups fail 166or it doesn't hear from borgmatic for a certain period of time. 167 168 169## Cronitor hook 170 171[Cronitor](https://cronitor.io/) provides "Cron monitoring and uptime healthchecks 172for websites, services and APIs", and borgmatic has built-in 173integration with it. Once you create a Cronitor account and cron job monitor on 174their site, all you need to do is configure borgmatic with the unique "Ping 175API URL" for your monitor. Here's an example: 176 177 178```yaml 179hooks: 180 cronitor: https://cronitor.link/d3x0c1 181``` 182 183With this hook in place, borgmatic pings your Cronitor monitor when a backup 184begins, ends, or errors. Specifically, after the <a 185href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup` 186hooks</a> run, borgmatic lets Cronitor know that it has started if any of the 187`prune`, `create`, or `check` actions are run. Then, if the actions complete 188successfully, borgmatic notifies Cronitor of the success after the 189`after_backup` hooks run. And if an error occurs during any action or hook, 190borgmatic notifies Cronitor after the `on_error` hooks run. 191 192You can configure Cronitor to notify you by a [variety of 193mechanisms](https://cronitor.io/docs/cron-job-notifications) when backups fail 194or it doesn't hear from borgmatic for a certain period of time. 195 196 197## Cronhub hook 198 199[Cronhub](https://cronhub.io/) provides "instant alerts when any of your 200background jobs fail silently or run longer than expected", and borgmatic has 201built-in integration with it. Once you create a Cronhub account and monitor on 202their site, all you need to do is configure borgmatic with the unique "Ping 203URL" for your monitor. Here's an example: 204 205 206```yaml 207hooks: 208 cronhub: https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d031 209``` 210 211With this hook in place, borgmatic pings your Cronhub monitor when a backup 212begins, ends, or errors. Specifically, after the <a 213href="https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/">`before_backup` 214hooks</a> run, borgmatic lets Cronhub know that it has started if any of the 215`prune`, `create`, or `check` actions are run. Then, if the actions complete 216successfully, borgmatic notifies Cronhub of the success after the 217`after_backup` hooks run. And if an error occurs during any action or hook, 218borgmatic notifies Cronhub after the `on_error` hooks run. 219 220Note that even though you configure borgmatic with the "start" variant of the 221ping URL, borgmatic substitutes the correct state into the URL when pinging 222Cronhub ("start", "finish", or "fail"). 223 224You can configure Cronhub to notify you by a [variety of 225mechanisms](https://docs.cronhub.io/integrations.html) when backups fail 226or it doesn't hear from borgmatic for a certain period of time. 227 228 229## PagerDuty hook 230 231In case you're new here: [borgmatic](https://torsion.org/borgmatic/) is 232simple, configuration-driven backup software for servers and workstations, 233powered by [Borg Backup](https://www.borgbackup.org/). 234 235[PagerDuty](https://www.pagerduty.com/) provides incident monitoring and 236alerting. borgmatic has built-in integration that can notify you via PagerDuty 237as soon as a backup fails, so you can make sure your backups keep working. 238 239First, create a PagerDuty account and <a 240href="https://support.pagerduty.com/docs/services-and-integrations">service</a> 241on their site. On the service, add an integration and set the Integration Type 242to "borgmatic". 243 244Then, configure borgmatic with the unique "Integration Key" for your service. 245Here's an example: 246 247 248```yaml 249hooks: 250 pagerduty: a177cad45bd374409f78906a810a3074 251``` 252 253With this hook in place, borgmatic creates a PagerDuty event for your service 254whenever backups fail. Specifically, if an error occurs during a `create`, 255`prune`, or `check` action, borgmatic sends an event to PagerDuty before the 256`on_error` hooks run. Note that borgmatic does not contact PagerDuty when a 257backup starts or ends without error. 258 259You can configure PagerDuty to notify you by a [variety of 260mechanisms](https://support.pagerduty.com/docs/notifications) when backups 261fail. 262 263If you have any issues with the integration, [please contact 264us](https://torsion.org/borgmatic/#support-and-contributing). 265 266 267## Scripting borgmatic 268 269To consume the output of borgmatic in other software, you can include an 270optional `--json` flag with `create`, `list`, or `info` to get the output 271formatted as JSON. 272 273Note that when you specify the `--json` flag, Borg's other non-JSON output is 274suppressed so as not to interfere with the captured JSON. Also note that JSON 275output only shows up at the console, and not in syslog. 276 277 278## Related software 279 280 * [Borgmacator GNOME AppIndicator](https://github.com/N-Coder/borgmacator/) 281 282 283### Successful backups 284 285`borgmatic list` includes support for a `--successful` flag that only lists 286successful (non-checkpoint) backups. This flag works via a basic heuristic: It 287assumes that non-checkpoint archive names end with a digit (e.g. from a 288timestamp), while checkpoint archive names do not. This means that if you're 289using custom archive names that do not end in a digit, the `--successful` flag 290will not work as expected. 291 292Combined with a built-in Borg flag like `--last`, you can list the last 293successful backup for use in your monitoring scripts. Here's an example 294combined with `--json`: 295 296```bash 297borgmatic list --successful --last 1 --json 298``` 299 300Note that this particular combination will only work if you've got a single 301backup "series" in your repository. If you're instead backing up, say, from 302multiple different hosts into a single repository, then you'll need to get 303fancier with your archive listing. See `borg list --help` for more flags. 304 305 306### Latest backups 307 308All borgmatic actions that accept an "--archive" flag allow you to specify an 309archive name of "latest". This lets you get the latest successful archive 310without having to first run "borgmatic list" manually, which can be handy in 311automated scripts. Here's an example: 312 313```bash 314borgmatic info --archive latest 315``` 316