1--- 2title: How to deal with very large backups 3eleventyNavigation: 4 key: Deal with very large backups 5 parent: How-to guides 6 order: 3 7--- 8## Biggish data 9 10Borg itself is great for efficiently de-duplicating data across successive 11backup archives, even when dealing with very large repositories. But you may 12find that while borgmatic's default mode of "prune, create, and check" works 13well on small repositories, it's not so great on larger ones. That's because 14running the default pruning and consistency checks take a long time on large 15repositories. 16 17### A la carte actions 18 19If you find yourself in this situation, you have some options. First, you can 20run borgmatic's pruning, creating, or checking actions separately. For 21instance, the following optional actions are available: 22 23```bash 24borgmatic prune 25borgmatic create 26borgmatic check 27``` 28 29(No borgmatic `prune`, `create`, or `check` actions? Try the old-style 30`--prune`, `--create`, or `--check`. Or upgrade borgmatic!) 31 32You can run with only one of these actions provided, or you can mix and match 33any number of them in a single borgmatic run. This supports approaches like 34skipping certain actions while running others. For instance, this skips 35`prune` and only runs `create` and `check`: 36 37```bash 38borgmatic create check 39``` 40 41Or, you can make backups with `create` on a frequent schedule (e.g. with 42`borgmatic create` called from one cron job), while only running expensive 43consistency checks with `check` on a much less frequent basis (e.g. with 44`borgmatic check` called from a separate cron job). 45 46 47### Consistency check configuration 48 49Another option is to customize your consistency checks. The default 50consistency checks run both full-repository checks and per-archive checks 51within each repository. 52 53But if you find that archive checks are too slow, for example, you can 54configure borgmatic to run repository checks only. Configure this in the 55`consistency` section of borgmatic configuration: 56 57```yaml 58consistency: 59 checks: 60 - repository 61``` 62 63Here are the available checks from fastest to slowest: 64 65 * `repository`: Checks the consistency of the repository itself. 66 * `archives`: Checks all of the archives in the repository. 67 * `extract`: Performs an extraction dry-run of the most recent archive. 68 * `data`: Verifies the data integrity of all archives contents, decrypting and decompressing all data (implies `archives` as well). 69 70See [Borg's check documentation](https://borgbackup.readthedocs.io/en/stable/usage/check.html) for more information. 71 72If that's still too slow, you can disable consistency checks entirely, 73either for a single repository or for all repositories. 74 75Disabling all consistency checks looks like this: 76 77```yaml 78consistency: 79 checks: 80 - disabled 81``` 82 83Or, if you have multiple repositories in your borgmatic configuration file, 84you can keep running consistency checks, but only against a subset of the 85repositories: 86 87```yaml 88consistency: 89 check_repositories: 90 - path/of/repository_to_check.borg 91``` 92 93Finally, you can override your configuration file's consistency checks, and 94run particular checks via the command-line. For instance: 95 96```bash 97borgmatic check --only data --only extract 98``` 99 100This is useful for running slow consistency checks on an infrequent basis, 101separate from your regular checks. 102 103 104## Troubleshooting 105 106### Broken pipe with remote repository 107 108When running borgmatic on a large remote repository, you may receive errors 109like the following, particularly while "borg check" is validating backups for 110consistency: 111 112```text 113 Write failed: Broken pipe 114 borg: Error: Connection closed by remote host 115``` 116 117This error can be caused by an ssh timeout, which you can rectify by adding 118the following to the `~/.ssh/config` file on the client: 119 120```text 121 Host * 122 ServerAliveInterval 120 123``` 124 125This should make the client keep the connection alive while validating 126backups. 127