1---
2title: How to deal with very large backups
3eleventyNavigation:
4  key: Deal with very large backups
5  parent: How-to guides
6  order: 3
7---
8## Biggish data
9
10Borg itself is great for efficiently de-duplicating data across successive
11backup archives, even when dealing with very large repositories. But you may
12find that while borgmatic's default mode of "prune, create, and check" works
13well on small repositories, it's not so great on larger ones. That's because
14running the default pruning and consistency checks take a long time on large
15repositories.
16
17### A la carte actions
18
19If you find yourself in this situation, you have some options. First, you can
20run borgmatic's pruning, creating, or checking actions separately. For
21instance, the following optional actions are available:
22
23```bash
24borgmatic prune
25borgmatic create
26borgmatic check
27```
28
29(No borgmatic `prune`, `create`, or `check` actions? Try the old-style
30`--prune`, `--create`, or `--check`. Or upgrade borgmatic!)
31
32You can run with only one of these actions provided, or you can mix and match
33any number of them in a single borgmatic run. This supports approaches like
34skipping certain actions while running others. For instance, this skips
35`prune` and only runs `create` and `check`:
36
37```bash
38borgmatic create check
39```
40
41Or, you can make backups with `create` on a frequent schedule (e.g. with
42`borgmatic create` called from one cron job), while only running expensive
43consistency checks with `check` on a much less frequent basis (e.g. with
44`borgmatic check` called from a separate cron job).
45
46
47### Consistency check configuration
48
49Another option is to customize your consistency checks. The default
50consistency checks run both full-repository checks and per-archive checks
51within each repository.
52
53But if you find that archive checks are too slow, for example, you can
54configure borgmatic to run repository checks only. Configure this in the
55`consistency` section of borgmatic configuration:
56
57```yaml
58consistency:
59    checks:
60        - repository
61```
62
63Here are the available checks from fastest to slowest:
64
65 * `repository`: Checks the consistency of the repository itself.
66 * `archives`: Checks all of the archives in the repository.
67 * `extract`: Performs an extraction dry-run of the most recent archive.
68 * `data`: Verifies the data integrity of all archives contents, decrypting and decompressing all data (implies `archives` as well).
69
70See [Borg's check documentation](https://borgbackup.readthedocs.io/en/stable/usage/check.html) for more information.
71
72If that's still too slow, you can disable consistency checks entirely,
73either for a single repository or for all repositories.
74
75Disabling all consistency checks looks like this:
76
77```yaml
78consistency:
79    checks:
80        - disabled
81```
82
83Or, if you have multiple repositories in your borgmatic configuration file,
84you can keep running consistency checks, but only against a subset of the
85repositories:
86
87```yaml
88consistency:
89    check_repositories:
90        - path/of/repository_to_check.borg
91```
92
93Finally, you can override your configuration file's consistency checks, and
94run particular checks via the command-line. For instance:
95
96```bash
97borgmatic check --only data --only extract
98```
99
100This is useful for running slow consistency checks on an infrequent basis,
101separate from your regular checks.
102
103
104## Troubleshooting
105
106### Broken pipe with remote repository
107
108When running borgmatic on a large remote repository, you may receive errors
109like the following, particularly while "borg check" is validating backups for
110consistency:
111
112```text
113    Write failed: Broken pipe
114    borg: Error: Connection closed by remote host
115```
116
117This error can be caused by an ssh timeout, which you can rectify by adding
118the following to the `~/.ssh/config` file on the client:
119
120```text
121    Host *
122        ServerAliveInterval 120
123```
124
125This should make the client keep the connection alive while validating
126backups.
127