1.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit!
2
3.. _borg_patterns:
4
5borg help patterns
6~~~~~~~~~~~~~~~~~~
7
8
9The path/filenames used as input for the pattern matching start from the
10currently active recursion root. You usually give the recursion root(s)
11when invoking borg and these can be either relative or absolute paths.
12
13So, when you give `relative/` as root, the paths going into the matcher
14will look like `relative/.../file.ext`. When you give `/absolute/` as
15root, they will look like `/absolute/.../file.ext`. This is meant when
16we talk about "full path" below.
17
18File paths in Borg archives are always stored normalized and relative.
19This means that e.g. ``borg create /path/to/repo ../some/path`` will
20store all files as `some/path/.../file.ext` and ``borg create
21/path/to/repo /home/user`` will store all files as
22`home/user/.../file.ext`. Therefore, always use relative paths in your
23patterns when matching archive content in commands like ``extract`` or
24``mount``. Starting with Borg 1.2 this behaviour will be changed to
25accept both absolute and relative paths.
26
27File patterns support these styles: fnmatch, shell, regular expressions,
28path prefixes and path full-matches. By default, fnmatch is used for
29``--exclude`` patterns and shell-style is used for the experimental
30``--pattern`` option.
31
32If followed by a colon (':') the first two characters of a pattern are
33used as a style selector. Explicit style selection is necessary when a
34non-default style is desired or when the desired pattern starts with
35two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
36
37`Fnmatch <https://docs.python.org/3/library/fnmatch.html>`_, selector `fm:`
38    This is the default style for ``--exclude`` and ``--exclude-from``.
39    These patterns use a variant of shell pattern syntax, with '\*' matching
40    any number of characters, '?' matching any single character, '[...]'
41    matching any single character specified, including ranges, and '[!...]'
42    matching any character not specified. For the purpose of these patterns,
43    the path separator (backslash for Windows and '/' on other systems) is not
44    treated specially. Wrap meta-characters in brackets for a literal
45    match (i.e. `[?]` to match the literal character `?`). For a path
46    to match a pattern, the full path must match, or it must match
47    from the start of the full path to just before a path separator. Except
48    for the root path, paths will never end in the path separator when
49    matching is attempted.  Thus, if a given pattern ends in a path
50    separator, a '\*' is appended before matching is attempted.
51
52Shell-style patterns, selector `sh:`
53    This is the default style for ``--pattern`` and ``--patterns-from``.
54    Like fnmatch patterns these are similar to shell patterns. The difference
55    is that the pattern may include `**/` for matching zero or more directory
56    levels, `*` for matching zero or more arbitrary characters with the
57    exception of any path separator.
58
59Regular expressions, selector `re:`
60    Regular expressions similar to those found in Perl are supported. Unlike
61    shell patterns regular expressions are not required to match the full
62    path and any substring match is sufficient. It is strongly recommended to
63    anchor patterns to the start ('^'), to the end ('$') or both. Path
64    separators (backslash for Windows and '/' on other systems) in paths are
65    always normalized to a forward slash ('/') before applying a pattern. The
66    regular expression syntax is described in the `Python documentation for
67    the re module <https://docs.python.org/3/library/re.html>`_.
68
69Path prefix, selector `pp:`
70    This pattern style is useful to match whole sub-directories. The pattern
71    `pp:root/somedir` matches `root/somedir` and everything therein.
72
73Path full-match, selector `pf:`
74    This pattern style is (only) useful to match full paths.
75    This is kind of a pseudo pattern as it can not have any variable or
76    unspecified parts - the full path must be given.
77    `pf:root/file.ext` matches `root/file.ext` only.
78
79    Implementation note: this is implemented via very time-efficient O(1)
80    hashtable lookups (this means you can have huge amounts of such patterns
81    without impacting performance much).
82    Due to that, this kind of pattern does not respect any context or order.
83    If you use such a pattern to include a file, it will always be included
84    (if the directory recursion encounters it).
85    Other include/exclude patterns that would normally match will be ignored.
86    Same logic applies for exclude.
87
88.. note::
89
90    `re:`, `sh:` and `fm:` patterns are all implemented on top of the Python SRE
91    engine. It is very easy to formulate patterns for each of these types which
92    requires an inordinate amount of time to match paths. If untrusted users
93    are able to supply patterns, ensure they cannot supply `re:` patterns.
94    Further, ensure that `sh:` and `fm:` patterns only contain a handful of
95    wildcards at most.
96
97Exclusions can be passed via the command line option ``--exclude``. When used
98from within a shell, the patterns should be quoted to protect them from
99expansion.
100
101The ``--exclude-from`` option permits loading exclusion patterns from a text
102file with one pattern per line. Lines empty or starting with the number sign
103('#') after removing whitespace on both ends are ignored. The optional style
104selector prefix is also supported for patterns loaded from a file. Due to
105whitespace removal, paths with whitespace at the beginning or end can only be
106excluded using regular expressions.
107
108To test your exclusion patterns without performing an actual backup you can
109run ``borg create --list --dry-run ...``.
110
111Examples::
112
113    # Exclude '/home/user/file.o' but not '/home/user/file.odt':
114    $ borg create -e '*.o' backup /
115
116    # Exclude '/home/user/junk' and '/home/user/subdir/junk' but
117    # not '/home/user/importantjunk' or '/etc/junk':
118    $ borg create -e '/home/*/junk' backup /
119
120    # Exclude the contents of '/home/user/cache' but not the directory itself:
121    $ borg create -e /home/user/cache/ backup /
122
123    # The file '/home/user/cache/important' is *not* backed up:
124    $ borg create -e /home/user/cache/ backup / /home/user/cache/important
125
126    # The contents of directories in '/home' are not backed up when their name
127    # ends in '.tmp'
128    $ borg create --exclude 're:^/home/[^/]+\.tmp/' backup /
129
130    # Load exclusions from file
131    $ cat >exclude.txt <<EOF
132    # Comment line
133    /home/*/junk
134    *.tmp
135    fm:aa:something/*
136    re:^/home/[^/]+\.tmp/
137    sh:/home/*/.thumbnails
138    # Example with spaces, no need to escape as it is processed by borg
139    some file with spaces.txt
140    EOF
141    $ borg create --exclude-from exclude.txt backup /
142
143.. container:: experimental
144
145    A more general and easier to use way to define filename matching patterns exists
146    with the experimental ``--pattern`` and ``--patterns-from`` options. Using these, you
147    may specify the backup roots (starting points) and patterns for inclusion/exclusion.
148    A root path starts with the prefix `R`, followed by a path (a plain path, not a
149    file pattern). An include rule starts with the prefix +, an exclude rule starts
150    with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern.
151
152    .. note::
153
154        Via ``--pattern`` or ``--patterns-from`` you can define BOTH inclusion and exclusion
155        of files using pattern prefixes ``+`` and ``-``. With ``--exclude`` and
156        ``--exclude-from`` ONLY excludes are defined.
157
158    Inclusion patterns are useful to include paths that are contained in an excluded
159    path. The first matching pattern is used so if an include pattern matches before
160    an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches
161    a directory, it won't recurse into it and won't discover any potential matches for
162    include rules below that directory.
163
164    Note that the default pattern style for ``--pattern`` and ``--patterns-from`` is
165    shell style (`sh:`), so those patterns behave similar to rsync include/exclude
166    patterns. The pattern style can be set via the `P` prefix.
167
168    Patterns (``--pattern``) and excludes (``--exclude``) from the command line are
169    considered first (in the order of appearance). Then patterns from ``--patterns-from``
170    are added. Exclusion patterns from ``--exclude-from`` files are appended last.
171
172    Examples::
173
174        # backup pics, but not the ones from 2018, except the good ones:
175        # note: using = is essential to avoid cmdline argument parsing issues.
176        borg create --pattern=+pics/2018/good --pattern=-pics/2018 repo::arch pics
177
178        # use a file with patterns:
179        borg create --patterns-from patterns.lst repo::arch
180
181    The patterns.lst file could look like that::
182
183        # "sh:" pattern style is the default, so the following line is not needed:
184        P sh
185        R /
186        # can be rebuild
187        - /home/*/.cache
188        # they're downloads for a reason
189        - /home/*/Downloads
190        # susan is a nice person
191        # include susans home
192        + /home/susan
193        # don't backup the other home directories
194        - /home/*
195        # don't even look in /proc
196        ! /proc
197
198.. _borg_placeholders:
199
200borg help placeholders
201~~~~~~~~~~~~~~~~~~~~~~
202
203
204Repository (or Archive) URLs, ``--prefix``, ``--glob-archives``, ``--comment``
205and ``--remote-path`` values support these placeholders:
206
207{hostname}
208    The (short) hostname of the machine.
209
210{fqdn}
211    The full name of the machine.
212
213{reverse-fqdn}
214    The full name of the machine in reverse domain name notation.
215
216{now}
217    The current local date and time, by default in ISO-8601 format.
218    You can also supply your own `format string <https://docs.python.org/3.7/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
219
220{utcnow}
221    The current UTC date and time, by default in ISO-8601 format.
222    You can also supply your own `format string <https://docs.python.org/3.7/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
223
224{user}
225    The user name (or UID, if no name is available) of the user running borg.
226
227{pid}
228    The current process ID.
229
230{borgversion}
231    The version of borg, e.g.: 1.0.8rc1
232
233{borgmajor}
234    The version of borg, only the major version, e.g.: 1
235
236{borgminor}
237    The version of borg, only major and minor version, e.g.: 1.0
238
239{borgpatch}
240    The version of borg, only major, minor and patch version, e.g.: 1.0.8
241
242If literal curly braces need to be used, double them for escaping::
243
244    borg create /path/to/repo::{{literal_text}}
245
246Examples::
247
248    borg create /path/to/repo::{hostname}-{user}-{utcnow} ...
249    borg create /path/to/repo::{hostname}-{now:%Y-%m-%d_%H:%M:%S} ...
250    borg prune --prefix '{hostname}-' ...
251
252.. note::
253    systemd uses a difficult, non-standard syntax for command lines in unit files (refer to
254    the `systemd.unit(5)` manual page).
255
256    When invoking borg from unit files, pay particular attention to escaping,
257    especially when using the now/utcnow placeholders, since systemd performs its own
258    %-based variable replacement even in quoted text. To avoid interference from systemd,
259    double all percent signs (``{hostname}-{now:%Y-%m-%d_%H:%M:%S}``
260    becomes ``{hostname}-{now:%%Y-%%m-%%d_%%H:%%M:%%S}``).
261
262.. _borg_compression:
263
264borg help compression
265~~~~~~~~~~~~~~~~~~~~~
266
267
268It is no problem to mix different compression methods in one repo,
269deduplication is done on the source data chunks (not on the compressed
270or encrypted data).
271
272If some specific chunk was once compressed and stored into the repo, creating
273another backup that also uses this chunk will not change the stored chunk.
274So if you use different compression specs for the backups, whichever stores a
275chunk first determines its compression. See also borg recreate.
276
277Compression is lz4 by default. If you want something else, you have to specify what you want.
278
279Valid compression specifiers are:
280
281none
282    Do not compress.
283
284lz4
285    Use lz4 compression. Very high speed, very low compression. (default)
286
287zstd[,L]
288    Use zstd ("zstandard") compression, a modern wide-range algorithm.
289    If you do not explicitely give the compression level L (ranging from 1
290    to 22), it will use level 3.
291    Archives compressed with zstd are not compatible with borg < 1.1.4.
292
293zlib[,L]
294    Use zlib ("gz") compression. Medium speed, medium compression.
295    If you do not explicitely give the compression level L (ranging from 0
296    to 9), it will use level 6.
297    Giving level 0 (means "no compression", but still has zlib protocol
298    overhead) is usually pointless, you better use "none" compression.
299
300lzma[,L]
301    Use lzma ("xz") compression. Low speed, high compression.
302    If you do not explicitely give the compression level L (ranging from 0
303    to 9), it will use level 6.
304    Giving levels above 6 is pointless and counterproductive because it does
305    not compress better due to the buffer size used by borg - but it wastes
306    lots of CPU cycles and RAM.
307
308auto,C[,L]
309    Use a built-in heuristic to decide per chunk whether to compress or not.
310    The heuristic tries with lz4 whether the data is compressible.
311    For incompressible data, it will not use compression (uses "none").
312    For compressible data, it uses the given C[,L] compression - with C[,L]
313    being any valid compression specifier.
314
315Examples::
316
317    borg create --compression lz4 REPO::ARCHIVE data
318    borg create --compression zstd REPO::ARCHIVE data
319    borg create --compression zstd,10 REPO::ARCHIVE data
320    borg create --compression zlib REPO::ARCHIVE data
321    borg create --compression zlib,1 REPO::ARCHIVE data
322    borg create --compression auto,lzma,6 REPO::ARCHIVE data
323    borg create --compression auto,lzma ...
324
325