bin/cpdup/cpdup.1

.\" (c) Copyright 1997-2010 by Matthew Dillon, Dima Ruban, and Oliver Fromme.
.\"    Permission to use and distribute based on the DragonFly copyright.
.\"    Supplied as-is, USE WITH CAUTION.
.\"
.Dd November 24, 2009
.Dt CPDUP 1
.Os
.Sh NAME
.Nm cpdup
.Nd mirror filesystems
.Sh SYNOPSIS
.Nm
.Op Fl C
.Op Fl v Ns Op Cm v Ns Op Cm v
.Op Fl d
.Op Fl n
.Op Fl u
.Op Fl I
.Op Fl f
.Op Fl F Ar ssh-arg
.Op Fl s0
.Op Fl i0
.Op Fl j0
.Op Fl l
.Op Fl q
.Op Fl o
.Op Fl m
.Op Fl H Ar path
.Op Fl M Ar file
.Op Fl V
.Op Fl VV
.Op Fl S
.Op Fl R
.Op Fl k
.Op Fl K Ar file
.Op Fl X Ar file
.Op Fl x
.Oo Oo Ar user Ns Li @ Oc Ns Ar host : Oc Ns Ar source_dir
.Oo Oo Ar user Ns Li @ Oc Ns Ar host : Oc Ns Ar target_dir
.Sh DESCRIPTION
The
.Nm
utility makes an exact mirror copy of the source in the destination, creating
and deleting files and directories as necessary.  UTimes, hardlinks,
softlinks, devices, permissions, and flags are mirrored.  By default,
.Nm
asks for confirmation if any file or directory needs to be removed from
the destination and does not copy files which it believes to have already
been synchronized (by observing that the source and destination files' sizes
and mtimes match).
.Nm
does not cross mount points in either the source or the destination.
As a safety measure,
.Nm
refuses to replace a destination directory with a file.
.Pp
The following options are available:
.Bl -tag -width flag
.It Fl C
If the source or target is a remote host, request that the
.Xr ssh 1
session be compressed.
This is the same as
.Fl F
.Fl C .
.It Fl v Ns Op Cm v Ns Op Cm v
Set verboseness.  By default
.Nm
does not report its progress except when asking for confirmation.  A single
.Fl v
will only report modifications made to the destination.
.Fl vv
will report directories as they are being traversed as well as
modifications made to the destination.
.Fl vvv
will cause all files and directories to be reported whether or not
modifications are made.
.It Fl d
Print directories as they are being traversed.
Useful to watch the progress;
this typically produces much less output than
.Fl vv .
.It Fl n
Go through the motions but don't actually make any changes to
the target.
.It Fl u
Causes the output generated by
.Fl v
and
.Fl d
to be unbuffered.
This can be useful for obtaining prompt progress updates through a pipe.
.It Fl I
will cause
.Nm
to print a summary at the end with performance counters.
.It Fl f
Forces file updates to occur even if the files appear to be the same.  If
the
.Fl H
option is used, this option will force a byte for byte comparison
between the original file and the file in the hardlink path, even if
all the stat info matches, but will still use a hardlink if they match.
.It Fl F Ar ssh-arg
Pass
.Ar ssh-arg
to ssh.  For example
.Dq Fl F Fl p222 .
Note the lack of a space.
.It Fl s0
Disable the disallow-file-replaces-directory safety feature.  This
safety feature is enabled by default to prevent user mistakes from blowing
away everything accidentally.
.It Fl i0
Do not request confirmation when removing something.
.It Fl j0
Do not try to recreate CHR or BLK devices.
.It Fl l
Line buffer verbose output.
.It Fl q
Quiet operation.
.It Fl o
Do not remove any files, just overwrite/add.
.It Fl m
Generate and maintain a MD5 checkfile called
.Pa \&.MD5.CHECKSUMS
in each directory on the source
and do an MD5 check on each file of the destination when the destination
appears to be the same as the source.  If the check fails,
the source is recopied to the destination.  When you specify a destination
directory, the MD5 checkfile is only updated as needed and may not be updated
even if modifications are made to a source file.  If you do not specify a
destination directory the
.Nm
command forcefully regenerates the MD5 checkfile for every file in the source.
.It Fl M Ar file
Works the same as
.Fl m
but allows you to specify the name of the MD5 checkfile.
.It Fl H Ar path
.Nm
will create a hardlink from a file found under
.Ar path
to the target instead of copying the source to the target if the file found
via
.Ar path
is identical to the source.
Note that a remote host specification should not be used for this option's
.Ar path ,
but the
.Ar path
will be relative to the target machine.
.Pp
This allows one to use
.Nm
to create incremental backups of a filesystem.  Create a direct
.Sq level 0
backup, and then specify the level 0 backup path with this option when
creating an incremental backup to a different target directory.
This method works so long as the filesystem does not hit a hardlink limit.
If the system does hit a hardlink limit,
.Nm
will generate a warning and copy the file instead.
Note that
.Nm
must record file paths for any hardlinked file while operating and therefore
uses a great deal more memory when dealing with hardlinks or hardlink-based
backups.  Example use:
.Pp
.Dl cpdup \-i0 \-s0 \-I \-H /backup/home.l0 /home /backup/home.l1
.Pp
WARNING: If this option is used
.Nm
must record the paths for all files it encounters while it operates
and it is possible that you may run the process out of memory.
.Pp
The file found via the hardlink path will be byte-by-byte compared with the
source if the
.Fl V
or
.Fl f
option is also used, otherwise only the stat info is checked to determine
whether it matches the source.
.It Fl V
This forces the contents of regular files to be verified, even if the
files appear to the be the same.  Whereas the
.Fl f
(force) option forces a copy regardless, this option will avoid rewriting
the target if everything matches and the contents are verified to be the
same.
.It Fl VV
This works the same as
.Fl V
but ignores mtime entirely, making it suitable for comparing HAMMER
master and slave filesystems or copies made without mtime retention.
.It Fl S
This places
.Nm
into slave mode and is used to initiate the slave protocol on a remote
machine.
This option is not intended to be used by humans.
.It Fl R
Place the slave into read-only mode.
Can only be used when the source is remote.
Useful for unattended backups via SSH keys.
.It Fl k
Generate and maintain a FSMID checkfile called
.Pa \& .FSMID.CHECK
in each directory on the target.
.Nm
will check the FSMID for each source file or directory against the checkfile
on the target and will not copy the file or recurse through the directory
when a match occurs.  Any source file or directory with the same name as the
checkfile will be ignored.  The FSMID will be re-checked after the copy
has been completed and
.Nm
will loop on that directory or file until it is sure it has an exact copy.
.Pp
Warning: FSMID is not always supported by a filesystem and may not be
synchronized if a crash occurs.
.Dx
will simulate an FSMID when
it is otherwise not supported by the filesystem, and users should be aware
that simulated FSMIDs may change state in such cases even if the underlying
hierarchy does not due to cache flushes.
Additionally, the FSMID may not reflect changes made to remote filesystems
by other hosts.  For example, using these options with NFS mounted sources
will not work well.
.It Fl K Ar file
Works the same as
.Fl k
but allows you to specify the name of the FSMID checkfile.
.It Fl x
Causes
.Nm
to use the exclusion file
.Pa \&.cpignore
in each directory on the source to
determine which files to ignore.  When this option is used, the exclusion
filename itself is automatically excluded from the copy.  If this option is
not used then the filename
.Pa \&.cpignore
is not considered special and will
be copied along with everything else.
.It Fl X Ar file
Works similarly to
.Fl x
but allows you to specify the name of the exclusion file.  This file is
automatically excluded from the copy.  Only one exclusion file may be
specified.
.Pp
When an absolute path is used, the same exclusive file is read for
every directory and may contain full paths or wildcarded paths based
on the full source path as specified on the cpdup command line.
In this situation, the exclusive file is read from the host running
the command, NOT from the source host (if remote).
.Pp
When a relative path is used (or
.Fl x
is specified), the exclusion file is only applicable to the directory
it resides in on the source host and only path elements (the directory
elements) are matched against it.
.El
.Sh REMOTE COPYING
.Nm
can mirror directory structures across machines and can also do third-party
copies.
This also works between machines that use different byte order.
.Xr ssh 1
sessions are used and
.Nm
is run on the remote machine(s) in slave mode.
You can use the
.Fl F
option to pass additional flags to the ssh command if necessary.
.Pp
The syntax of remote path specifications is similar to
.Xr scp 1 .
In particular, that means that a local path containing a colon must
be preceded by a slash to prevent it being considered a remote host:
.Ql foo:bar
causes
.Nm
to look for a directory called
.Ql bar
on host
.Ql foo ,
while
.Ql \&./foo:bar
denotes the directory
.Ql foo:bar
on the local machine.
.Pp
.Nm
also supports a
.Ql localhost:
prefix which is silently discarded but prevents any colons in the remainder
of the path from being interpreted as a host:path form.
this form can be used with relative filenames when you do not want colons in
the filename to be misinterpreted.
.Sh EXIT STATUS
.Ex -std
.Sh SEE ALSO
.Xr cp 1 ,
.Xr cpio 1 ,
.Xr scp 1 ,
.Xr ssh 1 ,
.Xr tar 1
.Sh HISTORY
The
.Nm
command was originally created to update servers at BEST Internet circa 1997
and was placed under the
.Fx
copyright for inclusion in the ports area in 1999.
The program was written by Matthew Dillon, Dima Ruban, and later
significantly improved by Oliver Fromme.
.Sh BUGS
.Xr UFS 5
has a hardlink limit of 32767.  Many programs, in particular CVS
with regards to its CVS/Root file, will generate a lot of hard links.
When using the
.Fl H
option it may not be possible for
.Nm
to maintain these hard links.  If this occurs,
.Nm
will be forced to copy the file instead of link it, and thus not be able
to make a perfect copy of the filesystem.
.Pp
When so-called sparse files (i.e. files with "holes") are copied,
the holes will be filled in the target files, so they occupy
more physical disk space than the source files.
.Pp
For compatibility reasons, the slave protocol is not as efficient
for writing remote files as it is for reading them.
Therefore it is recommended to run
.Nm
on the target machine when making remote copies,
so the source machine is remote.
If you do it the other way,
.Nm
will run somewhat slower.