RDIFF-BACKUP(1)

NAME
rdiff-backup - local/remote mirror and incremental backup

SYNOPSIS
rdiff-backup [options] [[[user@]host1.foo]::source_directory] [[[user@]host2.foo]::destination_directory] rdiff-backup {{ -l | --list-increments } | --remove-older-than time_interval | --list-at-time time | --list-changed-since time | --list-increment-sizes } [[[user@]host2.foo]::destination_directory] rdiff-backup --calculate-average statfile1 statfile2 ... rdiff-backup --test-server [user1]@host1.net1::path [[user2]@host2.net2::path] ...

DESCRIPTION
rdiff-backup is a script, written in python(1) that backs up one directory to another. The target directory ends up a copy (mirror) of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves symlinks, special files, hardlinks, permissions, uid/gid ownership (if it is running as root), and modification times. rdiff-backup can also operate in a bandwidth efficient manner over a pipe, like rsync(1). Thus you can use ssh and rdiff-backup to securely back a hard drive up to a remote location, and only the differences will be transmitted. Using the default settings, rdiff-backup requires that the remote system accept ssh connections, and that rdiff-backup is installed in the user's PATH on the remote system. For information on other options, see the section on REMOTE OPERATION. Note that you should not write to the mirror directory except with rdiff-backup. Many of the increments are stored as reverse diffs, so if you delete or modify a file, you may lose the ability to restore previous versions of that file.

OPTIONS

-b, --backup-mode
    Force backup mode even if first argument appears to be an increment or mirror file.

--calculate-average
    Enter calculate average mode. The arguments should be a number of statistics files. rdiff-backup will print the average of the listed statistics files and exit.

--check-destination-dir
    If an rdiff-backup session fails, running rdiff-backup with this option on the destination dir will undo the failed directory. This happens automatically if you attempt to back up to a directory and the last backup failed.

--current-time seconds
    This option is useful mainly for testing. If set, rdiff-backup will it for the current time instead of consulting the clock. The argument is the number of seconds since the epoch.

--exclude shell_pattern
    Exclude the file or files matched by shell_pattern. If a directory is matched, then files under that directory will also be matched. See the FILE SELECTION section for more information.

--exclude-device-files
    Exclude all device files. This can be useful for security/permissions reasons or if rdiff-backup is not handling device files correctly.

--exclude-filelist filename
    Excludes the files listed in filename See the FILE SELECTION section for more information.

--exclude-filelist-stdin
    Like --exclude-filelist, but the list of files will be read from standard input. See the FILE SELECTION section for more information.

--exclude-globbing-filelist filename
    Like --exclude-filelist but each line of the filelist will be interpreted according to the same rules as --include and --exclude.

--exclude-other-filesystems
    Exclude files on file systems (identified by device number) other than the file system the root of the source directory is on.

--exclude-regexp regexp
    Exclude files matching the given regexp. Unlike the --exclude option, this option does not match files in a directory it matches. See the FILE SELECTION section for more information.

--exclude-special-files
    Exclude all device files, fifos, sockets, and symlinks.

--force
    Authorize the updating or overwriting of a destination path. rdiff-backup will generally tell you if it needs this.

--include shell_pattern
    Similar to --exclude but include matched files instead. Unlike --exclude, this option will also match parent directories of matched files (although not necessarily their contents). See the FILE SELECTION section for more information.

--include-filelist filename
    Like --exclude-filelist, but include the listed files instead. See the FILE SELECTION section for more information.

--include-filelist-stdin
    Like --include-filelist, but read the list of included files from standard input. --include-globbing-filelist filename Like --include-filelist but each line of the filelist will be interpreted according to the same rules as --include and --exclude.

--include-regexp regexp
    Include files matching the regular expression regexp. Only files explicitly matched by regexp will be included by this option. See the FILE SELECTION section for more information.

--list-at-time time
    List the files in the archive that were present at the given time. If a directory in the archive is specified, list only the files under that directory.

--list-changed-since time
    List the files that have changed since the given time. See TIME FORMATS for the format of time. If a directory in the archive is specified, list only the files under that directory.

-l, --list-increments
    List the number and date of partial incremental backups contained in the specified destination directory. No backup or restore will take place if this option is given.

--list-increment-sizes
    List the total size of all the increment and mirror files by time. This may be helpful in deciding how many increments to keep, and when to --remove-older-than. Specifying a subdirectory is allowable; then only the sizes of the mirror and increments pertaining to that subdirectory will be listed.

-m, --mirror-only
    Do not create an rdiff-backup-data directory or make any increments. In this mode rdiff-backup is similar to rsync (but usually slower).

--no-change-dir-inc-perms
    Do not change the permissions of the directory increments to match the directories they represent.

--no-compare-inode
    This relatively esoteric option prevents rdiff-backup from flagging a file as changed when its inode changes. This option may be useful if you are backing up two different directories to the same rdiff-backup destination directory. The downside is that hard link information may get messed up, as the metadata file may no longer have the correct inode information.

--no-compression
    Disable the default gzip compression of most of the .snapshot and .diff increment files stored in the rdiff-backup-data directory. A backup volume can contain compressed and uncompressed increments, so using this option inconsistently is fine.

--no-compression-regexp  regexp
    Do not compress increments based on files whose filenames match regexp. The default is "(?i).*\.(gz|z|bz|bz2|tgz|zip|rpm|deb|jpg|gif|png|jp2|mp3|ogg|avi|wmv|mpeg|mpg|rm|mov)$"

--no-file-statistics
    This will disable writing to the file_statistics file in the rdiff-backup-data directory. rdiff-backup will run slightly quicker and take up a bit less space.

--no-hard-links
    Don't replicate hard links on destination side. Note that because metadata is written to a separate file, hard link information will not be lost even if the --no-hard-links option is given (however, mirror files will not be linked). If many hard-linked files are present, this option can drastically decrease memory usage.

--null-separator
    Use nulls (\0) instead of newlines (\n) as line separators, which may help when dealing with filenames containing newlines. This affects the expected format of the files specified by the --{include|exclude}-filelist[-stdin] switches as well as the format of the directory statistics file.

--parsable-output
    If set, rdiff-backup's output will be tailored for easy parsing by computers, instead of convenience for humans. Currently this only applies when listing increments using the -l or --list-increments switches, where the time will be given in seconds since the epoch.

--print-statistics
    If set, summary statistics will be printed after a successful backup If not set, this information will still be available from the session statistics file. See the STATISTICS section for more information.

-r, --restore-as-of restore_time
    Restore the specified directory as it was as of restore_time. See the TIME FORMATS section for more information on the format of restore_time, and see the RESTORING section for more information on restoring.

--remote-schema schema
    Specify an alternate method of connecting to a remote computer. This is necessary to get rdiff-backup not to use ssh for remote backups, or if, for instance, rdiff-backup is not in the PATH on the remote side. See the REMOTE OPERATION section for more information.

--remove-older-than time_spec
    Remove the incremental backup information in the destination directory that has been around longer than the given time. time_spec can be either an absolute time, like "2002-01-04", or a time interval. The time interval is an integer followed by the character s, m, h, D, W, M, or Y, indicating seconds, minutes, hours, days, weeks, months, or years respectively, or a number of these concatenated. For example, 32m means 32 minutes, and 3W2D10h7s means 3 weeks, 2 days, 10 hours, and 7 seconds. In this context, a month means 30 days, a year is 365 days, and a day is always 86400 seconds. rdiff-backup cannot remove-older-than and back up or restore in a single session. If you want to, for instance, backup a directory and remove old files in it, you must run rdiff-backup twice. Note that snapshots of deleted files are covered by this operation. Thus if you deleted a file two weeks ago, backed up immediately afterwards, and then ran rdiff-backup with --remove-older-than 10D today, no trace of that file would remain. Finally, file selection options such as --include and --exclude don't affect --remove-older-than.

--restrict path
    Require that all file access be inside the given path. This switch, and the following two, are intended to be used with the --server switch to provide a bit more protection when doing automated remote backups. They are not intended as your only line of defense so please don't do something silly like allow public access to an rdiff-backup server run with --restrict-read-only.

--restrict-read-only path
    Like --restrict, but also reject all write requests.

--restrict-update-only path
    Like --restrict, but only allow writes as part of an incremental backup. Requests for other types of writes (for instance, deleting path) will be rejected.

--server
    Enter server mode (not to be invoked directly, but instead used by another rdiff-backup process on a remote computer).

--ssh-no-compression
    When running ssh, do not use the -C option to enable compression. --ssh-no-compression is ignored if you specify a new schema using --remote-schema.

--terminal-verbosity [0-9]
    Select which messages will be displayed to the terminal. If missing the level defaults to the verbosity level.

--test-server
    Test for the presence of a compatible rdiff-backup server as specified in the following host::filename argument(s). The filename section will be ignored.

-v[0-9], --verbosity [0-9]
    Specify verbosity level (0 is totally silent, 3 is the default, and 9 is noisiest). This determines how much is written to the log file.

-V, --version
    Print the current version and exit

EXAMPLES
Simplest case---backup directory foo to directory bar, with increments in bar/rdiff-backup-data:

    rdiff-backup foo bar

This is exactly the same as previous example because trailing slashes are ignored:

    rdiff-backup foo/ bar/

Back files up from /home/bob to /mnt/backup, leaving increments in /mnt/backup/rdiff-backup-data. Do not back up directory /home/bob/tmp or any files in it.

    rdiff-backup --exclude /home/bob/tmp /home/bob /mnt/backup

The file selection options can be combined in various ways. The following command backs up the whole file system to /usr/local/backup. However, the entire /usr directory is skipped, with the exception of /usr/local, which is included, except for /usr/local/backup, which is excluded to prevent a circularity:

    rdiff-backup --exclude /usr/local/backup --include /usr/local --exclude /usr / /usr/local/backup

You can also use regular expressions in the --exclude statements. This will skip any files whose full pathnames contain the word "cache", or any files whose name is "tmp", "temp", "TMP", "tEmP", etc.

    rdiff-backup --exclude-regexp cache --exclude-regexp '(?i)/te?mp$' /home/bob /mnt/backup

After the previous command was completed, this command will list the backups present on the destination disk:

    rdiff-backup --list-increments /mnt/backup

If space is running out on the /mnt/backup directory, older incremental backups can be erased. The following command erases backup information older than a week:

    rdiff-backup --remove-older-than 7D /mnt/backup

The following reads the file important-data.2001-07-15T04:09:38-07:00.dir and restores the resulting directory important-data as it was on Februrary 14, 2001, calling the new directory "temp". Note that rdiff-backup goes into restore mode because it recognizes the suffix of the file. The -v9 means keep lots of logging information.

    rdiff-backup -v9 important-data.2001-07-15T04:09:38-07:00.dir temp

This command causes rdiff-backup to backup the directory /some/local-dir to the directory /whatever/remote-dir on the machine hostname.net. It uses ssh to open the necessary pipe to the remote copy of rdiff-backup. Here the username on the local machine and on hostname.net are the same.

    rdiff-backup /some/local-dir hostname.net::/whatever/remote-dir

This command logs into hostname.net as smith and restores the remote increment old-file on a remote computer to the current directory on the local computer:

    rdiff-backup smith@hostname.net::/foo/rdiff-backup-data/increments/bar/old-file.2001-11-09T12:43:53-04:00.diff

Backup foo on one remote machine to bar on another. This will probably be slower than running rdiff-backup from either machine.

    rdiff-backup smith@host1::foo jones@host2::bar

Test to see if the specified ssh command really opens up a working rdiff-backup server on the remote side.
    rdiff-backup --test-server hostname.net::/ignored

RESTORING
There are two ways to tell rdiff-backup to restore a file or directory. Firstly, you can run rdiff-backup on a mirror file and use the -r or --restore-as-of options. Secondly, you can run it on an increment file.

For example, suppose in the past you have run:

    rdiff-backup /usr /usr.backup

to back up the /usr directory into the /usr.backup directory, and now want a copy of the /usr/local directory the way it was 3 days ago placed at /usr/local.old.

One way to do this is to run:

    rdiff-backup -r 3D /usr.backup/local /usr/local.old

where above the "3D" means 3 days (for other ways to specify the time, see the TIME FORMATS section). The /usr.backup/local directory was selected, because that is the directory containing the current version of /usr/local.

Note that the option to --restore-as-of always specifies an exact time. (So "3D" refers to the instant 72 hours before the present.) If there was no backup made at that time, rdiff-backup restores the state recorded for the previous backup. For instance, in the above case, if "3D" is used, and there are only backups from 2 days and 4 days ago, /usr/local as it was 4 days ago will be restored.

The second way to restore files involves finding the corresponding increment file. It would be in the /backup/rdiff-backup-data/increments/usr directory, and its name would be something like "local.2002-11-09T12:43:53-04:00.dir" where the time indicates it is from 3 days ago. Note that the increment files all end in ".diff", ".snapshot", ".dir", or ".missing", where ".missing" just means that the file didn't exist at that time (finally, some of these may be gzip-compressed, and have an extra ".gz" to indicate this). Then running:

    rdiff-backup /backup/rdiff-backup-data/increments/usr/local.<time>.dir /usr/local.old

would also restore the file as desired.

If you are not sure exactly which version of a file you need, it is probably easiest to either restore from the increments files as described immediately above, or to see which increments are available with -l/--list-increments, and then specify exact times into -r/--restore-as-of.

TIME FORMATS
rdiff-backup uses time strings in two places. Firstly, all of the increment files rdiff-backup creates will have the time in their filenames in the w3 datetime format as described in a w3 note at http://www.w3.org/TR/NOTE-datetime. Basically they look like "2001-07-15T04:09:38-07:00", which means what it looks like. The "-07:00" section means the time zone is 7 hours behind UTC.

Secondly, the -r, --restore-as-of, and --remove-older-than options take a time string, which can be given in any of several formats:

1.  the string "now" (refers to the current time)

2.  a sequences of digits, like "123456890" (indicating the time in seconds after the epoch)

3.  A string like "2002-01-25T07:00:00+02:00" in datetime format

4.  An interval, which is a number followed by one of the characters s, m, h, D, W, M, or Y (indicating seconds, minutes, hourse, days, weeks, months, or years respectively), or a series of such pairs. In this case the string refers to the time that preceded the current time by the length of the interval. For instance, "1h78m" indicates the time that was one hour and 78 minutes ago. The calendar here is unsophisticated: a month is always 30 days, a year is always 365 days, and a day is always 86400 seconds.

5.  A date format of the form YYYY/MM/DD, YYYY-MM-DD, MM/DD/YYYY, or MM/DD/YYYY, which indicates midnight on the day in question, relative to the current timezone settings. For instance, "2002/3/5", "03-05-2002", and "2002-3-05" all mean March 5th, 2002.

REMOTE OPERATION
In order to access remote files, rdiff-backup opens up a pipe to a copy of rdiff-backup running on the remote machine. Thus rdiff-backup must be installed on both ends. To open this pipe, rdiff-backup first splits the filename into host_info::pathname. It then substitutes host_info into the remote schema, and runs the resulting command, reading its input and output.

The default remote schema is 'ssh %s rdiff-backup --server' meaning if the host_info is user@host.net, then rdiff-backup runs 'ssh user@host.net rdiff-backup --server'. The '%s' keyword is substituted with the host_info. Using --remote-schema, rdiff-backup can invoke an arbitrary command in order to open up a remote pipe. For instance,
    rdiff-backup --remote-schema 'cd /usr; %s' foo 'rdiff-backup --server'::bar
is basically equivalent to (but slower than)
    rdiff-backup foo /usr/bar

Concerning quoting, if for some reason you need to put two consecutive colons in the host_info section of a host_info::pathname argument, or in the pathname of a local file, you can quote one of them by prepending a backslash. So in 'a\::b::c', host_info is 'a::b' and the pathname is 'c'. Similarly, if you want to refer to a local file whose filename contains two consecutive colons, like 'strange::file', you'll have to quote one of the colons as in 'strange\::file'. Because the backslash is a quote character in these circumstances, it too must be quoted to get a literal backslash, so 'foo\::\\bar' evaluates to 'foo::\bar'. To make things more complicated, because the backslash is also a common shell quoting character, you may need to type in '\\\\' at the shell prompt to get a literal backslash (if it makes you feel better, I had to type in 8 backslashes to get that in this man page...). And finally, to include a literal % in the string specified by --remote-schema, quote it with another %, as in %%.

FILE SELECTION
rdiff-backup supports file selection options similar to (but different from) rsync(1). The system may appear complicated, but it is supposed to be flexible and easy-to-use. When rdiff-backup is run, it searches through the given source directory and backs up all the files specified by the file selection system. The file selection system comprises a number of file selection conditions, which are set using one of the following command line options: --exclude,--exclude-device-files,--exclude-filelist, --exclude-globbing-filelist, --exclude-filelist-stdin,--exclude-regexp,--exclude-special-files, --include, --include-filelist,--include-globbing-filelist, --include-filelist-stdin, and --include-regexp. Each file selection condition either matches or doesn't match a given file. A given file is excluded by the file selection system exactly when the first matching file selection condition specifies that the file be excluded; otherwise the file is included. For instance,

    rdiff-backup --include /usr --exclude /usr /usr /backup

is exactly the same as

    rdiff-backup /usr /backup

because the include and exclude directives match exactly the same files, and the --include comes first, giving it precedence. Similarly,

    rdiff-backup --include /usr/local/bin --exclude /usr/local /usr /backup

would backup the /usr/local/bin directory (and its contents), but not /usr/local/doc. The include, exclude, include-globbing-filelist, and exclude-globbing-filelist options accept extended shell globbing patterns. These patterns can contain the special patterns *, **, ?, and [...]. As in a normal shell, * can be expanded to any string of characters not containing "/", ? expands to any character except "/", and [...] expands to a single character of those characters specified (ranges are acceptable). The new special pattern, **, expands to any string of characters whether or not it contains "/". Furthermore, if the pattern starts with "ignorecase:" (case insensitive), then this prefix will be removed and any character in the string can be replaced with an upper- or lowercase version of itself. Remember that you may need to quote these characters when typing them into a shell, so the shell does not interpret the globbing patterns before rdiff-backup sees them. The --exclude pattern option matches a file iff:

1.  pattern can be expanded into the file's filename, or

2.  the file is inside a directory matched by the option.

Conversely, --include pattern matches a file iff:

1.  pattern can be expanded into the file's filename,

2.  the file is inside a directory matched by the option, or

3.  the file is a directory which contains a file matched by the option.

For example,

    --exclude /usr/local

matches /usr/local, /usr/local/lib, and /usr/local/lib/netscape. It is the same as --exclude /usr/local --exclude '/usr/local/**'.

    --include /usr/local

specifies that /usr, /usr/local, /usr/local/lib, and /usr/local/lib/netscape (but not /usr/doc) all be backed up. Thus you don't have to worry about including parent directories to make sure that included subdirectories have somewhere to go. Finally,

    --include ignorecase:'/usr/[a-z0-9]foo/*/**.py'

would match a file like /usR/5fOO/hello/there/world.py. If it did match anything, it would also match /usr. If there is no existing file that the given pattern can be expanded into, the option will not match /usr. The --include-filelist, --exclude-filelist, --include-filelist-stdin, and --exclude-filelist-stdin options also introduce file selection conditions. They direct rdiff-backup to read in a file, each line of which is a file specification, and to include or exclude the matching files. Lines are separated by newlines or nulls, depending on whether the --null-separator switch was given. Each line in a filelist is interpreted similarly to the way extended shell patterns are, with a few exceptions:

1.  Globbing patterns like *, **, ?, and [...] are not expanded.

2.  Include patterns do not match files in a directory that is included. So /usr/local in an include file will not match /usr/local/doc.

3.  Lines starting with "+ " are interpreted as include directives, even if found in a filelist referenced by --exclude-filelist. Similarly, lines starting with "- " exclude files even if they are found within an include filelist.
For example, if the file "list.txt" contains the lines:
    /usr/local
    - /usr/local/doc
    /usr/local/bin
    + /var
    - /var
then "--include-filelist list.txt" would include /usr, /usr/local, and /usr/local/bin. It would exclude /usr/local/doc, /usr/local/doc/python, etc. It neither excludes nor includes /usr/local/man, leaving the fate of this directory to the next specification condition. Finally, it is undefined what happens with /var. A single file list should not contain conflicting file specifications. The --include-globbing-filelist and --exclude-globbing-filelist options also specify filelists, but each line in the filelist will be interpreted as a globbing pattern the way --include and --exclude options are interpreted (although "+ " and "- " prefixing is still allowed). For instance, if the file "globbing-list.txt" contains the lines:
    dir/foo
    + dir/bar
    - **
Then "--include-globbing-filelist globbing-list.txt" would be exactly the same as specifying "--include dir/foo --include dir/bar --exclude **" on the command line. Finally, the --include-regexp and --exclude-regexp allow files to be included and excluded if their filenames match a python regular expression. Regular expression syntax is too complicated to explain here, but is covered in Python's library reference. Unlike the --include and --exclude options, the regular expression options don't match files containing or contained in matched files. So for instance

    --include '[0-9]{7}(?!foo)'

matches any files whose full pathnames contain 7 consecutive digits which aren't followed by 'foo'. However, it wouldn't match /home even if /home/ben/1234567 existed.

STATISTICS
Every session rdiff-backup saves various statistics into two files, the session statistics file at rdiff-backup-data/session_statistics.<time>.data and the directory statistics file at rdiff-backup-data/directory_statistics.<time>.data. They are both text files and contain similar information: how many files changed, how many were deleted, the total size of increment files created, etc. However, the session statistics file is intended to be very readable and only describes the session as a whole. The directory statistics file is more compact (and slightly less readable) but describes every directory backed up. It also may be compressed to save space. Statistics related options include --print-statistics and --null-separator. Also, rdiff-backup will save various messages to the log file, which is rdiff-backup-data/backup.log for backup sessions and rdiff-backup-data/restore.log for restore sessions. Generally what is written to this file will coincide with the messages diplayed to stdout or stderr, although this can be changed with the --terminal-verbosity option. The log file is not compressed and can become quite large if rdiff-backup is run with high verbosity.

BUGS
rdiff-backup uses the shell command mknod(1) to backup device files (e.g. /dev/ttyS0), so device files won't be handled correctly on systems with non-standard mknod syntax.

Files whose names are close to the maximum length (e.g. 235 chars if the maximum is 255) may be skipped because the filenames of related increment files would be too long.

The gzip library in versions 2.2 and earlier of python (but fixed in 2.3a1) has trouble producing files over 2GB in length. This bug will prevent rdiff-backup from producing large compressed increments (snapshots or diffs). A workaround is to disable compression for large uncompressable files.

AUTHOR
Ben Escoto <bescoto@stanford.edu>

Feel free to ask me questions or send me bug reports, but you may want to see the web page, mentioned below, first.

SEE ALSO
python(1), rdiff(1), rsync(1), ssh(1). The main rdiff-backup web page is at http://rdiff-backup.stanford.edu/. It has more information, links to the mailing list and CVS, etc.