Total recall - rsync full archive

rsync_full.sh.example

rsync --archive --verbose --acls --xattrs --atimes --hard-links --atimes --open-noatime --one-file-system \
  --progress --human-readable --info=name0,progress2 --stats --ipv6 example.com:/home/user/Foo/ /tank/Foo/ --dry-run


# --info=del0,remove0,misc1,name0,progress2
# --info=del1,remove1,misc1,name1,progress2   - show a bit more stuff as they happen on each file

# Quick overview:
#
#  --archive == -a, archive mode is -rlptgoD (but no -A,-X,-U,-N,-H):
#     aka -r, --recursive
#         -l, --links (copy symlinks as symlinks),
#         -p, --perms  (--executability, -E is unecassary with --perms)
#         -t, --times
#         -g, --group
#         -o, --owner
#         -D == same as --devices (char and block) --specials (sockets and fifos)
#
# additionally:
#   --acls == -A  (implies --perms, -p, but already in --archive)
#   --xattrs == -X
#   --atimes == -U
#   --crtimes == -N   (my rsync says: rsync: This rsync does not support --crtimes (-N), on CentOS 9 and Fedora 42)
#   --hard-links == -H
#
#   (not enabled: -N == --crtimes, preserve create times (newness), because it usually does not work)
# 
#   --one-file-system == -x, do not cross file system boundaries, on both sender and reader (will create empty dirs for mount points)
#   --open-noatime == on sender (also on receiver) side open files with O_NOATIME, to not modify atime just by reading
#   --progress - show files, disable if you have a lot of small files
#     alternatively use --info=progress2 or --info=name0,progress2 for total progress bar
#     alternatively just periodically send SIGINFO or SIGVTALRM signal to rsync process
#     (if you plan to rsync repeateadly, use --info=name1,progress2 maybe to show which files were copied / deleted)
#   --human-readable == -h, use human readable units for size and speed
#
# For full system backup purposes:
#   --numeric-ids, do not use username/groupname mapping, use just literal uid/gid (and run rsync from root I guess?)
#
# To taste:
#   --sparse, -S
#   --partial, keep partially transferred files (easier restart of partial big files)
#   (careful!) --inplace, when repeateadly syncing things like database files (combined with CoW, and snapshots in ZFS, btrfs)
#   (careful!) --append, for logs
#
#   --delete, delete files on target that were removed on the source side  (default is delete mode is --delete-during on modern rsync)
#   (if you are doing things like repo syncing, you probably want --delete-after or a bit more efficient --delete-delay , check your distro details)
#
# --ipv6 - use IPv6
#
# When using slow network connection, or copying big text files, consider --compress == -z, default is gzip, but zstd lz4 / zstd might be better.
#
# More advanced uses using --delay-..., where files are moved in two or three phases.
#
# --checksum  - recheck all files based on checksum, not just mtime and size
#
# --dry-run, -n

# DO NOT use these:  --copy-links, -L, --munge-links, --copy-dirlinks, -k, --keep-dirlinks, -K

# Note, this is useful for mostly one time rsync. For repeated syncing
# (including potential deletions in the source), like mirroring from one
# source to many, you might need to consider more steps, and more options.
# This includes things like --delete-afterm, and --safe-links
#  (so symlinks that have absolute target or relative pointing out
#   of the synced tree, are ignored).
#
# For repeated syncing, and when extra atomicity is required, consider:
#  1) doing zfs / btrfs clone, then rsyncing that, then flipping things or using a snapshots
#  2) Use --link-dest (see also /usr/share/rsync/scripts/atomic-rsync),
#     this will take existing directory, source, and destination (empty dir),
#     and will hard link all the files from existing directory to destination,
#     if source did not change them. Then at the end, flip entire
#     directory hierarchy (or instruct your webserver, or something to switch).
#     (If you want full atomicity, make destination directory be a symlink
#      to a directory with some suffix, so flip can be done truly atomic).
#
# Another good place too look for inspirations is the ftpsync script from Debian,
# which does 2-stage rsync (and automatically repeats until convergance),
# where bulk of bytes is transfered first (without deleting anything or updating
# indices), and in second stage, smaller indices are updated (by first downloading
# them all, and quickly flipping them in place by doing rename syscalls, and finally
# removing files to be removed).