ZFS on Proxmox: build, encrypt, mount, verify

A working reference for building a Proxmox VE host on ZFS with native encryption, handing datasets to unprivileged containers, and not losing data to the failure modes that do not announce themselves.

Written to be handed to an agent as implementation context. Every rule has a symptom, a cause and a check, so it can be verified rather than trusted. Commands are for OpenZFS 2.2+ on PVE 8/9.

1. The failure mode that governs everything

A mountpoint is a directory. When a dataset is not mounted, that directory still exists in whatever filesystem sits above it, and writes to paths under it land there.

Two consequences, both silent:

Misplaced writes. A write to /a/b/c while b is unmounted goes into a's filesystem as a literal b/c directory. If a is unencrypted and b is encrypted, that data is plaintext at rest, permanently, and everything looks correct until b gets mounted.

Buried mounts. Mounting a parent after its children hides every child. The children stay mounted (zfs get mounted says yes, /proc/self/mountinfo lists them) but nothing can reach their paths, and the directories in the parent appear empty in their place.

Both are ordinary Unix behaviour, not ZFS bugs, and neither produces an error. A backup running against a buried mount stores an empty tree and reports success.

Guards, in layers

Each layer catches what the previous one misses. Use all of them.

overlay=off on every pool root. The Linux default is on, which lets a mount silently cover a non-empty directory. Off, the mount fails and says so. Set at creation with -O overlay=off; children inherit.
chattr +i on the underlying directory. An immutable empty directory accepts a mount on top but refuses every write, including from root, while nothing is mounted. Apply it while the dataset is unmounted:
```
zfs umount tank/data
chattr +i /tank/data
zfs mount tank/data
```
lsattr -d shows the attribute only while unmounted; once mounted the path resolves to the dataset root, which has its own attributes.
is_mountpoint yes on every PVE directory storage that lives on a dataset. PVE marks the storage inactive when the path is not a mountpoint instead of writing backups and ISOs into the bare directory.
```
pvesm add dir backups --path /tank/backups --content backup --is_mountpoint yes
```
zfspool storages reference datasets and zvols by name, not by path, so they cannot leak this way.
RequiresMountsFor= in any systemd unit that reads or writes these paths, so the service refuses to start rather than operate on a bare directory.
Encrypt the pool root, so a misplaced write that gets past everything above is still ciphertext.

Mount order

Mount with zfs mount -a, never dataset by dataset. It sorts by mountpoint, so parents always precede children.
Pool roots hold only mountpoint directories. Keep data in children.

2. Encryption and keys

Encrypt at the pool root at creation. Every later dataset inherits and nothing can be forgotten:

zpool create -o ashift=12 \
  -O encryption=aes-256-gcm -O keyformat=raw -O keylocation=file:///etc/zfs/keys/tank.key \
  -O atime=off -O compression=lz4 -O xattr=sa -O acltype=posixacl \
  -O overlay=off -m /tank tank raidz1 <disk-by-id> <disk-by-id> <disk-by-id>

A key is loaded on the encryption root, never on an inheriting child. zfs load-key tank/child fails with "Keys must be loaded for encryption root of ...". List the roots:
```
zfs get -H -o value encryptionroot -t filesystem,volume | sort -u | grep -v '^-$'
```
keyformat=raw keys are 32 binary bytes and cannot be typed. Create them without ever displaying them, and keep them root-only:
```
mkdir -p -m 700 /etc/zfs/keys
dd if=/dev/urandom of=/etc/zfs/keys/tank.key bs=32 count=1 status=none
chmod 400 /etc/zfs/keys/tank.key
```
They load from a file (-L file:///path) or from a pipe (-L file:///dev/stdin).
Loading a key does not mount anything. Mounting is a separate step.
Keyfile on the boot disk protects any drive that leaves the chassis (RMA, resale, disposal). It does not protect against theft of the whole machine. Switching to a passphrase later rewrites only the wrapped key, not the data:
```
zfs change-key -o keyformat=passphrase -o keylocation=prompt tank
shred -u /etc/zfs/keys/tank.key
```
Native encryption leaves dataset names, sizes and snapshot names readable on a raw disk. Only the data and most metadata are ciphertext.
Keys and passphrases belong in the recovery kit, off the machine. A pool that restores perfectly into datasets nobody can unlock is not a backup.

3. Send and receive

zfs send of an encrypted dataset decrypts into the stream. Only zfs send -w (raw) keeps it encrypted.
Sending an encrypted dataset with properties (-p, and therefore -R) requires -w. A decrypted recursive copy is a per-dataset loop of plain sends, not one -R stream.
zfs send -Rw of an encrypted parent carries any unencrypted children along as unencrypted. A raw recursive send is not proof that the stream is ciphertext.
A stream sent with -p or -R carries the source's encryption=off. Received under an encrypted parent it lands unencrypted unless zfs recv -x encryption drops that property so the child inherits. This is how an unencrypted dataset appears inside an encrypted one years later.
Raw send works with the key unloaded. It is the way to copy a locked pool whose passphrase is unavailable. The copy stays locked by the original key: readable only with that passphrase, deletable without it.

Receive foreign or archival copies unmounted and without their mountpoints, then keep them from auto-mounting:

zfs send -Rw old@snap | zfs recv -u -x mountpoint tank/old-archive
for ds in $(zfs list -rH -o name tank/old-archive); do zfs set canmount=noauto "$ds"; done

Received replicas: readonly=on does not block zfs recv, so set it. It stops a host-side write from diverging the target and breaking the next incremental.

A long send belongs in a transient unit, not an ssh session:

systemd-run --unit=migrate --collect -p Type=oneshot \
  /bin/sh -c 'zfs send -Rw old@snap | zfs recv -u -x mountpoint tank/old-archive'
journalctl -u migrate

4. Boot: load keys, then mount everything

Manual procedure, in this order:

zfs load-key -a
zfs mount -a

Unattended boot with keyfiles, /etc/systemd/system/zfs-load-key.service:

[Unit]
Description=Load ZFS encryption keys
DefaultDependencies=no
Before=zfs-mount.service
After=zfs-import.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zfs load-key tank
ExecStart=/sbin/zfs load-key fast

[Install]
WantedBy=zfs-mount.service

Name the encryption roots explicitly. load-key -a also tries every raw archive with keylocation=prompt, blocks on a prompt that has no terminal, and fails the unit.

Two checks that must print nothing:

# encryption roots still locked
zfs list -H -o name,encryptionroot,keystatus -t filesystem,volume \
  | awk '$3=="unavailable" {print $2}' | sort -u | grep -v old-archive

# datasets that should be mounted and are not
zfs list -H -o name,mounted,canmount,mountpoint -t filesystem \
  | awk '$2=="no" && $3=="on" && $4 ~ /^\//'

Then pvesm status: every storage active. A directory storage showing inactive is is_mountpoint doing its job.

5. Datasets in unprivileged containers

Give a container a dataset with a mount point entry, one per dataset, parents first:

mp0: /tank/media,mp=/tank/media,ro=1,backup=0
mp1: /tank/media/photos,mp=/tank/media/photos,ro=1,backup=0

Verify from inside after every config change:

pct exec 111 -- findmnt -rno TARGET,OPTIONS | grep /tank

Per-mountpoint ro=1 is the reliable way to get read-only all the way down. Read-only cannot be added from inside the container: those mounts belong to the host's namespace, and mount -o remount,bind,ro returns EINVAL.

Container mounts are propagation peers of the host's

PVE mounts them shared. A bind mount created on the host under one of these paths propagates a copy into the running container, where it can land on top of an existing mount and bury the entire subtree: the same shadowing failure, one layer down.

Do not bind-mount these paths on the host while the container runs. If you must, restart the container afterwards.
Never umount one of these from inside the container. Propagation can carry the unmount outward to the host.
The fix for a container whose tree is buried is pct stop then pct start, which rebuilds the namespace from the config. Confirm afterwards:

pct exec 111 -- findmnt -rno TARGET | grep -c '^/tank'
pct exec 111 -- findmnt -rno TARGET | grep '^/tank' | sort | uniq -d   # empty

Reading host files from an unprivileged container

Default map is u 0 100000 65536: container uid 0 is host 100000. Files owned by host ids outside 100000-165535 appear as nobody (65534) and are readable only through their other permission bits.

To read them with their real ownership, carve the host ids into the map. For host uid 1000 and gid 1003, in /etc/pve/lxc/111.conf:

lxc.idmap: u 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 0 100000 1003
lxc.idmap: g 1003 1003 1
lxc.idmap: g 1004 101004 64532

The ranges must tile 0-65535 with no gap and no overlap, and the host side needs matching grants in /etc/subuid and /etc/subgid:

root:1000:1
root:1003:1

Container ids 1000 (uid) and 1003 (gid) now map to the same host ids. Any file the container created earlier under the old map (host 101000 and 101003) is orphaned; find them before switching:

find /tank -xdev \( -uid 101000 -o -gid 101003 \)

6. Shadow scan: finding hidden data

A non-recursive bind mount of a dataset shows that dataset alone, without the child mounts stacked on it, which exposes the directories underneath. Counting entries in those directories finds misplaced writes.

#!/bin/sh
# Counts entries hidden under child mountpoints. Runs in seconds.
# Stop containers that use these datasets first (section 5).
set -u
mkdir -p /mnt/shadow-check
zfs list -H -o name,mountpoint,mounted -t filesystem \
| while IFS="$(printf '\t')" read -r ds mp mounted; do
  case "$mp" in /*) ;; *) continue ;; esac
  [ "$mounted" = yes ] || continue
  mount --bind "$mp" /mnt/shadow-check || continue
  zfs list -H -o mountpoint -t filesystem | while read -r cmp; do
    case "$cmp" in "$mp"/*) ;; *) continue ;; esac
    rel=${cmp#"$mp"/}
    d="/mnt/shadow-check/$rel"
    [ -d "$d" ] || continue
    n=$(ls -A "$d" 2>/dev/null | wc -l)
    [ "$n" -gt 0 ] && echo "SHADOWED  $ds  at  $rel  ($n entries)"
  done
  umount /mnt/shadow-check
done
# pool roots sit on the root filesystem; check under them the same way
mount --bind / /mnt/shadow-check
zpool list -H -o name | while read -r p; do
  mp=$(zfs get -H -o value mountpoint "$p")
  case "$mp" in /*) ;; *) continue ;; esac
  n=$(ls -A "/mnt/shadow-check$mp" 2>/dev/null | wc -l)
  [ "$n" -gt 0 ] && echo "SHADOWED  rootfs  at  $mp  ($n entries)"
done
umount /mnt/shadow-check
rmdir /mnt/shadow-check

A parent whose children each have a mountpoint directory reports itself with a count equal to the number of children; that one is expected. Everything else is real.

Blind spots to close by hand:

Datasets with mountpoint=none or legacy have no expected path: zfs list -H -o name,mountpoint -t filesystem | awk '$2=="none" || $2=="legacy"'
Non-ZFS mounts hide data the same way and are not covered.
Snapshots hold the past; the scan sees only the present.

Cross-check with refer

zfs list -o name,used,refer,mountpoint -r tank

A dataset whose refer is far larger than what is visible under it is hiding something. A root that should hold only mountpoint directories reads a few hundred KB.

7. Moving misplaced data out

The source is under a mount, so it needs an unshadowed view. A snapshot of the parent gives one that is read-only and cannot be confused with the destination:

zfs snapshot tank@rescue
ls /tank/.zfs/snapshot/rescue/data/          # the misplaced tree, read-only
rsync -a /tank/.zfs/snapshot/rescue/data/ /tank/data/   # into the mounted child

Then delete the original through a bind mount of the parent, which is the only path that reaches it:

mount --bind /tank /mnt/shadow-check
rm -rf /mnt/shadow-check/data/*
umount /mnt/shadow-check
zfs destroy tank@rescue

Every older snapshot of the parent still holds the misplaced copy. Destroy those too, or the data stays. If the parent was unencrypted, the blocks were plaintext at rest and freed space cannot be wiped on flash; treat it as leaked and rotate whatever it contained.

8. Disks: wiping and replacing

A single overwrite is enough for a modern HDD; the multi-pass folklore predates perpendicular recording, and NIST SP 800-88 treats one pass as adequate. The gap is not what got overwritten, it is what an overwrite cannot reach:

Reallocated sectors. A retired sector keeps its contents and its LBA points elsewhere. smartctl -A /dev/sdX | grep -i realloc
Filesystem slop. ZFS keeps a reserve you cannot fill, plus metadata, labels and uberblocks.
SMR media cache. Shingled drives stage writes in a CMR region.
SSD over-provisioning and wear levelling. Zeros land on fresh pages; the old pages persist until the controller garbage-collects them. A fill on flash is close to theatre. Use blkdiscard, or the drive's crypto-erase.
L2ARC and SLOG. Cache and log vdevs hold whatever went through them. Blocks of encrypted datasets are stored there as ciphertext; blocks of unencrypted datasets as plaintext. A resilver does nothing about it. Check zpool status for cache and log vdevs and wipe them separately.

The version that avoids all of this: encrypt from the start, and wiping becomes destroying a key.

Disk-by-disk replacement

zpool status tank                       # layout, redundancy, cache/log vdevs
zpool replace tank <old> <new>          # both attached, keeps redundancy
zpool scrub tank                        # between replacements

Attach the new disk alongside rather than pulling the old one first. On raidz1 a detach-first replacement leaves the pool one failure from loss for the whole rebuild, and rebuilds of a full pool run for hours.

9. Cache and layout on a mixed host

For a host with spinning storage plus one NVMe:

ARC is the cache that matters. PVE caps zfs_arc_max at 10% of RAM, 16 GB at most. Raise it to what VMs leave free, live and persistently:
```
echo 51539607552 > /sys/module/zfs/parameters/zfs_arc_max
echo 'options zfs zfs_arc_max=51539607552' > /etc/modprobe.d/zfs.conf
update-initramfs -u -k all
```
The value is a ceiling, not a reservation; ARC shrinks under pressure.
L2ARC on an NVMe partition is the only cache that costs nothing when it fails: read-only, removable live, persistent across reboots on 2.x. Attach with zpool add tank cache <partition-by-id>.
SLOG helps synchronous writes only (NFS, iSCSI, sync=always). Leave room for a 16-32 GB partition and add it when a workload needs it.
special vdev places metadata and small blocks on flash permanently. It is a pool member: losing it loses the pool. Single-device special vdevs on a redundant pool are a mistake.
VMs belong on their own NVMe pool, not behind a cache. Fast writes happen because the data lives there.
A directory dataset on the NVMe pool (recordsize=1M, guarded like the rest) holds ISOs, templates and quick local backups; the HDD pool holds the real backups.

Partition layout that follows: NVMe p1 = VM pool, p2 = L2ARC for the HDD pool, 5-10% left unpartitioned for the controller.

10. Backups on top of this

The shadowing failure and backups interact badly, so design against it explicitly. A buried mount is indistinguishable from an empty directory: a backup walks it, finds nothing, stores an empty snapshot and reports success.

Guards, in order of value:

A canary file at the top of every source, and a backup that fails when it is absent. This is the only guard a backup tool can check cheaply.
A volume check on the run: fail when a source returns dramatically fewer files or bytes than the previous run, rather than storing the empty result.
RequiresMountsFor= in the unit that runs the backup.
is_mountpoint yes on the PVE storage that receives vzdump archives.
overlay=off, which turns the whole class of problem into a loud mount failure.

Other things that bite:

Compressed archives (vzdump .tar.zst and similar) deduplicate against nothing; each adds its full size. Proxmox Backup Server or an uncompressed archive lets a content-defined chunker find the overlap.
Locked datasets cannot be backed up at file level; there is no plaintext to read. The mechanism for those is zfs send -w.
A zvol has no directory tree. File-level tools cannot see it; it needs a snapshot plus zfs send, or the filesystems inside it mounted.
Keys and passphrases belong in the recovery kit, off the machine.

11. Checklist for a new host

[ ] Pools created with encryption at the root, ashift=12, overlay=off
[ ] acltype=posixacl and xattr=sa set at creation on file-serving pools
[ ] Pool roots hold no data; readonly=on once the children exist
[ ] Every dataset audited: zfs list -o name,encryption,encryptionroot -r
[ ] Underlying mountpoint directories chattr +i, applied while unmounted
[ ] Every PVE directory storage on a dataset has is_mountpoint yes
[ ] Key files 0400 root:root in a 0700 directory, never displayed
[ ] zfs-load-key.service enabled, naming each encryption root explicitly
[ ] Both boot verification checks print nothing; pvesm status all active
[ ] Keys and passphrases recorded outside this machine
[ ] Raw archives of foreign pools: canmount=noauto, excluded from key loading
[ ] zfs_arc_max sized to what VMs leave free, persisted through initramfs
[ ] Container mount points: one per dataset, ro=1, backup=0, parents first
[ ] Container mount count verified from inside after every config change
[ ] idmap ranges tile 0-65535; subuid/subgid granted; orphaned files checked
[ ] Shadow scan clean
[ ] Backup sources have canary files, and the backup fails when they are absent
[ ] Node notes state key paths, the passphrase procedure and the guards

mallendeo/zfs-pve-guide.md

Select an option

No results found