How to recover btrfs

chances.md

Confidence: ~95 % it’s just the filesystem, not the SSD

Evidence	What it means
SMART: 0 media/data-integrity errors, 0 error log entries, 5 % wear	Drive hardware is behaving normally.
NVMe logs: no aborts, resets, or PCIe AER errors shown	No transport or controller faults.
Btrfs stats: all *_io_errs counters at 0	The kernel never saw a write fail at the block layer.
Metadata chunk 93 % full at RO flip	Btrfs is documented to abort a transaction and remount RO when it can’t reserve metadata space.

A bad SSD almost always leaves fingerprints—CRC errors, aborted commands, or incrementing error logs—which you don’t have. The only smoking gun you do have is “metadata almost full”.

How to push certainty to 99 %

1. Run the NVMe built-in long self-test (3–5 h, non-destructive):

   sudo smartctl -t long /dev/nvme0
   sudo smartctl -a /dev/nvme0   # after it finishes

   A failing controller will report errors here.

2. Scrub after you’ve freed space and remounted RW:

   sudo btrfs scrub start -Bd /

   Scrub rereads every data/metadata block; it will surface silent corruption if any exists.

3. Watch dmesg while running heavy I/O (e.g., fio or copying large files). Any hidden PCIe or NAND issue tends to show up within minutes under load.

If those three come back clean, treat the incident as a pure Btrfs ENOSPC event.

Residual risk

• Firmware bugs can exist even in a “clean” drive, but they usually manifest as resets/aborts first, not silent RO flips.
• Bad RAM can corrupt metadata in flight; a full memtest is cheap insurance.

Bottom line

All signs point to recoverable metadata ENOSPC, not dying hardware. Free a few GB, run the metadata-only balance, scrub, and keep at least 2 GiB metadata headroom. If any of the checks above lights up afterwards, reassess—but odds are heavily in your favor.

how-to.md

Short answer

Reboot and fix it from a live-USB; don’t try heavy surgery while the root FS is mounted read-only.

---

Why

• All subvolumes are on the same Btrfs that’s currently ro.
• You’ll need RW access to delete data and run a balance.
• Doing that while the system is half-running (Docker, systemd, overlay2) risks another RO flip or a deadlock.

---

Minimal, safe procedure

Devices / names below match your layout. Substitute another mountpoint if you prefer.

1. Boot a recent Manjaro/Arch ISO (any kernel ≥ 6.8 is fine).

2. Unlock the LUKS container

   cryptsetup open /dev/nvme0n1p2 cryptroot

3. Mount the filesystem RW

   mount -o rw,noatime,ssd,space_cache=v2 /dev/mapper/cryptroot /mnt

4. Free a couple of GB fast

   # Pacman cache is the quickest win (~>1 GiB)
   rm -rf /mnt/var/cache/pacman/pkg/*
   
   # If you use Timeshift/Snapper, nuke old snapshots
   btrfs subvolume list /mnt | grep snapshot
   btrfs subvolume delete /mnt/@.snapshots/<ID>

5. Rebalance metadata only

   btrfs balance start -musage=50 -dlimit=0 /mnt

   (-dlimit=0 → touch data chunks only if needed to make space for metadata.)

6. Check usage

   btrfs fi df /mnt         # Metadata should drop to < 70 %

7. Clean exit

   umount /mnt
   cryptsetup close cryptroot
   reboot

---

If you must stay online (not recommended)

sudo mount -o remount,rw,device=/dev/mapper/luks-98e65cd2-8687-44f2-bee3-64ccc6124b70,clear_cache /
# If remount succeeds:
sudo rm -rf /var/cache/pacman/pkg/*
sudo btrfs balance start -musage=50 -dlimit=0 /

Abort immediately if the filesystem flips back to RO—then fall back to the live-USB method.

---

After you’re back up

# keep metadata healthy
systemctl enable --now btrfs-balance.timer      # monthly light balance
systemctl enable --now [email protected]      # weekly scrub

# docker often fills metadata; prune it
docker system prune -af

Always leave at least 2 GiB free metadata (≈ 25 % of an 8 GiB DUP chunk).