An interesting issue came up. I noticed a few days ago that there was some data that had 1/1 even though the options are set to min 1 but desired 2. I tried bcachefs data rereplicate but that did not seem to change anything.
Upon trying to boot the system and mount the FS, it seems to be failing because that replica isn't present?
Not sure why since all disks are present.
I guess there are 2 potential things going on: 1 rereplicate is not replicating all data to match desired options, and fsck fails if there are missing replicas (I think this is expected).
➜ ~ sudo bcachefs fsck -k /dev/nvme0n1
Running in-kernel offline fsck
bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): starting version 1.28: inode_has_case_insensitive opts=metadata_replicas=2,data_replicas=2,background_compression=zstd:15,foreground_target=cache.nvme1,background_target=hdd,promote_target=cache,degraded=yes,fsck,fix_errors=ask,noratelimit_errors,read_only
allowing incompatible features above 0.0: (unknown version)
features: lz4,zstd,journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): Using encoding defined by superblock: utf8-12.1.0
bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): insufficient devices online (0) for replicas entry user: 1/1 [4]
bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): shutting down
bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): shutdown complete
➜ ~ sudo bcachefs fsck /dev/nvme0n1
Running userspace offline fsck
starting version 1.28: inode_has_case_insensitive opts=metadata_replicas=2,data_replicas=2,background_compression=zstd:15,foreground_target=cache.nvme1,background_target=hdd,promote_target=cache,degraded=yes,fsck,fix_errors=ask,read_only
allowing incompatible features above 0.0: (unknown version)
Using encoding defined by superblock: utf8-12.1.0
insufficient devices online (0) for replicas entry user: 1/1 [4]dmesg when attempting to mount:
➜ ~ sudo mount -t bcachefs UUID=27cac550-3836-765c-d107-51d27ab4a6e1 /mnt/pool -o verbose,very_degraded,nofail,ro
mount: /dev/sdb:/dev/sdd:/dev/nvme0n1:/dev/sde:/dev/sdf: No such file or directory
[Jun 4 03:58] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): starting version 1.28: inode_has_case_insensitive opts=metadata_replicas=2,data_replicas=2,background_compression=zstd:15,foreground_target=cache.nvme1,background_target=hdd,promote_target=cache
allowing incompatible features above 0.0: (unknown version)
features: lz4,zstd,journal_seq_blacklist_v3,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
[ +0.000007] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): Using encoding defined by superblock: utf8-12.1.0
[ +0.000011] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): recovering from unclean shutdown
[ +0.000010] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): starting journal read
[Jun 4 03:59] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): journal read done on device nvme0n1, ret 0
[ +14.022487] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): journal read done on device sde, ret 0
[ +12.510415] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): journal read done on device sdd, ret 0
[ +1.193382] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): journal read done on device sdf, ret 0
[ +0.000049] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): journal read done, replaying entries 6680578-6681468
[ +0.016374] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): Journal keys: 5602 read, 4308 after sorting and compacting
[ +0.122078] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): accounting_read... done
[ +0.048840] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): alloc_read... done
[ +0.186456] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): snapshots_read... done
[ +0.215864] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): going read-write
[ +0.008090] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): journal_replay... done
[ +0.512474] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): resume_logged_ops... done
[ +0.001309] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): delete_dead_inodes... done
[ +0.001615] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): __bch2_inode_peek(): error looking up inum 1:4096: ENOENT_inode
[ +0.000813] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): error in recovery: ENOENT_inodeemergency read only at seq 6681533
[ +0.000778] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): bch2_fs_start(): error starting filesystem ENOENT_inode
[ +0.000025] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): shutting down
[ +0.000012] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): going read-only
[ +0.000008] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): finished waiting for writes to stop
[ +0.012218] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): flushing journal and stopping allocators, journal seq 6681533
[ +0.000052] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): flushing journal and stopping allocators complete, journal seq 6681533
[ +0.000031] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): unclean shutdown complete, journal seq 6681533
[ +0.001996] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): done going read-only, filesystem not clean
[ +0.006046] bcachefs (27cac550-3836-765c-d107-51d27ab4a6e1): shutdown complete
[ +0.145797] bcachefs: bch2_fs_get_tree() error: ENOENT_inode
Errors fixed