Adding a Replacement Disk to a Degraded Array

gistfile1.txt

# RAID5 Rebuild — Adding a Replacement Disk to a Degraded Array

## Ground rules

- Run commands **one at a time**. Do not paste blocks.
- Read each command back before pressing enter.
- Nothing here writes to the existing array members. The **only** dangerous command is
  `sgdisk --replicate` with its arguments reversed. Stage 0 exists to make that recoverable.
- Identify disks by **serial**, not by letter. Letters can shuffle across reboots.
- The rebuild runs in the kernel. Closing SSH does not stop it. It also survives a reboot
  via the write-intent bitmap.

Throughout, substitute:

- `NEW_DISK` — the blank replacement disk (no partition table, not in the array)
- `EXISTING_MEMBER` — any healthy disk already in the array, used as the partition template

---

## Stage 0 — Insurance

Back up partition tables and array metadata before touching anything.

```bash
mkdir -p /root/raid-rescue && cd /root/raid-rescue

# one backup per existing array member
sgdisk --backup=gpt-EXISTING_MEMBER.bin /dev/EXISTING_MEMBER

mdadm --detail /dev/md0 > md0-detail.txt
mdadm --examine /dev/EXISTING_MEMBER1 > md0-examine.txt
ls -l /dev/disk/by-id/ > by-id.txt

cat by-id.txt
```

**Record the exact by-id paths** for the new disk and for one existing member from that
output. They look like `ata-<MODEL>_<SERIAL>`, but underscore counts vary by model string —
copy them verbatim. You will use them in Stage 2.

**Recovery, if a live member's table is ever overwritten:**

```bash
sgdisk --load-backup=/root/raid-rescue/gpt-EXISTING_MEMBER.bin /dev/EXISTING_MEMBER
```

A partition table rewrite does not touch the data or the md superblock underneath it,
so this restores the member in seconds.

---

## Stage 1 — Verify the new disk is genuinely blank

```bash
wipefs -n /dev/NEW_DISK
mdadm --examine /dev/NEW_DISK
```

**Expected:**

- `wipefs -n` prints nothing (`-n` = dry run, no writes).
- `mdadm --examine` prints `mdadm: No md superblock detected`.

**STOP** if either output references the array's UUID. That would mean the disk was
previously an array member and was wiped in error. The correct action then is
`mdadm --re-add` (minutes, via the bitmap), not `--add` (full rebuild) — and the datacenter
should be told a healthy member was pulled. Escalate before proceeding.

---

## Stage 2 — Partition the new disk

This is the only destructive step. It writes to the new disk only.

**Argument order:** destination goes inside `--replicate=`, source is the trailing argument.
Use by-id paths so the serials appear in the command and you can read it back as
*"replicate onto the new disk, from the existing member."*

```bash
sgdisk --replicate=/dev/disk/by-id/<NEW_DISK_BY_ID> \
       /dev/disk/by-id/<EXISTING_MEMBER_BY_ID>
```

Assign fresh GUIDs — `--replicate` copies disk and partition GUIDs verbatim, and duplicates
confuse udev and `/dev/disk/by-partuuid`:

```bash
sgdisk -G /dev/disk/by-id/<NEW_DISK_BY_ID>
partprobe /dev/NEW_DISK
lsblk /dev/NEW_DISK
```

Confirm the new partition appears at the expected size.

**Verify it matches an existing member exactly, in sectors:**

```bash
blockdev --getsz /dev/NEW_DISK1
blockdev --getsz /dev/EXISTING_MEMBER1
```

These two numbers **must be identical**. Do not proceed if they differ.

---

## Stage 3 — Add to the array

```bash
mdadm --readwrite /dev/md0
mdadm --add /dev/md0 /dev/NEW_DISK1
cat /proc/mdstat
```

**Expected:** `mdadm: added /dev/NEW_DISK1`, then a recovery line:

```
md0 : active raid5 ...
      ... blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
      [>....................]  recovery =  0.1% (...) finish=1420.5min speed=160000K/sec
```

`[_UUU]` remains until recovery completes. That is normal.

The `--readwrite` clears the `auto-read-only` flag. The kernel clears it on `--add` anyway;
this is belt-and-braces.

---

## Stage 4 — Monitor

For 14 TB members, expect **20–40 hours**.

```bash
cat /proc/mdstat
```

Check drive temperatures periodically — the rebuild is the heaviest load these drives
will ever see:

```bash
for d in a b c d; do printf "sd$d "; smartctl -A /dev/sd$d | awk '/Temperature_Celsius/{print $10"C"}'; done
```

**Throttle if any drive exceeds ~55 °C.** Note this is `sync_speed_max`, not `min` —
`speed_limit_min` is only a floor for contended I/O and will not slow an idle rebuild:

```bash
echo 50000 > /sys/block/md0/md/sync_speed_max      # slow down
echo 200000 > /sys/block/md0/md/sync_speed_max     # restore default
```

---

## Stage 5 — Confirm and tidy up

```bash
cat /proc/mdstat          # want [4/4] [UUUU]
mdadm --detail /dev/md0   # State: clean — all members active sync, no "removed" slot
```

Check the config matches. The ARRAY line is keyed by UUID, which does not change when a
member is added, so this is usually verification rather than a required edit:

```bash
cat /etc/os-release | head -2
mdadm --detail --scan
grep -E 'ARRAY|MAILADDR' /etc/mdadm/mdadm.conf 2>/dev/null || grep -E 'ARRAY|MAILADDR' /etc/mdadm.conf
```

Rebuild initramfs only if the config actually changed:

- Debian / Ubuntu: `update-initramfs -u`
- RHEL / Rocky / Alma: `dracut -f`

Verify monitoring works — a degraded array should raise an alert, not wait for a ticket:

```bash
systemctl status mdmonitor
mdadm --monitor --scan --test --oneshot    # sends a test alert
```

---

## Risk note

The array has **zero redundancy** until the rebuild completes. The rebuild reads every
sector of every surviving member. A single unrecoverable read error during that window
loses the array. Confirm backups exist before starting, and check SMART on the surviving
disks — not just the new one — before committing to the rebuild.