Persistent SMART Monitoring for Munin & Dual-HBA Drive Renumbering Fix

README.md

Persistent SMART Monitoring for Munin & Dual-HBA Drive Renumbering Fix

Fixes Linux drive letter shifting (/dev/sda .. /dev/sdz) breaking Munin's smart_ wildcard plugin, eliminates dmesg Sense Key errors caused by non-ATA devices (e.g. STEC ZeusRAM NVRAM SLOGs), and provides non-destructive RRD telemetry migration on the Munin master.

---

1. The Problem

A. Non-Deterministic SCSI Drive Renumbering (sd_index_ida)

In the Linux kernel SCSI subsystem (drivers/scsi/sd.c), drive letters (sda, sdb, ..., sdz) are assigned on a first-come, first-served basis as SCSI targets report TEST UNIT READY.

On systems with:

1. Multiple SAS/SATA HBAs (e.g., dual Broadcom/LSI SAS3008 mpt3sas controllers),
2. Mixed storage media (spinning HDDs with spin-up delays vs. instant-on NVMe/SATA SSDs),
3. SAS expanders & hot-swap bays (where drives are discovered asynchronously or hot-plugged),

...drive letters inevitably shift across reboots or after drive replacements.

B. Why Munin's Default smart_ Wildcard Pattern Breaks

Munin conventionally uses symlinks matching kernel drive letters:

/etc/munin/plugins/smart_sda -> /usr/share/munin/plugins/smart_
/etc/munin/plugins/smart_sdb -> /usr/share/munin/plugins/smart_

When drive letters shuffle across reboots:

• Telemetry Corruption: On Day 1, smart_sda graphs a 4TB mechanical HDD (tracking spin-up time and load cycles). On Day 2 after a reboot, smart_sda points to a fast SSD (tracking wear indicators and NAND writes). The historical RRD graphs on the Munin master mix two completely different devices.
• dmesg Sense Key Flood from Non-ATA Devices: Enterprise SAS devices such as STEC ZeusRAM SLOGs or hardware RAID units adhere to SPC-4 SAS and do not implement standard ATA attribute tables. When Munin queries them via smartctl -a -A -i, the device firmware returns Sense Key : Recovered Error (ASC=0x80 ASCQ=0x0). The Linux SCSI driver logs this to dmesg every 5 minutes.
• Unmonitored Disks: If an admin manually unlinks smart_sdp to silence the ZeusRAM noise, a subsequent reboot may assign sdp to a real mechanical drive—leaving that drive completely unmonitored while ZeusRAM continues to be polled under a newly assigned letter like sdn.

---

2. The Solution: Persistent Naming via /dev/disk/by-id/

Munin's standard Python smart_ plugin natively supports persistent device identifiers:

def guess_full_path(hard_drive):
    for dev_dir in ('/dev', '/dev/disk/by-id'):
        full_path = os.path.join(dev_dir, hard_drive)
        if os.path.exists(full_path):
            return full_path
    return None

By pointing Munin symlinks directly to udev's persistent /dev/disk/by-id/ata-* symlinks:

ln -s /usr/share/munin/plugins/smart_ \
      /etc/munin/plugins/smart_ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6PP5S13

1. Reboot Invariance: Udev automatically updates the symlink target. Munin graphs on the master (<host>-smart_ata_<model>_<serial>-<metric>-g.rrd) are permanently anchored to the physical drive serial number forever.
2. Automatic Exclusion of Non-ATA Devices: Exotic SAS devices (like ZeusRAM) are registered under scsi-* or wwn-*, never ata-*. They are naturally excluded from ATA SMART polling.

---

3. Files Included in this Gist

• munin-reconcile-smart.sh: Boot-time reconciliation script deployed to /usr/local/sbin/munin-reconcile-smart. Scans /dev/disk/by-id/ata-*, ensures persistent links exist, prunes dead links, and purges volatile smart_sd* symlinks.
• munin-reconcile-smart.service: Systemd oneshot unit deployed to /etc/systemd/system/ and ordered Before=munin-node.service.
• migrate_munin_smart_rrds.py: Orchestration script to run on the Munin master or client. Backs up legacy RRD files and renames them to match the new sanitized smart_ata_* names without losing multi-year telemetry.

---

4. Quick Deployment Guide

Step 1: Install Reconciler on the Monitored Node

# 1. Install script
sudo cp munin-reconcile-smart.sh /usr/local/sbin/munin-reconcile-smart
sudo chmod +x /usr/local/sbin/munin-reconcile-smart

# 2. Install systemd unit
sudo cp munin-reconcile-smart.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable munin-reconcile-smart.service

# 3. Run initial reconciliation and restart munin-node
sudo /usr/local/sbin/munin-reconcile-smart
sudo systemctl restart munin-node

Step 2: Migrate Existing RRD Files on Munin Master

If you already have historical RRD files on the Munin master:

# Preview proposed renames (dry-run):
python3 migrate_munin_smart_rrds.py --node-host <node_ip_or_hostname> --munin-master <master_host>

# Apply renames with automatic backup:
python3 migrate_munin_smart_rrds.py --node-host <node_ip_or_hostname> --munin-master <master_host> --apply

On the next 5-minute poll, Munin master will detect the persistent smart_ata_* plugins, find the renamed RRD files, and seamlessly continue appending data points without a gap.

migrate_munin_smart_rrds.py

#!/usr/bin/env python3
"""
Migrate Munin SMART Monitoring to Persistent /dev/disk/by-id/ Names

1. Inspects the target node for whole-disk ATA devices in /dev/disk/by-id/ata-*.
2. Renames existing RRD files on the Munin master from legacy zamd-smart_sd<X>-<metric>-g.rrd
   to sanitized zamd-smart_ata_<sanitized_byid>-<metric>-g.rrd.
3. Automatically creates a timestamped backup before touching any RRD files.

Usage:
    python3 migrate_munin_smart_rrds.py --node-host 192.168.1.10 --munin-master munin-server.local
    python3 migrate_munin_smart_rrds.py --node-host 192.168.1.10 --munin-master munin-server.local --apply
"""

import argparse
import datetime
import os
import re
import subprocess
import sys

# Optional: Map drives with known past device swaps so history stays with the real physical drive
# e.g.: {"WD-WCC7K3PU2YLY": "sda", "WD-WCC7K1TN8756": "sdn"}
HISTORICAL_LINEAGE_OVERRIDE = {}

def run_cmd(cmd, check=True):
    """Run shell command locally, returning stdout."""
    res = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    if check and res.returncode != 0:
        raise RuntimeError(f"Command failed (exit {res.returncode}): {cmd}\nStderr: {res.stderr.strip()}")
    return res.stdout.strip()

def sanitize_munin_plugin_name(name):
    """Replicate Munin Master's _sanitise_plugin_name: s/[^_A-Za-z0-9]/_/g."""
    return re.sub(r"[^_A-Za-z0-9]", "_", name)

def get_node_disk_mapping(node_host, node_user="root"):
    """
    Inspect target node's /dev/disk/by-id/ata-* to construct mapping:
    dev (e.g. sdb) -> by-id (e.g. ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6PP5S13)
    """
    cmd = f"ssh {node_user}@{node_host} 'for d in /dev/disk/by-id/ata-*; do case \"$d\" in *-part*) continue ;; *) target=$(readlink -f \"$d\"); dev=$(basename \"$target\"); byid=$(basename \"$d\"); echo \"$dev $byid\" ;; esac; done'"
    out = run_cmd(cmd)
    disks = {}
    for line in out.strip().split("\n"):
        parts = line.strip().split()
        if len(parts) == 2:
            dev, byid = parts
            serial = byid.split("_")[-1] if "_" in byid else byid
            disks[dev] = {
                "dev": dev,
                "byid": byid,
                "serial": serial,
                "plugin_name": f"smart_{byid}",
                "sanitized_service": sanitize_munin_plugin_name(f"smart_{byid}")
            }
    return disks

def get_master_rrds(master_host, rrd_dir, node_name):
    """Fetch existing smart RRD files for the host on the Munin master."""
    cmd = f"ssh {master_host} sudo find '{rrd_dir}' -maxdepth 1 -name '{node_name}-smart_*.rrd'"
    out = run_cmd(cmd, check=False)
    if not out:
        return []
    return [line.strip() for line in out.split("\n") if line.strip()]

def plan_rrd_migrations(node_name, rrd_dir, node_disks, master_rrds):
    """
    Map existing <node>-smart_sd<X>-<field>-g.rrd to <node>-smart_ata_<sanitized>-<field>-g.rrd.
    """
    legacy_rrds = {}
    for rrd_path in master_rrds:
        filename = os.path.basename(rrd_path)
        m = re.match(rf"^{re.escape(node_name)}-smart_(sd[a-z]+)-(.*)\.rrd$", filename)
        if m:
            dev, suffix = m.groups()
            if dev not in legacy_rrds:
                legacy_rrds[dev] = []
            legacy_rrds[dev].append((suffix, rrd_path))

    migrations = []
    for info in node_disks.values():
        dev = info["dev"]
        serial = info["serial"]
        new_prefix = f"{node_name}-{info['sanitized_service']}"
        source_legacy_dev = HISTORICAL_LINEAGE_OVERRIDE.get(serial, dev)

        if source_legacy_dev in legacy_rrds:
            for suffix, src_path in legacy_rrds[source_legacy_dev]:
                dst_filename = f"{new_prefix}-{suffix}.rrd"
                dst_path = os.path.join(rrd_dir, dst_filename)
                desc = f"{source_legacy_dev} -> {info['byid']} ({serial})"
                migrations.append({
                    "src": src_path,
                    "dst": dst_path,
                    "desc": desc,
                    "dev": dev,
                    "legacy_dev": source_legacy_dev,
                    "byid": info["byid"]
                })

    return migrations, legacy_rrds

def main():
    parser = argparse.ArgumentParser(description="Migrate Munin SMART RRD files to persistent by-id naming.")
    parser.add_argument("--node-host", required=True, help="Hostname or IP of monitored node")
    parser.add_argument("--node-user", default="root", help="SSH user for monitored node (default: root)")
    parser.add_argument("--node-name", help="Munin host name on master (default: same as node-host without domain)")
    parser.add_argument("--munin-master", required=True, help="Hostname or IP of central Munin master server")
    parser.add_argument("--rrd-dir", default="/var/lib/munin/infosl", help="RRD directory on master")
    parser.add_argument("--apply", action="store_true", help="Apply renames on Munin master (default: dry-run)")
    args = parser.parse_args()

    node_name = args.node_name or args.node_host.split(".")[0]

    print("=" * 80)
    print("MUNIN SMART MONITORING: MIGRATION TO PERSISTENT BY-ID")
    print("=" * 80)
    print(f"Monitored Node: {args.node_user}@{args.node_host}")
    print(f"Munin Master:   {args.munin_master} (Path: {args.rrd_dir})")
    print(f"Mode:           {'APPLY (Production Changes)' if args.apply else 'DRY-RUN (Simulated)'}\n")

    print("[1/3] Inspecting ATA storage devices on node...")
    node_disks = get_node_disk_mapping(args.node_host, args.node_user)
    print(f"Found {len(node_disks)} ATA block devices:")
    for dev in sorted(node_disks.keys()):
        d = node_disks[dev]
        print(f"  - /dev/{dev:<4} -> {d['byid']:<45} [Serial: {d['serial']}]")

    print("\n[2/3] Inspecting existing Munin RRDs on master...")
    master_rrds = get_master_rrds(args.munin_master, args.rrd_dir, node_name)
    print(f"Found {len(master_rrds)} existing {node_name}-smart_*.rrd files on master.")

    migrations, legacy_rrds = plan_rrd_migrations(node_name, args.rrd_dir, node_disks, master_rrds)
    print(f"Planned RRD migrations: {len(migrations)} files.")

    print("\nSample RRD file migrations:")
    for m in migrations[:8]:
        print(f"  {os.path.basename(m['src'])}")
        print(f"    -> {os.path.basename(m['dst'])} [{m['desc']}]")
    if len(migrations) > 8:
        print(f"  ... and {len(migrations) - 8} more files.")

    if not args.apply:
        print("\n" + "=" * 80)
        print("DRY-RUN SUMMARY: Ready to execute.")
        print("Run with '--apply' to perform the migration.")
        print("=" * 80)
        return

    print("\n[3/3] Applying changes on Munin master...")
    timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    backup_dir = f"{args.rrd_dir}/backup-{node_name}-smart-{timestamp}"
    print(f"Creating backup directory on master: {backup_dir}")
    run_cmd(f"ssh {args.munin_master} sudo mkdir -p '{backup_dir}'")

    print("Backing up legacy RRD files...")
    run_cmd(f"ssh {args.munin_master} 'sudo cp -p {args.rrd_dir}/{node_name}-smart_*.rrd {backup_dir}/'")
    print(f"Backed up {len(master_rrds)} files to {backup_dir}.")

    print("Migrating RRD filenames on master...")
    rename_cmds = ["set -e"]
    for m in migrations:
        rename_cmds.append(f"mv -v '{m['src']}' '{m['dst']}'")
    rename_cmds.append(f"rm -f {args.rrd_dir}/{node_name}-smart_sd*.rrd")
    rename_cmds.append(f"chown -R munin:munin {args.rrd_dir}/{node_name}-smart_ata_*.rrd {backup_dir}")
    batch_script = "\n".join(rename_cmds) + "\n"

    p = subprocess.run(f"ssh {args.munin_master} sudo bash", shell=True, input=batch_script, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    if p.returncode != 0:
        raise RuntimeError(f"Rename batch failed:\n{p.stderr.strip()}")

    print(f"Successfully migrated {len(migrations)} RRD files on {args.munin_master}.")
    print("\n" + "=" * 80)
    print("MIGRATION COMPLETED SUCCESSFULLY")
    print("=" * 80)
    print(f"Backup saved to: {backup_dir}")

if __name__ == "__main__":
    main()

munin-reconcile-smart.service

[Unit]
Description=Reconcile Munin SMART disk plugins with persistent by-id links
Before=munin-node.service
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/munin-reconcile-smart
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

munin-reconcile-smart.sh

#!/usr/bin/env bash
# /usr/local/sbin/munin-reconcile-smart
# Reconciles Munin SMART plugins to use persistent /dev/disk/by-id/ata-* symlinks.
# Automatically purges volatile smart_sd* links and ignores SAS/SCSI ZeusRAM.

set -euo pipefail

PLUGIN_DIR="/etc/munin/plugins"
SHARE_PLUGIN="/usr/share/munin/plugins/smart_"
STATE_DIR="/var/lib/munin-node/plugin-state/root"

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting Munin SMART disk reconciliation..."

# 1. Ensure all whole-disk ATA drives have persistent symlinks
count_added=0
for disk in /dev/disk/by-id/ata-*; do
    [ -e "$disk" ] || continue
    case "$disk" in
        *-part*) continue ;; # Skip partition slices
    esac
    byid=$(basename "$disk")
    plugin="$PLUGIN_DIR/smart_$byid"
    if [ ! -L "$plugin" ]; then
        ln -sf "$SHARE_PLUGIN" "$plugin"
        echo "  [+] Linked $plugin -> $SHARE_PLUGIN"
        count_added=$((count_added + 1))
    fi
done

# 2. Remove obsolete / volatile smart_sd* symlinks
count_removed=0
for legacy in "$PLUGIN_DIR"/smart_sd*; do
    [ -e "$legacy" ] || continue
    rm -f "$legacy"
    echo "  [-] Removed volatile plugin: $legacy"
    count_removed=$((count_removed + 1))
done

# 3. Clean broken/dead smart_ata-* symlinks (removed drives)
for active in "$PLUGIN_DIR"/smart_ata-*; do
    [ -L "$active" ] || continue
    byid=$(basename "$active" | sed 's/^smart_//')
    if [ ! -e "/dev/disk/by-id/$byid" ]; then
        rm -f "$active"
        echo "  [-] Removed dead disk plugin: $active"
        count_removed=$((count_removed + 1))
    fi
done

# 4. Clean obsolete sd* state files
rm -f "$STATE_DIR"/smart-sd*.state 2>/dev/null || true

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Reconciliation complete. Added: $count_added, Removed: $count_removed."