Symbolicating modern Apple-silicon (arm64e/SPTM) kernel panics without a KDK: recover stripped XNU symbols via ipsw + blacktop signatures, and pick the KASLR slide by panic-machinery coherence.

kernel-panic-symbolicate.py

#!/usr/bin/env python3
"""
Kernel panic symbolicator for Apple Silicon (arm64e) macOS panics.

Released kernelcaches strip core-XNU symbols (kext C++ symbols survive, the
kernel proper does not), so the raw `lr:` backtrace in a `.panic` file resolves
to garbage adjacent-kext names. This recovers the real names by:

  1. Parsing the panic: backtrace LRs, the reported KASLR slides, the Fileset
     Kernelcache UUID, and the Darwin kernel version.
  2. Locating the on-disk Preboot kernelcache whose Fileset UUID matches the
     panic, and decompressing it with `ipsw kernel dec`.
  3. Recovering stripped XNU symbols with `ipsw kernel symbolicate` against the
     blacktop/symbolicator signature set for the matching kernel version.
  4. Merging those with the kext symbols already in the cache (`nm`).
  5. Resolving each frame at every reported slide and keeping the slide whose
     inner frames land on the panic/exception machinery (panic, sleh_synchronous,
     handle_*_fault, ...). That coherence check is what disambiguates the slide:
     on SPTM kernelcaches the kernel, text-exec, and collection slides differ,
     and only one maps the backtrace onto real functions.

Artifacts are cached under ~/.cache/kernel-panic-sym/<fileset-uuid>/ so repeat
runs are instant.

Requires: ipsw (brew install ipsw), nm (Xcode CLT), git (to fetch signatures).

Usage:
    kernel-panic-symbolicate.py <panic-file> [--signatures-repo DIR] [--json]
    kernel-panic-symbolicate.py --latest          # newest panic in DiagnosticReports
"""

import argparse
import bisect
import glob
import json
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path

CACHE_ROOT = Path.home() / ".cache" / "kernel-panic-sym"
SIG_REPO_DEFAULT = Path.home() / ".cache" / "symbolicator"
SIG_REPO_URL = "https://github.com/blacktop/symbolicator.git"

DIAG_DIRS = [
    Path("/Library/Logs/DiagnosticReports"),
    Path("/Library/Logs/DiagnosticReports/Retired"),
]

PREBOOT_GLOB = "/System/Volumes/Preboot/*/boot/*/System/Library/Caches/com.apple.kernelcaches/kernelcache"

# Symbols that only appear in the panic/exception path. A correct slide lands
# several backtrace frames on these; a wrong slide lands on none.
ANCHOR_SYMS = (
    "panic", "panic_trap_to_debugger", "handle_debugger_trap",
    "sleh_synchronous", "fleh_synchronous", "handle_uncategorized",
    "handle_kernel_abort", "handle_kernel_tag_check_fault", "Debugger",
)


def die(msg, code=1):
    print(f"error: {msg}", file=sys.stderr)
    sys.exit(code)


def have(tool):
    return shutil.which(tool) is not None


def run(cmd, **kw):
    return subprocess.run(cmd, capture_output=True, text=True, **kw)


# --- panic parsing ----------------------------------------------------------

def find_latest_panic():
    newest, newest_mtime = None, -1
    for d in DIAG_DIRS:
        for path in glob.glob(str(d / "panic-*.panic")):
            m = os.stat(path).st_mtime
            if m > newest_mtime:
                newest, newest_mtime = path, m
    return newest


def read_panic_string(path):
    """A .panic file is one JSON metadata line then a JSON body. The human
    backtrace + slide header live in body['panicString']. Fall back to raw
    text if the structure is unexpected."""
    raw = Path(path).read_text(errors="replace")
    try:
        first, rest = raw.split("\n", 1)
        body = json.loads(rest)
        ps = body.get("panicString")
        if ps:
            return ps
    except (ValueError, json.JSONDecodeError):
        pass
    return raw


def parse_panic(path):
    ps = read_panic_string(path)
    info = {"path": str(path), "panic_string": ps}

    m = re.search(r"panic\(.*?\):\s*(.+)", ps)
    info["reason"] = m.group(1).strip() if m else "(unknown)"

    m = re.search(r"Darwin Kernel Version (\d+)\.(\d+)", ps)
    info["kernel_ver"] = f"{m.group(1)}.{m.group(2)}" if m else None

    m = re.search(r"Fileset Kernelcache UUID:\s*([0-9A-Fa-f]{32})", ps)
    info["fileset_uuid"] = m.group(1).upper() if m else None

    slides = {}
    for label, key in [
        ("KernelCache slide", "kernelcache"),
        ("Kernel slide", "kernel"),
        ("Kernel text exec slide", "text_exec"),
    ]:
        m = re.search(rf"{re.escape(label)}:\s*(0x[0-9a-fA-F]+)", ps)
        if m:
            slides[key] = int(m.group(1), 16)
    info["slides"] = slides

    # Backtrace: take the LRs from the "Panicked thread" section onward.
    tail = ps[ps.find("Panicked thread"):] if "Panicked thread" in ps else ps
    info["backtrace"] = [int(x, 16) for x in re.findall(r"lr:\s*(0x[0-9a-fA-F]+)", tail)]
    return info


# --- kernelcache location + symbol generation -------------------------------

def macho_uuid(path):
    if not have("ipsw"):
        return None
    r = run(["ipsw", "macho", "info", str(path)])
    m = re.search(r"LC_UUID\s+([0-9A-Fa-f-]+)", r.stdout)
    return m.group(1).replace("-", "").upper() if m else None


def locate_kernelcache(fileset_uuid, cachedir):
    """Find the Preboot kernelcache matching the panic, decompress it, cache it."""
    cached = cachedir / "kernelcache.decompressed"
    if cached.exists():
        return cached
    tmp_src = cachedir / "kernelcache.im4p"
    dec = cachedir / "kernelcache.im4p.decompressed"  # ipsw's default output name
    for src in glob.glob(PREBOOT_GLOB):
        try:
            shutil.copy(src, tmp_src)
        except OSError:
            continue
        run(["ipsw", "kernel", "dec", str(tmp_src)])
        if dec.exists() and macho_uuid(dec) == fileset_uuid:
            dec.rename(cached)
            tmp_src.unlink(missing_ok=True)
            return cached
        if dec.exists():
            dec.unlink()
    tmp_src.unlink(missing_ok=True)
    return None


def ensure_signatures(repo, kernel_ver):
    if not repo.exists():
        print(f"  cloning signature DB -> {repo} ...", file=sys.stderr)
        repo.parent.mkdir(parents=True, exist_ok=True)
        r = run(["git", "clone", "--depth", "1", SIG_REPO_URL, str(repo)])
        if r.returncode != 0:
            die(f"could not clone signatures: {r.stderr.strip()}")
    kdir = repo / "kernel"
    if not kdir.exists():
        die(f"signature repo has no kernel/ dir at {kdir}")
    cand = kdir / kernel_ver
    if cand.exists():
        return cand
    # fall back to highest available minor of the same major
    major = kernel_ver.split(".")[0]
    minors = sorted(
        (p for p in kdir.glob(f"{major}.*") if p.is_dir()),
        key=lambda p: [int(x) for x in p.name.split(".")],
    )
    if minors:
        print(f"  note: exact sigs for {kernel_ver} absent; using {minors[-1].name}", file=sys.stderr)
        return minors[-1]
    die(f"no signatures for kernel major {major} in {kdir}")


def gen_xnu_syms(kc, sigdir, cachedir):
    out = cachedir / "xnu.syms"
    if out.exists():
        return out
    r = run(["ipsw", "kernel", "symbolicate",
             "--signatures", str(sigdir), "--flat", "-q",
             "-o", str(cachedir), str(kc)])
    produced = cachedir / (kc.name + ".syms")
    if produced.exists():
        produced.rename(out)
        return out
    # some versions name it differently; grab any fresh .syms
    cands = [p for p in cachedir.glob("*.syms") if p.name != "kext.syms"]
    if cands:
        cands[0].rename(out)
        return out
    die(f"ipsw kernel symbolicate produced no .syms ({r.stderr.strip()[:200]})")


def gen_kext_syms(kc, cachedir):
    out = cachedir / "kext.syms"
    if out.exists():
        return out
    for arch in ("arm64e", "arm64"):
        r = run(["nm", "-arch", arch, "-n", str(kc)])
        if r.returncode == 0 and r.stdout:
            with open(out, "w") as f:
                for ln in r.stdout.splitlines():
                    p = ln.split(None, 2)
                    if len(p) >= 3 and p[1] in ("t", "T"):
                        f.write(f"{p[0]} {p[2]}\n")
            return out
    die("nm produced no kext symbols")


# --- resolution -------------------------------------------------------------

def load_syms(*paths):
    table = {}
    for p in paths:
        for ln in Path(p).read_text(errors="replace").splitlines():
            parts = ln.split(None, 1)
            if len(parts) != 2:
                continue
            a = parts[0][2:] if parts[0].startswith("0x") else parts[0]
            try:
                addr = int(a, 16)
            except ValueError:
                continue
            table.setdefault(addr, parts[1].strip())
    addrs = sorted(table)
    return addrs, table


def resolver(addrs, table):
    def resolve(static_addr):
        i = bisect.bisect_right(addrs, static_addr) - 1
        if i < 0:
            return "??", 0
        base = addrs[i]
        return table[base], static_addr - base
    return resolve


def pick_slide(backtrace, slides, resolve):
    """Score each reported slide by how many frames land on panic/exception
    anchors. Highest score wins (ties -> smallest total offset)."""
    best = None
    for name, sl in slides.items():
        hits, tot = 0, 0
        for f in backtrace:
            sym, off = resolve(f - sl)
            if any(a in sym for a in ANCHOR_SYMS):
                hits += 1
            tot += off
        score = (hits, -tot)
        if best is None or score > best[0]:
            best = (score, name, sl)
    return best  # ((hits,-tot), name, slide)


def is_machinery(sym):
    return any(a in sym for a in ANCHOR_SYMS)


# --- main -------------------------------------------------------------------

def symbolicate(panic_path, sig_repo, as_json=False):
    if not have("ipsw"):
        die("ipsw not found. Install with: brew install ipsw")
    if not have("nm"):
        die("nm not found. Install Xcode Command Line Tools.")

    info = parse_panic(panic_path)
    if not info["backtrace"]:
        die("no kernel backtrace found in panic (userspace crash?)")
    if not info["fileset_uuid"]:
        die("panic has no Fileset Kernelcache UUID (not an arm64e kernel panic?)")
    if not info["slides"]:
        die("panic reports no KASLR slides")

    cachedir = CACHE_ROOT / info["fileset_uuid"]
    cachedir.mkdir(parents=True, exist_ok=True)

    kc = locate_kernelcache(info["fileset_uuid"], cachedir)
    if not kc:
        die(f"no on-disk kernelcache matches fileset UUID {info['fileset_uuid']} "
            "(OS updated since the panic?). Symbolication needs the matching kernelcache.")

    ver = info["kernel_ver"] or "25.4"
    sigdir = ensure_signatures(sig_repo, ver)
    xnu = gen_xnu_syms(kc, sigdir, cachedir)
    kext = gen_kext_syms(kc, cachedir)

    addrs, table = load_syms(xnu, kext)
    resolve = resolver(addrs, table)
    (hits, negtot), slide_name, slide = pick_slide(info["backtrace"], info["slides"], resolve)

    frames = []
    for i, f in enumerate(info["backtrace"]):
        sym, off = resolve(f - slide)
        frames.append({"i": i, "addr": f, "symbol": sym, "offset": off,
                       "machinery": is_machinery(sym)})

    # Faulting frames = everything below the LAST panic/exception-machinery
    # frame: the code that actually took the fault and its callers. (Stray
    # machinery-adjacent frames higher in the stack, e.g. Assert in the SP1
    # save path, are not part of this run.)
    last_mach = max((fr["i"] for fr in frames if fr["machinery"]), default=-1)
    fault_idx = {fr["i"] for fr in frames if fr["i"] > last_mach}
    fault_frames = [fr for fr in frames if fr["i"] in fault_idx]

    result = {
        "panic": info["path"],
        "reason": info["reason"],
        "kernel_ver": ver,
        "fileset_uuid": info["fileset_uuid"],
        "chosen_slide": f"0x{slide:x}",
        "chosen_slide_name": slide_name,
        "anchor_hits": hits,
        "frames": frames,
        "fault_frames": fault_frames,
        "symbol_source": {"xnu": str(xnu), "kext": str(kext), "kernelcache": str(kc)},
    }

    if as_json:
        print(json.dumps(result, indent=2, default=str))
        return result

    print(f"Panic:   {info['reason']}")
    print(f"Cache:   {info['fileset_uuid']}  (kernel {ver})")
    print(f"Slide:   0x{slide:x} ({slide_name}, {hits} machinery frames matched)")
    if hits == 0:
        print("  WARNING: no panic-machinery frames matched any slide — result is unreliable.")
    print()
    print(f"  {'fr':>2}  {'slid address':<20}  symbol + offset")
    for fr in frames:
        tag = "  <== fault path" if fr["i"] in fault_idx else ""
        print(f"  {fr['i']:>2}  0x{fr['addr']:016x}  {fr['symbol']}+{fr['offset']}{tag}")
    print()
    if fault_frames:
        print("Faulting call chain (the non-panic-machinery frames):")
        for fr in fault_frames:
            print(f"  {fr['symbol']}+{fr['offset']}")
    return result


def main():
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("panic", nargs="?", help="path to a .panic file")
    ap.add_argument("--latest", action="store_true",
                    help="use the newest panic in DiagnosticReports")
    ap.add_argument("--signatures-repo", default=str(SIG_REPO_DEFAULT),
                    help=f"blacktop/symbolicator checkout (default: {SIG_REPO_DEFAULT}; cloned if absent)")
    ap.add_argument("--json", action="store_true", help="machine-readable output")
    args = ap.parse_args()

    panic = args.panic
    if args.latest or not panic:
        panic = find_latest_panic()
        if not panic:
            die("no panic files found in DiagnosticReports")
        print(f"# latest panic: {panic}", file=sys.stderr)
    if not os.path.exists(panic):
        die(f"no such file: {panic}")

    symbolicate(panic, Path(args.signatures_repo), as_json=args.json)


if __name__ == "__main__":
    main()

symbolicating-arm64e-kernel-panics.md

Symbolicating modern Apple-silicon kernel panics without a KDK

arm64e, SPTM kernelcaches — macOS 26 / Darwin 25. Using only ipsw, the blacktop signature set, and the slides already in the panic header.

TL;DR

On recent Apple-silicon Macs the released kernelcache strips core-XNU symbols — kext C++ symbols survive, the kernel proper does not. So the raw lr: backtrace in a .panic file resolves to meaningless adjacent symbols: a "symbol salad" with GPU, video-codec, and Neural-Engine functions sharing one impossible call stack. This is how to get the real names back, and the one non-obvious trick that makes it work:

Pick the KASLR slide by panic-machinery coherence, not by arithmetic.

The payoff — turning this:

 7  __ZL22GenerateGrainLag2_neonILb0EEvP18AV1FilmGrainParams…+1920   ← "panic caller"?!
11  __ZN22ACMRMEnvironmentConfig20configureEnvironmentE…+616         ← faulting pc?!
13  __ZN11ANEHWDevice23ReadPerformanceCountersE…+236

into this:

 8  handle_kernel_tag_check_fault+1440
10  fleh_synchronous+72
11  cfil_info_action_timed_out+16      ← FAULTING pc
12  cfil_info_log+152
13  soflow_gc_expire+232
14  user_take_ast+948

Why the naive approach fails

A released kernelcache is an MH_FILESET. Run nm on it and you get hundreds of thousands of symbols — but they're overwhelmingly kext symbols (mangled C++ like __ZN24AGXFirmwareResourceStack…). The core kernel's functions (panic, sleh_synchronous, zalloc, the networking stack, …) have no symbol table entries. lldb confirms it: load the cache and it spits invalid string table offset for the kernel's stripped nlist.

So when you resolve a backtrace address that lands in core XNU, the nearest preceding symbol is whatever kext function happens to sit before it in memory. You get plausible-looking names that are completely wrong. The dead giveaway: unrelated subsystems (GPU + AV1 + ANE) in a single stack. A real kernel stack stays within a subsystem or threads through core XNU.

What the panic header gives you

A .panic file (bug_type 210) is one JSON metadata line then a JSON body; the human-readable header + backtrace live in body["panicString"]. The pieces you need:

• Backtrace — the lr: list under Panicked thread:.
• Fileset Kernelcache UUID — identifies exactly which on-disk kernelcache ran.
• Three KASLR slides — KernelCache slide, Kernel slide, Kernel text exec slide. On SPTM systems these differ, because the kernel, its text-exec segment, and the whole collection are slid independently.
• Kernel version — Darwin Kernel Version 25.4.0 → signature set 25.4.

The recipe

1. Match and decompress the kernelcache

The booted kernelcache is in Preboot (the .kc files under /System/Library/KernelCollections are not the fused, booted artifact). There are usually two; only one matches the panic's Fileset UUID.

for kc in /System/Volumes/Preboot/*/boot/*/System/Library/Caches/com.apple.kernelcaches/kernelcache; do
  cp "$kc" /tmp/kc.im4p
  ipsw kernel dec /tmp/kc.im4p                     # -> /tmp/kc.im4p.decompressed
  ipsw macho info /tmp/kc.im4p.decompressed | grep LC_UUID   # match against the panic
done

2. Recover stripped XNU symbols

ipsw kernel symbolicate pattern-matches the blacktop/symbolicator signature set against the cache and recovers the kernel symbol names — ~92% on a current build:

git clone --depth 1 https://github.com/blacktop/symbolicator ~/.cache/symbolicator
ipsw kernel symbolicate --signatures ~/.cache/symbolicator/kernel/25.4 \
     --flat -o /tmp /tmp/kc.im4p.decompressed       # -> /tmp/…decompressed.syms

Merge those with the kext symbols already in the cache (nm -arch arm64e -n) for a complete address→name table.

3. The slide gotcha — pick by coherence, not arithmetic

This is the part that cost me an hour. The intuitive move is to unslide each backtrace address with Kernel text exec base − Kernel text exec slide. It gives garbage. Apple references the reported "text exec slide" to the __TEXT (read-only) segment, not __TEXT_EXEC, so the math lands you in the wrong place and you get a fresh symbol salad — twice, if you're stubborn.

The robust fix: you have three candidate slides and a dense symbol table. Resolve the backtrace at each reported slide, and keep the one whose inner frames land on the panic/exception machinery — panic, panic_trap_to_debugger, sleh_synchronous, fleh_synchronous, handle_kernel_tag_check_fault, … A correct slide lights up many of these; a wrong slide lights up none. (Don't score by "small offset to nearest symbol" — with a 200k-symbol table every slide lands close to something. Score by hitting the specific functions that can only appear in a panic stack.)

On the example below, the winner was the KernelCache slide (0x46e74000), not the text-exec slide the header seems to point you at.

4. Read the stack

The faulting frames are the contiguous run below the last machinery frame — the code that actually took the fault and its callers. Everything above is the panic/exception plumbing.

Worked example

The panic that started this: Kernel tag check fault — Apple-silicon hardware memory-tagging (pointer tag 0xf3 vs the freed-and-retagged slot's 0xfe) catching a use-after-free in XNU's socket content-filter (CFIL) flow garbage collector:

user_take_ast
  → soflow_gc_expire              (socket-flow GC on return to userspace)
      → cfil_info_log
          → cfil_info_action_timed_out   ← freed cfil_info dereferenced
  → fleh_synchronous → sleh_synchronous → handle_kernel_tag_check_fault → panic

Trigger: an active NEFilterDataProvider content filter (here, Little Snitch Mini) makes XNU track every socket flow through CFIL; sustained socket churn over a ~6.7-day uptime eventually hit a latent UAF in content_filter.c. An Apple kernel bug, exercised — not caused — by the content filter. macOS 26.4.1 (25E253), Mac17,6.

A tag-check fault, incidentally, is the hardware catching a software memory-safety bug, not bad RAM — and panicString: Sleep 0 / Wake 0 rules out any wake-transition theory before you even reach for pmset.

The script

kernel-panic-symbolicate.py (attached) automates all of the above:

kernel-panic-symbolicate.py --latest        # newest panic in DiagnosticReports
kernel-panic-symbolicate.py /path/to.panic --json

It matches the cache by Fileset UUID, decompresses, recovers XNU symbols, auto-selects the coherent slide, and prints the annotated stack. Artifacts cache under ~/.cache/kernel-panic-sym/<uuid>/ (first run ~25s, repeats ~0.1s).

Requirements: ipsw (brew install ipsw), Xcode CLT (nm), git.

Credits

All the heavy lifting is blacktop's — ipsw for the kernelcache plumbing and the symbolicator signature set for XNU symbol recovery. The only thing here is the glue and the slide-by-coherence heuristic.