How ds4.c rethinks the KV cache: standard KV vs MLA vs disk-tier prefix caching

ds4-kv-cache-explained.md

How ds4.c rethinks the KV cache

A long-form explanation of how transformer KV caching normally works, and the two layered ideas — Multi-head Latent Attention in DeepSeek V4 Flash, and disk-tier prefix caching in ds4.c — that together turn "every cold start re-prefills your 25k-token system prompt" into "you pay that cost once, ever."

What a "normal" KV cache is and why it exists

When a transformer generates a token, every attention layer needs to look at all the tokens before it. Each previous token contributes a key (K) and value (V) vector per attention head, computed by multiplying that token's hidden state by the layer's K and V projection matrices. Without a cache, every new token would re-compute those projections for every past token — that's O(N²) per layer per step, which is unworkable past a few hundred tokens.

So inference servers cache K and V per token, per head, per layer the moment they're computed and reuse them on every subsequent step. Hence KV cache.

Sizing it. For a model with L layers, H heads, head dimension D, sequence length S, dtype B bytes:

KV bytes ≈ 2 × L × H × D × S × B
                ↑
              K and V

Concrete: Llama-3-70B (80 layers, 64 heads, 128 dim) at 32k context in fp16 ≈ 20 GB just for KV. At 1M context that's >600 GB — completely infeasible. KV cache is what kills long context, not the model weights.

The agent pain point

Two things make this worse for coding agents:

1. Stateless protocol. OpenAI- and Anthropic-style APIs are stateless: the client resends the whole conversation on every turn. The server has no memory of what it computed last call.
2. Big system preambles. Pi sends ~25k tokens of system prompt. Claude Code is similar.

So a "fresh" agent session looks like: parse 25k tokens → run them through 80+ layers to fill the KV → only then start generating. On standard servers, a restart wipes the cache, so you eat that prefill on every cold start. At ~250 t/s prefill (the M3 Max number from the ds4.c README) that's ~100 s of "the model is thinking" before any useful output. Daily friction.

vLLM and a few others added in-memory prefix caching to dedup within the same server's lifetime. Helpful, but: still RAM-only, still lost on restart, still bounded by RAM size.

What DeepSeek V4 Flash changes in the model itself

DeepSeek's architectures (V2 onwards) use Multi-head Latent Attention instead of standard multi-head attention. The trick: don't cache K and V per head. Cache a single small "latent" vector per token per layer, and project it back up to K and V on the fly during attention.

Effect on cache size: instead of 2 × H × D numbers per token per layer, you store something like D_compressed per token per layer — typically a 5–10× shrink. DS4 also has a separate compressed "indexer" structure used for routing.

In a real ds4-server log you can see it directly:

ds4-server: context buffers 2842.64 MiB (ctx=100000, ...,
            raw_kv_rows=2304, compressed_kv_rows=25002)

That's 2.8 GB for 100k tokens of KV. A standard fp16 KV for a model this size at 100k would be roughly an order of magnitude bigger. The README notes that the indexer alone at full 1M context is ~22 GB — meaning the whole KV at 1M is ~26 GB. Still significant, but feasible. Without MLA you couldn't dream of running 1M context locally at any quant.

This is the precondition. Without compressed KV, none of the disk-tier ideas would matter.

What ds4.c adds on top

Three layers of caching, each progressively cheaper:

┌───────────────────────────────────────────────────────────────┐
│ TIER 1 — live session in RAM                                  │
│   one active KV checkpoint, used by the current request       │
│   ~2.8 GB at ctx=100k                                         │
└───────────────────────────────────────────────────────────────┘
              ↓ on evict / continued interval / shutdown
┌───────────────────────────────────────────────────────────────┐
│ TIER 2 — disk KV cache (the new thing)                        │
│   ~/.cache/ds4-kv/<sha1>.kv files                             │
│   keyed by SHA1 of exact token IDs                            │
│   bounded by --kv-disk-space-mb (LRU eviction)                │
└───────────────────────────────────────────────────────────────┘
              ↓ if no hit
┌───────────────────────────────────────────────────────────────┐
│ TIER 3 — full prefill                                         │
│   the slow path                                               │
└───────────────────────────────────────────────────────────────┘

What goes in a Tier-2 file:

• KVC header
• The rendered text (purely for hexdump debugging — not the key)
• Checkpoint token IDs
• The next-token logits (so you can resume sampling without one extra decode step)
• The compressed KV rows for every layer
• The compressed indexer rows
• The raw sliding-window KV rows in logical order

Lookup is by token-ID SHA1, not by string. Each token ID is hashed as a little-endian u32. This dodges every flavor of tokenizer drift, BPE retokenization weirdness, and Unicode normalization difference. If two requests' token streams are byte-identical for the first N tokens, they share a cache entry; otherwise they don't, full stop.

What it looks like at runtime

Sequence of a session with the cache enabled:

Pi launches, sends turn 1 (≈ 25k token preamble + small user message)
  ↓
ds4-server:
  - hashes incoming token prefix
  - no .kv file matches → full prefill (~100 s)
  - generates response
  - writes "cold" snapshot at SHA1(first ~25k tokens) to disk

Pi sends turn 2 (preamble + previous turn + new user message)
  ↓
ds4-server:
  - hashes prefix → matches the snapshot from turn 1
  - loads compressed KV from disk into the live tier (~5–10 s for typical sizes)
  - extends with the new tokens only (small prefill)
  - generates

User restarts ds4-server overnight, comes back, starts a fresh Pi session
  ↓
ds4-server:
  - hashes incoming preamble
  - matches the on-disk snapshot from yesterday
  - loads it → starts generating in seconds, not minutes

That's the win. One cold prefill, ever, for a stable preamble.

Two design details worth understanding

1. Trim and align before saving. When the server saves a "cold" snapshot, it doesn't save every token — it trims the last 32 and aligns down to a 2048-token chunk boundary. Reason: BPE retokenization. If you tokenize "the system prompt" and later "the system prompt and now more", the boundary tokens of the first string can merge differently in the second string's tokenizer pass. Save the whole prefix, and the next request — which is the same prompt plus more text — won't actually have a matching token-ID prefix anymore, so the SHA1 misses. Trimming + aligning parks the saved boundary safely inside the stable region of the BPE output, where the next request will agree byte-for-byte.

2. read/write, not mmap. Even though loading a cache file feels like exactly the kind of thing you'd mmap, ds4.c uses ordinary I/O. Why: the process already mmaps the 82 GB GGUF as wired memory. Adding more VM mappings per cache hit risks bumping into the kernel's mapping-count limit, plus it would interact unpredictably with Metal residency. Paying explicit-I/O CPU cost is the cheap trade.

What this means for the practical numbers

Stage	Standard server	ds4.c (cold)	ds4.c (warm hit)
Pi first turn	~100 s prefill	~100 s prefill + save	—
Pi same session, turn 2+	small	small	small
Pi after restart	~100 s again	—	~5–10 s load
Pi new session, same preamble	~100 s again	—	~5–10 s load
Disk space cost	0	one ~few-GB file per stable preamble	same

The whole reason this is interesting is that it only became viable when the model itself shrank its KV (MLA). With a normal transformer's KV, persisting prefixes to disk would mean writing tens of gigabytes per session — slower than just re-prefilling. Compressed KV inverts the math: writing ~1 GB to a fast NVMe is faster than re-running the model over 25k tokens, even when you have a Metal GPU sitting right there.

Hence antirez's tagline: "the KV cache is actually a first-class disk citizen."