2x Peak Memory Pressure Analysis in Blob Store

2x.md

2× Peak Memory Pressure Analysis

Root Cause: Clay's Dual Storage

When a 2 GiB file is committed via |commit %base, Clay stores the content atom twice in its Nock-level state:

1. lat.ran ((map lobe page)) — the content-addressed object store, keyed by SHA-512 hash. Stores [%mime [mite [len atom]]] as a page.
2. mim.dom ((map path mime)) — the mime cache for unix-mounted desks. Stores [mite [len atom]] as a mime, keyed by path.

This is Clay's intentional design: lat.ran is the permanent content-addressable store; mim.dom is a fast-path cache for unix sync and %ergo delivery. Both hold the same 2 GiB atom. Since both are bob atoms (tiny 24-byte loom stubs pointing to .urb/bob/), the loom only dirties ~48 bytes for the stubs themselves. The 4.4 GiB dirty loom observed in vmmap is not the stubs — it is the atoms getting materialized during event processing.

Where Materialization Happens (the actual 2× problem)

The path during |commit %base processing in the serf:

1. %into arrives carrying a bob atom for the large file.
2. Clay calls page-to-lobe → (shax (jam page)) — jamming the page calls u3r_met/u3r_bytes on the bob atom → materializes 2 GiB into the loom temporarily, then frees it.
3. Clay runs Ford (checkout-changes) to type-check the file → may call retrieve functions on the bob atom again → materializes again.
4. Clay stores the bob atom in both lat.ran and mim.dom as bob stubs (fine — 24 bytes each).
5. u3m_save snapshots the loom. At this point only the two tiny bob stubs remain, but the loom pages dirtied by the jam buffer stay dirty until the next epoch snapshot cycle reclaims them via copy-on-write.

The kernel does not un-dirty loom pages just because the loom allocator frees a region — physical pages remain resident until the OS evicts them or the snapshot cycle reclaims them. This is why vmmap shows 4.4 GiB dirty after the commit even though the steady-state logical loom content is tiny.

Options

Option A: C-layer forward pointer (promote bob on first access)

When u3r_blob_load materializes a bob atom, rewrite the bob atom in-place as a forward pointer to the materialized normal atom:

• Set u3a_fwd_flag (bit 30 of len_w) alongside u3a_blob_flag
• Store the loom offset of the materialized normal atom in buf_w[0]
• All retrieve functions check u3a_is_fwd first, follow the pointer, skip disk I/O
• _me_lose: when a promoted bob is freed, decrement the target's refcount
• _ca_take: copy the fwd pointer, increment target refcount

This means the 2 GiB atom is loaded from disk once and reused for all accesses within the event. The jam in page-to-lobe, Ford's type-check, and the mim.dom update all share a single materialized copy.

Tradeoff: the materialized atom lives on the loom for the duration of the event, and is only freed when the last bob stub referencing it is freed. For a file in Clay's permanent state, this means the 2 GiB normal atom lives in the loom forever (until the file is deleted from Clay). Steady-state memory would be worse than the current behavior, which keeps only two tiny bob stubs.

Option A is not the right approach for permanent Clay state.

Option B: Hoon-level fix — bypass jam in page-to-lobe

If page-to-lobe could compute its lobe hash by streaming SHA-512 directly over the blob store file rather than via (jam page), the 2 GiB atom would never need to be materialized for the lobe computation. This is the correct architectural fix: the blob store file is already the canonical byte representation, so hashing it directly (with a u3_blob_hash_fd C function exposed as a jet) is semantically equivalent and avoids the jam entirely.

Requires a Hoon-level change to Clay's page-to-lobe arm and a new jet.

Option C: Per-event materialization cache

A C-level hash table (keyed by bob loom address) that stores the materialized normal atom, initialized at event start and cleared at event end. All u3r_blob_load calls check this cache first. This reduces materializations from O(calls) to O(1) per bob atom per event.

Tradeoff: requires threading event lifecycle callbacks through the noun layer. The cached atoms must be held with incremented refcounts for their duration. Adds complexity without fixing the fundamental page-to-lobe issue.

Option D: Accept the current behavior

Bob atoms in Clay's state are tiny stubs (24 bytes each). Materialization is temporary — freed after each retrieve operation. The steady-state loom after a snapshot is small. The 4.4 GiB dirty reading is a vmmap artifact of dirty-but-freed loom pages, not live data. Peak memory during the commit event is high, but short-lived, and the OS will page out freed loom regions under memory pressure.

Recommendation

The highest-impact fix is Option B (Hoon-level, bypass jam in page-to-lobe for blob-backed atoms). The lobe is a content hash; the blob store file is the content, so hashing it directly is both correct and avoids ever materializing the atom into the loom for hashing purposes.

Option C is a lower-risk incremental improvement that reduces repeated materialization within a single event, at the cost of added complexity.

Option D is defensible for now: the 2× peak is bounded (it occurs only during the commit event, not at steady state), and the OS handles memory pressure via paging. Since the vmmap dirty pages are freed loom regions — not live data — the actual working set after the event is small.

The right sequencing:

1. Ship the current implementation (Option D) — it works correctly.
2. Implement Option B as a follow-on: jet a page-to-lobe-blob path that calls a C function to stream-hash the blob file, bypassing jam for blob-backed pages.