Blob Store for Very Large Atoms (`ml/bob`)

1. Overview

Large atoms (> 32 MiB) are stored as content-addressed files in $pier/.urb/bob/<mug>/<seq> instead of being held inline in the loom. A bob atom is an indirect atom whose payload encodes only a (mug, seq) pair; the actual bytes live on disk. The loom holds a small u3a_blob metadata record per unique blob, reference-counted independently from the noun refcount.

Key invariants:

Urth (Earth-side) is the sole process that writes new files; it uses mkstemp(3) in bob/stg/ and sends [%blob-install path] to Mars.
Mars (Serf-side) stats, mugs, deduplicates, and renames the staged file into the canonical bucket. No blob bytes cross the IPC pipe.
A blob file is deleted only when both the loom-atom refcount and the event-log/lease refcount reach zero.

2. Architecture

2.1 Bob atom representation (`allocate.h`)

A bob atom is a standard u3a_atom with a sentinel bit in len_w:

/* Flag bit indicating the indirect atom is a blob reference. */
/* VERE64: bit 63 */ #define u3a_blob_flag  0x8000000000000000ULL
/* 32-bit:  bit 31 */ /* #define u3a_blob_flag  0x80000000u */

/* Mask to strip the flag from len_w / mug_h. */
#define u3a_blob_mask  (~u3a_blob_flag)

Layout of a bob atom on the loom:

field	value	meaning
`len_w`	`1 \| u3a_blob_flag`	one-word payload; flag set
`mug_h`	31-bit content mug	bucket directory name
`buf_w[0]`	`seq_w`	sequence number within bucket

Inline accessors (already implemented):

static inline c3_o  u3a_is_bob (u3_atom a);          // tests blob_flag
static inline c3_h  u3a_bob_mug(u3_atom a);          // strips flag from mug_h
static inline c3_w  u3a_bob_seq(u3_atom a);          // buf_w[0]

Constructor (already implemented, imprison.c):

u3_noun u3i_blob(c3_h mug_h, c3_w seq_w);

2.2 `u3a_blob` — loom-resident blob metadata (`allocate.h`) [todo]

typedef struct _u3a_blob {
  c3_w  use_w;   // refcount: event-log refs + active leases
  c3_h  mug_h;   // 31-bit content mug
  c3_w  seq_w;   // sequence number within bucket
  c3_d  siz_d;   // byte size
} u3a_blob;

Lives on the loom; allocated by Mars when a blob is committed. use_w is independent from the noun refcount (u3a_atom.use_w). A blob file is eligible for deletion only when both refcounts are zero.

2.3 `u3v_bank` — loom-resident blob bank (`vortex.h`) [todo]

typedef struct _u3v_bank {
  u3p(u3h_root) blb_p;   // HAMT: blob_id (u64) -> loom offset of u3a_blob
  u3p(u3h_root) res_p;   // HAMT: res_id  (u64) -> loom offset of u3v_lease
  c3_d          nxt_d;   // monotonic reservation counter
} u3v_bank;

blob_id encoding: ((c3_d)mug_h << 32) | (c3_d)seq_w

u3v_bank is added as a direct field of u3v_home (checkpointed in image.bin). u3v_home currently uses ~2800 bytes of a 4096-word (32 KB) page budget; there is ~29 KB of headroom.

The C-heap min-heap used for TTL expiry ordering is not checkpointed; it is rebuilt from res_p on every boot.

2.4 `u3v_lease` — active reservation (`vortex.h`) [todo]

typedef struct _u3v_lease {
  c3_d  res_d;        // reservation id (key in res_p)
  c3_d  exp_d;        // expiry time (Unix ms)
  c3_h  mug_h;        // blob mug (0 = not yet committed)
  c3_w  seq_w;        // blob seq (0 = not yet committed)
  c3_c  stg_c[4096];  // staging path (mkstemp result), used if mug_h==0
} u3v_lease;

A lease is created when Mars installs a blob. It holds one u3a_blob.use_w reference until the event-log ref-promotion step in _mars_fact claims it.

2.5 Dual refcounting

counter	owner	incremented by	decremented by
`u3a_atom.use_w`	loom allocator	normal noun reference operations	normal noun GC
`u3a_blob.use_w`	blob bank	lease creation; event-log commit	lease expiry; epoch chop GC

A blob file is deleted only when u3a_atom.use_w == 0 and u3a_blob.use_w == 0.

bob_free_f callback is removed from u3o_config. The loom allocator does not delete blob files when a bob atom is freed; that is the responsibility of the GC path at epoch chop.

3. Directory Layout

$pier/.urb/bob/
    stg/                  <- Urth owns; mkstemp(3) files written here
        tmpXXXXXX
    <mug>/                <- committed blobs; ONLY Mars writes (via rename)
        1
        2
        ...
        lock              <- ASCII decimal: next available seq number

stg/ is cleaned (all files deleted) on every boot.
Bucket directories are named by the 31-bit content mug (decimal or hex, TBD).
lock is a per-bucket lockfile whose content is the next available seq_w.

4. IPC Protocol

4.1 Writ type [todo]

// vere.h
typedef enum {
  ...
  u3_writ_blob = 5,   // new: blob install/ack
} u3_writ_type;

4.2 King → Serf: `%blob-install`

[%blob-install path]

field	type	description
`path`	`c3_c*`	filesystem path of staged file (Urth wrote data)

Mars performs: stat → mug → dedup → rename → allocate u3a_blob → create u3v_lease → send %blob-ack.

4.3 Serf → King: `%blob-ack` / `%blob-nack`

[%blob-ack  mug seq ttl]
[%blob-nack reason]
    ;; dedup hit: [%blob-nack %dup mug seq]

field	type	description
`mug`	`c3_h`	31-bit content mug of installed blob
`seq`	`c3_w`	sequence number within bucket
`ttl`	`c3_d`	expiry timestamp (Unix ms)
`reason`	noun	failure reason; `%dup mug seq` on dedup

4.4 King-side API [todo]

// lord.h / lord.c
void u3_lord_blob_install(u3_lord*    god_u,
                          const c3_c* pax_c,
                          void*       ptr_v,
                          void (*fun_f)(void*, c3_h, c3_w, c3_d));

fun_f is invoked with (ptr_v, mug, seq, exp_d) when %blob-ack arrives (or with mug=0, seq=0 on %blob-nack).

4.5 Wire format

Blob install/ack messages are framed using the existing newt protocol. Ram/Tap serialization (see §6) applies to all IPC nouns.

5. Ingestion Paths

All ingestion paths follow the same pattern:

Determine file/data size.
If size ≤ U3_BLOB_THRESH (32 MiB): use existing small-file path unchanged.
Otherwise: a. mkstemp("$pier/.urb/bob/stg/XXXXXX") → stg_path, fd b. posix_fallocate(fd, 0, siz_d) (Linux); F_PREALLOCATE (macOS); skip gracefully if unsupported. c. close(fd) d. Copy source data into stg_path (sendfile or mmap+write in 64 KB chunks). e. u3_lord_blob_install(god_u, stg_path, ctx, _on_blob_ack) f. Hold the pending ovum/request in a queue. — _on_blob_ack fires — g. u3i_blob(mug, seq) → build noun containing the bob atom. h. u3_auto_plan() (or equivalent) → enqueue normally.

5.1 `unix.c` [todo — refactor]

_unix_write_file_hard (already done): streams bob atom bytes from blob store to file in 64 KB chunks.

_unix_write_file_soft (already done): compares mug_h for bob atoms (content-hash shortcut).

Large file ingestion currently calls u3_blob_save_fd directly (done, but needs refactor to the mkstemp + u3_lord_blob_install pattern).

5.2 `mesa.c` [todo]

Large reassembled packets: mkstemp → write reassembled data → u3_lord_blob_install → wait for ack → u3i_blob → continue.

5.3 `http.c` [todo]

Large request bodies: mkstemp → stream body → u3_lord_blob_install → wait for ack → u3i_blob → continue.

5.4 `conn.c` [todo]

Same pattern as unix.c.

6. Ram / Tap Serialization

6.1 Wire format [done]

"RAM\0" (4 bytes) + 0x01 (1 byte) + bit-packed payload

Newt version byte:

0x01 = ram/tap (new)
0x00 = jam/cue (legacy, backward compat)

u3_newt_send_vers() is called on connection establishment to negotiate.

6.2 2-bit tag encoding [done]

tag	meaning	payload
00	atom	mat(len) + raw bits
01	blob-ref	mat(mug) + mat(seq)
10	cell	left subtree + right subtree
11	backref	mat(backreference offset)

6.3 Bob encoding/decoding [done]

Encode (u3s_tap_xeno): emit tag 01, then mat(mug), then mat(seq).
Decode (u3s_ram_xeno): read mug and seq; call u3i_blob(mug, seq).

6.4 Callsites replaced [done]

lord.c: all jam/cue replaced with ram/tap.
mars.c: all jam/cue replaced with ram/tap.
disk.c: u3_disk_etch uses ram; u3_disk_sift tries tap first, falls back to cue for legacy events.

7. Mars-side Blob Installation (`_mars_blob_install`) [todo]

Executed when Mars receives [%blob-install path]:

1.  stat(path)               -> siz_d
2.  mmap(path, PROT_READ)    -> _blob_mug(ptr, siz_d) -> mug_h
3.  Dedup scan in bob/<mug>/:
      for each existing seq file:
        mmap + byte-compare
        hit:  unlink(path), return existing (mug, seq)
        miss: continue
4.  _blob_lock_acquire(mug)  -> seq_w
    rename(path -> bob/<mug>/<seq_w>)
5.  Allocate u3a_blob on loom:
        .use_w = 1  (lease ref)
        .mug_h = mug_h
        .seq_w = seq_w
        .siz_d = siz_d
    Insert blob_id into bank.blb_p
6.  Create u3v_lease:
        .res_d  = bank.nxt_d++
        .exp_d  = now + TTL
        .mug_h  = mug_h
        .seq_w  = seq_w
        .stg_c  = ""  (already committed)
    Insert into bank.res_p + C-heap expiry heap
7.  _mars_gift([%blob-ack mug_h seq_w exp_d])

_blob_mug fix [done]: caps the first hash window at 0xFFFFFFFF bytes (avoids (c3_h)len_d == 0 for exact-4 GiB multiples). Also hashes the last window for tail sensitivity on files > 4 GiB.

8. GC and Lifecycle

8.1 Lease expiry sweeper [todo]

Runs at the start of _mars_work (before processing the next event):

while C-heap top exp_d <= now_ms:
  res_d  = heap.pop()
  lease  = res_p.get(res_d)
  if lease.mug_h == 0:
    unlink(lease.stg_c)     // staged but never committed
  else:
    blob_id = (mug_h << 32) | seq_w
    blob    = blb_p.get(blob_id)
    blob->use_w--
    if blob->use_w == 0:
      _blob_gc(blob)        // delete file, free u3a_blob, remove from blb_p
  res_p.erase(res_d)

8.2 Event-log ref promotion (`_mars_fact`) [todo]

After each successful event commit, walk the rammed noun for bob atoms. For each bob atom found:

blob->use_w++     // event-log ref
  -- find the lease for this (mug, seq) --
blob->use_w--     // release lease ref
remove lease from res_p + C-heap

Net effect: use_w stays at 1, now tracking the event-log reference.

8.3 Epoch chop GC [todo]

At u3_disk_chop():

Walk retired epoch's LMDB deeds; find bob-refs; decrement u3a_blob.use_w for each (event-log refs leaving).
Walk the loom snapshot for live bob atoms; note which are still reachable.
Any u3a_blob with use_w == 0 and u3a_atom.use_w == 0:
- Delete blob file: u3_blob_delete(pax_c, mug_h, seq_w)
- Free u3a_blob from loom
- Remove from bank.blb_p

8.4 Removal of `bob_free_f` [todo]

Remove bob_free_f field from u3o_config (options.h).
Remove _me_lose_north / _me_lose_south callback invocations (allocate.c).
Remove _disk_bob_free_cb and its registration (disk.c).

The loom allocator never directly deletes blob files.

9. Boot / Startup

9.1 `u3_blob_init` [done]

Called from u3_disk_make and u3_disk_load in disk.c. Creates $pier/.urb/bob/ and $pier/.urb/bob/stg/ if absent.

9.2 Fresh image — `_pave_home` [todo]

hed_u->ban_u.blb_p = u3h_new();   // empty HAMT
hed_u->ban_u.res_p = u3h_new();   // empty HAMT
hed_u->ban_u.nxt_d = 1;

9.3 Existing image — `_find_home` [todo]

HAMTs are loaded from snapshot as normal loom data.
Rebuild C-heap expiry min-heap by iterating res_p.
Expire any leases whose exp_d < now.
Delete all files in stg/ (left-over from prior session).

9.4 Epoch version [done]

// version.h
#define U3E_VER3    3    // adds u3v_bank + bob/ store
#define U3E_VERLAT  U3E_VER3

Migration: VER2 → VER3 is handled via epoch rollover in _disk_epoc_load (disk.c). No in-place data migration required; new epoch starts with empty u3v_bank.

10. Implementation Phases

Phase 0 — Foundation [done]

Phase 1 — Serialization [done]

u3s_ram_xeno / u3s_tap_xeno with 2-bit tag encoding (serial.c)
Ram wire format "RAM\0\x01" header (serial.c)
Newt protocol version 0x01/0x00 handling (newt.c)
u3_newt_send_vers() (newt.c)
lord.c: ram/tap serialization callsites
mars.c: ram/tap serialization callsites
disk.c: u3_disk_etch uses ram; u3_disk_sift tries tap first

Phase 2 — Epoch versioning [done]

U3E_VER3 = 3, U3E_VERLAT = U3E_VER3 (version.h)
VER2 → VER3 migration in _disk_epoc_load (disk.c)

Phase 3 — Loom structures [todo]

Add u3a_blob struct to allocate.h
Add u3v_lease struct to vortex.h
Add u3v_bank struct to vortex.h
Add u3v_bank ban_u field to u3v_home in vortex.h
_pave_home: init ban_u HAMTs and nxt_d=1
_find_home: rebuild C-heap from res_p; expire stale leases; clean stg/

Phase 4 — IPC and Mars handlers [todo]

Add u3_writ_blob = 5 to u3_writ_type enum (vere.h)
Implement u3_lord_blob_install() in lord.c
Handle %blob-ack / %blob-nack in king-side lord event loop (lord.c)
Implement _mars_blob_install() in mars.c (stat → mug → dedup → rename → allocate → lease → gift)
Dispatch %blob-install writ in _mars_work (mars.c)

Phase 5 — Lifecycle management [todo]

Lease expiry sweeper at start of _mars_work (mars.c)
Event-log ref promotion in _mars_fact (mars.c)
Epoch chop GC hook in u3_disk_chop (disk.c)
Remove bob_free_f from u3o_config (options.h)
Remove bob_free_f callback invocations from _me_lose_north / _me_lose_south (allocate.c)
Remove _disk_bob_free_cb and registration (disk.c)

Phase 6 — Ingestion path refactors [todo]

unix.c: replace u3_blob_save_fd with mkstemp + u3_lord_blob_install + async ack
mesa.c: blob install for large reassembled packets
http.c: blob install for large request bodies
conn.c: blob install for large payloads
Remove u3_blob_save and u3_blob_save_fd from public blob API (Mars-internal after refactor)

Appendix: Key Constants and Thresholds

constant	value	meaning
`U3_BLOB_THRESH`	`32 * 1024 * 1024`	bytes; atoms larger than this → blob
`u3a_blob_flag`	`0x8000000000000000ULL`	sentinel bit in `u3a_atom.len_w`
`u3a_blob_mask`	`~u3a_blob_flag`	strips flag from len/mug
`U3E_VER3`	`3`	epoch version with bob/ store
newt version `0x01`	—	ram/tap framing
newt version `0x00`	—	legacy jam/cue framing

matthew-levan/blob-store.md

Blob Store for Very Large Atoms (ml/bob)