Shared-Memory IPC for Large Nouns

mars-urth-ipc-plan .md

Design: Shared-Memory IPC for Large Nouns

Problem Summary

When a noun of size N crosses the king-serf boundary, the current system creates ~7 full copies in user-space:

1. King jam (u3s_jam_xeno): noun → bytes + hash table
2. Pipe transit: bytes → kernel → bytes
3. Newt decode memcpy: libuv buf → meat buffer
4. Serf cue (u3s_cue_xeno_with): bytes → noun + dictionary
5. Serf disk jam (u3qe_jam in-loom): noun → loom atom (worst offender)
6. Disk copy (u3r_bytes): loom atom → C-heap bytes
7. Serf IPC jam (u3s_jam_xeno): effects noun → bytes + hash table

The peak is at step 5 where the loom simultaneously holds the event noun (~N) AND its jammed representation as a loom atom (~N), plus hash tables. The in-loom jam at disk.c:141 via u3qe_jam(eve) is especially pathological because loom space is the scarcest resource and copy-on-write page faults compound the cost.

With structural sharing (hash-consing), the in-loom noun may be compact, but jam fully expands shared subtrees into a linear byte stream — so a 50MB noun in the loom could easily produce a 200MB+ jam.

Architecture Context

The king (Urth) and serf (Mars) are separate OS processes connected by Unix pipes (stdin/stdout), using the "newt" framing protocol (5-byte header + payload). The loom (noun memory arena) is mmap(MAP_ANON | MAP_PRIVATE) — not shared between processes. Every message must be serialized (jammed) on the send side and deserialized (cued) on the receive side.

Key Files

File	Role
pkg/vere/lord.c	King-side IPC manager, sends writs, receives pleas
pkg/vere/mars.c	Serf-side message handler, runs nock, manages loom
pkg/vere/newt.c	Wire protocol framing (5-byte header + payload)
pkg/vere/disk.c	Event log persistence (LMDB via book.c)
pkg/vere/main.c	Serf entry point, pipe setup (_cw_init_io)
pkg/vere/vere.h	Type definitions for writ, plea, moat, mojo, etc.
pkg/noun/serial.c	u3s_jam_xeno, u3s_cue_xeno_with (off-loom jam/cue)
pkg/ur/bitstream.c	ur_bsw_t bitstream writer (fibonacci-growing buffer)

Current Message Lifecycle (Poke Round Trip)

King → Serf:

lord.c:  _lord_writ_make()     → construct noun [%poke mil job]
lord.c:  u3s_jam_xeno(jar)     → off-loom jam, produces C-heap bytes
lord.c:  u3_newt_send()        → zero-copy handoff to libuv pipe write
newt.c:  _newt_write_cb()      → frees jammed bytes after write completes

Serf receives:

newt.c:  _newt_read_cb()       → libuv read buffer (up to 64KB chunks)
newt.c:  u3_newt_decode()      → memcpy into meat buffer (u3_meat)
mars.c:  u3_mars_kick()        → u3s_cue_xeno_with(bytes) → noun in loom
mars.c:  _mars_work()          → dispatch by tag
mars.c:  _mars_poke()          → nock computation (entirely in loom)
disk.c:  u3_disk_etch()        → u3qe_jam(eve) IN-LOOM → loom atom → C-heap copy
mars.c:  _mars_fact()          → u3s_jam_xeno(effects) → C-heap bytes for response
mars.c:  _mars_flush()         → u3_newt_send() response back to king

King receives response:

newt.c:  → meat buffer via memcpy
lord.c:  u3s_cue_xeno_with()   → effects noun in loom
lord.c:  _lord_plea_work()     → dispatch to pier callbacks

Existing Plea/Writ Types

Writs (king → serf): %poke, %peek, %live, %exit, %quiz Pleas (serf → king): %poke, %peek, %slog, %flog, %sync, %live, %ripe, %quiz

Existing Shared Memory Precedent

• trace.c:1151: shm_open + mmap(MAP_SHARED) for cross-process spin stack
• events.c: file-backed mmap(MAP_SHARED) for loom snapshot persistence
• urth.c:867: u3u_mmap_read() for reading rocks/pills via mmap

---

Design Overview

Two complementary changes:

1. Shared-memory IPC path for messages above 256MB jammed size — avoids pipe transit and extra memcpy/cue on the receiver
2. Eliminate double-jam on the serf — reuse jammed bytes for disk persistence instead of re-jamming in-loom

---

Part 1: New %page Message Type (Shared-Memory Large Messages)

Concept

For messages whose jammed representation exceeds 256MB, instead of writing bytes through the pipe, the sender:

1. Creates a POSIX shared memory object via shm_open
2. mmaps it and writes the jammed bytes into the shared region
3. Sends a small control message over the existing pipe with the shm name and length
4. The receiver shm_opens + mmaps the same object and reads directly

This eliminates:

• The kernel pipe copy (pipe buffers are 64KB, so 256MB = ~4000 read/write syscalls)
• The newt meat buffer memcpy (no _newt_mess_tail allocation + accumulation)
• Peak memory drops from 2× (pipe bytes + meat buffer) to 1× (single shared mapping)

Wire Protocol

Normal path (existing, < 256MB):
  Pipe: [%poke mil job]   -- jammed, sent via newt framing

Large path (new, >= 256MB):
  Pipe:   [%page name=@t len=@ud tag=@tas]   -- small control message
  SHM:    <raw jammed bytes of the inner message>

The pipe message is tiny (shm name + byte length + inner tag). The actual payload lives in shared memory. The inner tag (%poke, %peek, etc.) tells the receiver how to dispatch after cue.

Sender Flow (king sending a large poke)

 1. jar = _lord_writ_make(god_u, wit_u)      // construct [%poke mil job]
 2. len_d = u3s_jam_xeno(jar, &byt_y)        // jam to C-heap bytes

    if len_d < U3_PAGE_THRESHOLD (256MB):
      u3_newt_send(&god_u->inn_u, len_d, byt_y)  // existing path
    else:
 3.   shm_name = _page_shm_name()            // "/urbit-<pid>-<seq>"
 4.   fd = shm_open(shm_name, O_CREAT|O_RDWR, 0600)
 5.   ftruncate(fd, len_d)
 6.   ptr = mmap(NULL, len_d, PROT_WRITE, MAP_SHARED, fd, 0)
 7.   memcpy(ptr, byt_y, len_d)              // write jammed bytes to shm
 8.   c3_free(byt_y)                         // free C-heap copy
 9.   munmap(ptr, len_d)
10.   close(fd)
11.   ctrl = [%page shm_name len_d %poke]    // control message
12.   u3s_jam_xeno(ctrl, &ctl_len, &ctl_byt)
13.   u3_newt_send(&god_u->inn_u, ctl_len, ctl_byt)
14.   u3z(jar)

Receiver Flow (serf receiving a large poke)

1. u3_mars_kick() receives control message via pipe
2. cue control message → [%page shm_name len_d inner_tag]
3. fd = shm_open(shm_name, O_RDONLY, 0)
4. ptr = mmap(NULL, len_d, PROT_READ, MAP_SHARED, fd, 0)
5. jar = u3s_cue_xeno_with(sil_u, len_d, ptr)  // cue from mmap'd region
6. munmap(ptr, len_d)
7. close(fd)
8. shm_unlink(shm_name)                         // clean up shm object
9. dispatch(inner_tag, jar)                      // normal processing

The jammed bytes exist only once in physical memory (the shm object), mapped read-only by the receiver. No pipe transit, no meat buffer copy.

Further Optimization: Jam Directly into Shared Memory

Instead of jam to C-heap then memcpy to shm, modify the bitstream writer to target a pre-mapped shared memory region:

1. Create shm region with generous initial size
2. Init ur_bsw_t with bsw->bytes pointing to mmap'd region
3. u3a_walk_fore writes directly into shm
4. If region fills: ftruncate to larger size + mremap (or munmap+mmap)
5. On completion: ftruncate to actual size, send control message

The comment at bitstream.c:696-698 already anticipated this:

"this pattern should be easily adaptable to an alternate bitstream-writer implementation that flushes accumulated output periodically instead of reallocating the output buffer."

This eliminates the C-heap allocation entirely for large messages — jammed bytes are never on the C heap.

---

Part 2: Eliminate Double-Jam on the Serf (Disk Persistence)

Current Problem (disk.c:125-155)

u3_atom mat = u3qe_jam(eve);         // IN-LOOM jam → loom atom
c3_w  len_w = u3r_met(3, mat);
dat_y = c3_malloc(len_i);            // C-heap allocation
u3r_bytes(0, len_w, dat_y + 4, mat); // copy loom atom → C heap
u3z(mat);                            // free loom atom

Called from _disk_plan() → u3_disk_etch() on every event. For a large event, u3qe_jam creates a massive loom atom, doubling loom pressure.

Solution: Pass Jammed Bytes Through

Since the event was already jammed for IPC, preserve those bytes and pass them to disk persistence directly.

Approach: Replace In-Loom Jam with Off-Loom Jam

Change u3_disk_etch() to use u3s_jam_xeno (off-loom, C-heap output) instead of u3qe_jam (in-loom, loom atom output):

// BEFORE (disk.c:141):
u3_atom mat = u3qe_jam(eve);           // loom atom — BAD
c3_w  len_w = u3r_met(3, mat);
dat_y = c3_malloc(4 + len_w);
u3r_bytes(0, len_w, dat_y + 4, mat);
u3z(mat);

// AFTER:
c3_d  len_d;
c3_y* jam_y;
u3s_jam_xeno(eve, &len_d, &jam_y);     // C-heap bytes — no loom pressure
dat_y = c3_malloc(4 + len_d);
// write mug header...
memcpy(dat_y + 4, jam_y, len_d);
c3_free(jam_y);

This alone eliminates the worst memory spike: no more loom atom for disk persistence.

Further: Reuse IPC Jammed Bytes for Disk

For events received via IPC, the serf already has the raw jammed bytes (either from the meat buffer or from the shm mmap). Instead of re-jamming at all, pass these bytes directly to disk persistence.

Complication: The raw bytes from the king contain [%poke mil job] — the full writ wrapper. But disk persistence only needs job, and the serf adds a timestamp: job = u3nc(now, u3k(job)) at mars.c:602.

Resolution: Accept one off-loom jam for disk (of the timestamped [now job]), but use u3s_jam_xeno instead of u3qe_jam. This eliminates the loom atom while keeping the change simple. The off-loom jam writes to C-heap, which is plentiful compared to loom space.

Add a _disk_plan_raw() variant for when pre-jammed bytes are available:

static void
_disk_plan_raw(u3_disk* log_u,
               c3_h     mug_h,
               size_t   len_i,
               c3_y*    dat_y)     // already-jammed bytes (caller frees)
{
    u3_feat* fet_u = c3_malloc(sizeof(*fet_u));
    fet_u->eve_d = ++log_u->sen_d;
    fet_u->len_i = 4 + len_i;
    fet_u->hun_y = c3_malloc(fet_u->len_i);
    fet_u->hun_y[0] = mug_h & 0xff;
    fet_u->hun_y[1] = (mug_h >> 8) & 0xff;
    fet_u->hun_y[2] = (mug_h >> 16) & 0xff;
    fet_u->hun_y[3] = (mug_h >> 24) & 0xff;
    memcpy(fet_u->hun_y + 4, dat_y, len_i);
    fet_u->nex_u = 0;
    // enqueue as before...
}

And modify _mars_fact() to thread raw bytes through:

static void
_mars_fact(u3_mars* mar_u,
           u3_noun    job,
           c3_d       jam_len,     // NEW: pre-jammed event length (0 if none)
           c3_y*      jam_byt,     // NEW: pre-jammed event bytes (NULL if none)
           u3_noun    pro)
{
    if ( jam_byt ) {
        _disk_plan_raw(mar_u->log_u, mar_u->mug_h, jam_len, jam_byt);
    } else {
        _disk_plan(mar_u->log_u, mar_u->mug_h, job);
    }
    u3z(job);
    // ... jam and enqueue effects as before
}

---

Implementation Plan

Phase 1: Eliminate In-Loom Jam for Disk

Biggest win, simplest change. Estimated effort: small.

Files: disk.c, mars.c

1. Change u3_disk_etch() to use u3s_jam_xeno (off-loom) instead of u3qe_jam (in-loom)
2. Update the byte-copy logic to work with the u3s_jam_xeno output (C-heap c3_y* + c3_d length instead of loom atom)
3. Test with large commits to verify loom pressure reduction

Phase 2: Shared-Memory IPC (%page Message Type)

Core new feature. Estimated effort: medium.

Files: lord.c, mars.c, vere.h, new helper (or inline in lord/mars)

1. Add type definitions to vere.h:
   • #define U3_PAGE_THRESHOLD (256ULL << 20)
   • Shm helper function declarations
2. Implement shm helpers:
   • _page_shm_name(): generate unique name /urbit-<pid>-<seq>
   • _page_shm_create(name, len): shm_open + ftruncate + mmap
   • _page_shm_open(name, len): shm_open + mmap (read-only)
   • _page_shm_close(ptr, len, name): munmap + close + shm_unlink
3. King side (lord.c):
   • Modify _lord_send() / _lord_writ_send(): after jam, check len_d >= U3_PAGE_THRESHOLD; if so, write to shm and send %page control message instead
   • Add %page handling to _lord_on_plea() dispatch for large responses from serf
4. Serf side (mars.c):
   • Add %page handling to _mars_work() dispatch: open shm, cue from mmap, dispatch inner tag
   • Modify _mars_gift() / _mars_fact(): after jam, check threshold; if exceeded, write to shm and enqueue %page control message

Phase 3: Jam Directly into Shared Memory

Optimization to eliminate C-heap intermediate. Estimated effort: medium.

Files: pkg/ur/bitstream.c, pkg/noun/serial.c

1. Add ur_bsw_init_shm(): initialize bitstream writer targeting an mmap'd shm region
2. Add ur_bsw_grow_shm(): grow via ftruncate + remap instead of realloc
3. Add u3s_jam_xeno_shm(): variant that jams directly into shm, returns shm name and length
4. Wire into the %page send path in lord.c and mars.c
5. Handle the edge case where the jam turns out to be < 256MB (unlink shm, fall back to pipe path)

Phase 4: Reuse Jammed Bytes for Disk Persistence

Eliminate all redundant jams. Estimated effort: small-medium.

Files: mars.c, disk.c

1. Add _disk_plan_raw() to accept pre-jammed bytes
2. In u3_mars_kick(), preserve the raw bytes (meat buffer or shm pointer) across the cue + compute + persist cycle
3. In _mars_fact(), pass preserved bytes to _disk_plan_raw() when available
4. Handle lifetime management: ensure bytes outlive the disk write (for shm: keep mmap alive until disk commit completes; for meat: copy or refcount the buffer)

---

Memory Improvement Summary

For a commit of noun size N (jammed bytes):

Stage	Current Peak	After Phase 1	After All Phases
King send	2N (noun + jam)	2N	N (jam to shm)
Pipe transit	N (kernel bufs)	N	0 (shared memory)
Serf receive	2N (meat + cue)	2N	N (mmap + cue)
Serf disk jam	2N loom	N loom (off-loom)	0 (reuse bytes)
Overall peak	~4-5N	~2-3N	~1-2N

Phase 1 alone cuts the worst case roughly in half by eliminating the in-loom jam. The full implementation brings peak memory close to the theoretical minimum of ~N (the noun must exist in the loom at least once for computation).

---

Open Questions

1. Threshold check: 256MB is for the jammed byte length. We only know the size after jamming. Should we add a pre-jam size estimate (e.g., based on noun cell count), or is it acceptable to jam first and then decide the transport path?

2. Timestamp issue: The serf adds a timestamp to the event (mars.c:602) after receiving it, meaning the king's jammed bytes can't be reused verbatim for disk. Phase 4 handles this by accepting one off-loom jam for the timestamped event. An alternative is to move timestamping to the king side — is that desirable?

3. Cleanup on crash: If the king or serf crashes mid-transfer, shm objects may leak. We should add cleanup logic (e.g., scan for /urbit-<pid>-* on startup and unlink stale objects). Is this sufficient, or do we need more robust cleanup?