mars/urth ipc analysis

mars-urth-ipc-analysis.md

Mars/Urth IPC Analysis

Architecture

The king (Urth) and serf (Mars) are separate OS processes connected by Unix pipes (stdin/stdout), using the "newt" framing protocol (5-byte header + payload). The loom (noun memory arena) is not shared between processes — every message must be serialized (jammed) on the send side and deserialized (cued) on the receive side.

The Full Poke Lifecycle (e.g. |commit)

For a |commit of size N (the event noun) producing effects of size M:

King -> Serf (sending the event):

Step	Operation	Memory	Location
1	_unix_update_file() reads file into dat_y via read()	N bytes	C heap
2	u3i_bytes(len_ws, dat_y) creates noun in loom	N bytes	loom
3	c3_free(dat_y)	freed
4	Event noun job constructed and passed to u3_lord_work()	~N	loom
5	_lord_writ_make() wraps noun: [%poke mil job]	~N	loom
6	u3s_jam_xeno(jar) — traverses noun, produces byte buffer	~N	C heap
7	u3z(jar) — frees noun in loom	freed
8	u3_newt_send() — zero-copy handoff to libuv pipe write	~N	pipe buffer
9	_newt_write_cb() — frees jammed bytes after write	freed

King send peak: ~2N (noun in loom + jammed bytes on C heap, steps 5-6)

Serf receives and processes:

Step	Operation	Memory	Location
10	libuv read callback + newt decode: pipe -> meat buffer (memcpy)	~N	C heap
11	u3s_cue_xeno_with() — parses bytes, creates noun in loom	~N	loom
12	meat buffer freed	freed
13	_mars_poke() — nock computation in loom	~N+M	loom
14	u3_disk_etch() -> u3qe_jam(eve) — IN-LOOM jam of event for disk persistence	~N	loom (atom!)
15	u3r_bytes() + c3_malloc() — copies jammed atom to C heap	~N	C heap
16	u3z(mat) — frees in-loom jam atom	freed
17	u3s_jam_xeno(pro) — off-loom jam of effects for IPC response	~M	C heap
18	u3z(pro) — frees effects noun	freed
19	u3_newt_send() — sends jammed effects back to king	~M	pipe

Serf peak: at step 14, the loom contains BOTH the event noun (~N) AND its in-loom jam (~N as a loom atom), plus effects (M). That's 2N+M in loom alone.

But it gets worse. The u3qe_jam at step 14 is the standard in-loom jam which allocates a loom atom. For a large noun, this means:

• Original noun in loom: ~N
• Hash table for deduplication during jam: overhead proportional to noun cell count
• Growing slab for the output atom: up to ~N
• The jam bitstream is a loom atom, so it uses loom pages with copy-on-write overhead

Plus the u3s_jam_xeno also builds its own hash table and fibonacci-growing buffer.

King receives effects:

Step	Operation	Memory
20	newt decode: pipe -> meat buffer (memcpy)	~M
21	u3s_cue_xeno_with() — bytes -> noun in loom	~M
22	meat buffer freed	freed

Key Bottlenecks Identified

1. Double jam on the serf side (disk.c:141): The event job is jammed in-loom via u3qe_jam(eve) for disk persistence. This is separate from the off-loom u3s_jam_xeno used for IPC. The in-loom jam creates a loom atom which is particularly expensive — it consumes loom space (the most precious resource) and involves copy-on-write page faults.
2. No sharing of jammed bytes: The king already jammed the event to send it to the serf. The serf then cues it, runs nock, and jams the event again for disk persistence. The original jammed bytes from the king are discarded after cue.
3. No streaming: The bitstream writer (ur_bsw_t) allocates a single growing buffer. The comment at bitstream.c:696-698 explicitly notes it was designed to be adaptable to a streaming/flushing approach, but this was never implemented.
4. Pipe IPC requires full serialization: Every noun must be fully jammed into a contiguous byte buffer, written through the kernel pipe, then fully cued back. For a 100MB noun, this means ~100MB jammed + ~100MB cued, plus hash tables on both sides.
5. Peak loom pressure during disk etch: The u3_disk_etch function at disk.c:141 does u3_atom mat = u3qe_jam(eve) — this allocates the jammed representation inside the loom, doubling loom usage for the event data.

Total Memory Copies for a Large Commit

For a poke of size N, counting distinct full-size allocations:

1. File bytes read from disk (C heap) -> noun in loom (king side)
2. Noun jammed to bytes (C heap, via u3s_jam_xeno)
3. Pipe transit (kernel buffers)
4. Newt meat buffer (C heap, serf side, memcpy from pipe)
5. Noun cued into loom (serf side, via u3s_cue_xeno_with)
6. Event jammed as loom atom (serf side, via u3qe_jam — IN LOOM)
7. Loom atom copied to C heap (serf side, u3r_bytes + c3_malloc)
8. Effects jammed to bytes (C heap, via u3s_jam_xeno)
9. Pipe transit back
10. Meat buffer + cue on king side

That's at least 7 full copies of the event data in user-space alone, plus 2 kernel pipe transits, plus hash table overhead at each jam/cue step. The peak is worst on the serf side at step 6 where the loom holds both the noun and its jam simultaneously.