Skip to content

Instantly share code, notes, and snippets.

@Patrick-Erichsen

Patrick-Erichsen/lcm-behavior-eval-plan.html

Created May 27, 2026 18:34
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Patrick-Erichsen/1218bd734f4535c9ae83552694097862 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Patrick-Erichsen/1218bd734f4535c9ae83552694097862 to your computer and use it in GitHub Desktop.
Download ZIP
#pagedrop Crabpot LCM behavioral eval proof
Raw
lcm-behavior-eval-plan.html
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Crabpot LCM Behavioral Eval Proof</title>
<style>
:root {
color-scheme: light;
--ink: #18212f;
--muted: #5b6573;
--line: #d8dee8;
--page: #f7f9fc;
--panel: #ffffff;
--soft: #eef3f8;
--red: #b42318;
--red-bg: #fff0ed;
--green: #126a4a;
--green-bg: #e9f7ef;
--blue: #225ccc;
--blue-bg: #edf3ff;
--amber: #865200;
--amber-bg: #fff6df;
--shadow: 0 18px 50px rgba(24, 33, 47, 0.08);
}
* {
box-sizing: border-box;
}
body {
margin: 0;
background: var(--page);
color: var(--ink);
font-family:
Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
letter-spacing: 0;
}
.shell {
width: min(1180px, calc(100vw - 28px));
margin: 0 auto;
}
header {
background: #ffffff;
border-bottom: 1px solid var(--line);
}
.hero {
display: grid;
grid-template-columns: minmax(0, 1.35fr) minmax(280px, 0.65fr);
gap: 24px;
padding: 36px 0 26px;
align-items: end;
}
.eyebrow {
margin: 0 0 8px;
color: var(--blue);
font-size: 13px;
font-weight: 800;
text-transform: uppercase;
}
h1 {
margin: 0;
max-width: 820px;
font-size: clamp(30px, 5vw, 54px);
line-height: 1;
letter-spacing: 0;
}
.lede {
margin: 14px 0 0;
max-width: 780px;
color: var(--muted);
font-size: 18px;
line-height: 1.5;
}
.verdict,
.card,
.flow,
.table-wrap,
.codebox {
border: 1px solid var(--line);
border-radius: 8px;
background: var(--panel);
}
.verdict {
padding: 18px;
box-shadow: var(--shadow);
}
.verdict strong {
display: block;
margin-bottom: 8px;
font-size: 18px;
}
.verdict p,
.card p {
margin: 0;
color: var(--muted);
line-height: 1.5;
}
.status-row {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 12px;
padding: 18px 0 24px;
}
.metric {
border: 1px solid var(--line);
border-radius: 8px;
background: var(--panel);
padding: 14px;
}
.metric strong {
display: block;
margin-bottom: 3px;
font-size: 20px;
}
.metric span {
color: var(--muted);
font-size: 13px;
}
main {
padding: 24px 0 56px;
}
section {
margin-bottom: 26px;
}
.section-head {
display: flex;
justify-content: space-between;
gap: 18px;
margin-bottom: 12px;
align-items: flex-end;
}
h2 {
margin: 0;
font-size: 20px;
}
.section-head p {
margin: 0;
max-width: 700px;
color: var(--muted);
line-height: 1.45;
}
.grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 14px;
}
.card {
padding: 16px;
}
.card h3 {
margin: 0 0 8px;
font-size: 16px;
}
.badge {
display: inline-flex;
align-items: center;
min-height: 24px;
border-radius: 999px;
padding: 3px 9px;
font-size: 12px;
font-weight: 800;
}
.badge.red {
color: var(--red);
background: var(--red-bg);
}
.badge.green {
color: var(--green);
background: var(--green-bg);
}
.badge.blue {
color: var(--blue);
background: var(--blue-bg);
}
.badge.amber {
color: var(--amber);
background: var(--amber-bg);
}
.flow {
padding: 16px;
overflow: auto;
}
.flow svg {
display: block;
min-width: 780px;
width: 100%;
height: auto;
}
.table-wrap {
overflow: auto;
}
table {
width: 100%;
border-collapse: collapse;
min-width: 760px;
}
th,
td {
padding: 12px 14px;
border-bottom: 1px solid var(--line);
text-align: left;
vertical-align: top;
}
th {
background: var(--soft);
font-size: 13px;
}
tr:last-child td {
border-bottom: 0;
}
code,
pre {
font-family: "SFMono-Regular", Consolas, "Liberation Mono", monospace;
}
.codebox {
position: relative;
margin-bottom: 12px;
overflow: hidden;
}
.code-title {
display: flex;
justify-content: space-between;
gap: 10px;
padding: 10px 12px;
border-bottom: 1px solid var(--line);
background: var(--soft);
font-weight: 800;
font-size: 13px;
}
pre {
margin: 0;
padding: 14px;
overflow: auto;
color: #e9f1ff;
background: #111827;
font-size: 13px;
line-height: 1.5;
}
button {
border: 1px solid var(--line);
border-radius: 7px;
background: #ffffff;
color: var(--ink);
padding: 5px 9px;
font: inherit;
font-size: 12px;
font-weight: 800;
cursor: pointer;
}
.notes {
display: grid;
grid-template-columns: 1fr 1fr 1fr;
gap: 14px;
}
@media (max-width: 820px) {
.hero,
.status-row,
.grid,
.notes {
grid-template-columns: 1fr;
}
.section-head {
display: block;
}
.section-head p {
margin-top: 8px;
}
}
</style>
</head>
<body>
<header>
<div class="shell">
<div class="hero">
<div>
<p class="eyebrow">Crabpot behavioral eval POC</p>
<h1>LCM now has a red/green release-gate repro.</h1>
<p class="lede">
The POC installs OpenClaw and Lossless Claw in isolated temp state, starts a mock-provider gateway, drives two turns through `chat.send` and `agent.wait`, and asserts the latest assistant reply remembers the seeded fact.
</p>
</div>
<aside class="verdict">
<strong>Current verdict</strong>
<p>
`openclaw@2026.5.22` is red with `memory-recall-mismatch`. `openclaw@2026.5.26` is green with the same `lossless-claw@0.11.2` plugin.
</p>
</aside>
</div>
<div class="status-row">
<div class="metric"><strong>10/10</strong><span>focused harness tests passing</span></div>
<div class="metric"><strong>Red</strong><span>2026.5.22 + LCM 0.11.2</span></div>
<div class="metric"><strong>Green</strong><span>2026.5.26 + LCM 0.11.2</span></div>
<div class="metric"><strong>0 creds</strong><span>mock provider, temp npm config</span></div>
</div>
</div>
</header>
<main class="shell">
<section>
<div class="section-head">
<h2>What Runs</h2>
<p>Same scenario, two release targets. The historical profile is a known-failure guardrail; the forward profile is the release gate.</p>
</div>
<div class="flow">
<svg viewBox="0 0 920 230" role="img" aria-label="Behavior eval flow">
<defs>
<marker id="arrow" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto">
<path d="M0,0 L0,6 L8,3 z" fill="#5b6573"></path>
</marker>
</defs>
<rect x="20" y="44" width="130" height="72" rx="8" fill="#edf3ff" stroke="#225ccc"></rect>
<text x="85" y="74" text-anchor="middle" font-weight="800" fill="#18212f">Isolate</text>
<text x="85" y="96" text-anchor="middle" font-size="13" fill="#5b6573">HOME/state/npm</text>
<line x1="150" y1="80" x2="205" y2="80" stroke="#5b6573" stroke-width="2" marker-end="url(#arrow)"></line>
<rect x="210" y="44" width="150" height="72" rx="8" fill="#ffffff" stroke="#d8dee8"></rect>
<text x="285" y="74" text-anchor="middle" font-weight="800" fill="#18212f">Install</text>
<text x="285" y="96" text-anchor="middle" font-size="13" fill="#5b6573">OpenClaw + LCM</text>
<line x1="360" y1="80" x2="415" y2="80" stroke="#5b6573" stroke-width="2" marker-end="url(#arrow)"></line>
<rect x="420" y="44" width="150" height="72" rx="8" fill="#ffffff" stroke="#d8dee8"></rect>
<text x="495" y="74" text-anchor="middle" font-weight="800" fill="#18212f">Gateway</text>
<text x="495" y="96" text-anchor="middle" font-size="13" fill="#5b6573">mock-openai</text>
<line x1="570" y1="80" x2="625" y2="80" stroke="#5b6573" stroke-width="2" marker-end="url(#arrow)"></line>
<rect x="630" y="44" width="270" height="72" rx="8" fill="#e9f7ef" stroke="#126a4a"></rect>
<text x="765" y="74" text-anchor="middle" font-weight="800" fill="#18212f">Scenario</text>
<text x="765" y="96" text-anchor="middle" font-size="13" fill="#5b6573">chat.send -> agent.wait -> chat.history</text>
<rect x="120" y="154" width="285" height="46" rx="8" fill="#fff0ed" stroke="#b42318"></rect>
<text x="262" y="182" text-anchor="middle" font-weight="800" fill="#b42318">Red target: memory-recall-mismatch</text>
<rect x="515" y="154" width="285" height="46" rx="8" fill="#e9f7ef" stroke="#126a4a"></rect>
<text x="657" y="182" text-anchor="middle" font-weight="800" fill="#126a4a">Green target: latest assistant recalls value</text>
</svg>
</div>
</section>
<section>
<div class="section-head">
<h2>Evidence</h2>
<p>The green target uses the same published LCM plugin as the red target, so this isolates the OpenClaw release behavior.</p>
</div>
<div class="table-wrap">
<table>
<thead>
<tr>
<th>Target</th>
<th>OpenClaw</th>
<th>LCM</th>
<th>Observed behavior</th>
<th>Status</th>
</tr>
</thead>
<tbody>
<tr>
<td>Historical repro</td>
<td>`2026.5.22` (`a374c3a`)</td>
<td>`0.11.2`</td>
<td>Second `chat.history` still ended at the first assistant reply, so latest assistant did not contain `blue-lantern-42`.</td>
<td><span class="badge red">expected-failure</span></td>
</tr>
<tr>
<td>Forward gate</td>
<td>`2026.5.26` (`10ad3aa`)</td>
<td>`0.11.2`</td>
<td>Second turn persisted the new user message and assistant replied exactly `blue-lantern-42`.</td>
<td><span class="badge green">pass</span></td>
</tr>
</tbody>
</table>
</div>
</section>
<section>
<div class="section-head">
<h2>Commands</h2>
<p>Use the historical command as the red TDD target and the exact newer version as the green proof.</p>
</div>
<div class="codebox">
<div class="code-title">
<span>Historical red target</span>
<button data-copy="red">Copy</button>
</div>
<pre id="red">CRABPOT_EXECUTE_BEHAVIOR=1 npm run eval:behavior -- --execute --profile recent-lcm-2026-5-22 --runner local --timeout-ms 240000 --json --keep-temp</pre>
</div>
<div class="codebox">
<div class="code-title">
<span>Forward green target</span>
<button data-copy="green">Copy</button>
</div>
<pre id="green">CRABPOT_EXECUTE_BEHAVIOR=1 npm run eval:behavior -- --execute --profile forward-lcm-release-gate --openclaw-version 2026.5.26 --plugin npm:@martian-engineering/lossless-claw@0.11.2 --expect must-pass --runner local --timeout-ms 240000 --json --keep-temp</pre>
</div>
</section>
<section>
<div class="section-head">
<h2>Implementation Slices</h2>
<p>This is still a POC, but it is shaped so Crabpot can grow into category-based behavioral gates.</p>
</div>
<div class="grid">
<article class="card">
<h3>1. Profiles and scenarios</h3>
<p>`evals/profiles/*.json` picks OpenClaw, plugins, expectation mode, and failure classes. `evals/scenarios/*.json` owns the behavior checks.</p>
</article>
<article class="card">
<h3>2. Isolated execution</h3>
<p>The runner stages temp state, empty npm user config, mock auth profiles, mock model provider, and a loopback gateway.</p>
</article>
<article class="card">
<h3>3. Gateway behavior path</h3>
<p>The scenario drives the gateway directly with `chat.send`, waits with `agent.wait`, then inspects latest assistant text from `chat.history`.</p>
</article>
<article class="card">
<h3>4. Release expansion</h3>
<p>Next slice is multiple plugin categories: memory/context engines first, then skills/tools/channels with similarly tiny scenarios.</p>
</article>
</div>
</section>
<section>
<div class="section-head">
<h2>Risks</h2>
<p>The proof is useful now, but these are the sharp edges before making it a release blocker.</p>
</div>
<div class="notes">
<article class="card">
<span class="badge amber">npm tags</span>
<h3>Local npm policy can lie</h3>
<p>The runner now uses an empty temp npm user config because a local `before` cutoff made `latest` resolve to the old bad release.</p>
</article>
<article class="card">
<span class="badge blue">signal</span>
<h3>The eval checks behavior, not LLM quality</h3>
<p>The mock provider returns deterministic text. The test is whether OpenClaw persists and assembles the second turn correctly.</p>
</article>
<article class="card">
<span class="badge blue">scope</span>
<h3>Only LCM is covered</h3>
<p>The category framework exists, but the current runnable proof is one context-engine scenario. Top-N ClawHub expansion is still a follow-up.</p>
</article>
</div>
</section>
</main>
<script>
for (const button of document.querySelectorAll("button[data-copy]")) {
button.addEventListener("click", async () => {
const id = button.getAttribute("data-copy");
const text = document.getElementById(id).innerText;
await navigator.clipboard.writeText(text);
const old = button.textContent;
button.textContent = "Copied";
setTimeout(() => (button.textContent = old), 900);
});
}
</script>
</body>
</html>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment