Skip to content

Instantly share code, notes, and snippets.

@jmcarp

jmcarp/README.md

Last active July 31, 2026 02:32
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save jmcarp/728c7b4693446337708bb35902bd79f1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save jmcarp/728c7b4693446337708bb35902bd79f1 to your computer and use it in GitHub Desktop.
Download ZIP
oximeter collector perf harness (WIP, not yet checked in): shape x rounds x arms A/B on illumos, RSS/CPU from /proc, ClickHouse row counts, mdb umem + dtrace modes
Raw
README.md

oximeter perf harness

A/B the oximeter collector under synthetic load. Topology per measured run, all on one illumos box:

ch-dev (ClickHouse)  <--inserts--  oximeter standalone collector (the arm)
                                          ^ HTTP pull
                     N x oximeter-load-test producers -----------------------+

An arm is a label plus a prebuilt collector binary. Building is separate from measuring, so a build failure can't abort a matrix and a run measures exactly the bytes you built. The loadgen and ch-dev are shared across arms, never variables.

Layers

file scope
build.sh refs -> oximeter.<arm> + shared tools (TOOLS_REF) + manifest.tsv
orchestrate.sh lifecycle: spawn/teardown (sourced)
measure.sh observation: sample a pid, query ClickHouse, derive metrics (sourced)
run.sh one cell = one arm x one shape x one round, its own process
ab.sh the matrix: shapes x rounds x arms, plus aggregation

Prerequisites

Shell and platform tools only — no runtime to install. clickhouse on PATH (omicron ships one in out/clickhouse/); mdb and dtrace if you want the illumos introspection, which degrades to a no-op without them.

Flow

# 1. build (tens of minutes; run under nohup on the box)
nohup ./build.sh ~/bench-tree ~/bench/bin \
    main=main cache=jmcarp/oximeter-field-cache > ~/bench/build.log 2>&1 &

# 2. run the matrix
export PATH=$HOME/omicron/out/clickhouse:$PATH   # for the clickhouse client
ORCH_CHDEV_BIN=~/bench/bin/ch-dev \
ORCH_LOADGEN_BIN=~/bench/bin/oximeter-load-test \
nohup ./ab.sh ~/bench/bin ~/bench/out main cache > ~/bench/driver.log 2>&1 &

# 2b. restart a matrix that died partway (box reboot, ^C) without re-measuring
#     the cells that already finished
RESUME=1 ORCH_CHDEV_BIN=... ORCH_LOADGEN_BIN=... \
nohup ./ab.sh ~/bench/bin ~/bench/out main cache >> ~/bench/driver.log 2>&1 &

# or a single cell, with loadgen flags passed through verbatim
ORCH_CHDEV_BIN=... ORCH_LOADGEN_BIN=... OFFERED=60000 \
    ./run.sh ~/bench/bin/oximeter.cache /tmp/one 60 2 \
        --series-cardinality 1000 --interval-seconds 1 \
        --field component_kind:String:10 --field component_id:Uuid:1000

Outputs per cell: samples.tsv (epoch, rss_kb, cpu_time), counts.tsv, summary.tsv, meta.tsv, component logs, and mdb/dtrace artifacts. ab.sh writes rows.tsv plus a mean +/- half-range aggregate.

Things that will bite you

  • ab.sh needs the realistic loadgen (--field, --series-cardinality, --batches-per-poll). Older builds use --label-cardinality and will reject the shape table. run.sh forwards whatever you give it, so it works with either.
  • Field pools must multiply to >= --series-cardinality, or the loadgen refuses to start.
  • MEASURE_PROFILE is a mode, not a flag. 0 (default) takes mdb umem/depot snaps; 1 takes a dtrace CPU fold. Never both: mdb -p and dtrace -p both grab the target through /proc, and running them together wedges the collector.
  • ClickHouse readiness IS waited for; producers' is not. The standalone collector .unwrap()s its DB connection (bin/oximeter.rs) and panics if ClickHouse isn't up, so orch_start_clickhouse blocks until it's listening. Producers need no wait — each registers with Nexus from a background task that retries with backoff. Startup ramp still lands inside the measurement window, which is why RSS is a p50 over the window's tail (MEASURE_TAIL_FRAC).
  • run.sh sets set -m. Shells start background children with SIGINT ignored, and ch-dev removes its datadir only on SIGINT. Without job control, teardown falls through to -9 and leaks the datadir into swap-backed tmpfs. orch_start_clickhouse also reaps /tmp/ch-dev-* on start (guarded on no live server), since runs that die before teardown leak regardless — 59 orphaned dirs cost 14 GB of tmpfs once.
  • RSS is a max-statistic: its MEAN converges but its SPREAD does not shrink. It tracks the allocator's high-water arena, set by whichever startup transient aligned worst — and collection start times are jittered, so that alignment is random per run. Measured on identical binaries/shape, per-run RSS varies by +/-270 MB on an unmodified collector while the 3-round mean is stable to within 3% across three separate sessions. So report means, expect wide +/-, and do not add rounds hoping the spread tightens. (Verified not to be /tmp accumulation: spread was the same with 14 GB of leaked tmpfs and with none.) CPU and samples/cpu-sec are means and near noise-free (cores repeated to +/-0.00 across rounds). Trust CPU for small effects; treat RSS deltas under ~20% as unresolved.
  • A dead collector fails the run — the sampling loop notices the pid is gone, so a bad binary path or a mid-run crash surfaces immediately rather than as quiet zeroes.
  • rows_per_s uses the sampled window (N samples span N-1 seconds), so it reads ~1/dur high; negligible at dur=300. cpu_cores is exact.

Laptop->box sync/poll glue is intentionally not here; keep it in scratch/.

Raw
ab.sh
#!/bin/sh
# Matrix driver: shapes x rounds x arms, one run.sh process per cell.
#
# Loop order is the methodology: arms innermost so the arms being compared run
# adjacent in time, with their order alternating per round to cancel within-round
# drift.
#
# Usage: ab.sh <bindir> <outroot> <arm> [<arm>...]
# Arms name binaries built by build.sh ($bindir/oximeter.<arm>).
#
# Env: SHAPES (default all), ROUNDS (3), DUR (300), WORKERS, MEASURE_PROFILE,
# RESUME (set to re-use cells that already have a summary.tsv, so a matrix
# interrupted partway can be restarted without re-measuring what finished)
#
# Requires the realistic loadgen (--field / --series-cardinality /
# --batches-per-poll); the shape table below is written against that CLI.
set -u
BIN="$1"; OUTROOT="$2"; shift 2
ARMS="$*"
ROUNDS="${ROUNDS:-3}"
DUR="${DUR:-300}"
SHAPES="${SHAPES:-nominal 2x 4x highcard mgsfit mgsbig}"
_dir=$(dirname "$0")
ROWS="$OUTROOT/rows.tsv"
mkdir -p "$OUTROOT/runs"
# Real MGS field spec, measured on r3 (plans/oximeter-mgs-load-profile.md): 12
# named fields, ~480 distinct values, a sparse cross-product well above CARD.
MGS_FIELDS="--field rack_id:Uuid:1 --field gateway_id:Uuid:1 --field slot:U32:34 \
--field chassis_revision:U32:3 --field chassis_kind:String:3 \
--field chassis_model:String:4 --field chassis_serial:String:36 \
--field hubris_archive_id:String:5 --field component_kind:String:17 \
--field component_id:String:141 --field description:String:97 \
--field sensor:String:138"
# Per-shape: PROCS, CARD, BATCHES, INTERVAL, LIFETIME. series = PROCS*CARD,
# offered/s = PROCS*CARD*BATCHES/INTERVAL. Field pools must multiply to >= CARD
# or the loadgen rejects the shape.
_shape() {
FIELDS=""
case "$1" in
nominal) PROCS=10; CARD=1000; BATCHES=1; INTERVAL=1; LIFETIME=60 ;;
2x) PROCS=10; CARD=2000; BATCHES=1; INTERVAL=1; LIFETIME=60 ;;
4x) PROCS=20; CARD=2000; BATCHES=1; INTERVAL=1; LIFETIME=60 ;;
highcard) PROCS=20; CARD=4000; BATCHES=1; INTERVAL=4; LIFETIME=15 ;;
# Full 32-sled rack MGS: 2x4900 series, 10x stacking, ~9.8k samples/s.
# Lifetime ~31y = no churn (MGS series are fixed hardware); stay under
# the u64::MAX/2 ceiling that panics.
mgs) PROCS=2; CARD=4900; BATCHES=10; INTERVAL=10
LIFETIME=1000000000; FIELDS="$MGS_FIELDS" ;;
mgs-churn) PROCS=2; CARD=4900; BATCHES=10; INTERVAL=10
LIFETIME=60; FIELDS="$MGS_FIELDS" ;;
# Identical series set and field diversity to mgs, 10x lower sample rate
# (batches 1, not 10). Isolates allocation RATE from cardinality: tests
# whether the RSS win is throughput-gated via umem magazine growth.
mgs-lowrate) PROCS=2; CARD=4900; BATCHES=1; INTERVAL=10
LIFETIME=1000000000; FIELDS="$MGS_FIELDS" ;;
# 2.5x mgs-lowrate's rate (~2450/s) via interval alone — same producers,
# series, fields and per-collection burst. Probes where the RSS knee is
# between the 1k/s floor (-8%) and 9.5k/s (-68%).
mgs-mid) PROCS=2; CARD=4900; BATCHES=1; INTERVAL=4
LIFETIME=1000000000; FIELDS="$MGS_FIELDS" ;;
*) echo "unknown shape $1" >&2; return 1 ;;
esac
[ -n "$FIELDS" ] || FIELDS="--field component_kind:String:10 \
--field component_id:Uuid:$CARD --field address:IpAddr:100"
LOADGEN_ARGS="--series-cardinality $CARD \
--batches-per-poll $BATCHES \
--interval-seconds $INTERVAL \
--series-lifetime-seconds $LIFETIME \
$FIELDS"
OFFERED=$((PROCS * CARD * BATCHES * DUR / INTERVAL))
}
_reverse() { _r=""; for _w in $*; do _r="$_w $_r"; done; echo "$_r"; }
# Between cells the next ch-dev cannot bind the port until the previous
# clickhouse server has fully exited, and it panics rather than retrying.
_settle() {
_w=0
while pgrep -f "clickhouse server" >/dev/null 2>&1 && [ "$_w" -lt 90 ]; do
_w=$((_w + 1)); sleep 1
done
sleep 3
}
printf 'shape\tarm\tround\twall_s\trss_p50_mb\trss_peak_mb\tcpu_cores\trows\tseries\trows_per_s\tsamples_per_cpu_s\tdeliv_pct\n' > "$ROWS"
for _shape in $SHAPES; do
_shape "$_shape" || continue
_r=1
while [ "$_r" -le "$ROUNDS" ]; do
if [ $((_r % 2)) -eq 1 ]; then _order="$ARMS"; else _order=$(_reverse $ARMS); fi
for _arm in $_order; do
_label="$_shape-$_arm-r$_r"
echo "=== $_label ==="
# RESUME=1 re-uses a cell that already has a summary row instead of
# re-measuring it, so a matrix killed partway (box reboot, ^C) costs
# the in-flight cell rather than every completed one. rows.tsv is
# rebuilt in matrix order on every invocation, so it stays
# consistent with runs/ whether cells were measured or re-used.
if [ -n "${RESUME:-}" ] \
&& [ -s "$OUTROOT/runs/$_label/summary.tsv" ]
then
echo "resume: reusing $_label"
printf '%s\t%s\t%s\t%s\n' "$_shape" "$_arm" "$_r" \
"$(tail -1 "$OUTROOT/runs/$_label/summary.tsv")" >> "$ROWS"
continue
fi
# LOADGEN_ARGS unquoted on purpose: split into separate flags.
if OFFERED="$OFFERED" \
"$_dir/run.sh" "$BIN/oximeter.$_arm" "$OUTROOT/runs/$_label" \
"$DUR" "$PROCS" $LOADGEN_ARGS > "$OUTROOT/runs/$_label.row" 2>&1
then
printf '%s\t%s\t%s\t%s\n' "$_shape" "$_arm" "$_r" \
"$(tail -1 "$OUTROOT/runs/$_label/summary.tsv")" >> "$ROWS"
else
echo "$_label FAILED (see $OUTROOT/runs/$_label.row)" >&2
printf '%s\t%s\t%s\tFAILED\n' "$_shape" "$_arm" "$_r" >> "$ROWS"
fi
_settle
done
_r=$((_r + 1))
done
done
# column(1) is absent on some illumos images, so always keep a plain-cat path.
_show() { column -t "$1" 2>/dev/null || cat "$1"; }
echo
echo "=== per-run rows: $ROWS"
_show "$ROWS"
echo
echo "=== aggregate (mean +/- half-range across rounds) ==="
# cols in rows.tsv: 5=rss_p50_mb 7=cpu_cores 11=samples_per_cpu_s 12=deliv_pct
awk -F'\t' -v cols="5 7 11 12" 'NR>1 && $4!="FAILED" {
k = $1 "\t" $2
n[k]++
split(cols, c, " ")
for (i in c) {
v = $(c[i]) + 0
s[k,i] += v
if (n[k] == 1 || v < mn[k,i]) mn[k,i] = v
if (n[k] == 1 || v > mx[k,i]) mx[k,i] = v
}
}
END {
split(cols, c, " ")
printf "shape\tarm\tn\trss_p50_mb\tcpu_cores\tsamples_per_cpu_s\tdeliv_pct\n"
for (k in n) {
printf "%s\t%d", k, n[k]
for (i = 1; i <= length(c); i++)
printf "\t%.2f+/-%.2f", s[k,i]/n[k], (mx[k,i]-mn[k,i])/2
printf "\n"
}
}' "$ROWS" > "$OUTROOT/aggregate.tsv"
_show "$OUTROOT/aggregate.tsv"
Raw
build.sh
#!/bin/sh
# Build arm collector binaries plus the shared tools (loadgen, ch-dev) from git
# refs. Separate from the measurement path on purpose: a build failure should
# never abort a matrix, and a run measures exactly the bytes you built.
#
# Release builds take tens of minutes; run under nohup on the box.
#
# The run path is binary-only by design — do NOT `cargo run` the shared tools.
# Cargo spawns the binary as a child, so $! captures cargo's pid: `kill -INT`
# would never reach ch-dev and its datadir would leak, and producer pids would
# be uncontrollable. It would also pin the tools to whichever arm ref the tree
# happens to be on, making a shared component vary with the A/B variable.
#
# Usage: build.sh <worktree> <bindir> <arm>=<ref> [<arm>=<ref>...]
# e.g. build.sh ~/bench-tree ~/bench/bin main=main cache=jmcarp/oximeter-field-cache
#
# Env: REPO (default ~/omicron), TOOLS_REF (ref for loadgen + ch-dev),
# SHARED=rebuild to force rebuilding the tools
#
# Writes $bindir/oximeter.<arm>, oximeter-load-test, ch-dev, and manifest.tsv
# (arm -> ref -> commit) for result provenance.
set -eu
REPO="${REPO:-$HOME/omicron}"
TOOLS_REF="${TOOLS_REF:-jmcarp/oximeter-load-test-realistic}"
TREE="$1"; BIN="$2"; shift 2
mkdir -p "$BIN"
cd "$REPO"
git fetch origin
[ -d "$TREE" ] || git worktree add --detach "$TREE" HEAD
cd "$TREE"
# Keep the tools' provenance line if we end up skipping their rebuild below.
# Accepts the old `loadgen` label and normalizes it. -E because illumos grep has
# no \| in BREs.
_oldtools=$(grep -E '^(tools|loadgen)' "$BIN/manifest.tsv" 2>/dev/null \
| sed 's/^loadgen/tools/' || true)
: > "$BIN/manifest.tsv"
# Resolve to a sha and check that out: detaches without tripping over a branch
# already checked out in another worktree, and accepts origin-only refs.
_resolve() {
git rev-parse -q --verify "$1^{commit}" \
|| git rev-parse -q --verify "origin/$1^{commit}"
}
# Shared across arms, so never an A/B variable, and they change only when
# TOOLS_REF does. Built FIRST and skipped when already present: the checkout they
# need would otherwise invalidate the arms' crates in the shared target dir, so
# every later arm build would pay for it.
if [ "${SHARED:-}" = rebuild ] || [ ! -x "$BIN/oximeter-load-test" ] \
|| [ ! -x "$BIN/ch-dev" ]; then
_tsha=$(_resolve "$TOOLS_REF")
git checkout -f "$_tsha"
cargo build --release -p oximeter-load-test -p ch-dev
cp target/release/oximeter-load-test target/release/ch-dev "$BIN/"
printf 'tools\t%s\t%s\n' "$TOOLS_REF" "$_tsha" >> "$BIN/manifest.tsv"
echo "built shared tools from $TOOLS_REF ($_tsha)"
else
echo "shared tools present in $BIN, skipping (SHARED=rebuild to force)"
[ -n "$_oldtools" ] && printf '%s\n' "$_oldtools" >> "$BIN/manifest.tsv"
fi
for _spec in "$@"; do
_name=${_spec%%=*}; _ref=${_spec#*=}
_sha=$(_resolve "$_ref")
git checkout -f "$_sha"
cargo build --release -p oximeter-collector --bin oximeter
cp target/release/oximeter "$BIN/oximeter.$_name"
printf '%s\t%s\t%s\n' "$_name" "$_ref" "$_sha" >> "$BIN/manifest.tsv"
echo "built arm $_name from $_ref ($_sha)"
done
echo "BUILD COMPLETE -> $BIN"
cat "$BIN/manifest.tsv"
Raw
measure.sh
#!/bin/sh
# Measurement for an oximeter perf run. Source this file; don't execute it.
# Pure observer: samples a pid, queries ClickHouse, derives metrics. Lifecycle
# lives in orchestrate.sh, the tiers/rounds matrix in the driver.
#
# Deliberately shell + platform tools only, no extra runtime: the harness has to
# run on a bare box with no setup. psutil would give float CPU and byte RSS, but
# 1-second CPU granularity over a 300s window is 0.3% error against effects of
# 40-70%, which does not buy a language dependency.
#
# Config: MEASURE_CH_BIN (clickhouse), MEASURE_TAIL_FRAC (3 = p50 over the last
# third of samples, skipping startup ramp). Reads ORCH_CH_HOST/PORT.
#
# illumos introspection, MUTUALLY EXCLUSIVE modes rather than flags: mdb -p and
# dtrace -p both grab the target via /proc, and running them together wedges the
# collector. MEASURE_PROFILE=0 (default) takes mdb umem/depot snaps;
# MEASURE_PROFILE=1 takes a dtrace CPU fold instead. Both no-op elsewhere.
# MEASURE_STACKS_SH path to oximeter_cpu_stacks.sh (PROFILE=1)
# MEASURE_FLAME_DUR fold duration, default 30
# MEASURE_SNAP_AT seconds to snap at (PROFILE=0)
_measure_log() { echo "measure: $*" >&2; }
# cache_depot_contention is the validated alloc-pressure signal; snap twice so the
# driver can take a delta.
_measure_umem_snap() { # pid outdir tag
command -v mdb >/dev/null 2>&1 || return 0
echo '::umastat' | mdb -p "$1" > "$2/umastat.$3.txt" 2>&1
echo '::walk umem_cache | ::print umem_cache_t cache_name cache_depot_contention cache_magtype->mt_magsize' \
| mdb -p "$1" > "$2/depot.$3.txt" 2>&1
}
_measure_cpu_flame() { # pid outdir
[ -x "${MEASURE_STACKS_SH:-}" ] || return 0
DURATION="${MEASURE_FLAME_DUR:-30}" \
"$MEASURE_STACKS_SH" "$2/cpu.stacks" "$1" > "$2/flame.log" 2>&1
}
# Sample rss + cumulative CPU at ~1 Hz. Dies if the pid exits, which is also how
# a bad binary path or a mid-run crash surfaces.
measure_window() { # pid dur outdir
_pid="$1"; _dur="$2"; _out="$3"
mkdir -p "$_out"
: > "$_out/samples.tsv"
# Early points catch the startup ramp, where RSS does most of its growing;
# the later two bracket steady state for a depot-contention delta.
_snaps="${MEASURE_SNAP_AT:-5 15 30 $((_dur / 3)) $((_dur - 1))}"
_mid=$((_dur / 2))
_t=0
while [ "$_t" -lt "$_dur" ]; do
# Separate -o flags: illumos ps reads `rss=,time=` as one field with the
# header ",time=". Clobbers the positional params, already saved above.
set -- $(ps -o rss= -o time= -p "$_pid" 2>/dev/null)
[ -n "${1:-}" ] || { _measure_log "pid $_pid exited at t=${_t}s"; return 1; }
printf '%s\t%s\t%s\n' "$(date +%s)" "$1" "$2" >> "$_out/samples.tsv"
# An mdb snap stalls this loop for a beat; sample timestamps are wall
# clock, so the derived rates absorb it.
if [ "${MEASURE_PROFILE:-0}" = 0 ]; then
case " $_snaps " in
*" $_t "*) _measure_umem_snap "$_pid" "$_out" "$_t" ;;
esac
elif [ "$_t" -eq "$_mid" ]; then
_measure_cpu_flame "$_pid" "$_out" &
fi
_t=$((_t + 1)); sleep 1
done
_measure_log "sampled ${_dur}s of pid $_pid"
}
# The loadgen emits Cumulative<u64>, so all rows land in measurements_cumulativeu64.
measure_counts() { # outdir
_out="$1"
mkdir -p "$_out"
{
printf 'rows\t%s\n' "$(_measure_q 'SELECT count() FROM oximeter.measurements_cumulativeu64')"
printf 'series\t%s\n' "$(_measure_q 'SELECT uniqExact(timeseries_key) FROM oximeter.measurements_cumulativeu64')"
} > "$_out/counts.tsv"
}
_measure_q() {
"${MEASURE_CH_BIN:-clickhouse}" client \
--host "$ORCH_CH_HOST" --port "$ORCH_CH_PORT" --query "$1"
}
# One TSV row to stdout (also written to $outdir/summary.tsv):
# wall_s rss_p50_mb rss_peak_mb cpu_cores rows series rows_per_s
# samples_per_cpu_s deliv_pct
# OFFERED is the expected sample count (procs*card*batches*dur/interval); pass 0
# to skip deliv_pct.
measure_summarize() { # outdir [offered]
_out="$1"; _offered="${2:-0}"
# p50 rss over the tail, in MB.
_n=$(wc -l < "$_out/samples.tsv")
_tail=$((_n / ${MEASURE_TAIL_FRAC:-3}))
[ "$_tail" -gt 0 ] || _tail="$_n"
_p50=$(cut -f2 "$_out/samples.tsv" | tail -n "$_tail" | sort -n \
| awk '{v[NR]=$1} END {printf "%.0f", v[int((NR+1)/2)]/1024}')
_rows=$(awk -F'\t' '$1=="rows" {print $2}' "$_out/counts.tsv")
_series=$(awk -F'\t' '$1=="series" {print $2}' "$_out/counts.tsv")
awk -F'\t' -v p50="$_p50" -v rows="${_rows:-0}" -v series="${_series:-0}" \
-v offered="$_offered" '
# ps time= is [[DD-]HH:]MM:SS
function secs(s) {
n = split(s, a, ":")
if (n == 3) {
d = 0
if (split(a[1], b, "-") == 2) { d = b[1]; a[1] = b[2] }
return d*86400 + a[1]*3600 + a[2]*60 + a[3]
}
if (n == 2) return a[1]*60 + a[2]
return s + 0
}
NR == 1 { t0 = $1; c0 = secs($3) }
{ peak = ($2 > peak ? $2 : peak); t1 = $1; c1 = secs($3) }
END {
wall = t1 - t0
cpu = c1 - c0
cores = (wall > 0 ? cpu / wall : 0)
printf "%d\t%d\t%.0f\t%.2f\t%d\t%d\t%.0f\t%.0f\t%s\n",
wall, p50, peak/1024, cores, rows, series,
(wall > 0 ? rows / wall : 0),
(cpu > 0 ? rows / cpu : 0),
(offered > 0 ? sprintf("%.1f", 100 * rows / offered) : "-")
}
' "$_out/samples.tsv" | tee "$_out/summary.tsv"
}
Raw
orchestrate.sh
#!/bin/sh
# Lifecycle for an oximeter perf run: ch-dev + the collector under test + N
# oximeter-load-test producers. Source this file; don't execute it. Measurement
# lives in measure.sh, the tiers/rounds matrix in the driver.
#
# ClickHouse readiness IS waited for: the standalone collector `.unwrap()`s its DB
# connection (oximeter/collector/src/bin/oximeter.rs:144) and panics outright if
# ClickHouse isn't up. Producers need no wait — each registers with Nexus from a
# background task that retries with backoff. No input validation: a bad binary
# path fails when spawned.
#
# Startup ramp still lands inside the measurement window, so take steady state
# from its tail.
#
# Config: ORCH_CHDEV_BIN ORCH_LOADGEN_BIN ORCH_COLLECTOR_BIN ORCH_LOG_DIR
# ORCH_CH_PORT (9000) ORCH_NEXUS_ADDR ([::1]:12221)
# Exports: ORCH_CH_PID ORCH_COLLECTOR_PID ORCH_PRODUCER_PIDS ORCH_CH_HOST
# ORCH_CH_PORT
ORCH_CH_PORT="${ORCH_CH_PORT:-9000}"
ORCH_CH_HOST="::1"
ORCH_NEXUS_ADDR="${ORCH_NEXUS_ADDR:-[::1]:12221}"
ORCH_CH_PID=""
ORCH_COLLECTOR_PID=""
ORCH_PRODUCER_PIDS=""
_orch_log() { echo "orch: $*" >&2; }
orch_start_clickhouse() {
mkdir -p "$ORCH_LOG_DIR"
# Reap datadirs orphaned by earlier runs. ch-dev removes its own on SIGINT,
# but the -9 escalation in orch_teardown and any run that dies before
# teardown both skip that, and /tmp is swap-backed here — 59 leaked dirs cost
# 14GB of tmpfs once. Guarded on there being no live server, so this can
# never delete a concurrent run's datadir; runs are sequential anyway.
if ! pgrep -f "clickhouse server" >/dev/null 2>&1; then
rm -rf /tmp/ch-dev-* 2>/dev/null
fi
"$ORCH_CHDEV_BIN" run --port "$ORCH_CH_PORT" > "$ORCH_LOG_DIR/ch-dev.log" 2>&1 &
ORCH_CH_PID=$!
# ch-dev dies with "Reached EOF in ClickHouse log file without discovering
# ports" if the port is still held by a previous run's server.
_t=0
until grep -q "Native server listening" "$ORCH_LOG_DIR/ch-dev.log" 2>/dev/null; do
if ! kill -0 "$ORCH_CH_PID" 2>/dev/null; then
_orch_log "clickhouse died during startup (see $ORCH_LOG_DIR/ch-dev.log)"
ORCH_CH_PID=""; return 1
fi
_t=$((_t + 1))
if [ "$_t" -ge 60 ]; then
_orch_log "clickhouse not listening after ${_t}s"
return 1
fi
sleep 1
done
_orch_log "clickhouse up (pid $ORCH_CH_PID, port $ORCH_CH_PORT)"
}
# Standalone mode inits the ClickHouse schema and serves the mock Nexus the
# producers register with.
orch_start_collector() { # collector-binary [workers]
mkdir -p "$ORCH_LOG_DIR"
# unquoted: expands to nothing when no workers given.
env ${2:+TOKIO_WORKER_THREADS=$2} \
"${1:-$ORCH_COLLECTOR_BIN}" standalone \
--clickhouse "[$ORCH_CH_HOST]:$ORCH_CH_PORT" \
> "$ORCH_LOG_DIR/collector.log" 2>&1 &
ORCH_COLLECTOR_PID=$!
_orch_log "collector launched (pid $ORCH_COLLECTOR_PID)"
}
# Args after N are forwarded to each producer verbatim, as separate shell words.
# --nexus is injected here; don't pass it.
orch_start_producers() { # n loadgen-arg...
mkdir -p "$ORCH_LOG_DIR"
_n="$1"; shift
_i=0
while [ "$_i" -lt "$_n" ]; do
"$ORCH_LOADGEN_BIN" --nexus "$ORCH_NEXUS_ADDR" "$@" \
> "$ORCH_LOG_DIR/loadgen.$_i.log" 2>&1 &
ORCH_PRODUCER_PIDS="$ORCH_PRODUCER_PIDS $!"
_i=$((_i + 1))
done
ORCH_PRODUCER_PIDS="${ORCH_PRODUCER_PIDS# }"
_orch_log "started $_n producer(s): $*"
}
orch_stop_producers() {
[ -n "$ORCH_PRODUCER_PIDS" ] || return 0
kill $ORCH_PRODUCER_PIDS 2>/dev/null # unquoted: word-split the pid list
ORCH_PRODUCER_PIDS=""
_orch_log "stopped producers"
}
# Idempotent.
orch_teardown() {
orch_stop_producers
if [ -n "$ORCH_COLLECTOR_PID" ]; then
kill "$ORCH_COLLECTOR_PID" 2>/dev/null
ORCH_COLLECTOR_PID=""
fi
if [ -n "$ORCH_CH_PID" ]; then
# -INT is the only signal ch-dev cleans its datadir on. Requires the
# caller to `set -m`, else background children ignore SIGINT and we fall
# through to -9. Also frees the port before the next run.
kill -INT "$ORCH_CH_PID" 2>/dev/null
_n=0
while kill -0 "$ORCH_CH_PID" 2>/dev/null && [ "$_n" -lt 20 ]; do
_n=$((_n + 1)); sleep 1
done
kill -9 "$ORCH_CH_PID" 2>/dev/null
ORCH_CH_PID=""
fi
# ch-dev's grandchild, not our direct child.
pkill -f "clickhouse server" 2>/dev/null
_orch_log "teardown complete"
}
Raw
run.sh
#!/bin/sh
# One measured cell: one arm, one shape, one round. Its own process so the EXIT
# trap guarantees teardown even if the run dies. The shapes/rounds/arms matrix
# lives in the driver.
#
# Usage: run.sh <arm-binary> <outdir> <dur> <procs> [loadgen-arg...]
# Loadgen args are forwarded verbatim; don't pass --nexus.
#
# Env: OFFERED expected sample count, for deliv% (procs*card*batches*dur/interval)
# WORKERS TOKIO_WORKER_THREADS for the collector
# DRAIN post-load flush wait, default 15
# plus ORCH_* / MEASURE_* config (see orchestrate.sh, measure.sh)
set -u
# Job control, so `kill -INT` can actually reach the background children: a shell
# starts background jobs with SIGINT ignored, and ch-dev cleans up its datadir
# only on SIGINT (it has no SIGTERM handler). Without this, teardown always falls
# through to -9 and leaks the datadir into swap-backed tmpfs.
set -m
_dir=$(dirname "$0")
. "$_dir/orchestrate.sh"
. "$_dir/measure.sh"
ARM_BIN="$1"; OUT="$2"; DUR="$3"; PROCS="$4"; shift 4
ORCH_LOG_DIR="$OUT"
mkdir -p "$OUT"
# No set -e: the sourced libs kill pids that may already be gone, which would
# trip it. Failures we care about are checked explicitly.
trap orch_teardown EXIT INT TERM
{
printf 'arm_bin\t%s\n' "$ARM_BIN"
printf 'dur\t%s\n' "$DUR"
printf 'procs\t%s\n' "$PROCS"
printf 'offered\t%s\n' "${OFFERED:-0}"
printf 'workers\t%s\n' "${WORKERS:-default}"
printf 'profile\t%s\n' "${MEASURE_PROFILE:-0}"
printf 'loadgen_args\t%s\n' "$*"
printf 'started\t%s\n' "$(date +%FT%T)"
} > "$OUT/meta.tsv"
orch_start_clickhouse || exit 1
orch_start_collector "$ARM_BIN" "${WORKERS:-}"
orch_start_producers "$PROCS" "$@"
measure_window "$ORCH_COLLECTOR_PID" "$DUR" "$OUT" || exit 1
orch_stop_producers
sleep "${DRAIN:-15}" # let in-flight collections and the batcher flush
measure_counts "$OUT"
measure_summarize "$OUT" "${OFFERED:-0}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment