Two ways to compute "CPU wall time" for an Iris job subtree, validated against the marin production cluster.
Background: marin-community/marin#5637
adds iris job cpu-time, which on the controller answers via:
SUM(t.finished_at_ms - t.started_at_ms) over all leaf-job tasks
| dataset | status | n_cache_roots | n_arrays | n_sampled_docs | bos_id_guess | bos_frac | n_bugged | n_suspect | n_safe | n_gone | n_error | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| tokenized/validate | BUGGED | 3636 | 18550 | 349047 | 791 | 0.054 | 3489 | 0 | 147 | 0 | 0 | |
| tokenized/common_corpus_english-f7f46b | BUGGED | 4653 | 9653 | 381743 | 791 | 0.073 | 3809 | 0 | 843 | 0 | 1 | |
| tokenized/starcoder2_extras | BUGGED | 734 | 1069 | 5387 | 128000 | 1.000 | 13 | 3 | 718 | 0 | 0 | |
| tokenized/paloma | BUGGED | 64 | 790 | 5489 | 128000 | 1.000 | 24 | 0 | 40 | 0 | 0 | |
| tokenized/uncheatable_eval | BUGGED | 21 | 182 | 2100 | 128000 | 1.000 | 11 | 2 | 8 | 0 | 0 | |
| tokenized/nsf_awards-1a6caf | BUGGED | 1 | 26 | 100 | 15623 | 0.070 | 1 | 0 | 0 | 0 | 0 | |
| tokenized/data_efficiency | BUGGED | 4 | 20 | 400 | 791 | 0.070 | 4 | 0 | 0 | 0 | 0 | |
| tokenized/nemotron_cc_math_v1 | SUSPECT | 2 | 464 | 200 | 2 | 0.740 | 0 | 1 | 1 | 0 | 0 | |
| tokenized/tinystories-f8e445 | SUSPECT | 2 | 7 | 200 | 12805 | 0.855 | 0 | 2 | 0 | 0 | 0 |
Cluster: marin (config lib/iris/examples/marin.yaml)
Slice: marin-tpu-v4-reserved-2048-us-central2-b-20260414-2123-fd07e934
Symptom: Provider sync: 660 workers, 56 failed, 13015ms failed=[... 33× v4-2048 workers ...] at 22:51 UTC on 2026-04-14.
From controller logs filtered by fd07e934:
| #!/usr/bin/env python3 | |
| """Inspect Python wheel files: version info, file summary, dependencies, etc.""" | |
| import argparse | |
| import os | |
| import sys | |
| import zipfile | |
| from collections import Counter | |
| from dataclasses import dataclass, field | |
| from pathlib import Path |
| #!/usr/bin/env -S uv run --script | |
| # /// script | |
| # requires-python = ">=3.11" | |
| # dependencies = [ | |
| # "google-genai>=1.0", | |
| # "pydantic>=2.0", | |
| # ] | |
| # /// | |
| """ | |
| Agent-driven log scanner with two composable modes: grep and summarize. |
Iris runs user code in Docker containers on GCE worker VMs. When per-user credential isolation is enabled, each user's tasks should run as their designated GCP service account rather than the worker VM's native SA. The worker VM's SA has roles/iam.serviceAccountTokenCreator on each target user SA.
The challenge: how do we make all GCP client libraries inside the container — Python (google-cloud-storage, gcsfs, BigQuery client, etc.), gcloud CLI, and potentially Go/Java — use impersonated credentials transparently?
| #!/usr/bin/env -S uv run --script | |
| # /// script | |
| # requires-python = ">=3.11" | |
| # dependencies = [ | |
| # "pyarrow", | |
| # ] | |
| # /// | |
| """Benchmark: create N PyArrow tables with varied column types and concat them.""" |
| #!/usr/bin/env -S uv run --script | |
| # /// script | |
| # dependencies = [ | |
| # "pyarrow", | |
| # "fsspec", | |
| # "click", | |
| # "s3fs", | |
| # "gcsfs", | |
| # ] | |
| # /// |
| #!/usr/bin/env -S uv run --quiet --script | |
| # /// script | |
| # requires-python = ">=3.11" | |
| # dependencies = [] | |
| # /// | |
| """ | |
| gbranch - Manage branches with naming convention {USER}/{datestamp}-{name} | |
| Usage: |