When Karpathy released autoresearch [1], he showed you could hand an LLM a training script, let it propose changes, run the experiment, observe the result, and repeat. His target was pre-training. Same GPU, same environment, iterations in minutes. I wanted to test a harder claim: can the same loop work for RL fine-tuning, where each iteration needs its own GPU, rewards are sparse, and a single bad hyperparameter wastes an hour of A100 time?
The answer is yes, but the hard part isn't where I expected.
I built autoresearch-rl and pointed it at a GRPO fine-tuning task on Basilica A100s. One command: uv run autoresearch-rl experiment.yaml. It ran 15 iterations autonomously with 100% success rate: proposing hyperparameters, spawning ephemeral GPU containers, training, evaluating, keeping or discarding, and looping. No human in the loop. GSM8K pass@1 improved from 26% (baseline) to 36% across 15 iterations, with the LLM policy converging on the winning configuration by iteration 1. A second run on a supervised fine-tuning task hit 98.2% F1 in 6 iterations.
The real finding: the hard part of autonomous RL research isn't the search algorithm. It's the infrastructure. Spinning up isolated GPU environments per iteration, on demand, without babysitting.
In early 2025, Karpathy open-sourced autoresearch [1], a proof-of-concept where an LLM acts as an autonomous ML researcher. The loop is deceptively simple:
- The LLM reads the current training script and past results
- It proposes a modification (hyperparameter change or code edit)
- The experiment runs
- Results are logged
- Repeat
This maps naturally to a Markov Decision Process [10]:
- State: current code + experiment history + diagnostic logs
- Action: structured code diff or parameter configuration
- Reward: improvement (or not) on a scalar validation metric
- Transition: stochastic. Same parameters can yield different results due to initialization, data sampling, hardware variance
Neural Architecture Search [2] cast architecture design as an RL problem in 2017. Bayesian optimization [3] has been the workhorse of hyperparameter tuning since 2012. What Karpathy added is using modern LLMs as the policy directly. No surrogate model, no acquisition function, just natural language reasoning about what to try next based on the full experiment history. The LLM brings domain knowledge that a Gaussian process prior cannot encode: it knows learning rate warmup helps with transformers, that batch size and learning rate are coupled, that gradient clipping prevents spikes.
I wanted to build on this: a production-grade framework that generalizes the autoresearch loop for RL fine-tuning, with pluggable execution targets, multiple policy strategies, crash recovery, and the infrastructure to make it work without babysitting a GPU server.
Pre-training autoresearch runs on one GPU, same environment every time, iterations in minutes. RL post-training is operationally different: each iteration needs 5-10 minutes of A100 time, a clean CUDA environment (stale optimizer state from a previous run corrupts the next), and the reward signal is sparse enough that most iterations produce no measurable improvement. The cost of a bad hyperparameter at the 70B scale is hours of GPU time, not minutes.
This means the autoresearch loop for RL can't just be a script on a workstation. It needs to tolerate failures, cap compute costs, and provision GPUs on demand.
Here's what most autoresearch discussions skip: where does the GPU come from?
Karpathy's original assumes you're sitting at a machine with a GPU. Same hardware, same environment, iterations in minutes. RL fine-tuning breaks this assumption in five ways:
Isolation. Each iteration needs a clean environment. A botched GRPO run that corrupts model weights or leaves stale optimizer state shouldn't poison the next iteration. Shared-GPU setups require manual cleanup between runs, which is exactly the toil an autonomous loop should eliminate.
Resource contention. If your controller (the LLM reasoning about what to try next) and your training job compete for the same GPU, you're either running them sequentially (slow) or risking OOM kills. The controller needs fast inference on a small model; training needs 80GB of VRAM for gradient computation.
Heterogeneous compute. The controller can run on CPU. Training needs an A100 or H100. Running both on an A100 wastes money during the reasoning phase.
Cost efficiency. A100s run ~$3/hr. If your LLM policy spends 5 minutes reasoning about the next experiment, that's $0.25 of idle GPU time. Over 22 iterations, it adds up. The controller should provision GPU resources on demand, not hold them idle.
Reproducibility. If iteration 5 runs on an A100-40GB and iteration 12 runs on an A100-80GB, your memory-bound operations (batch size, gradient accumulation) aren't directly comparable. The hardware class needs to be deterministic across iterations.
The architecture I landed on separates the control plane from the compute plane:
flowchart LR
subgraph CPU["Controller (CPU)"]
Policy["LLM Policy"]
Loop["Experiment Loop"]
Tel["Telemetry"]
Ckpt["Checkpoint"]
end
subgraph GPU["Basilica (Ephemeral GPU)"]
Container["Fresh Container\nA100-80GB"]
Pip["pip install"]
Prepare["prepare.py"]
Train["train.py"]
end
CPU -- "spawn\nparams / diff" --> GPU
GPU -- "metrics" --> CPU
GPU -. "auto-deleted via TTL" .-> GPU
Each iteration spins up a fresh container with a known GPU class, runs the training job, emits metrics to stdout, and gets torn down. The controller parses metrics from container logs, makes the keep/discard decision, and proposes the next iteration. No shared state between iterations except what the controller explicitly carries forward.
Basilica provides the programmable GPU deployments that make this practical. You define a container spec (image, GPU requirements, environment variables, TTL) and it handles scheduling, provisioning, health checks, and cleanup. The properties that matter for autonomous RL:
- Per-iteration isolation: Fresh CUDA runtime, filesystem, and memory for every experiment. No state leaks.
- On-demand provisioning: GPU allocated only during training, released immediately after. The controller pays for CPU time (~$0.01/hr) while reasoning.
- Deterministic hardware class:
gpu_models: ["A100"]guarantees an A100 every iteration. The telemetry system records a hardware fingerprint (SHA-256 of OS + kernel + GPU name) to verify comparability across runs. - Auto-cleanup via TTL: If an experiment hangs (deadlocked NCCL, infinite loop in reward computation), the container is killed after
ttl_seconds. No orphaned GPU processes. - Immutable base image with runtime code overlay: The LLM-proposed training code is passed to each container as a base64-encoded environment variable (
AR_MODIFIED_SOURCE), leaving the base image (pytorch/pytorch:2.4.1-cuda12.4-cudnn9-devel) unchanged across all iterations. This eliminates the Docker build-and-push cycle from the iteration loop, reducing per-iteration overhead from minutes to seconds.
The framework decomposes the autonomous research loop into two layers: the pipeline inside each container, and the control loop that drives iteration.
The pipeline is deliberately simple. Two scripts, no shared imports, communicating only through the filesystem:
flowchart LR
subgraph Container["Ephemeral GPU Container"]
Prepare["prepare.py\n(frozen, runs once)"] -- "writes /app/data/*.jsonl" --> FS[("filesystem")]
FS -- "reads data" --> Train["train.py\n(mutable, per iteration)"]
Train -- "stdout" --> Metrics["eval_score=0.36\nloss=0.015"]
end
prepare.py runs once per container via prepare_cmd and produces data files. train.py reads that data, trains, and prints metrics to stdout. No Python import between them. This boundary is what makes the frozen/mutable contract enforceable: the LLM can modify train.py but can never touch the data pipeline.
The control loop wraps this pipeline in an autonomous search:
flowchart LR
LLM["LLM Policy\npropose params or diff"] --> Train["train.py\n(on ephemeral GPU)"]
Train --> Metrics["metrics"]
Metrics --> Decision{"keep /\ndiscard"}
Decision -- "repeat" --> LLM
The core loop runs a tight proposal-execute-evaluate cycle:
for iteration in range(max_iterations):
# 1. Propose: policy reasons about history and suggests next config
state = build_proposal_state(history, program, source)
proposal = policy.propose(state)
# 2. Execute: spawn ephemeral GPU container, run training
outcome = executor.execute(proposal, run_dir)
# 3. Evaluate: extract scalar metric, compare to best
value = extract_objective_metric(outcome.metrics)
score = value if direction == "min" else -value # lower always better internally
# 4. Keep or discard
if score < best_score:
save_version(versions_dir, iteration, outcome)
best_score, best_value = score, value
decision = "keep"
else:
decision = "discard"
# 5. Feedback to learnable policies
if isinstance(policy, Learnable):
reward = 1.0 if decision == "keep" else (-0.1 if failed else 0.0)
policy.record_reward(reward)
# 6. Stop guards
...Four stop guards prevent runaway compute (any one triggers termination):
| Guard | Condition | Default | Rationale |
|---|---|---|---|
| No-improvement streak | N consecutive iterations without beating best | 15 | Diminishing returns detected |
| Failure rate | Rolling failure rate exceeds threshold | 85% over last 8 | Configuration fundamentally broken |
| Wall-clock budget | Total elapsed time exceeds limit | 8 hours | Hard cost cap |
| Power-law forecast | Projected improvement is negligible | Auto | Based on scaling law observation that loss curves follow power laws [7] |
The power-law early stopping fits y = a * x^b + c to the score history and projects whether continued iterations are likely to produce meaningful improvement. This is motivated by the empirical observation in Kaplan et al. [7] that neural network loss curves exhibit smooth power-law behavior across many orders of magnitude. The same pattern appears at the meta-level of hyperparameter search progress.
The framework supports six policy types:
| Policy | Strategy | Use Case |
|---|---|---|
grid |
Exhaustive cartesian product | Small spaces (<100 configs) |
random |
Uniform sampling with fixed seed | Baseline comparison, large spaces |
static |
Fixed params, no search | Ablation studies |
llm |
LLM proposes hyperparameters | Medium spaces, benefits from domain reasoning |
llm_diff |
LLM proposes unified code diffs | Algorithmic changes beyond hyperparameters |
hybrid |
Params first, code diffs when stalled | Best of both worlds |
The LLM param policy maintains a multi-turn conversation with the LLM. Each iteration, it sends the full search space, the last 50 experiment results (with metrics, status, and rationale), any recent error logs, and the task description. The LLM responds with a JSON object of proposed parameters. This is not a one-shot suggestion. The LLM builds cumulative reasoning across iterations, recognizing patterns like "every time I increase batch_size past 2, the run times out" or "learning rate 3e-6 has produced the best results in 3 out of 4 attempts."
The actual prompt structure:
System: "You are a hyperparameter optimization assistant. Given a search
space and experiment history, propose the next set of hyperparameters
to try. Respond with ONLY a JSON object mapping parameter names to
values from the allowed choices."
User: [Task description from program.md]
[Objective: maximize eval_score]
[Current training script source]
[Search space: learning_rate: [3e-6, 5e-6, 1e-5], ...]
[Experiment history (last 50 iterations with params, metrics, status)]
[Recent errors (last 5 failed iterations, stderr tails)]
[Recent logs (last 3 successful iterations, stdout tails)]
"Respond with ONLY a JSON object."
For campaigns exceeding 50 iterations, the context window is managed by summarizing older entries into aggregate statistics (best metric value, failure rate, most common errors) while preserving full detail for the most recent 50. The LLM keeps the last 10 conversation turns (20 messages), so its reasoning accumulates across iterations without exceeding context limits.
An example of what the LLM sees and proposes at iteration 8, after the first "keep" at iteration 0:
History shows: lr=5e-6/batch=2/steps=30 gave eval_score=0.03 (keep),
lr=1e-5/batch=1/steps=30 gave eval_score=0.00 (discard),
lr=3e-6/batch=1/steps=50/gen=3 gave eval_score=0.01 (discard), ...
LLM response: {"learning_rate": 0.000003, "batch_size": 1,
"max_steps": 50, "num_generations": 2, "temperature": 1.0}
The LLM converged on 3e-6 after observing that 1e-5 consistently failed and 5e-6 plateaued. It reduced generations from 3 to 2 after noting that G=3 runs produced lower scores than G=2 runs, which matches the pattern in the results section. This is domain reasoning that grid or random search cannot perform.
The hybrid policy implements a three-phase strategy that addresses diminishing returns:
- Exploration (first N iterations): Search hyperparameters using LLM-guided proposals
- Code modification (when param search stalls): After M consecutive non-improvements, switch to LLM-generated code diffs against the mutable training script
- Rollback (if diffs fail): After K consecutive diff failures, revert to parameter search
Hyperparameter tuning finds the right learning rate and batch size in 8-10 iterations. Beyond that, the remaining gains require algorithmic changes: a different reward shaping function, a modified KL penalty coefficient schedule, a new sampling strategy. The hybrid policy automates this transition.
A critical safety property: prepare.py (data pipeline) is frozen. The LLM can never modify it. Only train.py is mutable. Every proposed diff is validated against a contract before execution:
def validate_diff_against_contract(diff: str, contract: ContractConfig) -> tuple[bool, str]:
touched = extract_touched_files_from_diff(diff)
for path in touched:
if path == contract.frozen_file:
return False, "frozen_file_mutation_blocked"
if path != contract.mutable_file:
return False, "out_of_scope_mutation_blocked"
return True, ""Without this, a creative LLM policy could "improve" eval_score by modifying the evaluation function to always return 1.0. Freezing the data pipeline and evaluation logic is a structural guarantee against this, stronger than any algorithmic safeguard.
Additionally, proposed code diffs undergo AST validation to ensure they produce syntactically valid Python. Invalid diffs trigger a correction retry: the validation error is sent back to the LLM for repair, up to 2 attempts before falling back.
Each training iteration follows this lifecycle on Basilica:
sequenceDiagram
participant C as Controller
participant B as Basilica API
participant G as GPU Container
C->>B: Create deployment (image, GPU spec, env vars, TTL)
B->>G: Schedule on A100
G->>G: pip install
G->>G: prepare.py
G->>G: train.py
C->>B: Poll logs (every 20s)
G-->>B: stdout: eval_score=0.04
B-->>C: Return metrics
C->>B: Delete deployment
B->>G: Terminate
Parameters are injected via environment variables: AR_PARAMS_JSON carries the full configuration as JSON, and individual AR_PARAM_<NAME> variables provide each value. For code modifications in diff mode, the modified source is base64-encoded into AR_MODIFIED_SOURCE, avoiding Docker image rebuilds entirely. The same base image serves every iteration. Only the parameters and code change.
The configuration for the GRPO experiment:
target:
type: basilica
prepare_cmd: [python3, /app/prepare.py]
train_cmd: [python3, /app/train.py]
timeout_s: 3600
basilica:
image: pytorch/pytorch:2.4.1-cuda12.4-cudnn9-devel
gpu_count: 1
gpu_models: ["A100"] # single GPU class for comparability
memory: "48Gi"
cpu: "8"
storage: /data
ttl_seconds: 3900
min_gpu_memory_gb: 24
setup_cmd: >-
pip install --no-cache-dir
transformers==4.47.1 datasets==3.2.0
accelerate==0.34.2 scipy- Model: Qwen2.5-0.5B-Instruct (494M parameters) [8]
- Algorithm: GRPO [5, 6]
- Dataset: GSM8K (7,473 train / 1,319 test samples) [9]
- Hardware: NVIDIA A100-SXM4-80GB via Basilica
- Budget: 8 hours wall time
- Policy: LLM-guided param search (DeepSeek-V3-0324)
- Evaluation: pass@1 on 100 GSM8K test problems, greedy decoding
The search space covers 180 possible configurations across learning rate, batch size, training steps, generation count, and temperature. The training script uses a pure PyTorch GRPO implementation because TRL's trainer deadlocked in containerized environments during early testing. The custom one emits metrics to stdout for the controller to parse.
15 iterations, 100% success rate, 4.6 GPU-hours of training across 8 hours elapsed. The LLM policy found the winning configuration by iteration 1:
| Iter | lr | steps | gen | temp | eval_score | time (s) | decision |
|---|---|---|---|---|---|---|---|
| 0 | 5e-6 | 50 | 3 | 1.0 | 0.34 | 761 | keep |
| 1 | 5e-6 | 80 | 3 | 0.8 | 0.36 | 1741 | keep (best) |
| 2 | 1e-5 | 80 | 3 | 1.0 | 0.03 | 1904 | discard |
| 3 | 3e-6 | 80 | 3 | 0.8 | 0.30 | 1031 | discard |
| 4 | 5e-6 | 50 | 3 | 1.0 | 0.36 | 800 | discard (tied) |
| ... | 0.23-0.35 | discard |
The baseline (Qwen2.5-0.5B-Instruct zero-shot) scored 26% pass@1 on GSM8K. The framework improved this to 36% pass@1, a +10 percentage point improvement, with the best configuration found in just 2 iterations. The remaining 13 iterations explored the space but never exceeded the best.
The step function in the chart tells the operational story: two "keep" decisions (green dots at iterations 0 and 1), then a plateau where no configuration beat the best. The stop guards would have terminated the campaign after 15 consecutive non-improvements, but the wall-clock budget expired first.
Key observations from the LLM policy's search:
- lr=1e-5 is catastrophic. The single 1e-5 run (iter 2) collapsed to 3% pass@1. The policy never proposed it again.
- lr=5e-6 dominated. Both kept iterations and the majority of competitive runs used 5e-6.
- More steps trend better. The best result used 80 steps (0.36) vs 50 for the first improvement (0.34).
- High variance is inherent. eval_score ranged from 0.03 to 0.36 across identical-looking configurations, reflecting the stochastic nature of GRPO with short step budgets and binary rewards.
A separate grid search run (13 iterations, same search space) provides a direct baseline:
| Metric | LLM Policy | Grid Search |
|---|---|---|
| Best eval_score | 0.36 (iter 1) | 0.35 (iter 7) |
| Iterations to best | 2 | 8 |
| Mean eval_score | 0.291 | 0.271 |
Final scores are close (0.36 vs 0.35, within eval noise at 100 samples). The meaningful difference is convergence speed: the LLM policy's prior knowledge about GRPO hyperparameters let it skip the low-performing region that grid search had to enumerate. Grid search needed 8 iterations and 4 incremental improvements (0.28 -> 0.31 -> 0.32 -> 0.35) to reach the same neighbourhood.
For a well-studied algorithm like GRPO, the LLM advantage is speed, not final quality. The stronger case for LLM-guided search is in novel settings where uninformed search has no good starting point.
Of the 8 hours elapsed, 4.6 hours was actual GPU training. The remainder was container startup (~90 seconds per iteration for scheduling, pip install, data preparation) and controller reasoning time. Pre-building a Docker image with dependencies baked in would cut the per-iteration overhead roughly in half.
The GSM8K experiment validated the framework on a well-studied task. To test a harder claim, that the framework adds value in genuinely novel settings where no published recipe exists, I pointed it at a task with no prior art: training a small LLM as a structured security judge.
The task: given a user message, output a structured verdict:
{"decision": "pass", "security_score": 0.12}decision is one of pass, block, or warning. security_score is a calibrated confidence between 0.0 (safe) and 1.0 (malicious). The model must learn three things simultaneously: valid JSON output, correct classification, and calibrated confidence scores.
The reward function reflects this multi-objective structure:
| Component | Weight | Signal |
|---|---|---|
| Valid JSON with correct schema | 0.3 | Format compliance |
| Correct decision (pass/block/warning) | 0.4 | Classification accuracy |
| Calibrated security_score (within 0.3 of expected) | 0.3 | Confidence calibration |
This is not a reward function you can look up in a paper. The weights (0.3/0.4/0.3) are a design choice that the framework itself should eventually learn to rebalance. That is exactly the kind of improvement that hybrid mode's code diff capability targets.
- Model: Qwen2.5-0.5B-Instruct with LoRA adapters (0.1% trainable parameters)
- Dataset: 19,186 prompt injection samples from 26 security benchmarks
- Evaluation: 110 held-out samples, greedy decoding, structured verdict parsing
- Algorithm: GRPO with multi-component reward
- Policy: Hybrid (param search → code diffs on stall → param fallback)
- Budget: 3.3 hours elapsed, 1.3 GPU-hours training, ~$4 cost
51 iterations, 48 successful, 7 kept improvements. The zero-shot baseline: ~20% decision accuracy (the model predicts the majority class for all inputs), ~96% JSON compliance (Qwen2.5 produces valid JSON with minimal prompting), and a composite eval_score of ~0.43. The campaign revealed a textbook reward hacking pattern and its resolution:
Phase 1 (iters 0-8): Format learning. The model immediately achieved ~96% JSON compliance. Decision accuracy stuck at ~20%. The composite eval_score climbed from 0.43 to 0.52, driven entirely by the format and score calibration components. The model learned to produce valid structured output while always predicting the same class.
Phase 2 (iters 9-11): Diff mode attempted. The hybrid policy detected param stall and switched to code diff mode. The 3 diff proposals did not produce viable modifications, and the policy fell back to param mode automatically, demonstrating the fallback mechanism working as designed.
Phase 3 (iters 12-34): Plateau. Param search continued but decision accuracy remained stuck at ~20%. The model had found a local optimum: maximize format compliance and score calibration (0.6 reward) while ignoring the harder classification signal (0.4 reward).
Phase 4 (iter 35): Breakthrough. A specific configuration (lr=1e-4, LoRA rank=8, 30 steps, gen=3, temp=0.7) broke through the plateau. Decision accuracy jumped from ~20% to 76.4% while maintaining 96% JSON compliance. The remaining 16 iterations never exceeded this result.
| Phase | Iterations | Best eval_score | Decision accuracy | What drove improvement |
|---|---|---|---|---|
| Format learning | 0-8 | 0.52 | ~20% | JSON compliance + score calibration |
| Diff attempts | 9-11 | - | - | No viable diffs; fell back to params |
| Plateau | 12-34 | 0.62 | ~20% | Score calibration only |
| Breakthrough | 35 | 0.635 | 76.4% | Classification finally learned |
The step function tells the story: slow climbing from 0.43 to 0.62 over 20 iterations (score calibration gains), a flat plateau from iter 20 to 34 where the model sat in a local optimum, then the single jump to 0.635 at iter 35 when the classification component finally engaged.
The breakthrough config wasn't exotic: lr=1e-4, rank=8, 30 steps. What made it special was the combination of learning rate and temperature that produced enough gradient signal from the decision component to escape the local optimum. This is the kind of configuration a grid search would eventually find, but at iter 35 out of 162 possible configurations, the LLM-guided search found it well before exhaustive enumeration would have.
This experiment demonstrates what the GSM8K experiment could not:
-
No prior knowledge helps. There is no published recipe for GRPO-training a 0.5B model as a structured security judge. The LLM policy cannot recall known-good hyperparameters. It must genuinely search.
-
The reward function is the bottleneck, not the hyperparameters. The 20% decision accuracy plateau lasted 34 iterations because the 0.3 format + 0.3 calibration components create a 0.6 reward floor that the model can reach without touching the harder 0.4 classification signal. This is a structural problem in the reward weighting, not a hyperparameter problem, and it is exactly the kind of problem that hyperparameter search alone cannot solve. The hybrid policy's code diff phase is designed for this: rebalancing reward weights by modifying
train.py. In this campaign, param search eventually found a configuration that broke through the plateau at iter 35. Whether diff mode could shorten that path by proposing reward rebalancing directly is the next experiment to run. -
Multi-component rewards with uneven difficulty create reward hacking by default. The model optimizes a single scalar reward composed of three components, but the components differ in difficulty. Format compliance and score calibration saturate quickly (combined 0.6 reward floor), while classification accuracy (0.4 weight) requires genuine learning. This asymmetry is what enables reward hacking [5, 6], and practitioners typically spend weeks debugging it manually. The framework surfaced the pattern automatically through the eval_score trajectory.
A separate experiment validated the framework on supervised fine-tuning: optimizing DeBERTa-v3-base for prompt injection detection on the same dataset. 9 iterations total (6 successful), best F1=0.982. This confirms the prepare.py → train.py → metrics → keep/discard loop works for both RL and supervised paradigms with only the training script changing.
| GSM8K (GRPO) | Security Judge (LoRA+GRPO) | DeBERTa (SFT) | |
|---|---|---|---|
| Task novelty | Well-studied | Novel (no published recipe) | Standard classification |
| Reward signal | Binary exact-match | Multi-component (0.3/0.4/0.3) | Cross-entropy loss |
| Output format | Free text | Structured JSON | Binary label |
| Training | Full model weights | LoRA adapters (0.1%) | Full model weights |
| Baseline | 26% pass@1 | ~20% decision accuracy | F1=0.826 |
| Best result | 36% pass@1 | 76.4% decision accuracy | F1=0.982 |
| Improvement | +10pp | +56pp | +15.6pp |
| Iterations | 15 | 51 (48 successful) | 9 (6 successful) |
| GPU-hours | 4.6 | 1.3 | 0.4 |
Same loop, same infrastructure, same stop guards. The only thing that changed between experiments was the training script and the YAML config.
An earlier run of this same experiment (22 iterations, episode 897e096800b8) returned a 0% baseline on a model that should score ~26%. The reward parser expected GSM8K's #### <number> delimiter, but the model's chat template produced answers in a different format. Both components passed their own tests. The bug lived at the integration boundary between prompt construction and reward extraction.
The fix was straightforward: use tokenizer.apply_chat_template() for prompt formatting and expand extract_answer to handle multiple answer formats (####, \boxed{}, trailing numbers). This raised the baseline from 0% to 26% and the best result from 4% to 36%. The lesson: any RL training pipeline that operates autonomously needs a preflight gate that confirms the reward signal is non-degenerate before committing GPU hours. A zero baseline on a capable model is an instrumentation problem, not a training problem.
The current run achieved 100% iteration success rate across 15 Basilica deployments. The earlier run (different prompt configuration) hit 77% (17/22), with failures from LLM inference provider timeouts and OOM on aggressive hyperparameter combinations. Both runs completed without manual intervention because the framework's failure_rate_limit acts as a circuit breaker.
The circuit breaker uses a rolling window of 8 iterations. Transient failures from scheduling jitter or upstream provider latency pass through without interrupting the campaign. But if failures cluster (indicating a systemic issue like a broken training script rather than routine noise), the breaker trips and the campaign halts. Whether the success rate is 77% or 100%, the operational pattern is the same: budget for failure at the iteration level, and invest the engineering effort in distinguishing recoverable noise from genuine degradation.
Hypothesis: The autoresearch paradigm can be extended from pre-training to RL post-training if you solve the per-iteration GPU provisioning problem.
Verdict: Correct, with important caveats.
Three experiments, three different paradigms (GRPO, LoRA+GRPO, supervised fine-tuning), same framework, same loop. The GSM8K campaign improved pass@1 from 26% to 36% in 15 iterations. The security judge campaign reached 76.4% decision accuracy in 51 iterations on a task with no published recipe. The DeBERTa campaign hit F1=0.982 in 9 iterations (6 successful). All ran start-to-finish with no human intervention.
Ephemeral GPU containers made the difference. Without per-iteration isolation, I would have spent most of the 8 hours debugging state contamination between runs instead of analyzing results. The security judge experiment (51 iterations across multiple session interruptions, with checkpoint/resume recovering cleanly each time) validated the operational resilience claim more convincingly than any unit test could.
Optimization scope is bounded by instrumentation quality. The framework automates the search loop, but it trusts the reward signal it receives. An earlier run demonstrated this clearly: a misconfigured reward parser produced a degenerate signal that 22 iterations could not recover from. The fix (proper chat template formatting) turned a 0% baseline into a 26% baseline and unlocked the 36% result. The preflight validation gate described in Section 7 catches this class of bug before GPU hours are spent.
Container startup overhead is the main latency cost. Each iteration pays ~90 seconds for scheduling, dependency installation, and data preparation. Over 15 iterations this is meaningful. Pre-baking dependencies into the base image would cut this roughly in half. A local-GPU setup would run ~3x faster per iteration by avoiding container overhead entirely, at the cost of isolation and reproducibility.
Search coverage improves with longer campaigns. The GSM8K campaign ran 15 iterations over a 180-configuration space. The security judge campaign ran 51 iterations over a 162-configuration space, and the breakthrough came at iter 35. Note that the LLM policy revisits promising regions, so the number of unique configurations explored is lower than the iteration count. Longer campaigns surface dynamics (like the reward hacking plateau) that short runs cannot reveal.
For anyone building on this:
-
Verify the reward function manually before launching. Run
train.pyonce, print raw model output, confirm your metric extraction works. 30 seconds of testing saves hours of compute. -
Pre-build Docker images with dependencies. Eliminating the per-iteration
pip installcuts 60 seconds per iteration. For 50+ iteration campaigns, this saves significant cost. -
Start with the LLM policy, graduate to hybrid. The LLM policy's prior knowledge (from training data containing ML papers, blog posts, and experiment reports) is remarkably effective for initial exploration. Switch to code diffs only when parameter search demonstrably stalls.
-
Set conservative stop guards. The no-improvement streak limit is your most important budget control. 15 is a reasonable default for parameter search; increase to 25+ if using code diffs, which need more iterations to find meaningful algorithmic improvements.
-
Use the telemetry ledger. Every iteration is logged to a TSV ledger with hardware fingerprints, budget mode, and comparability flags. This is your reproducibility record. If someone questions your results, the ledger + JSONL event trace provide a complete audit trail.
Prerequisites: Python 3.10+, uv (curl -LsSf https://astral.sh/uv/install.sh | sh). No local GPU required. All training runs on ephemeral Basilica containers.
Step 1: Install
git clone https://github.com/epappas/autoresearch-rl.git
cd autoresearch-rl
uv syncStep 2: Configure credentials
export BASILICA_API_KEY="..." # https://basilica.ai
export CHUTES_API_KEY="..." # LLM inference for the policy
export HF_TOKEN="..." # https://huggingface.co/settings/tokensStep 3: Run a campaign
The repo includes example configs under examples/. A minimal experiment config looks like this:
name: grpo-qwen
objective:
metric: eval_score
direction: max
target:
type: basilica
prepare_cmd: [python3, /app/prepare.py]
train_cmd: [python3, /app/train.py]
timeout_s: 3600
basilica:
image: pytorch/pytorch:2.4.1-cuda12.4-cudnn9-devel
gpu_count: 1
gpu_models: ["A100"]
memory: "48Gi"
ttl_seconds: 3900
setup_cmd: "pip install --no-cache-dir transformers datasets accelerate"
policy:
type: llm
params:
learning_rate: [0.000003, 0.000005, 0.00001]
batch_size: [1, 2]
max_steps: [30, 50]
llm_api_url: "https://llm.chutes.ai/v1"
llm_model: "deepseek-ai/DeepSeek-V3-0324"
llm_api_key_env: "CHUTES_API_KEY"
controller:
max_iterations: 10
no_improve_limit: 5
failure_rate_limit: 0.85
failure_window: 8
checkpoint_path: artifacts/checkpoint.json
telemetry:
trace_path: traces/events.jsonl
ledger_path: artifacts/results.tsv
artifacts_dir: artifacts/runs
versions_dir: artifacts/versionsuv run autoresearch-rl run experiment.yamlStep 4: Monitor and inspect
While the campaign is running (or after it completes):
# Campaign status: best value, iterations done, recent history
uv run autoresearch-rl status experiment.yaml --last 5
# The telemetry ledger is a plain TSV, readable with any tool
head -20 artifacts/results.tsvAfter completion, the artifact directory looks like this:
artifacts/
checkpoint.json # resumable campaign state
results.tsv # one row per iteration: params, metrics, decision
runs/
run-0000/ # per-iteration output
run-0001/
...
versions/
v0000/ # kept iterations only (beat previous best)
version.json # params, metrics, status
v0011/
traces/
events.jsonl # full event stream: proposals, outcomes, stop guard triggers
Step 5: Single-iteration mode (for debugging or external agents)
# Run one iteration with fixed parameters
uv run autoresearch-rl run-one experiment.yaml \
--params '{"learning_rate": 0.000003, "batch_size": 1, "max_steps": 50}'
# Run one iteration applying a code diff
uv run autoresearch-rl run-one experiment.yaml \
--diff /path/to/training_improvement.patchThe most interesting thing about autoresearch-rl isn't the policies or the telemetry. It's that infrastructure is the bottleneck for autonomous ML research, not algorithms. The LLM policy is good enough out of the box. The keep/discard logic is a comparison operator. The stop guards are simple counters. None of these are hard.
What's hard is spinning up a fresh A100 with the right CUDA version, injecting parameters, running a 10-minute training job, extracting metrics from logs, cleaning up, and doing it 51 times without human intervention. That's a systems engineering problem that happens to live inside an ML workflow. Ephemeral GPU containers turn it into a configuration problem: define your requirements, your training script, and your search space. The rest is a loop. And while this article focuses on RL post-training, the same pattern applies to pre-training sweeps, evolutionary search, or any workflow where each iteration needs isolated GPU compute.
[1] A. Karpathy, "autoresearch," GitHub, 2025. https://github.com/karpathy/autoresearch
[2] B. Zoph and Q. V. Le, "Neural Architecture Search with Reinforcement Learning," in Proc. ICLR, 2017.
[3] J. Snoek, H. Larochelle, and R. P. Adams, "Practical Bayesian Optimization of Machine Learning Hyperparameters," in Proc. NeurIPS, 2012.
[4] Z. Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models," arXiv:2402.03300, 2024.
[5] D. Amodei et al., "Concrete Problems in AI Safety," arXiv:1606.06565, 2016.
[6] DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning," arXiv:2501.12948, 2025.
[7] J. Kaplan et al., "Scaling Laws for Neural Language Models," arXiv:2001.08361, 2020.
[8] Qwen Team, "Qwen2.5 Technical Report," 2024.
[9] K. Cobbe et al., "Training Verifiers to Solve Math Word Problems," arXiv:2110.14168, 2021.
[10] R. S. Sutton and A. G. Barto, "Reinforcement Learning: An Introduction," 2nd ed., MIT Press, 2018.