epappas / autoresearch-rl.writeup.md

Last active March 26, 2026 14:53

Autonomous RL Fine-Tuning on Ephemeral GPUs: Extending Karpathy's autoresearch with autoresearch-rl

When Karpathy released autoresearch [1], he showed you could hand an LLM a training script, let it propose changes, run the experiment, observe the result, and repeat. His target was pre-training. Same GPU, same environment, iterations in minutes. I wanted to test a harder claim: can the same loop work for RL fine-tuning, where each iteration needs its own GPU, rewards are sparse, and a single bad hyperparameter wastes an hour of A100 time?

The answer is yes, but the hard part isn't where I expected.

tl;dr

I built autoresearch-rl and pointed it at a GRPO fine-tuning task on Basilica A100s. One command: uv run autoresearch-rl experiment.yaml. It ran 15 iterations autonomously with 100% success rate: proposing hyperparameters, spawning ephemeral GPU containers, training, evaluating, keeping or discarding, and looping. No human in the loop. GSM8K pass@1 improved from 26% (baseline) to 36% across

epappas / owasp-thread-modeling.md

Last active March 8, 2026 21:33

I Mapped Every OWASP Top 10 LLM Risk to a Transparent Security Proxy. Here's What Actually Works.

Threat-Modeling the OWASP Top 10 for LLM Applications

Samsung engineers pasted proprietary semiconductor code into ChatGPT. Three incidents in twenty days, followed by a company-wide ban. PromptArmor showed that a single message in a public Slack channel could make Slack's AI exfiltrate data from private channels the attacker never had access to. Mithril Security uploaded a surgically modified GPT-J model to Hugging Face that spread targeted misinformation while passing every standard evaluation benchmark. JFrog found about 100 models on Hugging Face with silent backdoors that established reverse shells on load.

The OWASP Top 10 for LLM Applications 2025 catalogues these and other LLM-specific threats. To understand how many of them are actually detectable at the infrastructure level, I built a transparent proxy for LLM traffic and tested it against all ten. The tooling is reasonable for some risks. For others, especially the ones

epappas / fine-tuning-security-judge.md

Last active March 5, 2026 10:43

I Fine-Tuned a 3B Model to Refuse Prompt Injections. Here's What Actually Worked.

We Fine-Tuned a 3B Model to Refuse Prompt Injections

If you're running LLMs in production, prompt injection is the attack you can't fully patch. Someone wraps "ignore your instructions" inside a polite customer support query, or buries a hijack command in a document your RAG pipeline retrieves, and your model follows it. The standard defenses (regex filters, classifier ensembles, guardrail APIs) catch the attacks they've been trained on. The ones they haven't seen walk right through.

We hit this wall ourselves. Together with George Politis, we've been running LLMTrace, an open-source security proxy that sits between applications and their LLM providers. It intercepts every request and runs it through an ensemble of detectors (regex patterns, a DeBERTa classifier, InjecGuard, jailbreak classifiers) at ~50ms overhead on the hot path. On known jailbreak datasets it hits 99% recall. We were reasonably confident in it until we ran [12,000+

epappas / MISTRAL_JUDGE_EXPLORATION.md

Last active February 28, 2026 18:34

Mistral LLM-as-a-Judge: Exploration & Plan for llmtrace

Mistral LLM-as-a-Judge: Exploration & Plan

Executive Summary

This document explores adding a fine-tuned Mistral model as an "LLM-as-a-Judge" analyzer to the llmtrace security pipeline. The judge evaluates prompts/responses with reasoning, producing structured findings that integrate with the existing ensemble voting system.

Key conclusions:

Model: Ministral-3 14B Instruct (Dec 2025) -- recommended by Mistral engineer
Training Stage 1 (SFT): QLoRA r=64, 4-bit NF4, TRL SFTTrainer (~$5-15 per run)
Training Stage 2 (GRPO, optional): Reinforcement learning for confidence calibration + evidence quality (~$15-75 per run, only if SFT model has calibration issues)

epappas / agent-harness-architecture.md

Last active May 14, 2026 02:35

The Agent Harness Is the Architecture (and Your Model Is Not the Bottleneck)

I keep hearing the same question at every engineering offsite, Slack thread, and investor pitch: "What's the best model right now -- GPT, Claude, or Gemini?" I spent the last several months building and debugging agent-based systems, and I think this is the wrong question entirely. The evidence is now overwhelming: what determines whether an AI agent succeeds in production is not the model underneath it, but the infrastructure wrapped around it.

I am going to lay out my hypothesis, test it against three independent case studies with published data, and show you exactly where the industry is converging. Every claim in this article is backed by a published source -- engineering blogs, peer-reviewed papers, or reporting from outlets with direct access.

My hypothesis: Agent harness engineering -- the design of context management, tool selection, error recovery, and state persistence -- is the primary determinant of agent relia

epappas / article.md

Last active February 20, 2026 16:02

Securing AI Agents: An Architecture for Systems You Can't Fully Control

I was experimenting with building a tech support agent, to provide first-line support to my customers over Discord. The security side seemed straightforward: treat it like any other API gateway, so I started building a security proxy for LLM traffic. Validate inputs, sanitize outputs, log everything. The same patterns we've used for REST APIs and microservices for a decade.

That assumption broke within the first week.

Here's what broke it. During early testing, before any detection pipeline existed, I fed the support agent a batch of test tickets. One ticket contained an indirect injection buried in the body: instructions telling the agent to query the full customer table and include the results in its summary. The agent complied. It had legitimate database_query access -- granted broadly so it could look up account details -- so the tool call succeeded. It returned 1,200 customer records in a

epappas / article.md

Last active February 17, 2026 21:38

I Tested the Major Open-Source Prompt Injection Detectors. Here's What Actually Works.

I needed a prompt injection detector for a transparent security proxy I was building. Simple requirement: intercept LLM API traffic, flag malicious inputs, don't block legitimate users. I started with the obvious approach -- find the best open-source model, plug it in, ship it.

That assumption lasted about two weeks.

After testing every DeBERTa variant, guard model, and regex ruleset I could get running against a 153-sample adversarial corpus pulled from 15+ sources (public datasets and internal samples), I can tell you: the assumption was wrong. Not because the models are bad. Because they each fail in a different, complementary way -- and understanding exactly how they fail turned out to be more important than finding the "best" one.

My hypothesis: an ensemble of imperfect detectors, properly calibrated, outperforms any single best model. The short version: 89% of true positives in my benchmark required multiple d

epappas / dual-plane-finality-article.md

Last active February 15, 2026 22:24

Dual-Plane Finality: Node Lifecycle Architecture for Decentralized Infrastructure (DRAFT)

Decentralized GPU compute marketplaces, where miners provide hardware and validators score performance to allocate token emissions, present a distinct architectural challenge in node lifecycle management: the separation of control-plane finality (network consensus and routing) from economic-plane finality (emission settlement and stake unlocking).

This architecture—Cordon → Drain → Remove—applies broadly to decentralized systems where nodes have both operational responsibilities (participating in consensus, serving requests) and economic commitments (locking stake, earning emissions, submitting proofs). Traditional infrastructure patterns like Kubernetes drain don't model this dual finality explicitly, because they don't need to—there's no economic layer to settle.

The architectural tension arises from a fundamental mismatch: distributed systems require atomic state transitions for operational consistency, but econo

epappas / node-decom-architecture-v2.md

Created February 14, 2026 14:19

Node Capacity Reduction Architecture v2: Cordon → Drain → Remove (Infra + DePIN) - Production-Hardened

Node Capacity Reduction Architecture: Cordon → Drain → Remove (Infra + DePIN)

Status: Reference Architecture (v2.0)
Audience: Platform / SRE / Protocol Engineering
Primary goal: Safely reduce capacity by removing a node from service without violating availability, durability, or economic commitments.

1. Scope and design goals

epappas / node-decom-review.md

Created February 14, 2026 13:46

Technical Review: Node Capacity Reduction Architecture (Cordon → Drain → Remove)

Technical Review: Node Capacity Reduction Architecture

Cordon → Drain → Remove (Infra + DePIN)

Reviewer: Claude (Technical Architecture Review)
Document Version: 8c44194c-7b66-444d-8fc2-1dc63cdf04db.md
Review Date: 2025-02-14
Review Type: Comprehensive Technical Peer Review

Evangelos Pappas epappas