Skip to content

Instantly share code, notes, and snippets.

@nibzard

nibzard/COLLAB-PLAN.md

Created April 21, 2026 21:36
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save nibzard/c7027a8e5e932ec9f537eea94b3c0c1b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save nibzard/c7027a8e5e932ec9f537eea94b3c0c1b to your computer and use it in GitHub Desktop.
Download ZIP
Steel × ClawBench Collaboration Plan
Raw
COLLAB-PLAN.md

Steel × ClawBench Collaboration Plan

Why This Makes Sense

ClawBench runs 153 tasks on live production websites (Uber Eats, Booking.com, Robinhood, etc.) inside local Docker containers with a DIY Chromium stealth layer. Steel solves exactly this problem as a managed service. The integration surface is minimal — browser-use already accepts a custom CDP endpoint, meaning the harness change is one line of code.

ClawBench is also the most credible AI browser agent benchmark published to date (arXiv 2604.08523, #3 Paper of the Day on Hugging Face). Association gives Steel academic legitimacy at zero technical cost to ClawBench.

Phase 1 — Technical Integration (Week 1–2)

Goal: Add a steel harness to ClawBench that replaces local Docker/Chromium with Steel Sessions.

What changes in ClawBench

File Change
Dockerfile.steel New harness layer: installs steel-sdk, browser-use, litellm
setup-steel.sh Generates LiteLLM config + sets STEEL_API_KEY, STEEL_SESSION_URL
run-steel.sh Creates Steel session via API, waits for CDP URL, launches agent
run-steel-agent.py Browser(cdp_url=steel_session.cdp_url) — one-line diff from browser-use harness
models/models.example.yaml Add steel_api_key field to model config schema

What Steel gets

  • CAPTCHA solving and proxy fingerprinting replace ClawBench's 7-layer DIY stealth — no maintenance burden for the research team
  • Sub-1s session starts vs. Docker cold boot
  • Session Viewer for debugging failed benchmark runs (complements existing 5-layer recording)
  • Parallel task execution: all 153 tasks can run concurrently across Steel sessions

Deliverable

A working --harness steel flag in ClawBench that any researcher can use by adding steel_api_key: <key> to their models.yaml. PR submitted to the ClawBench repo.

Phase 2 — Research Credits Program (Week 2–3)

Goal: Remove the cost barrier for running full benchmark suites on Steel.

Structure

  • Steel provides a dedicated research credit pool for ClawBench benchmark runs
  • Accessed via a special API key provisioned through research@steel.dev
  • Scoped to ClawBench task execution (rate-limited to prevent abuse)
  • ClawBench adds Steel to the CITATION.cff acknowledgements section and README

Why credits, not free tier

The Hobby free tier ($10/month, 100 browser hours) covers ~65 tasks at average session length. A full 153-task suite across 6 models = ~918 browser sessions. Credits targeted at this scale make Steel the path of least resistance for new model submissions.

Phase 3 — Leaderboard Sponsorship (Week 3–4)

Goal: Steel brand on the public leaderboard, which every model evaluation links to.

Structure

  • Steel logo + link in the Hugging Face Gradio leaderboard Space header
  • "Powered by Steel" badge on benchmark run results submitted via Steel harness
  • Optional: Steel-hosted mirror of the leaderboard at bench.steel.dev

What ClawBench gets

  • Hosting cost offset for the Gradio Space
  • Increased leaderboard credibility (infrastructure partner signals production-grade testing)

Phase 4 — Joint Content (Week 4–6)

Goal: Distribution for both parties via co-authored technical content.

Content pieces

  1. Blog post — "We ran 153 real-website tasks across 6 LLMs" — joint Steel + ClawBench byline. Steel handles infrastructure narrative; ClawBench handles benchmark methodology. Target: Steel blog + Hugging Face blog cross-post.

  2. awesome-web-agents listing — ClawBench is already tracking the steel-dev/awesome-web-agents PR as an outreach target. Approving it in exchange for the Steel harness integration is a natural trade. Fast-track the PR review.

  3. Discord/community announcement — ClawBench has an active Discord + 微信群. Steel announcement in those channels reaches the exact audience (agent developers who need browser infra).

Ongoing — Harness Maintenance

  • Steel owns the Dockerfile.steel, setup-steel.sh, run-steel.sh, and agent script
  • ClawBench core team is not responsible for Steel harness bugs
  • Steel gets early access to new task categories before public release (enables proactive testing of sessions against new site patterns)
  • Quarterly sync on upcoming ClawBench releases and Steel API changes

Outreach Template

To: ClawBench maintainers (reacher-z / ZJU-REAL)
Subject: Steel Sessions harness for ClawBench + research credits

ClawBench is the only benchmark that actually needs what Steel does — you're running tasks on live Uber Eats, Booking.com, and Robinhood. Your current Docker/Chromium stealth setup is solving the same problem we've already productionized: anti-bot detection, CAPTCHA solving, session management at scale.

We'd like to:

  1. Submit a --harness steel PR (browser-use already takes a CDP URL — it's one line)
  2. Provide research credits so running full 153-task suites costs nothing
  3. Sponsor the leaderboard

In return we'd love a co-authored blog post and a fast-track on the awesome-web-agents PR.

Happy to jump on a call or just start with the PR.

Success Metrics

Metric Target
Steel harness PR merged Week 2
First full benchmark run via Steel Week 3
awesome-web-agents listing live Week 4
Joint blog post published Week 6
Leaderboard submissions using Steel harness 2+ models by end of Q2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment