session-95920b6f-context1-deployment.md

Session 95920b6f — Deploying chromadb/context-1 on HF Dedicated Endpoints

Goal

Research chromadb/context-1 (a 20B MoE retrieval model based on openai/gpt-oss-20b) and deploy an optimized HF Dedicated Endpoint for it.

---

Did it use the HF Endpoint Deploy CLI?

No. The hf CLI was installed and available — Claude used it for hf whoami and repo operations — but it was unaware that hf endpoints deploy exists. Instead, it went straight to hand-crafted curl calls against https://api.endpoints.huggingface.cloud/v2/, reverse-engineering the JSON schema by inspecting existing endpoints.

The hf CLI has a full endpoints suite (hf endpoints deploy, update, delete, ls, describe, pause, resume, scale-to-zero) that would have handled most of the payload formatting, field casing, and lifecycle management automatically. Not knowing about it cost multiple iterations on wrong JSON casing (vllm vs vLLM), silently ignored commandLine fields, and manual status polling loops.

---

What it searched/researched about HF Endpoints

Step	What	How
1	Model page & specs	WebFetch on trychroma.com research page
2	HF model card & files	WebFetch on huggingface.co/chromadb/context-1, plus curl on config.json, generation_config.json, tokenizer_config.json
3	Available GPU providers	curl on /v2/provider API, filtered for GPUs ≥48GB VRAM
4	Existing endpoint format	Listed own existing endpoints to reverse-engineer the correct JSON payload schema (image field casing, env vars, etc.)
5	Optimal vLLM params (2nd round)	WebFetch re-read of both pages + spawned a sub-agent to web-search for "chromadb context-1 deployment vllm", "gpt-oss-20b vllm deployment", GitHub issues, etc.
6	Base model tokenizer	curl on openai/gpt-oss-20b/tokenizer_config.json to compare with context-1's broken tokenizer

Notably missing:

• Never ran hf endpoints --help or discovered the deploy CLI
• Never searched for HF Endpoints API docs or deployment guides
• Reverse-engineered the API schema by trial-and-error instead

---

What Was Attempted (chronologically)

1. Research — Fetched the Chroma research page and HF model card. Identified it as a ~42GB BF16 MoE model for long-context agentic retrieval (128K context).

2. First endpoint (1x A100, vLLM v0.16.0, 32K context) — Multiple failed curl attempts due to wrong JSON casing (vllm vs vLLM). Had to list existing endpoints to discover the correct format. Created successfully but user flagged the low context limit.

3. commandLine doesn't persist — Tried passing vLLM args via commandLine field in the image config. The API accepted it silently but didn't use it. Switched to environment variables.

4. Deep research round — User asked "did you do enough research?" Discovered critical requirements: need vLLM v0.10.2 (v0.11+ has blank responses), need VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1 for A100, and full 128K context.

5. Updated A100 endpoint (128K context) — OOM crash. 42GB model + KV cache for 128K tokens exceeded 80GB VRAM.

6. Scaled to 2x A100 with tensor parallelism — Crashed even earlier. TP likely incompatible with this architecture on vLLM v0.10.2.

7. Switched to 1x H200 (141GB VRAM) — API rejected region change on existing endpoint (can't change provider/region via PUT). Had to delete and recreate in us-east-2. Then vLLM v0.10.2 crashed (missing Hopper/H200 CUDA support). Upgraded to vLLM v0.16.0 — still failed.

8. Parallel attempt: SGLang + vLLM on A100 — Both failed with the same tokenizer error: ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported.

9. Root cause found — tokenizer_config.json specifies tokenizer_class: "TokenizersBackend" (gpt-oss-specific, requires transformers 5.3.0). vLLM ships transformers 4.x.

10. The fix — Duplicated repo to victor/context-1 via HF duplicate API, patched tokenizer_config.json to use PreTrainedTokenizerFast.

11. Final endpoint (chroma-context-1) — 1x A100 80GB, vLLM v0.10.2, pointing at fixed fork. Booted and worked.

12. Tested — Chat completions worked. Tool calling failed (parser not configured), but text generation was functional.

---

Pain Points

#	Pain Point	Impact
1	Tokenizer incompatibility — gpt-oss uses a custom tokenizer class not in standard transformers/vLLM	Root blocker. Every GPU/framework combo failed with the same error until the model repo was forked and patched.
2	Didn't know about hf endpoints deploy — went straight to raw curl	Wasted multiple cycles on wrong JSON casing, silent field ignoring, and manual polling. The CLI handles all of this.
3	OOM with 128K context on 1x A100	42GB model weights + 128K KV cache don't fit in 80GB. Had to iterate through GPU configs.
4	Tensor parallelism crash on 2x A100	Architecture/vLLM version incompatibility made multi-GPU a dead end.
5	vLLM version minefield	v0.10.2 needed for gpt-oss but lacks H200 support; v0.11+ has blank response bugs; v0.16.0 still crashed on H200.
6	HF Endpoints API quirks	Can't change region on existing endpoint (must delete + recreate). Endpoint names linger after deletion causing conflicts. commandLine field silently ignored.
7	Insufficient initial research	First deploy used 32K context, missing the model's core value prop (long-context retrieval). Required a user prompt to go deeper.
8	Log retrieval difficulty	Endpoint logs were sometimes empty or hard to parse, making crash diagnosis slow. The OOM vs tokenizer confusion cost several iterations.
9	Tool calling not working	Even after successful deployment, the tool-call parser wasn't configured — left unresolved.

---

HF Dedicated Endpoints: Agent-Readiness Assessment

Based on what an AI agent (Claude) actually experienced trying to deploy a model end-to-end:

✅ What works well for agents

• hf endpoints CLI exists — Full lifecycle management (deploy, update, delete, ls, describe, pause, resume, scale-to-zero) is available. Claude just didn't know about it.
• REST API exists — Endpoints can be created, updated, deleted, and monitored programmatically
• Provider listing — /v2/provider (or hf endpoints catalog) gives structured GPU inventory with pricing
• Status polling — State machine (initializing → running/failed) is clear and pollable
• Logs endpoint — /v2/endpoint/{ns}/{name}/logs exists for debugging
• OpenAI-compatible serving — Once running, the /v1/chat/completions API is standard

⚠️ Friction points for agents

• Agent didn't discover the CLI — The hf endpoints deploy command exists but Claude (a frontier model) didn't know about it and never ran --help to discover it. This suggests the CLI's discoverability could be improved, or that AI model training data doesn't yet include it.
• commandLine silently ignored — The REST API accepts a commandLine field but doesn't apply it. No error, no warning. Agent wasted a cycle before switching to env vars.
• Env vars as proxy for CLI flags — vLLM/SGLang flags must be passed as env vars (MAX_MODEL_LEN, TRUST_REMOTE_CODE, etc.) rather than a command line, and the mapping isn't documented in the endpoint API.
• Endpoint name conflicts after deletion — Deleting an endpoint doesn't immediately free the name, causing create failures that require inventing a new name.
• Region immutability — Can't change region via PUT. Must delete + recreate, which compounds with the name-conflict issue.
• Logs sometimes empty — Failed endpoints sometimes return no logs, leaving the agent blind to the failure cause.
• No model compatibility check — No pre-flight validation that a model's tokenizer/architecture is compatible with the chosen framework version. Agent burned 6 endpoint create/destroy cycles before diagnosing a tokenizer issue.

❌ Blockers for fully autonomous agent deployment

• No recommended config endpoint — No API to ask "what GPU/framework/version should I use for model X?" Agent must know or research vLLM version compatibility, GPU memory requirements, attention backend workarounds, etc.
• No tokenizer/model validation — The platform doesn't check if tokenizer_class is loadable by the chosen framework before starting a multi-minute deploy cycle.
• No structured error taxonomy — Failures are raw container logs. An agent can't distinguish OOM from tokenizer errors from CUDA incompatibility without parsing free-text logs.
• Tool calling setup is opaque — For agentic models (like context-1), configuring --tool-call-parser requires framework-specific knowledge not exposed by the endpoint API.
• No fork/patch workflow — When a model needs a compatibility fix (like the tokenizer patch), there's no "deploy with override" option. Agent had to duplicate the entire repo just to change one JSON field.

Overall verdict

HF Dedicated Endpoints has the tooling for agent-driven deployment (hf endpoints deploy CLI, REST API, status polling). But the agent didn't know the CLI existed and the platform doesn't guide agents toward success: no pre-flight model validation, no recommended configs, no structured errors. The result was 7 failed endpoints before finding a working config — a process that required forking a model repo to fix a single JSON field.

---

Final Working Config

• Endpoint: chroma-context-1 on 1x A100 80GB (us-east-1)
• Model: victor/context-1 (fork with fixed tokenizer)
• Framework: vLLM v0.10.2
• Context: 128K tokens (default from model config)
• Key env vars: VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1, VLLM_USE_V1=1
• Total endpoints created and destroyed: 7 (before the one that worked)