DO: Iterate ON the Space. Push early, verify against live URL. Stream logs. Read logs first, act once, make one targeted fix. Use cheapest iteration rung. Set all three cache env vars at module top, before any import. Measure @spaces.GPU(duration=) empirically. Call c.view_api() before any predict() call. python3 -m py_compile app.py is the maximum local check before pushing. Always ship 3–5 curated gr.Examples and cache them (cache_examples=True, cache_mode="lazy", with fn=/outputs= set) — find good inputs, not placeholders.
NEVER: Sleep-poll. Build mock modes, SKIP_MODEL_LOAD env vars, Playwright harnesses, or local Gradio servers. Restart before reading the error. Issue concurrent uploads. Restart while uploading. Stack restarts while runtime.sha is still flipping. Use integer device IDs (.to(0), device_map={"": 0}, set_device(0)). Pin torch or torchaudio. Proxy another community ZeroGPU Space via gradio_client. Wrap a gradio_client proxy call in @spaces.GPU. Stack @spaces.GPU and @app.api on the same function. Edit files in dev-mode SSH expecting them to survive restarts.
hf spaces search "<model name or task>" --sdk gradio --limit 10 # search reference Space first
hf repos create <ns>/<name> --type space \
--space-sdk <gradio|streamlit|docker|static> [--flavor zero-a10g] --exist-ok
hf upload <ns>/<name> . . --type space \
--exclude '.git/*' --exclude '__pycache__/*' --exclude '.venv/*' --exclude '*.pyc' \
--commit-message 'init'
hf spaces logs <ns>/<name> --build --follow
hf spaces logs <ns>/<name> --follow
hf spaces logs <id> --build --tail 200 # BUILD_ERROR: find the FIRST error
hf spaces logs <id> --tail 200 # RUNTIME_ERROR or stuck APP_STARTINGREADME frontmatter (required fields):
---
title: ...
emoji: 🚀
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.10.0 # omit for docker/static
app_file: app.py # gradio/streamlit only
short_description: ... # <= 60 chars
startup_duration_timeout: 1h # default 30m; bump for big LLMs
---hardware: in frontmatter is silently ignored. Set hardware via --flavor at create time OR hf spaces settings <id> --hardware zero-a10g. A silently wrong hardware shows up later as RuntimeError: Found no NVIDIA driver.
| Rung | When | Command |
|---|---|---|
| 1. Hot-reload | Pure Python edit, Gradio >= 6.1, no new deps | hf spaces hot-reload <id> -f app.py |
| 2. Dev SSH | Diagnostics only (edits don't persist) | ssh -i ~/.ssh/id_rsa <subdomain>@ssh.hf.space '<cmd>' |
| 3. Targeted upload | Code-only, non-Gradio file, gr.Server |
hf upload <ns>/<name> --include '<file>' && hf spaces logs <id> --follow |
| 4. Full rebuild | requirements.txt, Dockerfile, frontmatter, hardware |
hf spaces logs <id> --build --follow |
| 5. Factory reboot | Container in inconsistent state, last resort | hf spaces restart <id> --factory-reboot |
Hot-reload poisoning: --factory-reboot after a hot-reload-only commit fails with could not read Username for https://huggingface.co. Push any normal hf upload commit first, then restart. After upload, runtime.sha lags; do NOT restart again until it flips.
States: BUILDING -> APP_STARTING -> RUNNING.
hf spaces dev-mode <ns>/<name> # triggers rebuild; PRO/Team/Enterprise only
hf spaces info <ns>/<name> --format json \
| python3 -c "import json,sys; print(json.load(sys.stdin)['runtime']['raw']['domains'][0]['domain'])"
git add . && git commit && git push # from inside SSH to persist editsTreat all SSH edits as throwaway. Use dev mode for: reading logs, pip list, ad-hoc imports, curl localhost:7860. Not as an editor.
ZeroGPU is Gradio-only. Flavor: zero-a10g. Actual GPU: NVIDIA RTX PRO 6000 Blackwell.
Stub probe (required for zero-a10g Spaces):
@spaces.GPU(duration=1)
def _zerogpu_probe(): return "ready"Without at least one @spaces.GPU function, the Space RUNTIME_ERRORs immediately.
Cache env vars + import order (set at top of app.py, before any import):
import os
os.environ.setdefault("HF_HOME", "/data/.cache/huggingface") # or /tmp on non-bucket spaces
os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
import spaces # before torch; avoids libcudart.so.X errorsNeMo/RNNT exception (import order matters):
os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")
import spaces # immediately after NUMBA_DISABLE_CUDA, before any nemo/torch importLoad model on CPU at startup; inside @spaces.GPU do model.to("cuda") then infer then model.to("cpu").
Model loading: Load at module level including .to("cuda"). Only compute goes inside @spaces.GPU.
model = Model.from_pretrained(..., torch_dtype=torch.bfloat16).to("cuda")
# NOT device_map="cuda" for plain from_pretrained - crashes with "Found no NVIDIA driver"
# device_map="cuda" is OK only with bitsandbytes quantization_config (Pattern B)Exception: frameworks that touch CUDA at import (pynvml, NCCL, vllm, NVFP4 quantizers) must defer construction into the first @spaces.GPU call with duration set high enough to cover load + inference.
Multiple checkpoints:
BUNDLES = {}
for family in ("image", "video"):
BUNDLES[family] = load(family).to("cuda")Never unload/reload between requests.
Duration: Never hardcode without measuring. Ship placeholder, instrument with time.perf_counter(), run 2-3 calls (include at least one cold start, which is 1.5-3x slower), set duration = round(measured_max * 1.4). Too-high surfaces as "duration above maximum" at call time. Too-low silently truncates. Neither shows at deploy.
Callable duration (use *args, **kwargs to absorb gr.Progress positional arg):
def _estimate(prompt, history, max_new_tokens, *args, **kwargs):
return min(240, 60 + int(max_new_tokens / 64))
@spaces.GPU(duration=_estimate)
def chat(prompt, history, max_new_tokens, ..., progress=gr.Progress(track_tqdm=True)): ...Two-ceiling: @spaces.GPU(duration=) and @app.api(time_limit=) both apply; lower wins. Put them on SEPARATE functions:
@spaces.GPU(duration=60)
def _run_gpu(prompt): return inference(prompt)
@app.api(name="generate", concurrency_limit=1, time_limit=180)
def generate(prompt: str) -> str: return _run_gpu(prompt)Size: default "large" = 48 GB, 1x quota. Use size="xlarge" only when exceeding 48 GB:
@spaces.GPU(duration=120, size="xlarge") # xlarge = 96 GB, 2x quota
def heavy_fn(): ...bf16 costs params_B * 2 GB (27B=54 GB overflows large). NF4 4-bit costs params_B * ~0.55 GB (27B=~15 GB fits large, 70B=~40 GB fits large).
Flash-attn / xformers: bitsandbytes works on ZeroGPU. Legacy flash-attn 1.x/2.x, triton, and xformers do not build cleanly; flash-attn 3 is officially recommended. Wrap unconditional top-level imports:
try:
from flash_attn import flash_attn_func; HAS_FA = True
except ImportError:
flash_attn_func = None; HAS_FA = Falsexformers SDPA shim (for models with hard import xformers.ops): install at boot before model imports.
xformers.ops.memory_efficient_attention = _meff # full shim code in §4 item 5 of full gistMisc: torch.compile NOT supported on ZeroGPU; use AoTI (torch >= 2.8). Gemma2 / transformers >= 4.50: attn_implementation="eager". Thinking models: max_new_tokens >= 512; suppress with chat_template_kwargs={"enable_thinking": False} (not enable_thinking= directly, raises TypeError in transformers >= 5). ONNX: use onnxruntime-gpu; rewrite custom ops to opset-20. TorchScript: torch._C._set_graph_executor_optimize(False).
Gradio slow startup: set GRADIO_SSR_MODE=false via env var (NOT launch(ssr_mode=False), ignored on HF) AND startup_duration_timeout: 1h in README YAML. Both required.
hf spaces variables add <id> --env GRADIO_SSR_MODE=falsegr.Examples: use cache_examples=True, cache_mode="lazy". "eager" runs every example at startup and burns ZeroGPU daily quota. Cache is keyed by file path, not content hash; bump a cache_version constant to wipe stale cache.
Streamlit: port 8501 only. Docker: set app_port: in README YAML if not 7860 (EXPOSE not auto-read); build with --platform=linux/amd64. Static: set app_build_command: npm run build and app_file: dist/index.html in README YAML.
Pin everything except torch/torchaudio (base layer). Add torchvision unpinned if needed. Include accelerate whenever using device_map=. No local paths or editable installs. Non-pip-installable model code: include the directory in the upload; it lands in /home/user/app/. CUDA-extension build failures: vendor a same-named pure-PyTorch shim in the Space root. Verify with python3 -m py_compile fast_hadamard_transform.py before pushing. Build hangs = pip backtracking; read --build logs and pin the conflicting transitive dep.
- Search reference Space:
hf spaces search "<model name or task>" --sdk gradio --limit 10. Read itsapp.py+requirements.txt. Cap pre-push research at one reference Space. - Decide SDK + hardware. Write minimal
app.py+requirements.txt+ README frontmatter. - Push immediately:
hf repos create --exist-okthenhf uploadthenhf spaces logs --build --follow. - Once
RUNNING: verify withgradio_client(see §11). - Iterate via cheapest rung in §2.
- On error: read the FIRST error, make ONE targeted fix, use smallest rung.
Pattern A (Inference Provider proxy): stateless chat, any model size, zero VRAM, no @spaces.GPU, cpu-basic hardware.
gr.load("models/<org>/<model>", accept_token=button, provider="fireworks-ai")README needs hf_oauth: true and hf_oauth_scopes: [inference-api].
Pattern B (NF4 4-bit on ZeroGPU): custom inference, tool use, multimodal, stateful.
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True,
device_map="cuda", torch_dtype=torch.bfloat16, quantization_config=bnb,
attn_implementation="sdpa", low_cpu_mem_usage=True).eval()Streaming (non-negotiable for chat UX):
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True, timeout=120)
Thread(target=model.generate, kwargs=dict(**inputs, streamer=streamer, ...)).start()Pattern B minimum requirements.txt:
gradio>=6.10
spaces>=0.41
transformers>=4.57
accelerate>=1.10
bitsandbytes>=0.48
sentencepiece
Before deploying Pattern B: set GRADIO_SSR_MODE=false and startup_duration_timeout: 1h. Full Pattern B code in full gist §9.
demo = app # HF runtime expects this name
@spaces.GPU(duration=60)
def _run_gpu(prompt): return inference(prompt) # separate function
@app.api(name="generate", concurrency_limit=1, time_limit=180)
def generate(prompt: str) -> str: return _run_gpu(prompt) # never stack @app.api + @spaces.GPUHot-reload (rung 1) does NOT work with gr.Server. Files in @app.api routes are plain dicts: isinstance(v, dict) and (v.get("path") or v.get("name")).
Persistent storage (HF Buckets):
hf buckets create <ns>/<bucket-name>
hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data/data is ephemeral by default; bucket mount makes it durable. Do NOT load model weights from /data; bucket I/O is S3-paced and stalls past any @spaces.GPU duration cap. Use bucket for user-generated content only.
hf spaces info <id> --expand runtime --format json | python3 -c \
"import json,sys; r=json.load(sys.stdin)['runtime']; print(r['stage'], r.get('hardware','?'), '| requested:', r.get('requested_hardware','?'))"
# expect: RUNNING zero-a10g | requested: zero-a10gfrom gradio_client import Client, handle_file
import os
c = Client("ns/name", token=os.environ["HF_TOKEN"], # token=, not hf_token=
httpx_kwargs={"timeout": 600}) # >= @spaces.GPU duration + 60
print(c.view_api()) # always call first; never guess api_name
result = c.predict(handle_file("input.png"), "prompt", api_name="/generate")
# Streaming:
job = c.submit("prompt", api_name="/chat")
for chunk in job: print(chunk, end="")
# Custom @app.get/post routes (not in view_api()):
import httpx
r = httpx.post(f"{base}/your_route", json={...}, headers=headers, timeout=600)No Playwright. No mock-mode local servers. No anonymous calls to private Spaces (returns 404, not an app bug).
hf jobs logs <id> --follow # stream; never sleep+pollzero-a10g flavor name is historical. Actual GPU is RTX PRO 6000 Blackwell (sm_120). Compile AOTI artifacts for sm_120, not A10G sm_86.
BUILDING > 5min -> logs --build --tail 500, find FIRST error
APP_STARTING 10-25min (big LLM) -> NORMAL: model download/load; tail logs to see progress
APP_STARTING forever (small app) -> logs --tail 500; usually missing import or model OOM
RUNTIME_ERROR right after long -> startup timed out: (1) set startup_duration_timeout: 1h
APP_STARTING, sparse logs in README YAML AND (2) set GRADIO_SSR_MODE=false env var
RUNNING but 404 from public URL -> Space is private; auth gradio_client with token=
SSH hangs / Permission denied -> check ~/.ssh/id_rsa, key on /settings/keys, PRO/Team plan
@spaces.GPU "duration above max" -> lower the value
ZeroGPU init spinner / GPU abort -> CUDA touched before allocation; for NeMo/RNNT set
before app logs anything NUMBA_DISABLE_CUDA=1 then 'import spaces' immediately;
for others reorder so 'import spaces' precedes torch
libcudart.so.X at startup -> reorder: import spaces before import torch
Factory reboot fails "could not -> previous commit was hot-reload-only; push any normal
read Username..." hf upload commit first, then restart
runtime.sha stuck on old commit -> container still loading; poll SHA, do NOT re-restart
Stale example after asset regen -> gr.Examples cache keyed by path not hash; bump cache_version
| Symptom | Fix | Ref |
|---|---|---|
Space on cpu-basic despite hardware: in README YAML |
--flavor at create or hf spaces settings <id> --hardware zero-a10g; frontmatter silently ignored |
§1 |
RuntimeError: Found no NVIDIA driver inside @spaces.GPU |
cpu-basic hardware (above) OR device_map='cuda' on plain from_pretrained; use .to('cuda') |
§1/§4 |
factory-reboot fails could not read Username |
Push any normal hf upload commit first, then restart |
§2 |
runtime.sha stuck after upload |
Container loading; do NOT issue another restart | §2 |
| SSH Permission denied or hangs | Key on /settings/keys, PRO/Team plan, subdomain looked up dynamically |
§3 |
| SSH edits lost after restart | git add . && git commit && git push from inside SSH session |
§3 |
RUNTIME_ERROR after long APP_STARTING |
startup_duration_timeout: 1h in README YAML AND GRADIO_SSR_MODE=false env var |
§5/§7 |
RUNNING but 404 |
Private Space; token= (not hf_token=) in gradio_client |
§11 |
duration above maximum at call time |
Lower @spaces.GPU(duration=) value; not visible at deploy time |
§4 |
| ZeroGPU init spinner or aborted before app logs | NeMo: NUMBA_DISABLE_CUDA=1 then import spaces immediately; others: import spaces before torch |
§4 |
libcudart.so.X at startup |
import spaces before import torch |
§4 |
| xformers CUDA extension mismatch | SDPA shim before model imports; full shim in full gist §4 item 5 | §4 |
Gemma2 sdpa_mask / _vmap_for_bhqkv failures |
attn_implementation='eager' in from_pretrained |
§4 |
gr.Examples stale output after in-place asset regen |
Cache keyed by path not hash; bump cache_version |
§5 |
gr.Examples eager burns ZeroGPU quota at startup |
cache_examples=True, cache_mode='lazy' |
§5 |
TypeError: N+1 positional args in callable duration |
Add *args, **kwargs to duration function signature |
§4 |
Files in @app.api routes are plain dicts not FileData |
isinstance(v, dict) and (v.get('path') or v.get('name')) |
§10 |
AOTI artifacts for sm_86 fail on ZeroGPU |
Compile for sm_120; zero-a10g name is historical |
§12 |
| ONNX custom op errors | onnxruntime-gpu; rewrite to opset-20; dynamic batch dim |
§4 |
enable_thinking= raises TypeError in transformers >= 5 |
chat_template_kwargs={"enable_thinking": False} in apply_chat_template |
§4 |
gradio_client proxy in @spaces.GPU burns quota |
Never wrap HTTP-only proxy in @spaces.GPU; proxy Space uses cpu-basic |
§9 |
Hot-reload has no effect with gr.Server |
Use hf upload (rung 3) or real commit |
§10 |
HF_HOME set but cache fails silently |
Set all three: HF_HOME, HF_MODULES_CACHE=/tmp/hf_modules, MPLCONFIGDIR=/tmp/matplotlib, before any import |
§4 |