spark_fixes_grok.md

Practical fixes that actually work, drawn from the same NVIDIA forum threads, Reddit reports, GitHub issues, and dual-Spark recipes people are using right now.

1. Kill the most common progressive leak (multi-node / TP=2)

Add these two environment variables to every vLLM launch (or docker-compose / start script):

UCX_MEM_MMAP_HOOK_MODE=none
UCX_RCACHE_MAX_UNRELEASED=1024

This stops the unbounded UCX RDMA registration cache growth that was draining UMA at ~14 MB per request and causing the slow freeze. Multiple dual-Spark DeepSeek users confirmed free memory stays flat and the host no longer wedges after adding them.

2. Give the allocator real headroom

Set --gpu-memory-utilization to 0.78–0.82 (0.85+ is where many people start seeing the #17-style crashes under agentic bursts).
Keep --max_num_seqs conservative (4–6 at 256k–512k; drop further if you go to 1M).
Cap --max-num-batched-tokens at 4096–8192. Larger values increase per-request pressure on the unified pool.

3. Make OOM recoverable instead of a full host freeze

# Reserve ~2–3 GB so the kernel can still run the OOM killer
sudo sysctl -w vm.min_free_kbytes=2097152   # or 3145728 for ~3 GB

# Turn swap completely off (on UMA, swap turns a clean kill into a long livelock)
sudo swapoff -a

Install and configure earlyoom so it kills the vLLM/Python process early instead of letting the whole box hang:

sudo apt install earlyoom
# Edit /etc/default/earlyoom
EARLYOOM_ARGS="-m 3 -s 10 --prefer 'vllm|python|triton' --avoid 'sshd|systemd|dockerd'"

Optional but highly recommended: install Dropbear SSH on a second port as a tiny recovery channel when OpenSSH dies.

4. Software / driver hygiene

Prefer the 580.x driver series (590 had confirmed memory-leak regressions on GB10).
Use a recent vLLM build that includes the SM12x Triton recompile fix (the per-shape constexpr → runtime change that stopped the host-side RSS leak of 100–200 MB/h). Older 0.26.1-rc1 builds still bleed.
After any heavy session or before long runs, flush page cache if needed:
```
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
```

5. Workload-side habits that reduce the tax

Prefer single-session or low-concurrency agentic work when possible.
Avoid dual weight-copy engines or anything that keeps large extra resident copies.
Monitor free memory / earlyoom logs; treat <5–7 GB free as a warning zone on these boxes.

Expected result after applying the above

Most people who hit the progressive freezes and #17 crashes report the problem drops from “constant under agentic load” to rare or near-zero. The UMA architecture still requires more careful tuning than a discrete 96 GB card, but the combination of UCX fixes + headroom + earlyoom + recent vLLM turns the “constant tax” into something manageable for 256k–512k single-session (and even 1M with lower concurrency).

Start with the two UCX variables + lower utilization + earlyoom/min_free_kbytes — those three changes alone fixed the majority of the reported dual-Spark DeepSeek cases.

alexellis/spark_fixes_grok.md

Select an option

No results found