What was done

~19 tok/s decode · ~60 tok/s prefill · ~30 GB VRAM · 256k context

Poolside’s Laguna S 2.1 is a 118B total / ~8B active MoE coding model (256 experts, top-10 + shared, hybrid SWA). Official GGUFs: poolside/Laguna-S-2.1-GGUF.

People have been showing hybrid llama.cpp runs on 24 GB cards. That works — but the default “all experts on CPU” placement leaves a lot on the table. On a 5090 (32 GB) the win is packing full layers (experts included) into VRAM with auto-fit, not pure --cpu-moe.

This is a measured recipe from one box, not marketing.

Hardware


GPU	NVIDIA GeForce RTX 5090 (32 GB, SM_120 / Blackwell)
RAM	62 GB system + 128 GB swap
OS	Ubuntu, CUDA 13.1
Model	`laguna-s-2.1-Q4_K_M.gguf` (~75 GB on disk)
Runtime	Poolside `llama.cpp` branch `laguna`

You need ~50–64+ GB system RAM (more is happier). The GGUF alone is ~75 GB; whatever doesn’t fit in VRAM stays on the host.

The placement mistake (and the fix)

Slow path — all experts on host

llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -c 262144 -ngl 999 --cpu-moe \
  -fa on --no-mmap -t $(nproc) -b 4096 -ub 4096 \
  -ctk q8_0 -ctv q8_0

Attention/non-expert on GPU, every expert on CPU. Fits easily (~18 GB VRAM) but decode/prefill are host-bound.

Fast path — pack full layers into VRAM (use this)

llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -c 262144 \
  -ngl auto --fit on --fit-target 2048 \
  -fa on --jinja --no-mmap \
  -t $(nproc) -b 4096 -ub 4096 \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.7 --top-p 0.95 --top-k 20 \
  --host 0.0.0.0 --port 8095

Do not set -ngl 999 / all without --cpu-moe. That tries to put the entire 75 GB on the GPU and OOMs.
-ngl auto + --fit fills free VRAM with as many full layers (including experts) as will fit and leaves the rest on host.

--fit-target 2048 leaves ~2 GB free (display / driver headroom on a desktop).

Measured results (same prompts, same box)

Short coding prompts, thinking off, temp 0.2.

Mode	VRAM	Swap (peak class)	Prefill (cold)	Decode (cold)	Decode (warm)
`--cpu-moe` + `-ngl 999`	~18 GB	~60 GB	1.3 t/s	8.2 t/s	12.8 t/s
`-ngl auto` + `--fit`	~30 GB	~22 GB	~58 t/s	~19 t/s	~18 t/s

Rough speedups on this machine:

Prefill: ~40–45×
Decode: ~1.5–2× → lands in a fully usable interactive range (~18–19 tok/s)

GPU util during decode went from single-digit % to ~15%+ — still hybrid, but no longer “CPU thrashing while the 5090 naps.”

Correctness spot-checks (merge sorted lists, longest palindrome, binary-search infinite-loop fix) all passed.

Build (CUDA / SM_120)

Official Laguna support is on Poolside’s fork (upstream PR: ggml-org/llama.cpp#25165). Stock mainline may not load architecture = laguna yet.

git clone --branch laguna --single-branch \
  https://github.com/poolsideai/llama.cpp.git
cd llama.cpp

# One-liner fix seen on GCC 15: std::isfinite needs <cmath>
# (if build fails in common/speculative.cpp — add #include <cmath>)

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DGGML_NATIVE=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build -j$(nproc) --target llama-server
# binary: build/bin/llama-server  (+ shared libs in same dir)
export LD_LIBRARY_PATH="$PWD/build/bin${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

Download weights:

hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-Q4_K_M.gguf \
  --local-dir ./Laguna-S-2.1-GGUF

Optional DFlash draft (~2.2 GB) for speculative decode after the base path is stable:

hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-DFlash-BF16.gguf \
  --local-dir ./Laguna-S-2.1-GGUF

Minimal smoke test

curl -sS http://127.0.0.1:8095/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role":"user","content":"Write merge_sorted(a,b) in Python, O(n+m). Code only."}],
    "temperature": 0.2,
    "max_tokens": 400,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content, .timings'

Watch:

watch -n1 'nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv; free -h'

Healthy packing run: ~30 GB VRAM, moderate swap (not 60+ GB), decode ~15–20 t/s.

Why this works

MoE sparsity — only ~8B params active per token, but all expert weights must be addressable.
Hybrid SWA — 36/48 layers sliding-window (512) + 8 KV heads → 256k context is cheap on KV.
Layer packing > expert exile — putting some full MoE layers on the GPU beats keeping all experts on the host when you have 32 GB.
--no-mmap — forces resident weight pages (helps once loaded; needs RAM/swap budget).
Large -b / -ub — prefill batching (4096) matters a lot for prompt throughput.

NVFP4 (~71 GB) is great on multi-GPU / 128 GB unified (DGX Spark recipe). It does not magically fit a 32 GB card full-GPU. GGUF hybrid is the single-consumer-GPU path.

Caveats (read these)

62 GB RAM is tight. Expect some swap. 96–128 GB host RAM would be much smoother (closer to viral “1k prefill” demos).
Agent UIs (e.g. full tool schemas) can send 5–10k+ token system prompts. At ~60 t/s prefill that’s still 1–3 minutes to first token on a cold session — not a hang. Same-session prefix cache helps a lot after turn 1.
Thinking mode can burn hundreds–thousands of tokens before visible answer. For latency tests: "chat_template_kwargs": {"enable_thinking": false}.
GGUF ships configured for 256k. Native training goes to 1M; Poolside documents YaRN overrides and quality caveats for >256k.
First load of 75 GB can take ~1 minute (disk + page-in) before /v1/models is ready even if /health answers early.
Desktop safety: leave a couple GB VRAM free (--fit-target 2048) so the display doesn’t wedge under memory pressure.

4090 / 3090 note

Same recipe applies with less headroom:

Prefer -ngl auto --fit over pure --cpu-moe if anything fits.
Expect lower layer count on GPU → closer to the slow column, still better than all-experts-host if fit places any expert layers on-device.
Stay under VRAM ceiling; don’t force -ngl 999 without fit.

TL;DR

Laguna S 2.1 Q4_K_M + poolside llama.cpp (laguna branch)
  -ngl auto --fit on --fit-target 2048
  -fa on --no-mmap -b 4096 -ub 4096 -ctk q8_0 -ctv q8_0
  -c 262144

RTX 5090 32GB + 62GB RAM:
  ~30 GB VRAM · ~18–19 tok/s decode · ~40–60 tok/s prefill
  vs all-experts-on-CPU: ~45× prefill, ~2× decode

The VRAM limit isn’t fake — but putting the right tensors in it matters more than the “all MoE on CPU” cheat code. Pack full layers with auto-fit; keep the rest on host. That’s what made 118B-class agentic coding actually feel usable on one 5090. Big win at any rate.

solatticus/poolside_118B_5090RTX.md

Select an option

No results found