🚀 The 2026 M4 Max Local-First AI Guide (v2 — llama.cpp Edition)

Ditching Ollama: Direct llama.cpp for 2.4x Faster Inference

This document is the updated setup for a high-performance, local AI development environment on an M4 Max (36GB RAM). It replaces the Ollama backend with llama-server (upstream llama.cpp), delivering a 2.4x speedup on the fast-coder slot and 1.9x on the think-brain slot, with correct model templating and full control over every flag.

What changed from v1: Ollama is gone as the inference backend. llama-swap stays as the proxy but now talks directly to llama-server instead of spawning Ollama processes. Every flag you set now actually applies.

🏗️ 1. The Architecture

The setup is a three-layer stack: clients talk to LiteLLM (unified API with fallback), which routes to either llama-swap (local GPU) or cloud APIs. llama-swap hot-swaps models on demand so only one model is in GPU memory at a time.

Your client (VS Code / Kilo Code / Cursor / custom scripts)
        ↓ OpenAI API at :4000
    LiteLLM proxy (routing, fallback, auth)
        ↓                    ↓
  localhost:8080          Dashscope API
  llama-swap              (cloud fallback)
        ↓
  llama-server (llama.cpp)
        ↓
  Metal / Apple GPU (M4 Max)

Why LiteLLM? Provides a single endpoint for all models (local + cloud), automatic fallback when local is busy/down, load balancing, and API key management. Clients don't need to know whether they're hitting local or cloud.

Why llama-swap? Hot-swaps between models (fast-coder vs think-brain) on a single GPU. Only one model is loaded at a time, with TTL-based eviction.

Why not Ollama? Ollama maintains its own internal fork of llama.cpp that has diverged significantly from upstream. In benchmarks on M4 Max with GLM-4.7-Flash, Ollama delivers ~25 t/s where llama-server delivers ~62 t/s — a 2.4x gap. Additionally, Ollama silently ignores environment variables like OLLAMA_KV_CACHE_TYPE in favour of its own global settings, making precise tuning impossible.

🧰 2. Prerequisites

Before you start, install these tools:

# Xcode command line tools (provides clang, make, etc.)
xcode-select --install

# Homebrew (if not already installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# cmake — required to build llama.cpp from source
brew install cmake

# Hugging Face CLI — required to download gated models (GLM-4.7 needs license acceptance)
brew install huggingface-cli
# or: pip install huggingface-hub[cli]

# Python 3.12+ with pip
brew install python@3.12

# uv (fast Python package manager) — used to run LiteLLM
brew install uv

Hardware: Apple Silicon Mac with ≥32GB unified memory. This guide is tested on M4 Max 36GB. M4 Pro 24GB will work with one model at a time but can't fit the larger Qwen3.5-35B at full context.

🛠️ 3. Hardware & OS Tuning

Unlock Wired GPU Memory

Run once per boot (or add to a startup script):

sudo sysctl iogpu.wired_limit_mb=32768

This allows Metal to claim up to 32GB of your 36GB as wired GPU memory. Without it, macOS caps GPU memory conservatively and you'll see slowdowns on large contexts.

Note: The launchctl setenv OLLAMA_* commands from v1 are no longer needed — llama-server receives all flags directly via the config.

Build llama.cpp from Source

The Homebrew build lags behind upstream by several weeks. Build from source to get the latest Metal optimisations and bug fixes:

git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp-src
cd ~/llama.cpp-src

cmake -B build \
  -DGGML_METAL=ON \
  -DGGML_BLAS=ON \
  -DGGML_BLAS_VENDOR=Apple \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)

Build takes ~5 minutes on M4 Max. Keep it current with:

cd ~/llama.cpp-src && git pull && cmake --build build --config Release -j$(sysctl -n hw.logicalcpu)

📦 4. Model Downloads

mkdir -p ~/models
hf auth login  # Required for GLM-4.7 (accept license at huggingface.co/THUDM/GLM-4.7)

# fast-coder: GLM-4.7-Flash — 18GB
hf download unsloth/GLM-4.7-Flash-GGUF \
  GLM-4.7-Flash-Q4_K_M.gguf \
  --local-dir ~/models/

# think-brain: Qwen3.5-35B-A3B — 21GB
hf download unsloth/Qwen3.5-35B-A3B-GGUF \
  Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
  --local-dir ~/models/

# Draft model for speculative decoding (future use): Qwen3.5-2B — 2GB
hf download unsloth/Qwen3.5-2B-GGUF \
  Qwen3.5-2B-Q8_0.gguf \
  --local-dir ~/models/

Memory at rest: Models never co-load. llama-swap evicts based on TTL. Peak GPU usage is ~22GB (think-brain) or ~20GB (fast-coder), well within the 32GB ceiling.

📥 5. Install llama-swap

llama-swap is a lightweight Go proxy that hot-swaps llama-server instances. Install via Homebrew:

brew tap mostlygeek/llama-swap
brew install llama-swap

Verify:

llama-swap --version
# llama-swap version: 198 (or similar)

Create the config directory:

mkdir -p ~/llama-swap

Note: llama-swap is a single static binary with no dependencies. The Homebrew tap tracks stable releases. You can also build from source via go install github.com/mostlygeek/llama-swap@latest if you prefer.

📄 6. llama-swap Config

Save as ~/llama-swap/config-llamacpp.yaml:

healthCheckTimeout: 600
logToStdout: "both"

models:
  "fast-coder":
    # GLM-4.7-Flash: DeepSeek2 MoE architecture, 30B total / 3B active
    # Strongest agentic coding + tool use at this size class
    # n-gram speculative decoding active (no draft model needed, uses context history)
    cmd: "llama-server --model /Users/YOUR_USERNAME/models/GLM-4.7-Flash-Q4_K_M.gguf --port ${PORT} --host 127.0.0.1 --ctx-size 65536 --n-gpu-layers 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 2048 --ubatch-size 512 --threads 4 --parallel 1 --mlock --jinja --no-mmap --repeat-penalty 1.0 --temp 0.7 --top-p 1.0 --min-p 0.01 --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 16 --draft-max 64"
    checkEndpoint: "/health"
    healthCheckEndpoint: "/health"
    ttl: 600

  "think-brain":
    # Qwen3.5-35B-A3B: Hybrid SSM/MoE, 35B total / 3B active
    # Deep reasoning, long context, hybrid thinking mode
    # Note: speculative decoding unsupported for SSM architecture (upstream fix pending)
    cmd: "llama-server --model /Users/YOUR_USERNAME/models/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf --port ${PORT} --host 127.0.0.1 --ctx-size 32768 --n-gpu-layers 99 --flash-attn on --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --threads 4 --parallel 1 --mlock --jinja --no-mmap --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5"
    checkEndpoint: "/health"
    healthCheckEndpoint: "/health"
    ttl: 3600

Apply your username:

sed -i '' "s/YOUR_USERNAME/$USER/g" ~/llama-swap/config-llamacpp.yaml

🚀 7. Startup

You need two processes running: llama-swap (model proxy) and LiteLLM (API gateway).

Terminal 1: llama-swap

#!/bin/bash
# ~/scripts/start-llm.sh

# Unlock GPU memory ceiling
sudo sysctl iogpu.wired_limit_mb=32768

# Start llama-swap
llama-swap --config ~/llama-swap/config-llamacpp.yaml --listen 127.0.0.1:8080

chmod +x ~/scripts/start-llm.sh
~/scripts/start-llm.sh

Terminal 2: LiteLLM

#!/bin/bash
# ~/scripts/start-litellm.sh

source /path/to/your/.venv/bin/activate  # or skip if uvx is available globally
uvx --with websockets litellm --config ~/litellm_config.yaml --port 4000

Or without a venv:

uvx --with websockets litellm --config ~/litellm_config.yaml --port 4000

Tip: Run both in a tmux/screen session or use the launchd approach in Section 16 for auto-start on boot.

🔀 8. LiteLLM Setup (Unified API Gateway)

LiteLLM sits in front of llama-swap and cloud APIs, providing a single OpenAI-compatible endpoint with automatic fallback, load balancing, and auth.

Install

# Option 1: Run via uvx (no permanent install needed)
uvx --with websockets litellm --config ~/litellm_config.yaml --port 4000

# Option 2: Install permanently
pip install 'litellm[proxy]'

Config

Save as ~/litellm_config.yaml:

model_list:
  # ── Local models (via llama-swap on :8080) ──────────────────
  - model_name: local-llama
    litellm_params:
      model: openai/fast-coder
      api_base: http://localhost:8080/v1
      api_key: local-no-key-needed

  - model_name: think-brain
    litellm_params:
      model: openai/think-brain
      api_base: http://localhost:8080/v1
      api_key: local-no-key-needed

  # ── Cloud fallback models ───────────────────────────────────
  - model_name: qwen3.5-flash
    litellm_params:
      model: openai/qwen3.5-flash
      api_base: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
      api_key: "os.environ/DASHSCOPE_API_KEY"  # Set in your shell

  - model_name: qwen-turbo
    litellm_params:
      model: openai/qwen-turbo-latest
      api_base: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
      api_key: "os.environ/DASHSCOPE_API_KEY"

litellm_settings:
  fallbacks:
    - local-llama: [qwen3.5-flash]       # If local GPU is busy → cloud
    - think-brain: [qwen3.5-flash]        # If think-brain is busy → cloud
  request_timeout: 120
  routing_strategy: least-busy

general_settings:
  master_key: sk-litellm-local            # Auth key for the proxy

Key points:

local-llama and think-brain route to llama-swap on port 8080
Cloud models use Dashscope (Alibaba Cloud) — set DASHSCOPE_API_KEY in your environment
fallbacks ensure that if the local GPU is busy, requests automatically route to cloud
routing_strategy: least-busy picks the model with fewest in-flight requests
master_key is the auth token clients use — change it if exposing beyond localhost

Cloud alternatives: Replace the Dashscope models with any OpenAI-compatible provider (OpenRouter, Together, Groq, etc.) by changing api_base and api_key.

Dashboard note: LiteLLM has a web dashboard (/ui) but it requires a PostgreSQL + Prisma backend. For local use, skip it — it's not worth the setup overhead.

⚙️ 9. Key Flag Decisions

Flag	Value	Why
`--n-gpu-layers 99`	99	Offload all layers to Metal. On unified memory, always do this.
`--flash-attn on`	on	Essential at 32K+ context. Without it, attention becomes the bottleneck.
`--cache-type-k`	`q8_0` (fast-coder) / `q4_0` (think-brain)	KV cache quantisation. Saves 2-4GB RAM. q8_0 for quality, q4_0 for the larger model where headroom matters.
`--parallel 1`	1	Single request at a time. Required for speculative decoding compatibility.
`--mlock`	—	Prevents OS from swapping model weights to disk during long sessions.
`--no-mmap`	—	Avoids a macOS hang at model load time.
`--jinja`	—	Enables correct Jinja2 chat templates. Required for tool use and GLM/Qwen thinking mode.
`--repeat-penalty 1.0`	1.0	GLM-specific: default repeat penalty causes looping. Must be 1.0.
`--min-p 0.01`	0.01	GLM-specific: Z.ai's recommended sampling floor.
`--threads 4`	4	Optimal for Apple Silicon. More threads compete with GPU scheduling.
`--spec-type ngram-mod`	ngram-mod	Draftless speculative decoding using context history. Free speed on repeated code patterns.

🧠 10. Model Selection Rationale

fast-coder: GLM-4.7-Flash vs Qwen3.5-9B

Tested both on M4 Max 36GB:

Model	TG t/s	Architecture	Speculative
GLM-4.7-Flash Q4_K_M	62 t/s	DeepSeek2 MoE	n-gram ✅
Qwen3.5-9B UD-Q4_K_XL	48 t/s	Hybrid SSM	❌ rejected

GLM wins on both speed and agentic coding benchmarks (SWE-bench Verified: 59.2% vs ~22% for Qwen3.5 equivalent). The deepseek2 architecture has no SSM layers, which is why n-gram speculative decoding works where it doesn't for Qwen3.5.

think-brain: Qwen3.5-35B-A3B vs Qwen3.5-27B "dense"

The 27B is not actually dense — it has 64 SSM layers vs the 35B-A3B's 40, making it 3.6x slower despite smaller parameter count:

Model	TG t/s	Layers	Speculative
Qwen3.5-35B-A3B UD-Q4_K_XL	59 t/s	40 (sparse MoE)	❌ SSM
Qwen3.5-27B UD-Q4_K_XL	16 t/s	64 (dense SSM)	❌ SSM

The 35B-A3B wins decisively. The MoE's sparse activation (only 3B active per token from 35B total) gives it far more speed than the 27B's dense computation across 64 layers.

📊 11. Benchmarks vs Ollama

Same model (GLM-4.7-Flash), same prompt ("Write a binary search in Python"), 256 completion tokens, M4 Max 36GB:

fast-coder

Backend	Wall clock	Effective TG	Internal TG
Ollama 0.17.7	10.1s	~25 t/s	n/a
llama-server 8430	4.2s	~61 t/s	62 t/s
Speedup	2.4x	2.4x	—

think-brain (Qwen3.5-35B-A3B)

Backend	Wall clock	Internal TG
Ollama 0.17.7	21.5s	~12 t/s
llama-server 8430 (cold)	15.1s	59 t/s
llama-server 8430 (warm)	~5s	59 t/s
Speedup (warm)	~4x	~5x

llama-bench baseline (GLM-4.7-Flash, no server overhead)

| model                          | size    | params | backend  | fa | test  | t/s           |
| deepseek2 30B.A3B Q4_K - Med   | 17.05G  | 29.94B | BLAS,MTL | 1  | pp512 | 959.86 ± 1.48 |
| deepseek2 30B.A3B Q4_K - Med   | 17.05G  | 29.94B | BLAS,MTL | 1  | tg128 | 71.58 ± 1.29  |

🔧 12. Thinking Mode Control

Both models support hybrid thinking/non-thinking modes. Control per-request:

GLM-4.7-Flash

Thinking is on by default. Disable via chat_template_kwargs:

payload = {
    "model": "fast-coder",
    "messages": [...],
    "chat_template_kwargs": {"enable_thinking": False}
}

Or trigger fast mode in the system message: add /no_think context.

Qwen3.5-35B-A3B

Thinking is on by default in the config. Disable per-request:

payload = {
    "model": "think-brain",
    "messages": [{"role": "user", "content": "Your question /no_think"}],
    ...
}

Important: In llama.cpp 8430+, assistant response prefill (</think> tricks) is explicitly blocked when thinking mode is enabled. Use chat_template_kwargs instead.

🧩 13. Client Config

With LiteLLM running, point all clients at http://localhost:4000/v1 with the master key. Clients don't need to know about llama-swap, cloud APIs, or fallback logic — LiteLLM handles it all.

VS Code / Cursor / Continue

{
  "baseUrl": "http://localhost:4000/v1",
  "apiKey": "sk-litellm-local",
  "model": "local-llama"
}

Available model names: local-llama, think-brain, qwen3.5-flash, qwen-turbo

Kilo Code

Point both Kilo-Fast and Kilo-Think profiles at http://127.0.0.1:4000/v1 with API key sk-litellm-local. Use model IDs local-llama and think-brain respectively.

Python / curl

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-litellm-local" \
  -d '{
    "model": "local-llama",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Direct Access (bypass LiteLLM)

If you want to skip LiteLLM and talk to llama-swap directly (no fallback, no auth):

curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "fast-coder", "messages": [{"role": "user", "content": "Hello"}]}'

✅ 14. Verification & Smoke Tests

After starting both llama-swap and LiteLLM, verify the full stack is working:

# 1. Check llama-swap is running
curl -s http://localhost:8080/healthz
# Expected: "ok" or {"status":"ok"}

# 2. Check LiteLLM is running
curl -s http://localhost:4000/health
# Expected: {"status":"healthy"}

# 3. List available models through LiteLLM
curl -s http://localhost:4000/v1/models \
  -H "Authorization: Bearer sk-litellm-local" | python3 -m json.tool
# Expected: list of local-llama, think-brain, qwen3.5-flash, qwen-turbo

# 4. Test local inference (cold start — first request loads the model, ~10-15s)
time curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-litellm-local" \
  -d '{"model": "local-llama", "messages": [{"role": "user", "content": "Write a hello world in Python"}], "max_tokens": 100}' \
  | python3 -m json.tool
# Expected: valid completion, ~15s cold / ~2s warm

# 5. Test fallback (optional — requires DASHSCOPE_API_KEY set)
curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-litellm-local" \
  -d '{"model": "qwen3.5-flash", "messages": [{"role": "user", "content": "ping"}], "max_tokens": 10}'

🔍 15. Troubleshooting

Problem	Cause	Fix
llama-server hangs on model load	macOS mmap bug with large GGUFs	Add `--no-mmap` flag (already in our config)
`port already in use` on 8080	Previous llama-swap still running	`lsof -ti:8080 \| xargs kill`
`port already in use` on 4000	Previous LiteLLM still running	`lsof -ti:4000 \| xargs kill`
LiteLLM returns 401	Wrong or missing API key	Add `-H "Authorization: Bearer sk-litellm-local"`
Slow first response (~15-30s)	Cold model load via llama-swap	Normal — subsequent requests use warm model. TTL keeps it loaded.
`CUDA out of memory` / Metal OOM	Context too large for available RAM	Reduce `--ctx-size` or use more aggressive KV cache quantisation (`q4_0`)
GLM output loops / repeats	Wrong repeat penalty	Ensure `--repeat-penalty 1.0` for GLM-4.7-Flash
Thinking mode ignored	Using old llama.cpp build	Update llama.cpp to build 8430+. Use `chat_template_kwargs` not prefill tricks.
LiteLLM fallback not triggering	Timeout too short / model name wrong	Check `model_name` matches exactly in config. Increase `request_timeout`.
`uvx: command not found`	uv not installed	`brew install uv` or `pip install uv`
Cloud models 403/429	API key invalid or rate limited	Verify `DASHSCOPE_API_KEY` env var is set and valid

🔄 16. Auto-Start on Boot (Optional)

Create launchd agents so llama-swap and LiteLLM start automatically on login:

llama-swap

cat > ~/Library/LaunchAgents/com.local.llama-swap.plist << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.local.llama-swap</string>
  <key>ProgramArguments</key>
  <array>
    <string>/opt/homebrew/bin/llama-swap</string>
    <string>--config</string>
    <string>/Users/YOUR_USERNAME/llama-swap/config-llamacpp.yaml</string>
    <string>--listen</string>
    <string>127.0.0.1:8080</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
  <key>StandardOutPath</key>
  <string>/tmp/llama-swap.log</string>
  <key>StandardErrorPath</key>
  <string>/tmp/llama-swap.err</string>
</dict>
</plist>
EOF

sed -i '' "s/YOUR_USERNAME/$USER/g" ~/Library/LaunchAgents/com.local.llama-swap.plist
launchctl load ~/Library/LaunchAgents/com.local.llama-swap.plist

GPU memory unlock: The sudo sysctl iogpu.wired_limit_mb=32768 command requires root and doesn't persist across reboots. Add it to a root-level launchd daemon or run it manually after each reboot.

⚠️ 17. Known Limitations & What's Coming

Item	Status	Notes
Speculative decoding (Qwen3.5)	⏳ Pending	Blocked by hybrid SSM architecture. PR #20075 merged upstream, not yet stable.
N-gram speculative decoding (GLM)	✅ Active	~2 t/s marginal improvement on short prompts; better on long coding sessions with repeated patterns.
Draft model for GLM	❌ No candidate	No small GLM-family model exists. GLM-4.5-Air is 106B — too large.
Metal Tensor API	❌ M4 only	`tensor API disabled for pre-M5` — this feature is M5/A19 only, giving another 20-30% on next gen hardware.
Multi-turn prompt caching	✅ Active	Context checkpointing working in build 8430. Warm requests return in ~4s.

📊 18. Memory Reference

State	GPU Memory
Idle	~0 GB
fast-coder loaded (GLM-4.7-Flash)	~20 GB (weights 17GB + KV 2GB + compute 310MB)
think-brain loaded (Qwen3.5-35B-A3B)	~22 GB (weights 21GB + KV 180MB + RS 63MB + compute 489MB)
Danger zone (swapping begins)	32 GB+

Mission: Maximum intelligence without latency. Privacy without compromise.

adamteale/M4 Max Local-First AI Guide (v2 — llama.cpp Edition).md