DiegoRBaquero/readme.md

Last active September 24, 2026 20:30

Star (17) You must be signed in to star a gist
Fork (1) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/DiegoRBaquero/f53ab22ae978226c86158a60dad8199d.js"></script>
Save DiegoRBaquero/f53ab22ae978226c86158a60dad8199d to your computer and use it in GitHub Desktop.

Download ZIP

Running Claude Code with a local LLM

Raw

Running Claude Code with a local LLM

1. Download and install oMLX (macOS-native MLX server with smart caching)

https://github.com/jundot/omlx/releases

2. Download the model

Go to model downloader

Multiple options, depending on your RAM

35B parameters with 3 billion active:

unsloth/Qwen3.6-35B-A3B-UD-MLX-3bit - 17.4 GB (36GB+ RAM ideal)
unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit - 21.6 GB (48GB+ RAM ideal)
unsloth/Qwen3.6-35B-A3B-MLX-8bit - 37.7GB GB (64GB+ RAM ideal)

27B billion parameters

unsloth/Qwen3.6-27B-UD-MLX-4bit - 26.2GB (48GB+ RAM ideal)
unsloth/Qwen3.6-27B-UD-MLX-6bit - 30.5GB (64GB+ RAM ideal)
unsloth/Qwen3.6-27B-UD-MLX-8bit - 34.7GB (64GB+ RAM ideal)

3. Configure oMLX settings

Go to model settings
Pin and default model to the downloaded one
Open the model's settings
Enable TurboQuant KV Cache in 3.5-bit
Go to global settings
Turn on Fallback to Default Model
Set Hot Cache Limit (In-Memory Cache) to 10%
Set Cold Cache Limit (SSD Cache) to 10%
Increase Max Context Window to 256000
Increase Max Tokens to 64000
Save

4. Configure Claude Code

Add "CLAUDE_CODE_ATTRIBUTION_HEADER": "0" in env key inside ~/.claude/settings.json (Ref)

Example:
```
  {
    "env": {
      "CLAUDE_CODE_ATTRIBUTION_HEADER": "0"
    }
  }
```

5. Configure oMLX Claude's settings

Go to Dashboard
Scroll down to Claude Code with oMLX
Set Qwen3.6-35B-A3B-UD-MLX-4bit for all three tiers
Enable Context Scaling for Claude Code and set Target Context Size to 64000
Run the displayed Command
Optional for lighter work: Add --bare

6. Share you findings and optimizations!

imamin01 commented Jun 18, 2026

I am running a "Pro M5 Pro" + 24GB. Which coding model would be suitable with oMLX + ClaudeCode ? TIA

christ-off commented Jun 22, 2026

@imamin01 have a look at real user benchmarks
https://omlx.ai/benchmarks?sort=memory_gb&order=asc&context=65536&page=50

yin-bro commented Sep 10, 2026

I am running a "Pro M5 Pro" + 24GB. Which coding model would be suitable with oMLX + ClaudeCode ? TIA

Model: Qwen3.5-9B-mlx-lm-mxfp4

Hermes Agent: https://hermes-agent.nousresearch.com/docs/guides/local-llm-on-mac

yin-bro commented Sep 10, 2026

I am running a "Pro M5 Pro" + 24GB. Which coding model would be suitable with oMLX + ClaudeCode ? TIA

Model: Qwen3.5-9B-mlx-lm-mxfp4

Hermes Agent: https://hermes-agent.nousresearch.com/docs/guides/local-llm-on-mac

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment