Listing below the latest (popular) releases as of June 2026
Download from hugging face:
| Min RAM | Type | Model | Link |
|---|---|---|---|
| 32 | MOE | Qwen3.6 A3B 35B | https://huggingface.co/Qwen/Qwen3.6-35B-A3B |
| 32 | MOE | Gemma4 A4B 26B | https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized |
| 32 | Dense | Qwen3.6 27B | https://huggingface.co/Qwen/Qwen3.6-27B |
| 32 | Dense | Gemma4 31B | https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized |
| 16 | Dense | Qwen3.5 9B | https://huggingface.co/Qwen/Qwen3.5-9B |
| 16 | Dense | Gemma4 12B | https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-unquantized |
| 8 | MOE | LFM2.5 A1B 8B | https://huggingface.co/LiquidAI/LFM2.5-8B-A1B |
| 8 | Dense | Gemma4 E4B | https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-unquantized |
| 4 | Dense | Bonsai 8B | https://huggingface.co/prism-ml/Bonsai-8B-gguf |
| 4 | Dense | LFM2.5 1.2B | https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking-GGUF |
RAM min numbers are recommmended to run a Q4 quant with sufficient space for long context and rest of your apps, system needs
There are several more options, you can find what fits at:
- https://github.com/AlexsJones/llmfit
- https://llmfit.io/ (different project)
- https://llmfitcheck.com/
llama.cpp: https://github.com/ggml-org/llama.cpp
Others:
- MLX: Apple Silicon native inference engine
- Lemonade: AMD native inference engine
- OpenWebUI: https://github.com/open-webui/open-webui
- OpenWebUI Desktop: https://github.com/open-webui/desktop
Others:
- LM Studio: https://lmstudio.ai/
- Msty: https://msty.ai/
Pi: https://pi.dev/
To connect to local models, place this in .pi/agent/models.json
{
"providers": {
"name-of-your-server": {
"baseUrl": "http://localhost:port/v1",
"api": "openai-completions",
"apiKey": "nonerequired",
"models": [
{
"id": "model-name-in-inference-engine",
"name": "Pretty Model Name",
"reasoning": true,
"input": [
"text"
],
"contextWindow": 128000,
"maxTokens": 32000,
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
}
}
]
}
}
}Others:
- Claude Code: https://code.claude.com/docs/en/overview
Using Claude Code with local LLM:
Place this in .claude/settings.json
{
"env": {
"ANTHROPIC_BASE_URL": "http://localhost:port",
"ANTHROPIC_MODEL": "qwen3.6:27b",
"ANTHROPIC_SMALL_FAST_MODEL": "qwen3.6:27b",
"ANTHROPIC_API_KEY": "sk-no-key-required",
"ANTHROPIC_AUTH_TOKEN": "",
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
"DISABLE_COST_WARNINGS": "1",
"CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
"CLAUDE_CODE_DISABLE_1M_CONTEXT": "1",
"CLAUDE_CODE_MAX_OUTPUT_TOKENS": "32768",
"CLAUDE_CODE_AUTO_COMPACT_WINDOW": "65536",
"CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "90",
"CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS": "1",
"CLAUDE_CODE_DISABLE_FAST_MODE": "1",
"CLAUDE_CODE_MAX_RETRIES": "3",
"CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY": "1",
"DISABLE_TELEMETRY": "1",
"CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY": "1",
"ENABLE_TOOL_SEARCH": "auto",
"DISABLE_AUTOUPDATER": "1",
"DISABLE_ERROR_REPORTING": "1",
"DISABLE_FEEDBACK_COMMAND": "1"
}
}
- llama-swap: inference engine orchestration - https://github.com/mostlygeek/llama-swap
- Homebrew (for mac): package installer - https://brew.sh/
- Local AI news: https://old.reddit.com/r/LocalLLaMA/