Skip to content

Instantly share code, notes, and snippets.

@mbohun

mbohun/NOTES.md

Created June 6, 2026 10:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mbohun/844ac73f1ab677f8d4a4aa3a10102d9c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mbohun/844ac73f1ab677f8d4a4aa3a10102d9c to your computer and use it in GitHub Desktop.
Download ZIP
AI models Ollama llama.cpp
Raw
NOTES.md

AI models for local deployment

Company Model(s) Parameters Local Deployment Method License
DeepSeek DeepSeek-V3, DeepSeek-R1 Various Ollama, vLLM, llama.cpp Open weights
Mistral AI Ministral 3, Devstral Small 2 14B, 24B Ollama, vLLM Apache 2.0
Google (DeepMind) Gemma 4 12B 12B Ollama, vLLM, SGLang, MLX, llama.cpp, LiteRT-LM Apache 2.0
Meta Llama 3 (8B Instruct, 70B) 8B, 70B Ollama, llama.cpp Llama license
IBM Granite 4.0 (series) 350M to 3B+ Ollama, vLLM Apache 2.0
xAI Grok 2 270B llama.cpp (GGUF quantized) Grok Community License
H Company Holo 3.1 (vision-language for agents) 0.8B, 4B, 9B, 35B-A3B GGUF, FP8, NVFP4 quantizations Apache 2.0
Alibaba (Qwen) Qwen 3.5 family 0.8B to 72B Ollama, vLLM, llama.cpp Apache 2.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment