| Model | Strength | Size | Command |
|---|---|---|---|
| DeepSeek-R1 70B | Reasoning, analysis, complex problems | 42GB | ollama run deepseek-r1:70b |
| Qwen3 32B | Fast general-purpose, good at following instructions | 20GB | ollama run qwen3:32b |
| Gemma 3 27B | Natural writing, prose, tone-matching | 17GB | ollama run gemma3:27b |
All three total 79GB -- you can keep two loaded simultaneously with room to spare. DeepSeek works best solo given its size.
# Install Ollama if you haven't
brew install ollama
# Pull all three
ollama pull deepseek-r1:70b
ollama pull qwen3:32b
ollama pull gemma3:27b
# Start chatting
ollama run qwen3:32bAdvantages:
- Zero cost per token -- run as many queries as you want
- Fully offline -- works on planes, in the mountains, wherever
- Complete privacy -- nothing leaves your machine
- No rate limits, no content filters, no "I can't help with that"
Trade-offs:
- These are strong but not frontier-level -- think GPT-4-class, not Claude Opus
- Slower than API calls -- expect 10-30 tokens/sec depending on model and prompt length
- No tool use, no web search, no file reading -- you paste everything in
- Quality degrades as the conversation gets long
Local models lose coherence faster than cloud models as context fills up.
For a new topic, exit (/bye) and start a new session. Don't try to have
a 50-message conversation.
Don't use DeepSeek for simple tasks -- it's slower and will "overthink." Use Qwen3 for quick stuff, DeepSeek when you actually need it to reason through something, Gemma when you care about how the output reads.
Local models are more sensitive to vague prompts than cloud models. "Summarize this" will get you a wall of text. "Summarize this in 3 bullet points, one sentence each" gets you what you want.
These models can't read files, URLs, or attachments. Copy-paste the relevant text into the chat. For long documents, paste the section you care about rather than the whole thing.
ollama run gemma3:27b --system "You are a concise writing assistant. Keep all responses under 200 words unless asked otherwise."Ollama serves an OpenAI-compatible API on localhost:11434. Any tool that supports OpenAI's API can point at your local models:
Base URL: http://localhost:11434/v1
API Key: ollama (or anything -- it's not checked)
Model: qwen3:32b
This works with LM Studio, Open WebUI, and most AI-powered apps that let you configure a custom endpoint.
Default temperature works fine for most tasks. For creative writing, bump it up. For factual/structured output, drop it:
ollama run gemma3:27b --temperature 0.3If you prefer a ChatGPT-style interface, install LM Studio from https://lmstudio.ai. It can load models from Ollama's store directly and gives you a cleaner experience for longer conversations. It also lets you tweak parameters (temperature, top-p, context length) with sliders instead of CLI flags.
With 128GB unified memory, you have room to play:
| Loaded Models | RAM Used | Headroom |
|---|---|---|
| Qwen3 32B + Gemma 27B | ~37GB | Plenty for everything else |
| DeepSeek 70B alone | ~42GB | Comfortable |
| All three loaded | ~79GB | Tight but works |
Ollama auto-unloads models after 5 minutes of inactivity by default. To keep a model loaded longer:
# Keep loaded for 1 hour
curl http://localhost:11434/api/generate -d '{"model": "qwen3:32b", "keep_alive": "1h"}'
# Keep loaded until you manually unload
curl http://localhost:11434/api/generate -d '{"model": "qwen3:32b", "keep_alive": -1}'If you see model tags like q4_K_M or q8_0, that's quantization --
compressing the model to use less memory at some quality cost.
| Quant | Quality | Size | When to Use |
|---|---|---|---|
| Q8 | Near-original | Large | When it fits in memory |
| Q6_K | Very good | Medium-large | Sweet spot for big models |
| Q4_K_M | Good | Medium | Default for most Ollama models |
| Q3_K | Noticeable loss | Small | When nothing else fits |
| Q2_K | Significant loss | Smallest | Last resort |
The models recommended above are already at sensible defaults (Q4 for DeepSeek 70B, Q4 for Qwen3 32B, Q4 for Gemma 27B). With 128GB you could pull higher quants if you want slightly better quality:
ollama pull qwen3:32b-q8_0 # 34GB, better quality
ollama pull gemma3:27b-q8_0 # 28GB, better quality| What | Command |
|---|---|
| List downloaded models | ollama list |
| Remove a model | ollama rm codellama:34b |
| See what's loaded in memory | ollama ps |
| Pull a newer version | ollama pull qwen3:32b |
| Run with custom context length | ollama run qwen3:32b --num-ctx 16384 |
| Check Ollama version | ollama --version |
-
Unsloth fine-tunes: If you find a fine-tuned model on HuggingFace that was trained with Unsloth, you can run it locally if there's a GGUF version available. Download the GGUF and create a custom Ollama model:
# Create a Modelfile echo 'FROM ./your-model.gguf' > Modelfile ollama create my-custom-model -f Modelfile ollama run my-custom-model
-
Open WebUI: Self-hosted ChatGPT-like interface with conversation history, multiple model support, and document upload. Runs in Docker:
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main
Then open http://localhost:3000 and point it at your Ollama instance.
-
Higher quants: With your RAM headroom, experiment with Q8 versions of Qwen3 and Gemma for better output quality.