Running a Local LLM with Ollama — Install & Learning Guide

A hands-on starter guide for experimenting with a small open-weight LLM on your Mac. Written for Apple Silicon (M-series). Everything here runs locally, offline, and costs nothing to run.

0. First, a clear-eyes note on "training"

What you're doing	What it changes	Tool	Difficulty
Running a model	Nothing — pure inference	Ollama `run`	Trivial
Customizing behavior (system prompt, personality, temperature, context size)	How the model behaves — not what it knows	Ollama Modelfile	Easy
RAG (feeding it your own documents at query time)	What it can reference, per query	Ollama + a small app/library	Moderate
Fine-tuning (QLoRA/LoRA)	What the model knows — actually adjusts weights	Separate tools (Unsloth, Axolotl, MLX), then import into Ollama	Advanced

For learning, do the first two first. True fine-tuning is a separate pipeline (covered lightly in Section 7) and you almost never need it to start understanding how LLMs work. Most of the "aha" moments come from Modelfiles and parameter tweaking, which cost you nothing and take minutes.

1. The quickest, safest toolset

Everything below is open-source or first-party, widely used, and installs cleanly on macOS:

Ollama — the core. Downloads, manages, and runs open-weight models with one command. Handles Apple Metal GPU acceleration automatically, no config.
Homebrew (optional but recommended) — the standard macOS package manager, makes install/updates clean and reversible.
A terminal — the built-in Terminal app is fine.
(Optional) Open WebUI — a local ChatGPT-style web interface for Ollama if you'd rather click than type. Runs in Docker.

Safety notes:

Ollama runs entirely on your machine. Prompts and models never leave your computer.
By default Ollama listens only on localhost (127.0.0.1) — not exposed to your network. Leave it that way unless you deliberately want remote access.
Only pull models from Ollama's official library (ollama.com/library) or verified publishers. Model files are data, but treat unknown third-party GGUF files with the same caution as any download.

2. Install Ollama (macOS, ~5 minutes)

Option A — Homebrew (recommended, easiest to update/remove):

brew install ollama

Option B — Direct download: grab the app from ollama.com/download, unzip, and drag Ollama to your Applications folder. It auto-starts.

Verify it's working:

ollama --version

If you installed via Homebrew and the command isn't found, start the background service once:

ollama serve

(Leave that running in one terminal tab, or let the app run in the background, then use a second tab for the commands below.)

3. Run your first model (~2 minutes + download time)

Start small. A 3–4B model is plenty to learn on and runs comfortably on 8GB+ RAM; a 7–9B model is noticeably sharper if you have 16GB+.

# A small, capable starting model — check the exact current tag at ollama.com/library
ollama run llama3.2:3b

Tag names drift over time. Model families update (e.g. newer Llama, Qwen, and Gemma releases). Before pulling, glance at ollama.com/library for the current small-model tags. Good small choices to look for: a Llama 3.x 3B, a Qwen 3B–8B, a Gemma small, or Phi — all lightweight and beginner-friendly.

The first run downloads the model (a few GB), then drops you into a chat prompt. Type a question. Type /bye to exit.

Useful commands:

ollama list           # models you've downloaded
ollama pull <model>   # download without running
ollama rm <model>     # delete a model to free disk space
ollama ps             # what's currently loaded in memory

Pick a size that fits your RAM (rough guide):

Your Mac RAM	Comfortable model size (Q4 quantization)
8 GB	3–4B
16 GB	7–9B
32 GB+	13–14B, or a small MoE

4. Learn the knobs — parameters that shape output

The fastest way to build intuition is to change one setting and watch the output change. Inside a running session you can set these live:

/set parameter temperature 0.2   # lower = focused/deterministic; higher = creative/random
/set parameter num_ctx 4096       # context window: how much text it can "remember" at once
/set parameter top_p 0.9          # nucleus sampling — another creativity dial
/set parameter seed 42            # fix the seed for reproducible outputs

Experiments worth doing:

Ask the same question at temperature 0 vs temperature 1.2. Notice how low temp is repeatable and high temp is varied (sometimes wrong/weird).
Set seed to a fixed number and re-run the same prompt at temp 0 — you'll get identical output. This teaches you that the model is deterministic given fixed inputs; "randomness" is a sampling choice, not magic.
Shrink num_ctx very small and give it a long passage, then ask about the beginning — watch it "forget." This is the context window made tangible.

5. Build your first custom model — a Modelfile

A Modelfile is a tiny text file (Dockerfile-style) that bakes a base model + a system prompt + parameters into a new named model. This is the single best learning exercise.

Create a file named Modelfile (no extension):

FROM llama3.2:3b

# Bake in a persistent persona / instructions
SYSTEM """
You are a concise, patient tutor explaining how large language models work.
Answer in plain language. When you use a technical term, define it in one short clause.
Keep answers under 200 words unless asked to go deeper.
"""

# Bake in default parameters
PARAMETER temperature 0.4
PARAMETER num_ctx 4096

Build and run it:

ollama create llm-tutor -f ./Modelfile
ollama run llm-tutor

You now have a reusable custom model. Notice: you changed how it behaves without touching a single weight — this is the key conceptual distinction from fine-tuning. Try editing the SYSTEM block (make it a pirate, a Socratic questioner, a code reviewer) and rebuilding to feel how much personality comes from the prompt alone.

6. (Optional) A friendlier interface

If you'd rather use a browser UI than the terminal, Open WebUI gives you a local chat interface wired to Ollama. It needs Docker Desktop installed, then:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000. It stays local, and it's handy for saving conversations and switching between your models.

7. When you're ready: actual fine-tuning (a preview, not required)

Only pursue this once Modelfiles and RAG feel limiting — which for pure learning may be never. The honest sequence of "make the model better at my task":

Prompt/Modelfile first — free, instant, gets you 80% of the way.
RAG — retrieve your own documents at query time; best when you need the model to reference specific knowledge (like your own notes) without retraining.
Fine-tuning (QLoRA) — actually adjust weights. Worth it only with good training data, a clear quality gap, and a repetitive high-volume task.

The typical fine-tuning path on a Mac or a rented GPU:

Prepare a dataset (prompt/response pairs, usually JSONL).
Train a LoRA/QLoRA adapter using Unsloth, Axolotl, or Apple's MLX framework. This produces a small adapter file, not a whole new model — cheaper and faster than full training.
Import the adapter into Ollama via a Modelfile using the ADAPTER directive alongside FROM.

This is a genuine project (hours to days, some GPU cost or a long Mac run), which is exactly why it's last. Start with Sections 2–5.

8. Suggested learning path (a weekend)

Install Ollama, run a 3B model, chat with it. (Section 2–3)
Play with temperature and seed; make output repeatable then random. (Section 4)
Shrink the context window and watch it forget — feel what "context" means. (Section 4)
Write a Modelfile, create llm-tutor, then remix its persona a few times. (Section 5)
Try two different model families of the same size (e.g. a Llama vs a Qwen) on the same prompts — notice how base models differ. (Section 3)
Only if curious: read one RAG tutorial, then one QLoRA tutorial, before deciding whether fine-tuning is worth it. (Section 7)

By step 4 you'll have a working mental model of inference, sampling, context windows, and prompt-vs-weights — which is most of the conceptual core of how LLMs actually work.

Quick command cheat sheet

brew install ollama              # install
ollama run llama3.2:3b           # download + chat (check current tag first)
ollama list                      # list downloaded models
ollama pull <model>              # download only
ollama rm <model>                # delete a model
ollama ps                        # what's loaded in memory
ollama create <name> -f Modelfile# build a custom model
/set parameter temperature 0.2   # (inside a session) change a knob
/bye                             # exit a chat session

Model tags, sizes, and versions evolve; verify current small-model names at ollama.com/library before pulling. This guide is for personal learning.

soverby/Ollama-Local-LLM-Guide.md

Select an option

No results found