llama.cpp + CUDA 13.2 on Ubuntu 24.04 with GTX 1650 Ti - setup notes and learnings for 4GB VRAM

llamacpp-learning-setup-summary.md

llama.cpp + CUDA Setup — What I Learned

System: Ubuntu 24.04, GTX 1650 Ti (4GB VRAM)

Hardware Limits

• 4GB VRAM fits Llama 3.2 3B (Q4_K_M) fully on GPU with 4K context
• 8K context = partial CPU offload (slower), 16K+ = mostly CPU
• TinyLlama 1.1B runs easily with full offload
• No tensor cores on GTX 1650 Ti -- warning is cosmetic
• 7B models don't fit at Q4 (4.5 GB), Q2 is degraded quality

Driver and CUDA

• Driver version determines max CUDA toolkit version (nvidia-smi shows it)
• Driver 595.58.03 -> CUDA 13.2
• Install: sudo apt install nvidia-driver-595, reboot
• CUDA toolkit from NVIDIA repo: cuda-toolkit package
• Path: /usr/local/cuda/bin -> add to PATH and LD_LIBRARY_PATH

llama.cpp Build

• cmake .. -DGGML_CUDA=ON -- auto-detects GPU and CUDA
• Build time: ~5-10 minutes on 8 cores
• llama-server needs LD_LIBRARY_PATH pointing to build/bin/ where libllama-common.so.0 lives
• Start: llama-server -m model.gguf --port 8080 -c 4096

Models (downloaded)

Model	File	Size
TinyLlama 1.1B	~/models/tinyllama.gguf	638 MB
Llama 3.2 3B	~/models/llama3.2-3b.gguf	1.9 GB

opencode Config

• Config at ~/.config/opencode/opencode.json
• Cloned from github.com/syntaxhacker/opencode-config.git
• Uses NVIDIA API + MiMo providers (not local llama -- context too small for coding)
• Local llama server can be used for chat but opencode sends 11K+ token requests

Web UI

• http://localhost:8080 -- llama.cpp built-in web interface
• Current model: TinyLlama (switch by changing -m path)

Node.js

• Installed via nvm: v24.15.0 LTS, npm 11.12.1
• Default set in bashrc

Startup

• Script: ~/start-llama.sh
• After reboot: bash ~/start-llama.sh

Git

• SSH key: ~/.ssh/id_ed25519 -- authenticated to GitHub as syntaxhacker

Next Models to Try

• Qwen3.5 4B -- ~2.8 GB, best quality for 4GB VRAM
• Phi-4 Mini -- ~2.5 GB, best for coding