| Component | Details |
|---|---|
| Host type | QEMU/KVM virtual machine |
| CPU | Intel Core (Broadwell, no TSX, IBRS) — 8 vCPUs (8 sockets × 1 core × 1 thread) |
| RAM | 31 GB total, ~10 GB available |
| GPU | Cirrus Logic GD 5446 (emulated VGA — no CUDA, no GPU inference) |
| OS | Ubuntu 24.04.4 LTS (Noble Numbat) |
| Kernel | 6.8.0-111-generic |
| Ollama | v0.23.1 |
| Inference backend | CPU-only (llama.cpp via Ollama) |
Note: The Cirrus Logic GD 5446 is QEMU's default emulated VGA adapter. It provides no compute capability. All inference runs on the host CPU.
| Model | Params | Quantization | Size on Disk |
|---|---|---|---|
llama3.2:latest |
3.2B | Q4_K_M | ~1.9 GB |
deepseek-coder-v2:16b |
15.7B | Q4_0 | ~8.3 GB |
gpt-oss:20b |
20.9B | MXFP4 | ~12.9 GB |
Prompt: "Write a haiku about benchmarking." (33 prompt tokens, 14 generated tokens)
| Metric | Value |
|---|---|
| Prefill speed | 63.5 tok/s |
| Decode speed | 28.0 tok/s |
| Time to first token (TTFT) | 519.9 ms |
| Total wall time | 4,097.7 ms |
prompt_tps = r['prompt_eval_count'] / (r['prompt_eval_duration'] / 1e9)
decode_tps = r['eval_count'] / (r['eval_duration'] / 1e9)Tokens per second alone is insufficient. There are two distinct phases:
| Metric | What it measures | Bottleneck | Matters most for |
|---|---|---|---|
| TTFT (time to first token) | Prompt/prefill processing | Compute (FLOPS) | Interactive latency |
| Decode tok/s | Token generation throughput | Memory bandwidth | Response speed, batch workloads |
An H100 wins on both axes but for different reasons: prefill is compute-bound (H100 has 3,958 TFLOPS BF16), decode is memory-bandwidth-bound (H100 HBM3: 3.35 TB/s).
| Metric | CPU (current) | H100 (estimated) | Speedup |
|---|---|---|---|
| Decode tok/s | ~28 | ~1,500–3,000+ | ~50–100× |
| TTFT | ~520 ms | ~5–20 ms | ~25–100× |
Estimates based on publicly reported H100 llama.cpp benchmarks for 3–7B models.
| Tool | Best for | Notes |
|---|---|---|
llama-bench (llama.cpp) |
Apples-to-apples hardware comparison | Tests pp (prefill) and tg (decode) at varying batch sizes; bypasses HTTP layer |
ollama run --verbose |
Quick spot checks | Same timing fields as the API response |
| LLMPerf | Concurrent load testing | Measures latency + throughput under multiple simultaneous requests |
| MLPerf Inference | Formal industry comparison | Overkill for single-node; used by cloud providers |
# Install llama.cpp, then:
llama-bench -m model.gguf -p 512 -n 128 -r 5
# Compare pp (prompt processing) and tg (token generation) columns- No GPU offload — all matrix multiplications run on CPU cores
- Emulated vCPUs — Broadwell vCPUs in a VM have lower memory bandwidth than bare metal
- Memory pressure — 20 GB of 31 GB RAM already in use; the 12–13 GB
gpt-oss:20bmodel nearly saturates available RAM
Even moving to a bare-metal CPU (e.g., AMD EPYC with high memory bandwidth) would yield a meaningful improvement before reaching for an H100.