Local LLM benchmark on NVIDIA Jetson Orin Nano Super (8GB) — agentic tool-calling, not tok/s. 8 models x 5 runs. The best agentic model is 1.9GB and also the fastest.

agentic_bench.py

#!/usr/bin/env python3
"""Agentic tool-calling benchmark for small models on a Jetson Orin Nano Super.

Differences from the July/August harness this replaces:

1. **Talks to Ollama's /api/chat directly** instead of driving an agent CLI.
   The question is which *model* is capable; routing through an agent runtime
   measures the runtime's scaffolding as much as the model.
2. **N runs per model, reporting a rate.** The previous harness ran each task
   once. Its own notes say a 2B model scored 2/5 and 3/5 on identical clean
   runs — so single-run scores were never evidence. Anything reported here is
   k/N.
3. **Records why a task failed**, not just that it did: no tool call at all,
   wrong tool, malformed arguments, or a claim of success with nothing on disk.
   Those need different fixes.

Every check has an oracle that a plausible-sounding answer cannot satisfy —
an unguessable token, a file on disk, a value only the hardware knows.

Usage:  python3 agentic_bench.py --models a,b,c --runs 5
"""
import argparse, json, os, random, re, shutil, string, subprocess, time, urllib.request

OLLAMA = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434")
WORK = os.path.expanduser("~/agentic-bench")

TOOLS = [
    {"type": "function", "function": {
        "name": "read_file",
        "description": "Read a file from disk and return its contents.",
        "parameters": {"type": "object", "properties": {
            "path": {"type": "string", "description": "Absolute path to read"}}, "required": ["path"]}}},
    {"type": "function", "function": {
        "name": "write_file",
        "description": "Write text to a file on disk.",
        "parameters": {"type": "object", "properties": {
            "path": {"type": "string"}, "content": {"type": "string"}}, "required": ["path", "content"]}}},
    {"type": "function", "function": {
        "name": "run_shell",
        "description": "Run a shell command and return its stdout.",
        "parameters": {"type": "object", "properties": {
            "command": {"type": "string"}}, "required": ["command"]}}},
]


def exec_tool(name, args):
    try:
        if name == "read_file":
            with open(args["path"]) as f:
                return f.read()
        if name == "write_file":
            path = args["path"]
            os.makedirs(os.path.dirname(path), exist_ok=True)
            with open(path, "w") as f:
                f.write(str(args.get("content", "")))
            return f"wrote {len(str(args.get('content','')))} bytes to {path}"
        if name == "run_shell":
            r = subprocess.run(args["command"], shell=True, capture_output=True, text=True, timeout=30)
            return (r.stdout or "") + (r.stderr or "")
    except Exception as e:
        return f"ERROR: {e}"
    return f"ERROR: no such tool {name}"


def chat(model, prompt, max_turns=8, timeout=300):
    """Run one agent loop. Returns (final_text, tool_calls_made, error)."""
    messages = [{"role": "user", "content": prompt}]
    calls = []
    for _ in range(max_turns):
        body = json.dumps({
            "model": model, "messages": messages, "tools": TOOLS, "stream": False,
            "options": {"temperature": 0},
        }).encode()
        req = urllib.request.Request(f"{OLLAMA}/api/chat", data=body,
                                     headers={"Content-Type": "application/json"})
        try:
            with urllib.request.urlopen(req, timeout=timeout) as r:
                data = json.load(r)
        except Exception as e:
            return "", calls, f"http:{str(e)[:80]}"
        msg = data.get("message", {}) or {}
        messages.append(msg)
        tcs = msg.get("tool_calls") or []
        if not tcs:
            return (msg.get("content") or ""), calls, None
        for tc in tcs:
            fn = (tc.get("function") or {})
            name = fn.get("name", "")
            raw = fn.get("arguments", {})
            args = raw if isinstance(raw, dict) else (json.loads(raw) if raw else {})
            calls.append(name)
            messages.append({"role": "tool", "content": str(exec_tool(name, args))[:4000]})
    return "", calls, "max_turns"


def why(calls, expect):
    if not calls:
        return "no_tool_call"
    if expect not in calls:
        return f"wrong_tool({','.join(calls[:2])})"
    return "tool_ran_wrong_result"


def one_run(model):
    """Five tasks. Returns dict task -> (passed, reason)."""
    shutil.rmtree(WORK, ignore_errors=True)
    os.makedirs(WORK, exist_ok=True)
    token = "ZEPHYR-" + "".join(random.choices(string.ascii_uppercase + string.digits, k=10)) + "-QUARTZ"
    open(f"{WORK}/secret.txt", "w").write(token + "\n")
    open(f"{WORK}/nums.txt", "w").write("17\n25\n8\n")
    out = {}

    text, calls, err = chat(model, f"Read the file {WORK}/secret.txt and reply with ONLY the exact word it contains.")
    out["read"] = (token in text, err or (None if token in text else why(calls, "read_file")))

    _, calls, err = chat(model, f"Create a file at {WORK}/out.txt containing exactly: HELLO_FROM_AGENT")
    ok = os.path.exists(f"{WORK}/out.txt") and open(f"{WORK}/out.txt").read().strip() == "HELLO_FROM_AGENT"
    out["write"] = (ok, err or (None if ok else why(calls, "write_file")))

    _, calls, err = chat(model, f"Read the numbers in {WORK}/nums.txt, add them together, and write ONLY the total into {WORK}/sum.txt")
    ok = os.path.exists(f"{WORK}/sum.txt") and "50" in open(f"{WORK}/sum.txt").read()
    out["chain"] = (ok, err or (None if ok else why(calls, "write_file")))

    text, calls, err = chat(model, "Run the shell command 'uname -m' and reply with only its output.")
    ok = "aarch64" in text
    out["shell"] = (ok, err or (None if ok else why(calls, "run_shell")))

    text, calls, err = chat(model, "What is 2+2? Answer with just the number. Do not use any tools.")
    ok = bool(re.search(r"(^|[^0-9])4([^0-9]|$)", text)) and not calls
    out["restraint"] = (ok, err or (None if ok else ("used_tool" if calls else "wrong_answer")))
    return out


ap = argparse.ArgumentParser()
ap.add_argument("--models", required=True)
ap.add_argument("--runs", type=int, default=5)
ap.add_argument("--out", default=os.path.expanduser("~/agentic-bench-results.json"))
a = ap.parse_args()

TASKS = ["read", "write", "chain", "shell", "restraint"]
results = {}
for model in [m.strip() for m in a.models.split(",") if m.strip()]:
    print(f"\n### {model}", flush=True)
    tally = {t: 0 for t in TASKS}
    reasons = {t: [] for t in TASKS}
    times = []
    for i in range(a.runs):
        t0 = time.time()
        try:
            r = one_run(model)
        except Exception as e:
            print(f"  run {i+1}: EXCEPTION {str(e)[:100]}", flush=True)
            continue
        times.append(round(time.time() - t0, 1))
        for t in TASKS:
            ok, reason = r[t]
            if ok:
                tally[t] += 1
            elif reason:
                reasons[t].append(reason)
        print(f"  run {i+1}/{a.runs}: " + " ".join(f"{t}={'P' if r[t][0] else '.'}" for t in TASKS)
              + f"  {times[-1]}s", flush=True)
    subprocess.run(["ollama", "stop", model], capture_output=True)
    results[model] = {
        "runs": a.runs,
        "pass_rate": {t: f"{tally[t]}/{a.runs}" for t in TASKS},
        "score_mean": round(sum(tally.values()) / max(a.runs, 1), 2),
        "median_secs": sorted(times)[len(times) // 2] if times else None,
        "failure_reasons": {t: sorted(set(v)) for t, v in reasons.items() if v},
    }
    json.dump(results, open(a.out, "w"), indent=1)
    print(f"  => {results[model]['score_mean']}/5 mean, median {results[model]['median_secs']}s", flush=True)

print("\n=== SUMMARY ===")
for m, r in sorted(results.items(), key=lambda kv: -kv[1]["score_mean"]):
    print(f"{m:26} {r['score_mean']}/5  " + "  ".join(f"{t}:{r['pass_rate'][t]}" for t in TASKS))
print(f"\nwritten to {a.out}")

jetson-orin-nano-agentic-llm-benchmark.md

Local LLM Benchmark on NVIDIA Jetson Orin Nano Super (8GB) — Agentic Tool-Calling, Not Just tok/s

Last updated: 3 August 2026 · Measured on real hardware. 8 models × 5 runs × 5 tasks.

The short version: you don't need a bigger box — you need a smaller, better model. The best agentic model we tested is 1.9 GB, passes 5/5, and is also the fastest.

---

⚠️ Correction — 3 August 2026

An earlier version of this gist said close to the opposite, and it was wrong. It claimed the fastest model "fails most of the work", that qwen3.5:2b scored 2–3/5, and that four of five models failed multi-step chaining.

That didn't survive replication. The original ran each task once, through an agent CLI. Re-run five times per model against the inference API directly, qwen3.5:2b scores 5/5, and every 64K-context model passed chaining. The old numbers were measuring the agent runtime's scaffolding rather than the models.

Ironically the old version already said "report a rate over N ≥ 5 or do not report" — and then reported single runs. This version follows its own advice.

If these numbers don't reproduce for you, please say so in the comments. That's why the harness is published.

---

Every "best local LLM for X GB" list ranks models by tokens per second. If you're running an agent — something that reads files, runs commands, chains steps — tok/s tells you almost nothing. A model can be quick at finishing a sentence and useless at doing a job.

So we measured the thing that matters, on one box, with tests that a plausible-sounding answer cannot pass.

Hardware: NVIDIA Jetson Orin Nano Super, 8 GB unified memory, 67 TOPS, JetPack R36.5 / Ubuntu 22.04, MAXN_SUPER with clocks pinned, Ollama. Freshly booted with nothing else running — a loaded box produces garbage numbers, which we learned the hard way (see below).

We build and sell this box pre-configured — clawbox.com — but everything here reproduces on any Orin Nano Super. Harness at the bottom.

---

Method

Five tasks, each with an objective oracle:

1. read — read a file containing an unguessable random token. A hallucinating model can't produce it.
2. write — write exact contents to a file. Verified on disk, not from what the model claims.
3. chain — read numbers, sum them, write the total elsewhere. Three dependent steps.
4. shell — run a command whose answer only the hardware knows.
5. restraint — a trivial question where calling a tool at all is the failure.

Every model runs all five tasks, five times. Scores are pass rates. This matters more than it sounds: single-run scoring is exactly how the first version of this test got the answer backwards.

Models loaded as 64K-context variants (PARAMETER num_ctx 65536), temperature 0, with:

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_MAX_LOADED_MODELS=1

---

Results

Model	Size	Score (N=5)	read	write	chain	shell	restraint	Median run
granite4:micro-h	1.9 GB	5.0 / 5	5/5	5/5	5/5	5/5	5/5	16.4 s
granite4:tiny-h	4.2 GB	5.0 / 5	5/5	5/5	5/5	5/5	5/5	16.8 s
cogito:3b ¹	2.2 GB	5.0 / 5	5/5	5/5	5/5	5/5	5/5	41.2 s
qwen3.5:2b	2.7 GB	5.0 / 5	5/5	5/5	5/5	5/5	5/5	45.9 s
qwen3.5:4b	3.4 GB	5.0 / 5	5/5	5/5	5/5	5/5	5/5	88.2 s
nemotron-3-nano:4b	2.8 GB	4.8 / 5	4/5	5/5	5/5	5/5	5/5	49.8 s
llama3.2:3b ¹	2.0 GB	3.0 / 5	5/5	5/5	0/5	5/5	0/5	20.8 s
hermes3:3b ¹	2.0 GB	1.0 / 5	0/5	0/5	0/5	0/5	5/5	21.8 s

¹ Run at 32K context — see the memory note below. Median run = wall-clock for one full five-task pass, including every tool round-trip.

For agents, the "efficient" power mode is the slow one

The best published throughput sweep of this board (smolhub — 8 models x 4 power modes, raw data on HuggingFace) finds 25 W is the pareto sweet spot: 35-47% more tok/s than 15 W and better tok/J than MAXN. So we re-ran this whole suite at 25 W with clocks pinned, to check whether our MAXN numbers were leaving performance unclaimed.

They weren't. For agentic work, 25 W came out 12% slower on every model:

Model	MAXN_SUPER	25 W	Penalty
granite4:micro-h	16.4 s	18.4 s	+12%
granite4:tiny-h	16.8 s	18.7 s	+11%
qwen3.5:2b	45.9 s	50.7 s	+10%
qwen3.5:4b	88.2 s	105.4 s	+20%
nemotron-3-nano:4b	49.8 s	54.8 s	+10%

Both results are right — they measure different shapes of work. A throughput sweep measures decode at fixed prompt and generation lengths, and decode is bound by memory bandwidth, where 25 W's efficiency wins. An agent loop is the opposite shape: every tool result re-processes a growing context, so it is prefill-heavy with short generations, and prefill is compute-bound — which is exactly what MAXN's higher clocks buy.

Scores were identical in both modes. Power mode changes how long the work takes, not whether the model gets it right. If you are running an agent on this board, use MAXN.

What this measures — and what it doesn't

Several models score 100% here while purpose-built tool-calling suites put capable small models nearer 40-55% (BFCL and similar). That is not a contradiction and it is not us marking our own homework — it is a different question, and worth knowing before quoting any number above.

• These are foundational tasks, not a difficulty ceiling. Four of the five are single tool calls; one is a two-hop chain. Dedicated suites add parallel calls, deeply nested arguments and long multi-turn state. A perfect score here means a model clears the basics reliably — not that it matches a frontier model.
• A pass rate is not an accuracy score. No partial credit, no argument-quality scoring, no measurement of recovery after a bad call. The per-task columns are there so you can see which thing broke.
• Ollama costs throughput, and we used it anyway. Independent testing on this same board puts llama.cpp 36-74% ahead of Ollama for small models. We measured through Ollama because it is what most people actually run. Treat the seconds as realistic rather than optimal; the ranking is unaffected, since every model faced identical conditions.

What this actually says

granite4:micro-h wins on both axes at once. Perfect score, fastest, 1.9 GB — leaving roughly 6 GB free for context and everything else on an 8 GB box. It's our shipping default, and it earned that by measurement rather than assumption.

Five of eight models score a clean 5/5. Small models on cheap edge hardware are genuinely capable of multi-step tool use in 2026. That wasn't true eighteen months ago, and it's the finding that surprised us most.

Size predicts almost nothing. The 1.9 GB model beat the 4.2 GB one on speed and tied it on score. The 3.4 GB model took 5× longer than the 1.9 GB one to reach the same result.

hermes3:3b fails in an instructive way. It doesn't refuse or answer wrongly — it writes the tool call into the message body as prose instead of emitting a structured call, so nothing parses it. Its single pass is the restraint task, which it passes by accident: the correct move there is to not call a tool, and it can't.

---

Things that will save you an afternoon

At 64K context, Llama-3.2-derived models run out of memory on 8 GB. llama3.2:3b, cogito:3b and hermes3:3b all die with cudaMalloc failed allocating a 2.28 GB KV cache. They're fine at 32K. Granite-4 (hybrid attention) and Qwen-3.5 don't have the problem at all. So when a model won't load, check context length before you blame parameter count.

gemma3:4b cannot call tools. Not "does it poorly" — Ollama refuses outright: registry.ollama.ai/library/gemma3:4b does not support tools. Fine chat model, cannot be an agent. Plenty of "best small model" lists still recommend it for agent work.

Nanbeige4.2-3B needs a newer llama.cpp than Ollama ships. Stock Ollama fails with unknown model architecture: 'nanbeige'. The GGUFs are fine — the model uses weight-shared depth loops (num_loops: 2: 22 layers run twice rather than 44 stacked), which needs its own architecture handler. llama.cpp merged support in PR #25994 on 27 July 2026; Ollama v0.32.5 shipped that same day without it. Built from current llama.cpp source, it loads and runs.

Never benchmark on a loaded box, and never trust a single run. Both produced wrong answers here. An early run scored a model 0/5 purely from memory pressure — it had ~1.4 GB free. The same model scored differently on identical clean runs. These are stochastic systems: report a rate over N ≥ 5, or don't report.

---

Reproduce it

agentic_bench.py is in this gist. Point it at any Ollama endpoint:

python3 agentic_bench.py --models granite4:micro-h,qwen3.5:2b --runs 5

It reports a pass rate per task and, on failure, why — no tool call at all, wrong tool, or a claim of success with nothing on disk. Those are different problems and need different fixes.

Results, corrections and extra models welcome in the comments.

---

Measured by ID Robots, who build the ClawBox — an Orin Nano Super with the agent stack pre-installed. The numbers reproduce on any Orin Nano Super.