apollo-mg

God damnit buun, there’s no binaries on your site: A Pragmatic Guide to Local Agentic LLMs

I have been getting so wrapped up in testing models, engines, harnesses, and everything else out there LLM-related lately. It’s never-ending. But it occurred to me; I haven’t actually looked at how you get from:

1. An X post sounding like a fun idea to try this weekend at home, to
2. Actually having something working on your computer that’s legitimately useful.

So how do you do it?

Let’s take buun’s fork of llama.cpp for example. You might have seen it on X recently when Clem posted, asking “is this useful?”. I am personally fortunate enough to work with buun on a nearly daily basis, but what if I didn’t? I hope he doesn’t hate me.

apollo-mg

#!/bin/bash
# Launch buun-llama-cpp with dynamic VBR KV cache + inline MTP speculative decoding.
#
# Companion to: "God damnit buun, there's no binaries on your site"
# Tested on: RX 9070 XT (gfx1201, ROCm 7.2), buun-llama-cpp 3823c9eb6,
#            Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf
#
# FLAG NOTES (comments live here, not inline -- a "# comment" after a trailing
# backslash silently breaks the line continuation and truncates the command):
#

apollo-mg

The P100 has been doing silently noisy math in llama.cpp for years. Three lines fix it for free.

---

TL;DR

llama.cpp's CUDA backend has a FAST_FP16_AVAILABLE flag: "this GPU is fast at fp16, so do quality-sensitive math in fp16." The GTX 10-series (sm_61) was exempted from it long ago. The Tesla P100 (sm_60) never was — because GP100 is the one Pascal chip with fast fp16 hardware. Hardware can, therefore software did. Nobody measured what it cost.

apollo-mg

Llama-Server is Throwing Away Your Perfectly Good KV Caches, and How to Fix It

Introduction

I watched 2.49 GB of state restore from disk in 1.23 seconds — and then get thrown away. llama-server's slot save/restore promises exactly what long-context work on budget hardware needs: park a session on disk, bring it back later without paying the prefill tax again. And the restore itself works perfectly. But across a process restart, the feature was functionally useless: the first query after restoring discarded the entire rehydrated state and re-prefilled from scratch. The reason turned out to be a single piece of metadata that lived only in process memory.

Testbed: Qwopus3.6-27B Q6_K (hybrid architecture), 2× Tesla P100 layer-split, turbo4 KV quantization.

The Discovery

apollo-mg

Sovereign AI Architecture: The "L3 Spine" Gatekeeper

Executive Summary

In complex, localized AI architectures (like Project Apollo's multi-agent swarm), utilizing massive GPU VRAM for simple intent routing is computationally inefficient. This proof-of-concept demonstrates an air-gapped, zero-VRAM "Gatekeeper" node by pinning a hyper-quantized 135M parameter LLM strictly to a CPU's L3 V-Cache.

By leveraging native Linux CPU pinning (taskset) and rigorous grammar constraints (GBNF), we achieve deterministic, zero-hallucination JSON output at GPU-like bandwidths (~136 Tokens Per Second) while leaving the primary accelerator (RX 9070 XT) completely untouched.

Core Technical Concept

1. Model: SmolLM2-135M-Instruct-Q4_K_M (~60MB working footprint).
2. Hardware: AMD Ryzen 7 5700X3D (96MB L3 Cache).

apollo-mg

🧪 SOVEREIGN MODEL TEST LAB

Objective: Document empirical test data, failure modes, and VRAM footprints to determine the optimal model for specific agent workflows on the RX 9070 XT.

🏆 Current Resident Kings

1. The Logic Core: qwen3.5-9b-heretic:4bit

• Architecture: Qwen 3.5 9B (Dense)
• Format: GGUF Q4_K_M (Ollama)
• VRAM Footprint: ~5.5 GB (Leaves room for ~100k context)
• Speed (GFX1201): ~50-60 tokens/sec

apollo-mg

🛸 RDNA 4 (GFX1201) AI MASTER LIST

Last Updated: March 9, 2026 | Environment: ROCm 7.2 / Poachers Special Ed (PyTorch 2.9.1 / Triton 3.5.1)

🟢 1. THE "GREEN ZONE" (Verified Working Bare-Metal)

• Flash Linear Attention (FLA): ASCENDED. Liberated from Docker; running bare-metal via Triton kernels.
• 4-bit Resident Vision: CONFIRMED. Qwen 3.5 4B running in 4.7GB VRAM with ~27-40s prefill.
• Dual-Core Residency: VERIFIED. Logic (DeepSeek-R1 14B @ 51 tok/s) and Vision (Qwen 3.5 4B) running simultaneously in 16GB VRAM.
• Triton 3.5.1 + PyTorch 2.9.1: Stable native pairing for GFX1201.
• Unsloth 4-bit Native: Works perfectly once vLLM/CUDA dependency checks are bypassed.

apollo-mg

🛸 RDNA 4 (GFX1201) AI MASTER LIST

Last Updated: March 9, 2026 | Environment: ROCm 7.2 / Poachers Special Ed (PyTorch 2.9.1 / Triton 3.5.1)

🟢 1. THE "GREEN ZONE" (Verified Working Bare-Metal)

• Flash Linear Attention (FLA): ASCENDED. Liberated from Docker; running bare-metal via Triton kernels.
• 4-bit Resident Vision: CONFIRMED. Qwen 3.5 4B running in 4.7GB VRAM with ~27-40s prefill.
• Dual-Core Residency: VERIFIED. Logic (DeepSeek-R1 14B @ 51 tok/s) and Vision (Qwen 3.5 4B) running simultaneously in 16GB VRAM.
• Triton 3.5.1 + PyTorch 2.9.1: Stable native pairing for GFX1201.
• Unsloth 4-bit Native: Works perfectly once vLLM/CUDA dependency checks are bypassed.

apollo-mg

The NVCC Trap: Why Causal-Conv1d Fails on Native RDNA 4 (and how to bypass it)

Date: March 9, 2026 Hardware: AMD Radeon RX 9070 XT (gfx1201) Software: ROCm 7.2.0, PyTorch 2.12.0 (Nightly)

The Problem

As of early 2026, many frontier models (like Qwen 3.5 Unified Vision and Mamba-2) rely on `causal-conv1d`. On AMD hardware, attempting to install this package results in immediate failure, forcing the model into a "slow-path" fallback that pulls up to 320W and utilizes high CPU overhead for simple vision tasks.

The Forensic Breakdown

During a live engineering session on an RDNA 4 rig, we identified three fatal layers of hardcoding in the `dao-ailab/causal-conv1d` (v1.6.0) installer:

apollo-mg

RDNA 4 (GFX1201) Technical Post-Mortem: TileLang Kernel Forging & The Wave32 Barrier

Executive Summary

This gist documents the first known attempt to use TileLang for custom kernel forging on RDNA 4 hardware (specifically the AMD Radeon RX 9070 XT, gfx1201). While TileLang is a powerful "Blacksmith's Kit" for AMD Instinct (CDNA) hardware, our research reveals critical architectural barriers when targeting consumer RDNA 4 cards.

🏁 The Success: General Purpose Compute

We successfully compiled and executed a custom "Buffer Copy" smoke test kernel on the RX 9070 XT using TileLang's JIT backend and ROCm 7.2.

Key Finding: The core TileLang compiler and ROCm JIT pipeline are functional for standard memory operations and non-matrix compute on RDNA 4.