virt.md

virt.md

QEMU virt: device-tree emission for a CBQRI capacity controller

Date: 2026-06-15 Repos: ~/dev/qemu (implementation), ~/dev/linux (consumer, branch dfustini/atl-sc-cbqri-dt)

Goal

Let the QEMU virt machine boot in device-tree mode and have the kernel's CBQRI cache capacity-allocation platform driver (drivers/resctrl/cbqri_capacity.c, CONFIG_RISCV_CBQRI_CAPACITY) probe a modeled CBQRI capacity controller, so the DT driver path can be tested without the Atlantis HAPS lab machine.

Scope for this iteration: a single shared L2 cache with one capacity controller backing it. This mirrors the Atlantis sc_cbqri node exactly and is the smallest change that exercises the driver end to end.

Background

What the DT driver needs

The platform driver binds the generic riscv,cbqri-capacity-controller compatible and, on probe, reads:

• reg -> the CBQRI capacity-controller register block (base, size).
• riscv,cbqri-rcid -> RCID count.
• riscv,cbqri-cache -> phandle to the cache node it governs.

The driver does not read the cache geometry from that node. It resolves the phandle through the kernel's cacheinfo: it walks online CPUs looking for a cacheinfo leaf at the cache's level whose fw_token equals the cache device_node, and from that leaf takes the resctrl domain id (cache_id) and the shared_cpu_map. This only works if the FDT carries a real cache node (cache-level, cache-unified, etc.) and every sharing hart points at it via next-level-cache.

Reference DT (Atlantis, arch/riscv/boot/dts/tenstorrent/atlantis-soc.dtsi):

• 8 harts, each with next-level-cache = <&l2_cache>.
• l2_cache: cache-level = <2>, cache-unified, cache-size = <0xc00000>.
• sc_cbqri: reg = <0x0 0xa21a00c0 0x0 0xf40>, riscv,cbqri-rcid = <16>, riscv,cbqri-cache = <&l2_cache>.

What QEMU has today

• It models the controllers: hw/riscv/cbqri_capacity.c, hw/riscv/cbqri_bandwidth.c, include/hw/riscv/cbqri.h.
• Controllers are instantiated as dynamic sysbus devices on the platform bus, for example -device riscv.cbqri.capacity,...,mmio_base=0x04820000. The device self-maps its MMIO at the absolute mmio_base in realize (sysbus_mmio_map), so the address is deterministic.
• Controllers are exposed only through ACPI/RQSC. gatherCbqriDetails() in hw/riscv/virt-acpi-build.c walks vs->platform_bus_dev->parent_bus->children to build the RQSC table.
• virt.c emits no DT node for the controllers and no cache topology at all: create_fdt_socket_cpus() writes no next-level-cache, and there is no L2/L3 node anywhere in the generated FDT.

So two pieces are missing on the DT side: a CPU cache hierarchy and the controller node.

Approach

Chosen: reuse the existing -device instantiation model and emit the DT by walking the platform bus, the same source of truth the ACPI path already trusts. Rejected alternatives: a -M virt,cbqri=on machine flag (diverges from the -device model the test scripts use, adds a machine property); a hw/core/sysbus-fdt.c binding (runs in the platform-bus context with no knowledge of the cache phandle created in virt.c, so it cannot emit riscv,cbqri-cache cleanly, and the cache hierarchy would still have to come from virt.c).

Timing

virt.c builds the base FDT early in create_fdt(), but finalizes it in finalize_fdt(), which runs from the machine-done notifier (virt_machine_done) after all -device instances are realized. finalize_fdt() calls create_fdt_sockets() -> create_fdt_socket_cpus(), which builds the CPU nodes, and s->platform_bus_dev is populated by then. Both the cache wiring and the controller walk happen in this one late stage, the same point at which gatherCbqriDetails() reads the devices for RQSC.

Components

Component 1: shared L2 cache node and next-level-cache

Location: create_fdt_sockets() / create_fdt_socket_cpus() in hw/riscv/virt.c.

• Allocate one phandle from the existing phandle counter for a shared L2.
• Emit one node /cpus/l2-cache:
  • compatible = "cache"
  • cache-level = <2>
  • cache-unified (empty property)
  • cache-size, cache-sets, cache-block-size (mirror Atlantis: 0xc00000 / 512 / 64)
  • phandle
• In the per-CPU loop, add next-level-cache = <l2_phandle> to every /cpus/cpu@N node.

Result: all harts reference the same cache node, so cacheinfo reports one L2 domain whose shared_cpu_map covers every online CPU. Scoped to sockets=1; the single shared node is created once and its phandle reused across the CPU loop.

Component 2: controller DT node

New helper create_fdt_cbqri(RISCVVirtState *s, uint32_t l2_phandle), called from finalize_fdt() after create_fdt_sockets().

• Walk s->platform_bus_dev->parent_bus->children; for each child of type TYPE_RISCV_CBQRI_CC, read generically (no coupling to the device's private struct):
  • base = object_property_get_uint(obj, "mmio_base", &error_abort)
  • rcid = object_property_get_uint(obj, "max_rcids", &error_abort)
  • size = memory_region_size(sysbus_mmio_get_region(SYS_BUS_DEVICE(child), 0))
• Emit /soc/cache-controller@<base>:
  • compatible = "riscv,cbqri-capacity-controller" (the generic compatible the driver's of_match table binds; no vendor string needed on QEMU)
  • reg = <base size> (two address cells, two size cells, matching /soc #address-cells/#size-cells = 2)
  • riscv,cbqri-rcid = <rcid>
  • riscv,cbqri-cache = <l2_phandle>

For the single-L2 scope every capacity controller points at the one L2 node.

Component 3: DT-mode run script

New script ~/dev/qemu/run-dt-sock-1cc.sh, derived from run-acpi-sock-1bc.sh:

• Drop the EDK2 pflash blockdevs and the pflash0=/pflash1= machine options; boot the default OpenSBI plus -kernel ${LX}/arch/riscv/boot/Image so QEMU's generated FDT is passed straight to the kernel (no -dtb).
• Keep the socket serial plumbing, -smp, -m, rootfs blockdev, and a single -device riscv.cbqri.capacity,...,mmio_base=0x04820000 line.
• Keep aia=aplic-imsic.

Kernel side

No kernel changes. The consumer branch dfustini/atl-sc-cbqri-dt already carries the platform driver and binding; build with CONFIG_RISCV_CBQRI_CAPACITY=y and DT support.

Verification

1. Boot the DT script; the kernel log shows the driver probe: cbqri-capacity: ... registered L2 capacity controller at <base> (cache_id=..., rcid=...).
2. /sys/fs/resctrl mounts and exposes the L2 cache-allocation resource with a single domain whose CPU mask covers all harts.
3. A schemata write round-trips: setting a capacity bitmask is observable in the QEMU device (allocation register state).

Out of scope

• L3 or multiple cache levels, per-cluster cache topology.
• Bandwidth-controller DT emission.
• Multiple distinct caches mapped to distinct controllers. The platform-bus walk extends to these naturally later, but they are not built now.

Files touched

• ~/dev/qemu/hw/riscv/virt.c: cache hierarchy (L1 + shared L2) + next-level-cache; new create_fdt_cbqri(); call site in finalize_fdt().
• ~/dev/qemu/run-dt-sock-1cc.sh: new DT-mode launcher.

Implementation notes (deviations found during implementation)

Two things diverged from the initial design and were resolved:

1. Prerequisite build fix. The b4/riscv-rqsc base did not compile: after a QEMU rebase, device_class_set_props() became a macro that, at -O2, rejects a legacy terminator by evaluating the array's last element, which only works when the array is const. The CBQRI capacity and bandwidth property arrays were still static Property (non-const), leaving the qemu_build_not_reached() path reachable. Fixed by making both arrays const (its own commit), matching every other device.

2. L1 cache properties are required, not just the L2 node. RISC-V populate_cache_leaves() only creates a cache leaf for a node that carries a *-cache-size property, while the generic of_count_cache_leaves() assumes a split L1 (counts 2 leaves) for a CPU node with no cache properties. With only the L2 node and next-level-cache, the two disagree, cache_setup_of_node() bails with -ENOENT, no cacheinfo is built, and the controller's shared_cpu_map comes out empty (driver returns -EINVAL). The fix is to emit split L1 i/d cache properties (i/d-cache-size/sets/block-size) on each CPU node alongside next-level-cache, as the working Atlantis DT does. This lives in the same commit as the L2 node (the cache hierarchy is one logical change), gated on CBQRI-CC presence.

Verification result

Confirmed end-to-end on QEMU virt (DT mode, 8 harts): driver logs registered L2 capacity controller at 0x4820000 (cache_id=0, rcid=64); cpu0 cacheinfo shows L2 Unified shared 0-7; /sys/fs/resctrl/info/L2 reports num_closids=64, cbm_mask=ffff; schemata shows L2:0=ffff with cpus_list=0-7; writing L2:0=0x0f reads back L2:0=f.