Linux kernel SUNVNET RX-race fix for Debian sparc64 running as Solaris LDom guest

README.md

Linux kernel SUNVNET RX-race fix for Debian sparc64 running as Solaris LDom guest

Bug description

Permanent virtual-network channel deadlock on sun4v (lost RX interrupt).

Affected software: drivers/net/ethernet/sun/sunvnet_common.c — mainline Linux, all sun4v (LDoms) guests using sunvnet / ldmvsw. Verified on Debian sid kernel 7.1.4-1 (7.1.4+deb14-sparc64-smp)

Hardware setup used for tests: UltraSPARC T2+ machine (SPARC Enterprise T5240), Hypervisor 1.10.7.f, guest of Oracle VM Server for SPARC 3.5.0.3.3

Symptom

Under sustained transfer, a vnet interface stalls completely and permanently in both directions. Nothing is logged, no error/drop counters increment on either end, and the channel never recovers on its own. Only an interface bounce (ip link set ethX down && ip link set ethX up, sometimes needed twice — the race can also strike the recovery VIO handshake) restores traffic. On the test system the stall reproduced within seconds to minutes of bulk traffic (~5 race hits per second under load; the stock driver deadlocks on the first).

Solaris vsw peer signature during a stall (kstat, per affected channel): dring_data_msgs_sent = dring_data_acks_rcvd + N (N = 1..6), both frozen; reverse direction balanced. Guest signature: the channel's RX IRQ count in /proc/interrupts frozen while the peer demonstrably keeps writing.

Root cause

The sun4v LDC RX interrupt (cookie VIRQ) is generated only when the RX queue makes an empty → non-empty transition while the VIRQ is valid. An event that occurs while the VIRQ is invalid is not latched by the hypervisor.

sunvnet's NAPI flow:

1. ldc_rx (hard IRQ) → sunvnet_event_common → vio_set_intr(rx_ino, HV_INTR_DISABLED) (= sun4v_vintr_set_valid off) → napi_schedule (sunvnet_common.c:935).
2. NAPI: sunvnet_poll_common → vnet_event_napi → ldc_read → read_nonraw until it observes rx_head == rx_tail (ldc.c:1756). [T1]
3. napi_complete_done → vio_set_intr(rx_ino, HV_INTR_ENABLED) (sunvnet_common.c:923). [T2] No queue re-check follows.

A message written by the peer inside (T1, T2) makes the queue's only empty → non-empty transition while the VIRQ is invalid: no interrupt, event not latched. After T2 the queue is non-empty and never transitions again, so no interrupt can ever fire. The peer's ACKs for guest TX sit unread in the same dead RX queue, so the guest's VIO TX window exhausts too — the stall is bidirectional. The Solaris vsw suppresses further DRING_DATA doorbells while any are unacked, so nothing on the wire ever re-raises the channel.

Deploying the RX-race fix

Applies to Debian kernel 7.1.4+deb14-sparc64-smp (linux-source-7.1 7.1.4-1). Only two modules are rebuilt: sunvnet_common.ko and sunvnet.ko. My running kernel has CONFIG_MODVERSIONS and CONFIG_MODULE_SIG_FORCE off, so a plain rebuild with matching vermagic loads cleanly.

Build

cd /usr/src
tar xJf linux-source-7.1.tar.xz          # if not already unpacked
cd linux-source-7.1
cp /boot/config-$(uname -r) .config
make olddefconfig
make -j8 modules_prepare

# Fix vermagic by hand:
echo "#define UTS_RELEASE \"$(uname -r)\"" > include/generated/utsrelease.h

# Apply the patch
patch -p1 < /root/sunvnet-rx-race-fix/sunvnet-rx-race-fix.patch

# No Module.symvers exists without a full kernel build, so modpost must be
# told to warn (not fail) on symbols resolved at load time from vmlinux:
make M=drivers/net/ethernet/sun modules KBUILD_MODPOST_WARN=1

modinfo -F vermagic drivers/net/ethernet/sun/sunvnet_common.ko
# must print proper kernel version string

Install persistently (survives reboot)

updates/ takes precedence over kernel/ in depmod's search order, so the stock .ko.xz files stay untouched as an automatic fallback:

mkdir -p /lib/modules/$(uname -r)/updates
cp /usr/src/linux-source-7.1/drivers/net/ethernet/sun/sunvnet_common.ko \
   /usr/src/linux-source-7.1/drivers/net/ethernet/sun/sunvnet.ko \
   /lib/modules/$(uname -r)/updates/
depmod -a
modinfo -n sunvnet_common     # must print the /updates/ path

Then reboot the machine.

sunvnet is not in the initramfs (root is on sunvdc), so no update-initramfs is needed; verify with lsinitramfs /boot/initrd.img-$(uname -r) | grep sunvnet if in doubt.

Verify

modinfo -n sunvnet_common                 # /lib/modules/.../updates/... path
cat /sys/module/sunvnet_common/parameters/rx_race_recovered
# The parameter only exists in the patched module. It counts recovered
# lost-RX-interrupt events; expect it to grow under bulk traffic (~5/s on a
# T2+). Each increment is a deadlock that did not happen.
dmesg | grep "recovered lost RX"

Roll back

rm /lib/modules/$(uname -r)/updates/sunvnet_common.ko \
   /lib/modules/$(uname -r)/updates/sunvnet.ko
depmod -a
# then reboot, or repeat the detached reload script (it will load stock)

Known caveats

• A kernel package upgrade installs a new /lib/modules/<newver>/ tree; the fix must be rebuilt/reinstalled for the new kernel (or the fix accepted into the distro/upstream kernel).

sunvnet-rx-race-fix.patch

--- a/drivers/net/ethernet/sun/sunvnet_common.c
+++ b/drivers/net/ethernet/sun/sunvnet_common.c
@@ -29,6 +29,7 @@
 #include <net/icmp.h>
 #include <net/route.h>

+#include <asm/hypervisor.h>
 #include <asm/vio.h>
 #include <asm/ldc.h>

@@ -911,6 +912,46 @@
 	return npkts;
 }

+/* Count of RX interrupts recovered by the post-enable queue re-check in
+ * sunvnet_poll_common(), readable at
+ * /sys/module/sunvnet_common/parameters/rx_race_recovered.  Every
+ * increment is a would-have-been permanent channel deadlock.
+ */
+static unsigned long rx_race_recovered;
+module_param(rx_race_recovered, ulong, 0444);
+MODULE_PARM_DESC(rx_race_recovered,
+		 "RX interrupts recovered by post-enable LDC queue re-check");
+
+/* The sun4v LDC RX interrupt is generated only on the queue's
+ * empty->non-empty transition, and only while the VIRQ is valid; the
+ * hypervisor does not latch an event that occurs while the VIRQ is
+ * invalid.  A message that lands between the final ldc_read() seeing an
+ * empty queue and vio_set_intr(HV_INTR_ENABLED) below therefore never
+ * raises an interrupt - and never will, because the queue stays
+ * non-empty from then on.  Re-check the queue after re-enabling.
+ *
+ * sun4v_ldc_rx_get_state() is not exported to modules, so make the
+ * hypervisor call directly (HV_FAST_LDC_RX_GET_STATE per asm/hypervisor.h;
+ * fast traps clobber only %o0-%o5).
+ */
+static long vnet_ldc_rx_pending(unsigned long channel_id)
+{
+	register unsigned long o0 asm("o0") = channel_id;
+	register unsigned long o1 asm("o1");
+	register unsigned long o2 asm("o2");
+	register unsigned long o3 asm("o3");
+	register unsigned long o5 asm("o5") = HV_FAST_LDC_RX_GET_STATE;
+
+	__asm__ __volatile__("ta %5"
+			     : "+r" (o0), "=r" (o1), "=r" (o2), "=r" (o3),
+			       "+r" (o5)
+			     : "i" (HV_FAST_TRAP)
+			     : "o4", "cc", "memory");
+	if (o0 != HV_EOK)
+		return -1;
+	return o1 != o2;
+}
+
 int sunvnet_poll_common(struct napi_struct *napi, int budget)
 {
 	struct vnet_port *port = container_of(napi, struct vnet_port, napi);
@@ -921,6 +962,15 @@
 		napi_complete_done(napi, processed);
 		port->rx_event &= ~LDC_EVENT_DATA_READY;
 		vio_set_intr(vio->vdev->rx_ino, HV_INTR_ENABLED);
+
+		if (vnet_ldc_rx_pending(vio->vdev->channel_id) > 0 &&
+		    napi_schedule_prep(napi)) {
+			vio_set_intr(vio->vdev->rx_ino, HV_INTR_DISABLED);
+			rx_race_recovered++;
+			pr_info_ratelimited("sunvnet: recovered lost RX interrupt on channel %lu\n",
+					    vio->vdev->channel_id);
+			__napi_schedule(napi);
+		}
 	}
 	return processed;
 }