Enabling CUDA P2P on RTX 5060 Ti (Blackwell) with Open GPU Kernel Modules Patch (Guide and Troubleshooting)

how_to_patch_nvidia_open_kernel_p2p.md

Enabling CUDA P2P on RTX 5060 Ti (Blackwell) with Open GPU Kernel Modules Patch

Goal

Enable CUDA Peer-to-Peer (P2P) communication between 4x RTX 5060 Ti GPUs on an old Xeon E5 v4 / X99 platform where NVIDIA reports P2P as unsupported.

Target workload:

• SGLang TP=4
• vLLM TP=4
• PCIe-only topology (no NVLink)
• PHB interconnect between all GPUs

---

Hardware

Platform

• Xeon E5 v4 (Broadwell)
• X99 / C610 chipset
• Ubuntu 22.04
• Linux kernel 5.15

GPUs

• GPU0 = RTX 3090
• GPU1-4 = RTX 5060 Ti 16GB (PCI ID 10de:2d04)

Topology:

nvidia-smi topo -m

Output:

GPU0 PHB GPU1
GPU0 PHB GPU2
GPU0 PHB GPU3
GPU0 PHB GPU4

GPU1 PHB GPU2
GPU1 PHB GPU3
GPU1 PHB GPU4
...

All GPU pairs communicate through PHB.

---

Initial Problem

Tensor Parallel inference showed:

• Low GPU utilization
• ~80W power draw on 180W cards
• Little difference between small and large batches

Suspected NCCL communication bottleneck.

---

Baseline P2P Status

Proprietary Driver

nvidia-smi topo -p2p r

Output:

GPU1 GPU2 CNS
GPU1 GPU3 CNS
GPU1 GPU4 CNS

CNS = Chipset Not Supported

---

Open Kernel Modules

After reinstalling NVIDIA driver using:

MIT/GPL Open Kernel Modules

Output changed to:

GPU1 GPU2 GNS
GPU1 GPU3 GNS
GPU1 GPU4 GNS

GNS = GPU Not Supported

---

CUDA P2P Test (Before Patch)

Using CUDA sample:

CUDA_VISIBLE_DEVICES=1,2,3,4 ./p2pBandwidthLatencyTest

Output:

Device=0 CANNOT Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
...

Connectivity matrix:

1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1

Bandwidth:

~6.7 GB/s

Latency:

~14.3 us

---

Installing the Patch

Repository:

https://github.com/aikitoria/open-gpu-kernel-modules

Build:

./install.sh

Compilation completed successfully.

---

Critical Gotcha

Initially the patch appeared to do nothing.

Reason:

The patched module was installed to:

/lib/modules/<kernel>/kernel/drivers/video/nvidia.ko

But the system was still loading:

/lib/modules/<kernel>/updates/dkms/nvidia.ko

Verification:

modinfo -n nvidia

Loaded:

updates/dkms/nvidia.ko

Patched:

kernel/drivers/video/nvidia.ko

Different srcversion:

Loaded:

F9594124BDD9B20293DA033

Patched:

5133AA53FEA92ECFF8E5016

Therefore the patched module was NOT being used.

---

Fix

Backup DKMS modules:

sudo mkdir -p /root/nvidia-dkms-backup

sudo mv /lib/modules/$(uname -r)/updates/dkms/nvidia*.ko \
         /root/nvidia-dkms-backup/

Rebuild module dependencies:

sudo depmod -a

Reboot:

sudo reboot

Verify:

modinfo -n nvidia

Expected:

/lib/modules/<kernel>/kernel/drivers/video/nvidia.ko

Verify srcversion:

cat /sys/module/nvidia/srcversion

Expected:

5133AA53FEA92ECFF8E5016

---

Results After Patch

NVIDIA Topology

nvidia-smi topo -p2p r

Output:

GPU0 GPU1 OK
GPU0 GPU2 OK
GPU0 GPU3 OK
GPU0 GPU4 OK

GPU1 GPU2 OK
GPU1 GPU3 OK
GPU1 GPU4 OK

GPU2 GPU3 OK
GPU2 GPU4 OK

GPU3 GPU4 OK

All GPU pairs now report:

OK

---

CUDA P2P Test (After Patch)

CUDA_VISIBLE_DEVICES=1,2,3,4 ./p2pBandwidthLatencyTest

Output:

Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
...

Connectivity matrix:

1 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1

---

Performance Comparison

Unidirectional P2P Bandwidth

Before:

~6.75 GB/s

After:

~71 GB/s

Improvement:

10.5x

---

Bidirectional P2P Bandwidth

Before:

~8.2 GB/s

After:

~142 GB/s

Improvement:

17x

---

P2P Latency

Before:

~14.3 us

After:

~0.40 us

Improvement:

~35x lower latency

---

NCCL Verification

NCCL logs:

Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1

Important:

isAllCudaP2p = 1

NCCL now recognizes CUDA peer access.

Communication paths:

via SHM/direct/direct

instead of host-memory fallback.

---

Docker Gotcha

After purging NVIDIA packages:

sudo apt purge 'nvidia*'

Docker GPU support broke:

failed to discover GPU vendor from CDI

Cause:

Removed:

• nvidia-container-toolkit
• libnvidia-container
• nvidia-container-runtime

Reinstall:

sudo apt install \
  nvidia-container-toolkit \
  libnvidia-container1 \
  libnvidia-container-tools

Then:

sudo systemctl restart docker

---

SGLang Notes

Keep:

--enable-p2p-check

Verify NCCL logs:

isAllCudaP2p 1

Experiment with:

SGLANG_USE_CUDA_IPC_TRANSPORT: "1"

Previously it was disabled because CUDA P2P was unavailable.

Now CUDA P2P is fully enabled.

---

Final Conclusion

The patch successfully enabled CUDA P2P on:

• RTX 5060 Ti (GB206)
• Blackwell consumer GPUs
• Xeon E5 v4 / X99
• PHB-only topology
• PCIe Gen3 x8

Observed improvements:

Metric	Before	After
P2P Status	GNS	OK
CUDA Peer Access	No	Yes
Unidir BW	6.7 GB/s	71 GB/s
Bidir BW	8.2 GB/s	142 GB/s
Latency	14.3 us	0.4 us

NCCL now detects:

isAllCudaP2p = 1

which means TP=4 inference can use direct GPU-to-GPU communication rather than host-memory fallback.