MAVLink forwarding mutex benchmark: before/after results for PR #26894

mavlink_forwarding_bench_report.md

MAVLink Forwarding Mutex Benchmark Report

Context

PR #26894 reworked MAVLink signing. During review, @dakejahl noted that forward_message() acquires mavlink_module_mutex on every forwarded message, and asked whether this mutex could be removed from the hot path.

This benchmark measures the wall-clock cost of that mutex and the improvement from removing it.

Methodology

Instrumentation

PX4's built-in perf_counter_t uses hrt_absolute_time(), which in lockstep SITL is simulated time with 4ms resolution. Sub-microsecond operations read as 0us. To get real measurements, we added wall-clock counters using clock_gettime(CLOCK_MONOTONIC) directly in mavlink_main.cpp:

• fwd_msg total: Entire forward_message() body, including metadata extraction, early exits, lock, iteration, and pass_message()
• fwd_msg lock: Only the locked section (mutex acquire through iteration and pass_message() calls)

Each counter tracks call count, cumulative nanoseconds, and max single-call nanoseconds. Atomics (px4::atomic<uint64_t>) ensure thread safety without adding their own lock overhead.

Traffic Generation

A standalone C++ tool (test_mavlink_forwarding_bench.cpp) using MAVSDK 3.15 sends DEBUG_FLOAT_ARRAY messages at a configurable rate. This message type has no target fields, so PX4 treats it as broadcast and always enters forward_message(), forwarding to all other instances.

• SITL: Two UDP connections (ports 14540 + 14550), two receiver threads competing for the mutex
• Hardware: Single USB CDC serial connection

Tests ran for 30-60 seconds at 200 Hz (and 500 Hz for SITL stress test). The benchmark tool automatically reads perf counters from the board via the MAVLink shell protocol after the run completes.

What the Mutex Protects

mavlink_module_mutex guards mavlink_module_instances[MAVLINK_COMM_NUM_BUFFERS] (max 6 slots). In forward_message(), the lock covers:

1. Iterating the instance array
2. Calling component_was_seen() on each instance's receiver
3. Calling pass_message() (which has its own per-instance _message_buffer_mutex)

Why Removing It Is Safe

• Instances can only be destroyed via destroy_all_instances(), which stops all receiver threads before modifying the array
• A receiver thread calling forward_message() is alive, so its own instance can't be torn down mid-call
• Pointer reads from the array are naturally atomic on aligned architectures (ARM, x86)
• pass_message() has its own per-instance mutex for the ring buffer
• component_was_seen() on MavlinkReceiver is read-only on per-instance data

Results

SITL (macOS arm64, 2 connections, 60s)

200 Hz/connection (400 msg/s total)

Metric	With Mutex	Without Mutex	Delta
fwd_msg total (avg)	648 ns	563 ns	-85 ns (-13%)
fwd_msg lock (avg)	327 ns	262 ns	-65 ns (-20%)
fwd_msg total (max)	29,000 ns	30,000 ns	~same

500 Hz/connection (1000 msg/s total)

Metric	With Mutex	Without Mutex	Delta
fwd_msg total (avg)	469 ns	407 ns	-62 ns (-13%)
fwd_msg lock (avg)	243 ns	179 ns	-64 ns (-26%)
fwd_msg total (max)	30,000 ns	31,000 ns	~same

CubeOrange+ (STM32H7, 1 connection, 200 Hz, 30s)

Metric	With Mutex	Without Mutex	Delta
fwd_msg total (avg)	4,521 ns	2,948 ns	-1,573 ns (-35%)
fwd_msg lock (avg)	986 ns	824 ns	-162 ns (-16%)
fwd_msg total (max)	1,000,000 ns	2,000,000 ns	-

Pixhawk 6X-RT (i.MX RT1176, 1 connection, 200 Hz, 30s)

Metric	With Mutex	Without Mutex	Delta
fwd_msg total (avg)	791 ns	162 ns	-629 ns (-80%)
fwd_msg lock (avg)	163 ns	165 ns	~same
fwd_msg total (max)	1,000,000 ns	1,000,000 ns	~same

Summary across platforms

Platform	With Mutex	Without	Improvement
SITL (macOS arm64)	648 ns	563 ns	-13%
CubeOrange+ (STM32H7)	4,521 ns	2,948 ns	-35%
Pixhawk 6X-RT (i.MX RT1176)	791 ns	162 ns	-80%

Observations

1. Hardware impact is significantly larger than SITL. Uncontended pthread_mutex on macOS is very cheap (~20-40 ns). On NuttX, mutex maps to interrupt-disable with priority inheritance, which is substantially more expensive.
2. RT1176 shows the biggest win (80%). The mutex was the dominant cost in forward_message() on this chip. Without it, the function takes only 162 ns.
3. STM32H7 is the slowest overall at ~3-5 us per forwarded message. The mutex accounts for about 35% of the cost there, with the rest being the loop iteration, component_was_seen(), and pass_message().
4. Max latency spikes of 1-2 ms on hardware are from NuttX scheduling (higher-priority tasks preempting the receiver thread), not mutex contention.
5. No meaningful flash savings. The change removes one LockGuard call (~40 bytes of code).

Test environments

Platform	Board	MCU	NuttX	Toolchain
SITL	macOS arm64	-	-	AppleClang 21.0
H7	CubeOrange+	STM32H7x5 rev V	11.0.0	arm-none-eabi-gcc 13.2.1
RT1176	Pixhawk 6X-RT	i.MX RT1170 rB0	11.0.0	arm-none-eabi-gcc 13.2.1

Files

• Instrumentation: src/modules/mavlink/mavlink_main.cpp (wall-clock BenchCounter struct)
• Traffic generator: test/mavsdk_tests/test_mavlink_forwarding_bench.cpp
• Build: test/mavsdk_tests/CMakeLists_bench.txt
• Runner: test/mavsdk_tests/run_forwarding_bench.sh
• PR: #26927