Load balancing via reverse proxy, rather than SO_REUSEPORT

1.readme.md

Why a Reverse Proxy Can Beat SO_REUSEPORT on Tail Latency

Summary

When several server workers listen on one port with SO_REUSEPORT, the kernel normally assigns each new TCP connection using a hash. This is fast and scalable, but the hash does not know which worker is busy.

A reverse proxy can expose each worker as a separate upstream endpoint and send new work to the worker with the fewest active connections (least_conn). Because this decision uses current load, it can reduce accidental queues and improve p99 response time.

This does not mean a reverse proxy is always faster. The expected benefit is mainly lower tail latency under concurrency, bursts, or uneven request durations.

A small example

Suppose 16 tasks arrive together and are assigned independently to 8 single-task workers.

With random assignment, each task chooses one of the 8 workers with probability $1/8$. For one particular worker, the number of assigned tasks $X$ follows

[ X \sim \operatorname{Binomial}(16, 1/8). ]

The average is only two tasks per worker, but

[ P(X \ge 4) = 1 - \sum_{k=0}^{3} \binom{16}{k} \left(\frac18\right)^k \left(\frac78\right)^{16-k} \approx 13.0%. ]

There are 8 workers, so the probability that at least one worker receives 4 or more tasks is approximately 79.2%. The worker counts are not independent, so this value is computed from their joint distribution rather than by simply multiplying 13.0% by 8.

In contrast, a load-aware dispatcher assigning each new task to a least-loaded worker produces

2, 2, 2, 2, 2, 2, 2, 2

in this simplified example.

Random assignment does not reduce total capacity, but it can leave one worker with a queue while another worker is idle. Requests in the unlucky queue become the slow tail. This is why average response time may look acceptable while p99 becomes much worse.

Why SO_REUSEPORT cannot correct the imbalance

The normal kernel selection happens when a TCP connection is created. It is based on a hash, not on application state such as:

• active request count;
• worker queue length;
• event-loop delay;
• remaining work.

Once selected, the connection belongs to that worker. HTTP keep-alive can preserve this choice across multiple requests, so short-term imbalance may last longer than one request.

A reverse proxy sees separate upstream endpoints. With least_conn, it can avoid a worker that already has more active connections. It is not a perfect estimate of remaining work, but it provides useful feedback that ordinary SO_REUSEPORT lacks.

Suggested server feature

Uvicorn and Gunicorn could keep their current shared-listener mode while optionally supporting one endpoint per worker, for example:

127.0.0.1:8001
127.0.0.1:8002
127.0.0.1:8003
127.0.0.1:8004

or one Unix-domain socket per worker. A reverse proxy could then list these endpoints separately and apply least_conn or another load-aware policy.

The process manager should manage endpoint allocation, startup, health, graceful shutdown, and configuration output. The feature should be optional: a single shared port remains convenient when no load-aware proxy is present.

Limits of the argument

The benefit will be smaller when load is low, workers can efficiently handle many concurrent requests, or active connection count poorly represents actual work. An extra proxy hop also has a cost. Performance should therefore be measured for the target workload.

The claim is narrower and practical:

When worker capacity is limited and requests can queue, load-aware routing can avoid random worker imbalance and substantially improve p99 latency compared with hash-based SO_REUSEPORT distribution.

References

• Linux SO_REUSEPORT socket-selection implementation
• nginx least_conn documentation