kolu CI: GitHub 403 API rate-limit on shared pool NAT IP 219.65.110.2 (unauthenticated 60/hr) — admin

403-gist.md

kolu CI: GitHub HTTP 403 "API rate limit exceeded" on the shared pool NAT IP

The kolu-ci-* Incus pool boxes all egress through one shared NAT IP: 219.65.110.2. Unauthenticated requests from that IP to api.github.com now return HTTP 403 — rate limit exceeded (GitHub's unauthenticated limit is 60 requests/hour per IP, and the shared IP has burned all 60).

Raw evidence (from inside kolu-ci-1)

$ curl -s -D - -o /dev/null https://api.github.com
HTTP/2 403
server: Varnish
x-ratelimit-limit: 60
x-ratelimit-remaining: 0
x-ratelimit-used: 60
x-ratelimit-resource: core
x-ratelimit-reset: 1782986747

$ curl -s https://api.github.com
{"message":"API rate limit exceeded for 219.65.110.2. (But here's the good news:
Authenticated requests get a higher rate limit. Check out the documentation for
more details.)","documentation_url":".../#rate-limiting"}

$ curl -s https://api.ipify.org        # egress IP GitHub sees
219.65.110.2

General connectivity from the box is otherwise fine:

github.com        -> HTTP 200
cache.nixos.org   -> HTTP 200
DNS               -> resolves

Why it takes down CI

ci/pu/lease.sh's egress health-probe is an unauthenticated curl -sf https://api.github.com. The 403 makes curl -sf (fail-on-≥400) exit non-zero, so lease.sh marks every box NOEGRESS and skips it → the whole linux CI pool is unusable, even though the boxes are healthy.

For the admin — the 403 is the thing to look at

The unauthenticated 60/hr limit is per egress IP, and every CI box shares 219.65.110.2, so the budget is exhausted almost continuously. Options, in rough order of durability:

1. Authenticate GitHub egress from the boxes — put a GITHUB_TOKEN (a PAT / app token) in the box env so api.github.com (and gh) use the authenticated 5000/hr limit instead of 60/hr. This is the real fix — the boxes make authenticated calls anyway during CI.
2. Give the pool a dedicated egress IP (or a small pool of them) instead of sharing one NAT IP with everything else behind that gateway, so the 60/hr budget isn't drained by neighbours.
3. (probe-only mitigation, filed separately) repoint lease.sh's probe off unauthenticated api.github.com to github.com / cache.nixos.org (not unauth-rate-limited) so a rate-limited api.github.com can't false-negative the pool — but that only unblocks the lease gate, not any real authenticated api.github.com usage during a run.

Shared NAT IP: 219.65.110.2 · reset epoch seen: 1782986747.