If you self-host GitHub Actions runners on the same machine as a production Kubernetes cluster, you've got a quiet problem waiting to happen. A heavy CI job, say a Rust cargo build that spawns a dozen rustc processes and a memory-hungry linker, can grab every core and most of the RAM on the box. When that happens your databases don't crash in any obvious way. They just get slow, start missing health checks, and begin flapping. Nobody gets paged for "the linker used 40 gigs," but everyone notices the API timing out.
I ran into exactly this and fixed it with plain systemd resource control on cgroup v2. No nested containers, no extra daemons. Here's what I learned.
One bare-metal box with 12 logical cores (6 physical plus hyperthreading) and 64 GB of RAM. A k3s cluster runs the real work: app pods, Postgres, that sort of thing. Two self-hosted runners sit on the same host and build Rust projects all day.
The rule I wanted to enforce was simple. Production always wins. CI can have whatever is left over, and it should use it well, but it must never take resources the cluster needs.
The defaults will bite you. Mine did, in three ways.
First, no memory limit at all on the runners. A runaway build allocates until the kernel OOM killer steps in, and the OOM killer has no idea your Postgres matters more than a linker. It picks by heuristics, and sometimes it picks wrong.
Second, per-runner limits that quietly add up. "Each runner gets 16 GB" feels safe right up until two jobs run at once and together eat 32. Independent caps don't add up to a global budget, they just sit there next to each other.
Third, conflicting drop-in files. systemd merges drop-ins alphabetically, so if cpu.conf says 300 percent and a later limits.conf says 600, the second one wins. You end up running at double the limit you think you set. The lesson there is to always check the value the kernel is actually enforcing, not the file you edited last.
cgroup v2 through systemd gives you a handful of controls, and the whole game is knowing which one does what.
CPUQuota is a hard ceiling. Set it to 800 percent and the cgroup gets 8 cores at most, even when the box is idle.
CPUWeight is proportional and only matters when there's contention. A low weight means this group yields to higher-weight groups when they both want CPU at the same moment.
MemoryHigh is a soft limit. Cross it and the process gets throttled and put under reclaim pressure. It slows down, but it doesn't die.
MemoryMax is the hard wall. Cross that and something inside this cgroup gets OOM killed, but only inside this cgroup. It can't reach pods in another one.
Two things clicked for me once I understood these.
CPUWeight is the real protection, not CPUQuota. A quota on its own still lets CI grab everything up to the cap while production waits in the run queue. Weight is what actually makes the scheduler prefer production when both want the CPU. Give CI a low weight like 20 and leave production at the default 100. Under load CI gets roughly a fifth of the contested time and backs off on its own. When the box is quiet it can still burst up to its quota.
And you usually want MemoryHigh doing the everyday work, with MemoryMax just as a backstop. When a build crosses MemoryHigh it gets throttled and slowed instead of killed. MemoryMax only exists to save the host as a last resort, and when it fires it kills something inside the CI cgroup, never your database. I set MemoryHigh about 10 to 15 percent below MemoryMax so the gentle brake always comes first.
This was the change that actually fixed things. With two runners, you want a shared slice capping their combined usage, and looser per-runner caps inside it just for fairness.
The slice that bounds both runners together:
# /etc/systemd/system/ci-runners.slice
[Unit]
Description=CI runners slice (bounded so production never starves)
[Slice]
# 12 cores total. Cap the pair at 8 combined, leave ~4 for the cluster and system.
CPUQuota=800%
# Proportional. Production stays at the default 100, so CI yields under contention.
CPUWeight=20
# 64 GB total. Reserve ~26 GB for the cluster and page cache.
MemoryHigh=32G
MemoryMax=36G
# Never let CI touch swap. Swap thrash starves everything through I/O
# even when the RAM numbers look fine.
MemorySwapMax=0
IOWeight=20Then each runner goes into the slice with caps that are on purpose larger than half of it:
# /etc/systemd/system/<runner-service>.service.d/limits.conf
[Service]
Slice=ci-runners.slice
CPUQuota=500%
MemoryHigh=18G
MemoryMax=20G
MemorySwapMax=0
TasksMax=8192
# On OOM, kill the build process, not the long-lived runner agent.
OOMPolicy=continue
Restart=always
RestartSec=10The reason the per-runner caps exceed half the slice is that a single job, when it's the only thing building, should be able to stretch out to 5 cores and 20 GB. But two jobs at once are still clamped by the slice to 8 cores and 36 GB between them. You get good throughput for the common case and a firm ceiling on the total.
systemctl daemon-reload
systemctl restart <runner-service> # heads up: this kills any running jobDon't trust the file you just wrote. Read the enforced value straight from the cgroup:
base=/sys/fs/cgroup/ci.slice/ci-runners.slice
cat $base/cpu.max # "800000 100000" means 8 cores
cat $base/cpu.weight
cat $base/memory.max
cat $base/memory.highThis is how you catch the alphabetical-merge trap. The config file says one thing, the kernel enforces another, and only the cgroup tells you the truth.
One operational note. Restarting the service kills whatever job is in flight. If you can, wait until there's no Runner.Worker process around before you apply, or do it in a quiet window.
Limits are a safety net, not a cure. For Rust most of the memory pain is at link time, and it scales with how many linkers run at the same time. Two cheap changes go a long way.
Switch the linker to mold, or at least lld. The default bfd and gold linkers are slow and hungry, and mold uses far less of both time and memory. Then cap build parallelism with CARGO_BUILD_JOBS so you don't get a dozen codegen units and several linkers all peaking at once. While you're there, drop fat LTO and trim debuginfo in CI, since both are big memory multipliers you rarely need for a check build.
Fix the build and you mostly stop approaching the wall in the first place, which means the cgroup caps only ever matter for the genuinely pathological case.
Capping CI bounds the aggressor. If you want to go further you can also guarantee the victim. Put the cluster in its own slice with a MemoryMin, which is a working set the kernel promises never to reclaim and is perfect for keeping a database's hot pages resident, plus a high CPUWeight and a high IOWeight. Now it's belt and suspenders. CI can't take too much, and production is promised a floor.
Putting CI and production on the same box is a fine idea and a good way to use spare capacity, as long as you make the priority explicit to the kernel instead of hoping it sorts itself out. One slice bounds the whole CI fleet so concurrent jobs can't add up past your budget. CPUWeight makes production win the races while CPUQuota just sets the ceiling. MemoryHigh throttles and MemoryMax kills, but only ever inside the CI cgroup, so production is never the one that dies. No swap for CI. Check the cgroup, not the config file. And lighten the builds so none of it matters most days.
Production keeps its floor, CI gets the leftovers and uses them well, and nothing fights.