Skip to content

Instantly share code, notes, and snippets.

@StephanDollberg

StephanDollberg/cluster-upgrade-analysis.md

Created March 13, 2026 11:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save StephanDollberg/11d763e996f9d911d4acb70f08d9f22f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save StephanDollberg/11d763e996f9d911d4acb70f08d9f22f to your computer and use it in GitHub Desktop.
Download ZIP
Cluster cjk99o0a3v32vef0rqtg upgrade analysis: v25.3.10 → v26.1.1-rc2 (March 11, 2026)
Raw
cluster-upgrade-analysis.md

Cluster cjk99o0a3v32vef0rqtg Upgrade Analysis

Cluster: cjk99o0a3v32vef0rqtg (preprod) Upgrade: v25.3.10 → v26.1.1-rc2 Date: March 11, 2026 ~14:42-14:45 UTC

Upgrade Timeline

All 3 nodes restarted simultaneously (not rolling) at 14:42-14:45 UTC on March 11. Additional restarts occurred through March 13 01:04 UTC.

Node Restart Timeline

Node Restarts
rp-...-a-0 Mar 11 14:42, Mar 12 00:27, 01:05, 15:02, 15:40
rp-...-a-1 Mar 11 14:43, Mar 12 13:00, 13:40, 15:19, Mar 13 00:26
rp-...-a-2 Mar 11 14:45, Mar 12 00:43, 13:18, Mar 13 00:42, 01:04

Comparison Windows

Window Period Description
Pre-upgrade Mar 11 08:00-14:00 UTC v25.3.10, 6h, 0 under-replicated replicas
Post-upgrade stable Mar 13 02:00-12:00 UTC v26.1.1-rc2, 10h, all nodes stable since 01:04, 0 under-replicated replicas

The post-upgrade window (Mar 13 02:00-12:00) was chosen because all three nodes had completed their final restarts by 01:04, there were zero under-replicated replicas, and the workload (produce/fetch rate, partition count) is identical to pre-upgrade.

An earlier clean window (Mar 12 02:00-12:00) also exists, though some nodes had not yet completed their final restarts.

Partition count remained constant at 467 throughout.

Overall Metrics Comparison

Metric Pre (v25.3.10) Post (v26.1.1-rc2) Change
Produce rate 41,516 rec/s 41,415 rec/s -0.2%
Fetch rate 41,612 rec/s 41,526 rec/s -0.2%
Request bytes 130.3 MB/s 131.2 MB/s +0.7%
Avg CPU busy 57.8% 53.9% -6.8%
IO Write ops/s 18,207 18,344 +0.8%
IO Read ops/s 740 754 +1.9%
Kafka RPC p99 latency 319.2 ms 314.9 ms -1.3%
Internal RPC p99 latency 8.56 ms 8.72 ms +1.9%
Cloud storage upload 57.1 MB/s 56.7 MB/s -0.8%

Produce Latency

Aggregate

Metric Pre (Mar 11 08-14) Post (Mar 13 02-12) Change
Produce avg latency 3.468 ms 3.077 ms -11.3%
Produce p50 latency 2.890 ms 2.466 ms -14.7%
Produce p95 latency 8.029 ms 7.744 ms -3.6%
Produce p99 latency 16.126 ms 15.518 ms -3.8%
Produce request rate 1,992 req/s 2,027 req/s +1.8%

Per-Node

Node Pre p50 Post p50 Change Pre p99 Post p99 Change
a-0 2.809 ms 2.391 ms -14.9% 15.364 ms 14.782 ms -3.8%
a-1 2.985 ms 2.404 ms -19.5% 17.749 ms 15.930 ms -10.3%
a-2 2.872 ms 2.599 ms -9.5% 16.548 ms 16.734 ms +1.1%

Hourly Produce Average Latency Trend

Mar 11 08:00  3.442ms
Mar 11 09:00  3.380ms
Mar 11 10:00  3.510ms
Mar 11 11:00  3.511ms
Mar 11 12:00  3.486ms
Mar 11 13:00  3.478ms
Mar 11 14:00  3.709ms  <-- UPGRADE
Mar 11 15:00  3.264ms
Mar 11 16:00  3.225ms
Mar 11 17:00  3.183ms
Mar 11 18:00  3.180ms
Mar 11 19:00  2.996ms
Mar 11 20:00  2.968ms
Mar 11 21:00  2.921ms
Mar 11 22:00  2.960ms
...
Mar 13 02:00  2.744ms
Mar 13 03:00  2.852ms
Mar 13 04:00  2.909ms
Mar 13 05:00  2.923ms
Mar 13 06:00  3.157ms
Mar 13 07:00  3.211ms
Mar 13 08:00  3.253ms
Mar 13 09:00  3.326ms
Mar 13 10:00  3.216ms
Mar 13 11:00  3.185ms

Hourly Produce p99 Latency Trend

Mar 11 08:00  15.884ms
Mar 11 09:00  15.700ms
Mar 11 10:00  16.127ms
Mar 11 11:00  16.279ms
Mar 11 12:00  16.058ms
Mar 11 13:00  16.707ms
Mar 11 14:00  22.417ms  <-- UPGRADE (transient spike)
Mar 11 15:00  22.905ms
Mar 11 16:00  23.326ms
Mar 11 17:00  23.587ms
Mar 11 18:00  22.889ms
Mar 11 19:00  14.930ms
Mar 11 20:00  14.833ms
Mar 11 21:00  14.531ms
Mar 11 22:00  14.762ms
...
Mar 13 02:00  14.169ms
Mar 13 03:00  14.723ms
Mar 13 04:00  15.252ms
Mar 13 05:00  15.340ms
Mar 13 06:00  15.794ms
Mar 13 07:00  16.185ms
Mar 13 08:00  16.079ms
Mar 13 09:00  16.432ms
Mar 13 10:00  15.883ms
Mar 13 11:00  15.303ms

Note: Histogram bucket boundaries are at powers of 2 (2.047ms, 4.095ms, 8.191ms, 16.383ms), so quantile estimates carry some interpolation error. The sum/count average (-11.3%) is exact.

CPU Efficiency: Scheduler Runtime Breakdown

Scheduling Group Pre (s/s) Post (s/s) Change Abs Diff
produce 0.619 0.521 -15.9% -0.098
fetch 0.707 0.609 -13.9% -0.098
main 0.700 0.614 -12.3% -0.086
raft_send 0.236 0.202 -14.4% -0.034
raft_recv 0.392 0.345 -12.0% -0.047
admin 0.017 0.008 -53.1% -0.009
archival_upload 0.204 0.190 -7.2% -0.015
kafka 0.251 0.247 -1.6% -0.004
log_compaction 0.077 0.083 +8.0% +0.006
ts_read 0.226 0.225 -0.4% -0.001
raft_hb 0.006 0.004 -32.2% -0.002
cache_background_reclaim 0.011 0.010 -8.0% -0.001
cluster 0.009 0.008 -10.8% -0.001
TOTAL 3.455 3.065 -11.3% -0.390

Three new scheduling groups appeared post-upgrade (cloud_topics_compaction, cloud_topics_metastore, cloud_topics_reconciler) but consume negligible CPU.

Per-Node CPU

Node Pre CPU Post CPU Change
a-0 52.9% 51.7% -2.3%
a-1 60.9% 53.8% -11.7%
a-2 59.5% 56.6% -4.9%

Key Findings

  1. Produce latency improved: Median latency dropped ~15%, average dropped ~11%, p99 dropped ~4%. The improvement was immediate after the upgrade.

  2. CPU efficiency improved: Total scheduler runtime decreased 11.3% with identical throughput. The largest savings came from the produce, fetch, main, and raft scheduling groups.

  3. Throughput unchanged: Produce rate (~41,500 rec/s), fetch rate (~41,500 rec/s), and request bytes (~131 MB/s) are effectively identical pre and post upgrade.

  4. I/O unchanged: Both read and write I/O ops are within ~2% of pre-upgrade levels.

  5. Tail latency note: p99 produce latency showed a transient spike (15ms → 23ms) in the first ~4 hours after upgrade (14:00-18:00 Mar 11), then settled to ~15.5ms which is slightly better than the pre-upgrade ~16.1ms.

  6. Node a-1 benefited most: Saw the largest p50 improvement (-19.5%), p99 improvement (-10.3%), and CPU reduction (-11.7%).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment