Date: 2026-06-04
Istio: v1.28.5 · Kubernetes: v1.34.6 (all spokes) · Harness: a4b8a18
Topology: 1 hub (rosa-001, no mesh) + 10 mesh spokes (rosa-002 … rosa-011), multi-primary / multi-network. rosa-002 is the source/primary; mesh size N = rosa-002 plus N−1 remotes.
istiod: pinned 3 replicas/spoke (autoscaleEnabled=false), no istiod HPA. CPU request 250m (O5 fix), mem req 2Gi / limit 8Gi.
Status: 4 of 5 suites complete and clean. The 5th (churn-dataplane) is re-running on a harness fix (see §6); partial results through mesh-size 3 are clean. This report will be finalized when that sweep completes.
| Dimension | Range exercised |
|---|---|
| Mesh size (clusters) | 1 → 10 |
| istiod replicas / spoke | 3 (pinned) |
| Connected proxies (control-plane suite, max) | 20 |
| Connected proxies (churn suite, source / remote max) | 36 / 216 |
| Services in registry | 10 per cluster |
| Sidecar scoping modes | none, namespace, explicit (Sidecar API) |
| Data-plane load | 10 / 100 / 500 / 1000 QPS, 8 connections |
| Churn intensity | scale 1↔5 replicas across 5 deployments; rates 1 / 5 / 10 ops/s |
| Pods scheduled (control-plane suite, max) | 136 / 750 allocatable (~18%) |
Headline reads:
- Control plane is comfortable at 10 clusters — istiod CPU peaks ~0.4 cores, heap ~110–190 Mi, 0 restarts across all 30 combinations.
- Sidecar scoping cuts per-proxy config ~88–89% (3.8 MB → 0.4–0.5 MB), stable across mesh size.
- Cross-cluster data-plane overhead is ~1 ms at p50 (local ~2.4 ms vs remote ~3.5 ms) and 100% success (pct_200 = 1.0) at every QPS level up to 1000.
- Config/endpoint propagation to remote sidecars is the dominant latency (tens of seconds at mesh 10) — expected for full-mesh EDS fan-out, and the main scaling signal.
Measures time for a new endpoint to propagate: P1 (local xDS push), P2 (remote istiod EDS), P3 (remote sidecar apply). Pre-warm + label-flip topology (O1 fix) so P3 measures propagation, not pod boot.
Parameters: source rosa-002; mesh 1→10; iterations 10/size; poll 250 ms; timeout 120 s; settle 5 s; backer hashicorp/http-echo:1.0. Sweep 20260604T024318Z-1085945 — 10/10 sizes, 0 errors (~1h47m).
| Mesh | P1 local xDS p50 / p99 (ms) | P2 remote istiod EDS p50 / p99 (ms) | P3 remote sidecar p50 / p99 (ms) |
|---|---|---|---|
| 1 | 2027 / 2106 | — (no remote) | — (no remote) |
| 2 | 2013 / 2072 | 2012 / 2079 | 18215 / 46424 |
| 10 | 2043 / 2296 | 2106 / 2533 | 13243 / 62909 |
Read: local xDS push is flat ~2.0–2.3 s regardless of mesh size; remote istiod EDS ~2.1 s; the long pole is remote sidecar application (p50 ~13–18 s, p99 up to ~63 s at mesh 10) as the endpoint fans out across all clusters. P1 histogram-convergence rows report 0-100 ms where valid. No TIMEOUT_P3, no skew outliers (improvement over the prior workaround pass).
On the P3 magnitude: these seconds-to-tens-of-seconds values are real (raw per-iteration P3 at mesh 10 ranges ~2 s to ~40 s, varying by remote cluster within a single iteration), not an aggregation artifact — verified against the raw TSVs. P3 is the dominant scaling signal and the high variance is the thing most worth scrutinizing in deeper analysis; the harness is reporting it faithfully. The non-monotonicity between mesh sizes (e.g. mesh-2 avg > mesh-10 avg) is sample-size driven — mesh 2 has n=10 vs mesh 10's n=90.
Measures istiod convergence and push amplification while 5 deployments scale 1↔5. Parameters: mesh 1→10; 5 deployments; scale 1→5; 5 iterations/size; churn intensity 5; istiod 3 replicas. Sweep 20260604T043213Z-1465754 — 10/10 stages, 0 errors.
| Mesh | local_avg (ms) | remote_reach_avg (ms) | remote_eds_avg (ms) | src→rmt pushes | src / rmt proxies | amplification |
|---|---|---|---|---|---|---|
| 1 | 2442 | — | — | 98 / 0 | 33 / 0 | 0.1 |
| 2 | 2558 | 7817 | 2989 | 328 / 515 | 36 / 28 | 0.5 |
| 3 | 2836 | 8224 | 4303 | 362 / 1049 | 24 / 48 | 1.1 |
| 4 | 2855 | 9360 | 4839 | 407 / 1395 | 24 / 72 | 0.8 |
| 5 | 3101 | 9629 | 4849 | 391 / 1959 | 24 / 96 | 0.4 |
| 6 | 3077 | 10486 | 4775 | 442 / 2291 | 24 / 120 | 0.5 |
| 7 | 3270 | 10531 | 4915 | 293 / 1828 | 24 / 144 | 0.3 |
| 8 | 3483 | 11993 | 4172 | 358 / 2670 | 24 / 168 | 0.4 |
| 9 | 3326 | 10179 | 3529 | 290 / 2632 | 24 / 192 | 0.6 |
| 10 | 3682 | 11154 | 3913 | 328 / 2980 | 24 / 216 | 0.3 |
Read: local convergence grows gently (~2.4 → 3.7 s) with mesh size; remote reach plateaus ~10–12 s; push amplification stays ≤ ~1.1 (no push storm). Remote proxy count scales linearly (24 → 216 = 24/cluster × remotes), confirming correct full-mesh wiring. Queue p99 0-100 ms throughout.
istiod CPU/mem/heap and config-dump size across mesh size × sidecar scoping. Parameters: mesh 1→10; 10 services; 3 replicas; 1 namespace; scopings none/namespace/explicit; 3 config-dump samples; settle 60 s. Sweep 20260604T072535Z-51665 — 30/30 combos, 0 FailedScheduling, 0 restarts (O5 fix validated at full 10×3).
| Mesh | Scoping | istiod CPU avg (m) | mem avg (Mi) | heap inuse (Mi) | conv p99 | proxies | cfg dump (MB) |
|---|---|---|---|---|---|---|---|
| 1 | none / ns / explicit | 139 / 95 / 83 | 354 / 329 / 329 | 187 / 154 / 169 | ≤500 / ≤100 / ≤100 | 11 | 3.8 / 0.4 / 0.4 |
| 5 | none / ns / explicit | 271 / 173 / 195 | 350 / 326 / 332 | 179 / 175 / 169 | ≤500 / ≤100 / ≤100 | 11 | 3.8 / 0.5 / 0.5 |
| 10 | none / ns / explicit | 402 / 310 / 313 | 354 / 340 / 344 | 158 / 166 / 157 | ≤500 / ≤100 / ≤100 | 11 | 3.9 / 0.5 / 0.5 |
Sidecar-scoping effect on per-proxy config size (lower is better):
| Mesh | none (MB) | namespace (MB) | explicit (MB) | reduction |
|---|---|---|---|---|
| 1 | 3.8 | 0.4 | 0.4 | 89.3% |
| 5 | 3.8 | 0.5 | 0.5 | 88.2% |
| 10 | 3.9 | 0.5 | 0.5 | 87.0% |
Read: istiod CPU rises modestly with mesh size and is highest with no scoping (402 m at mesh 10) vs ~310 m with scoping; memory is flat ~330–354 Mi; heap inuse ~150–190 Mi. Scoping sidecars cuts per-proxy config ~87–89% and trims istiod CPU. Convergence p99 stays in the 0–500 ms band; 0 istiod restarts across all 30 combos. (Capacity-percentage fields read unknown for this sweep — a provenance gap fixed under #45/PL36, not a measurement failure.)
fortio request latency, local vs cross-cluster (remote) targets. Parameters: mesh 1→10; QPS 10/100/500/1000; 8 connections; 30 s/level; 5 s warm-up; settle 30 s; fortio/fortio:1.69.5. Sweep 20260604T114908Z-831116 — 10×4 QPS, pct_200 = 100% everywhere.
Representative p50 / p99 (ms), local vs remote:
| Mesh | QPS | local p50 / p99 | remote p50 / p99 |
|---|---|---|---|
| 2 | 1000 | 1.97 / 2.99 | 3.17 / 4.78 |
| 5 | 1000 | 2.24 / 3.00 | 2.90 / 9.18 |
| 10 | 10 | 3.42 / 17.51 | 3.75 / 6.98 |
| 10 | 100 | 2.77 / 4.61 | 3.57 / 6.10 |
| 10 | 500 | 2.43 / 5.79 | 3.02 / 5.65 |
| 10 | 1000 | 2.37 / 3.06 | 2.77 / 6.36 |
Read: cross-cluster adds ~0.5–1.0 ms at p50 (local ~2.0–2.8 ms vs remote ~2.8–3.6 ms), flat across mesh 1→10. p99 stays single-digit ms at scale; the occasional higher local p99 at 10 QPS is small-sample tail noise (n=1). Every level achieved 100% success (pct_200 = 1.0) and full target QPS.
Measures added data-plane tail latency (Δp99) while the mesh churns under live fortio load. Parameters: mesh 1→10; churn rates 1/5/10 ops/s; 5 deployments; scale 1→5; QPS 200; 8 connections; baseline 60 s + churn 60 s/combo. Sweep 20260604T170554Z-2030208 (in progress).
The original run (…1174742) lost ~40% of its matrix to a setup/teardown race introduced by the O8 deploy-once-per-mesh-size refactor: per-mesh-size namespace teardown (fortio + sidecar-injected churn pods) exceeded its 180 s budget → CLEANUP_TIMEOUT; the next mesh-size's setup then collided with the still-Terminating namespace → SETUP_FAILED for all its combos. The failure path absorbed the remaining teardown, so the following mesh-size recovered — a deterministic odd-✓ / even-✗ oscillation (mesh sizes 2, 4, 6 lost entirely).
Fix: (A) setup now waits out any pre-existing Terminating namespace before applying (delay, not destroy); (B) cleanup fast-drains sidecar pods (5 s grace) before deleting the namespace; shared timeout 180 → 240 s. Reviewed by the 7-agent scale-test review — all six reviewers approved, round 1, zero blocking findings; lesson captured as PL37. Validated live: the always-failing even mesh-sizes now pass; fast-drain handled every teardown, with the setup-wait as the safety net for rare slow-finalize cases.
| Mesh | Churn rate | baseline p50 / p99 (ms) | churn p50 / p99 (ms) | Δp99 (ms) | xDS pushes | Status |
|---|---|---|---|---|---|---|
| 1 | 1 | 2.33 / 3.54 | 2.71 / 19.53 | 15.99 | 7117 | OK |
| 1 | 5 | 2.32 / 3.49 | 2.72 / 19.85 | 16.36 | 40282 | OK |
| 1 | 10 | 2.20 / 3.00 | 2.94 / 19.80 | 16.80 | 84026 | OK |
| 2 | 1 | 2.33 / 3.95 | 3.09 / 21.63 | 17.67 | 8047 | OK |
| 2 | 5 | 2.35 / 3.95 | 2.99 / 19.21 | 15.26 | 49886 | OK |
| 2 | 10 | 2.43 / 3.99 | 3.05 / 24.93 | 20.94 | 101956 | OK |
| 3 | 1 | 2.35 / 3.98 | 2.92 / 17.46 | 13.49 | 9101 | OK |
| 3 | 5 | 2.40 / 4.15 | 3.20 / 19.91 | 15.77 | 48347 | OK |
| 3 | 10 | 2.46 / 3.98 | 3.24 / 20.07 | 16.09 | 99…k | OK |
(Δp99 = churn p99 − baseline p99, the harness's churn-sensitivity metric; verified = churn_p99 − baseline_p99 on every combo. Mesh-sizes 4–10 pending.)
Read so far: idle p99 ~3–4 ms; under continuous churn p99 rises to ~17–25 ms, i.e. Δp99 ~13–21 ms of added tail latency, flat across mesh size and churn rate; p50 barely moves (~2.3 → 3.2 ms). xDS push volume scales cleanly with churn rate (~7 k at rate 1 → ~100 k at rate 10). 0 failures, 0 restarts across all completed combos — the fix is holding at full scale.
| # | Suite | Sweep | Result |
|---|---|---|---|
| 1 | Configuration / endpoint propagation | …024318Z-1085945 |
✅ 10/10 sizes, 0 errors |
| 2 | Service/endpoint churn convergence | …043213Z-1465754 |
✅ 10/10 stages, 0 errors |
| 3 | Control-plane resource scaling | …072535Z-51665 |
✅ 30/30 combos, 0 restarts |
| 4 | Data-plane latency | …114908Z-831116 |
✅ 10×4 QPS, pct_200 = 100% |
| 5 | Churn under data-plane load | …170554Z-2030208 |
🔄 re-running on PR #50 fix — clean through mesh 3 |
Notable fixes applied this campaign: O5 (istiod CPU request, unblocks full 10×3 control-plane scheduling) · O1 (propagation pre-warm so P3 measures propagation, not boot) · O10 / PR #50 (churn-dataplane cleanup-cascade).
Generated 2026-06-04 from the per-suite sweep summaries under tests/<suite>/results/. Cluster Kubernetes version v1.34.6 on all spokes; Istio v1.28.5; harness commit a4b8a18.