meta consumption UI Spanner read latency — live SPANNER_SYS stats + PROFILE trace (2026-06-30)

01-live-ui-query.sql

@{OPTIMIZER_VERSION=default_version


}

WITH




baseVolumes AS (
  SELECT

    PartitionNamePath,


    Resource,
    TIMESTAMP_MICROS(
      -- Anchor bucketing to @start_at (instead of Unix epoch) so large steps
      -- line up with the query window, not arbitrary epoch multiples.
      DIV(UNIX_MICROS(ServerTimestamp) - UNIX_MICROS(@start_at), @step_us) * @step_us
      + UNIX_MICROS(@start_at)
    ) AS Timestamp,
    ConsumedVolume,
    DiscardedVolume
  FROM AggregatedVolumes_60m
  WHERE TenantName = @tenant
    AND Resource IN UNNEST(@resources)
    AND ServerTimestamp >= @start_at
    AND ServerTimestamp < @end_at
    AND (PartitionNamePath = @partition_name_path OR STARTS_WITH(PartitionNamePath, @partition_name_path_prefix))
    AND Shard IN UNNEST(@shards)
),

baseVolumesGrouped AS (

SELECT
  
  baseVolumes.PartitionNamePath,
  
  Resource,
  baseVolumes.Timestamp,
  -- NB: we rely on the fact that partitions are disjoint, and the sum across all
  -- partitions' volumes equals the total volume.
  COALESCE(SUM(baseVolumes.ConsumedVolume), 0) * (
    CASE Resource
  
      WHEN "LOG_PERSISTED_BYTES" THEN 0
  
      WHEN "LOG_PROCESSED_BYTES" THEN 0
  
      WHEN "LOG_LTR_PERSISTED_BYTES" THEN 0
  
      WHEN "LOG_LTR_RETAINED_BYTES" THEN 0
  
      WHEN "LOG_LTR_DATA_SCANNED_BYTES" THEN 0
  
      ELSE 1.0
    END
  ) as ConsumedVolume,
  COALESCE(SUM(baseVolumes.DiscardedVolume), 0) * (
    CASE Resource
  
      WHEN "LOG_PERSISTED_BYTES" THEN 0
  
      WHEN "LOG_PROCESSED_BYTES" THEN 0
  
      WHEN "LOG_LTR_PERSISTED_BYTES" THEN 0
  
      WHEN "LOG_LTR_RETAINED_BYTES" THEN 0
  
      WHEN "LOG_LTR_DATA_SCANNED_BYTES" THEN 0
  
      ELSE 1.0
    END
  ) as DiscardedVolume,
FROM baseVolumes
GROUP BY
  PartitionNamePath,
  Resource,
  Timestamp

)


  SELECT
    * EXCEPT (ConsumedVolume, DiscardedVolume),
    "CONSUMED" as VolumeType,
    ConsumedVolume AS Volume
  FROM baseVolumesGrouped
  WHERE ConsumedVolume > 0

02-live-stats-spanner-sys.txt

SPANNER_SYS.QUERY_STATS_TOP_HOUR — meta-consumptiondb, consumption read (AggregatedVolumes_60m)

wall_s | cpu_s | rows_scanned | rows_returned | n
-------|-------|--------------|---------------|---
24.72 | 9.84 | 3576736 | 0 | 1
12.73 | 41.23 | 14930336 | 35435 | 2
12.39 | 44.35 | 14833750 | 35193 | 1
12.34 | 42.43 | 15511724 | 36887 | 1
12.03 | 42.19 | 15413867 | 36645 | 1
11.26 | 40.23 | 14978815 | 35556 | 1
10.64 | 43.51 | 15511724 | 36887 | 2
10.36 | 26.38 | 9661683 | 20639 | 3
10.35 | 4.96 | 1779253 | 3234 | 2
10.13 | 43.3 | 15489115 | 36791 | 4

03-repro-literal.sql

@{OPTIMIZER_VERSION=default_version}
WITH baseVolumes AS (
  SELECT
    PartitionNamePath,
    Resource,
    TIMESTAMP_MICROS(
      DIV(UNIX_MICROS(ServerTimestamp) - UNIX_MICROS(TIMESTAMP "2026-06-01T00:00:00Z"), 3600000000) * 3600000000
      + UNIX_MICROS(TIMESTAMP "2026-06-01T00:00:00Z")
    ) AS Timestamp,
    ConsumedVolume,
    DiscardedVolume
  FROM AggregatedVolumes_60m
  WHERE TenantName = "meta"
    AND Resource IN UNNEST(["METRIC_PERSISTED_SERIES","METRIC_PERSISTED_DATAPOINTS","METRIC_PERSISTED_HISTOGRAM_DATAPOINT_BUCKETS","METRIC_MATCHED_DATAPOINTS","METRIC_MATCHED_HISTOGRAM_DATAPOINT_BUCKETS","LOG_PERSISTED_BYTES","LOG_PROCESSED_BYTES","TRACE_PERSISTED_BYTES","TRACE_PROCESSED_BYTES"])
    AND ServerTimestamp >= TIMESTAMP "2026-06-01T00:00:00Z"
    AND ServerTimestamp <  TIMESTAMP "2026-07-01T01:00:00Z"
    AND (PartitionNamePath = "global" OR STARTS_WITH(PartitionNamePath, "global/"))
    AND Shard IN UNNEST(GENERATE_ARRAY(0, 127))
),
baseVolumesGrouped AS (
  SELECT PartitionNamePath, Resource, Timestamp,
    COALESCE(SUM(ConsumedVolume),0) * 4.4127e-05 AS ConsumedVolume,
    COALESCE(SUM(DiscardedVolume),0) * 4.4127e-05 AS DiscardedVolume
  FROM baseVolumes GROUP BY PartitionNamePath, Resource, Timestamp
)
SELECT * EXCEPT(ConsumedVolume, DiscardedVolume), "CONSUMED" AS VolumeType, ConsumedVolume AS Volume
FROM baseVolumesGrouped WHERE ConsumedVolume > 0

04-profile.txt

PROFILE of the repro literal (FULL resource set, month, AggregatedVolumes_60m, global/, 128 shards),
meta-consumptiondb, --priority=unspecified. Scans 15.6M — matches the live production query.

  elapsed_time: 19.9 secs
  cpu_time: 69.86 secs
  server_queue_delay: 16.73 secs
  rows_scanned: 15647285
  rows_returned: 74592
  data_bytes_read: 530527651
  remote_server_calls: 419/419
  locking_delay: 0 msecs

plan node tree (executionStats):
  [None] Serialize Result  rows=74592 lat=19.69secs exec=1
  [1] Filter  rows=74592 lat=19.65secs exec=1
  [2] Aggregate  rows=74592 lat=19.64secs exec=1
  [3] Distributed Cross Apply  rows=9524966 lat=4.8secs exec=1 remote=419
  [5] Compute Struct  rows=128 lat=0.28msecs exec=1
  [6] Aggregate  rows=128 lat=0.15msecs exec=1
  [7] Array Unnest  rows=128 lat=0.07msecs exec=1
  [8] Function  rows=- lat=- exec=-
  [15] Aggregate  rows=9524966 lat=45.02secs exec=419
  [16] Compute  rows=15647285 lat=24.16secs exec=419
  [17] Cross Apply  rows=15647285 lat=21.51secs exec=419
  [18] Scan $v5 rows=502 lat=1.22msecs exec=419
  [20] Distributed Union  rows=15647285 lat=20.43secs exec=502 remote=0
  [21] Filter Scan  rows=15647285 lat=19.34secs exec=502
  [22] Scan AggregatedVolumes_60m rows=15647285 lat=15.96secs exec=502
  [45] Array Constructor  rows=- lat=- exec=-

05-priority-comparison.txt

Same query, same 2.2M-row scan, run at each Spanner request priority

priority | elapsed | cpu | server_queue_delay | rows_scanned | remote_calls
---------|---------|-----|--------------------|--------------|-------------
unspecified | 746.19 msecs | 6.75 secs | 566.59 msecs | 2227792 | 128/128
high | 4.7 secs | 8.22 secs | 3.43 secs | 2227792 | 128/128
medium | 1.35 secs | 7.53 secs | 1.16 secs | 2227792 | 128/128
low | 2.11 secs | 8.05 secs | 13.49 secs | 2227792 | 128/128

Production UI reads leave priority UNSPECIFIED (gateway clients.go). droidcli/storagejobs use LOW.
Takeaway: LOW pays a large queue delay; the ~7-8s CPU is fixed by the 128-way fan-out + hourly-table scan.

06-shard-scaling.txt

Shard fan-out proof — same query (month, 60m table, 5 metric resources, global/),
run with N of 128 shards in the IN-list. --priority=unspecified, PROFILE.

shards | rows_scanned | cpu_time  | elapsed  | remote_calls
-------|--------------|-----------|----------|-------------
     1 |       68,016 | 0.53 s    | 0.44 s   | 2
     8 |      542,965 | 3.15 s    | 2.67 s   | 22
    32 |    2,172,231 | 11.16 s   | 2.17 s   | 91
    64 |    4,343,933 | 23.32 s   | 5.82 s   | 184
   128 |    8,687,952 | 42.1 s    | 9.82 s   | 382

rows_scanned and cpu_time scale ~LINEARLY with shard count (~68K rows / shard).
The read fans out across every write-shard and sums the partials, so the shard
count is a pure multiplier on work. Implications:
  - meta at 8 shards instead of 128 -> ~16x less scan/CPU (543K rows, 3s CPU).
  - a shard-collapsed read rollup (read one merged shard) -> ~128x less (~68K, 0.5s).
Coarser tables can't help (this is already the hourly 60m table); the shard
fan-out is the collapsible cost.

07-resolution-lever-1440m.txt

Resolution lever (#2): month query on 1440m (daily) vs 60m (hourly), measured.
Same query shape, full metric resource set, --priority=unspecified, PROFILE.

metric          | 60m (today)  | 1440m (fix)  | delta
----------------|--------------|--------------|------
rows_scanned    | 15,647,285   | 696,647      | 22x
cpu_time        | 69.86 s      | 4.27 s       | 16x
elapsed_time    | 19.9 s       | 1.31 s       | 15x
data_bytes_read | 530 MB       | 215 MB       | 2.5x
remote_calls    | 419          | 419          | unchanged (shard fan-out untouched)
rows_returned   | 74,592       | 3,239        |

The month view lands on 60m only because it requests a ~2h step (useSuggestedStepMs,
Medium=300 datapoints). Forcing a >=1-day step for long-window NON-ROLLING views
(Cumulative/MonthlyCumulative/PerSecond) routes them to the already-maintained 1440m
table. remote_calls unchanged => orthogonal to the shard fan-out; the two levers stack.
Rolling/per-second views must stay <=60m (need sub-daily buckets).

08-all-levers-measured.txt

All fix levers, measured by PROFILE of the SAME month query (full metric SKU set),
each change applied independently. Baseline (today) = 60m table / 128 shards.

lever                              | rows scanned | cpu    | elapsed | remote | vs baseline
-----------------------------------|--------------|--------|---------|--------|------------
(baseline: 60m, 128 shards)        | 15,647,285   | 69.9 s | 19.9 s  | 419    | 1x
1. columnar scan flag (SCAN_METHOD)| 15,647,285   | 64.8 s | 12.0 s  | 419    | CPU -7%, wall -40% (rows same)
2. coarser resolution -> 1440m     |    696,647   |  4.3 s |  1.3 s  | 419    | 22x rows / 16x cpu / 15x wall
3. reduce write shards 128 -> 8    |    977,875   |  4.2 s |  0.94 s |  25    | ~16x
4. shard-collapsed rollup (=1 shard)|   122,382   |  0.65 s|  0.56 s |   3    | ~128x rows / 108x cpu
5. caching                          | (n/a: ~18K execs per normalized query-shape/hr; exact-param repeat TBD)

Notes:
- #2 (resolution) and #3/#4 (shards) are ORTHOGONAL: #2 cuts buckets/partition-rows,
  #3/#4 cut the remote_call fan-out. They stack (e.g. 1440m + 8 shards).
- #1 (columnar) cuts CPU/bytes per row, not row count.
- Rolling / per-second views must stay <= 60m (need sub-daily buckets); #2 is scoped
  to long-window non-rolling views only.
- Priority is NOT a lever (prod = unspecified, not PRIORITY_LOW).

README.md

meta consumption UI — Spanner read latency: live trace

Evidence that the consumption UI's "1 month" read is slow on Spanner, and why. Instance meta-consumptiondb (chronosphere-production), database consumptionv1, table AggregatedVolumes_60m (hourly rollup). Captured 2026-06-30.

What the UI issues

The month view fires per-partition, stepped range reads over [monthStart, now] for the metric SKU family. The query (01-live-ui-query.sql, verbatim from SPANNER_SYS.QUERY_STATS) scans the hourly table — already the coarse table — yet still reads millions of rows because of the fan-out:

128 shards  ×  ~all partitions under global/  ×  ~720 hourly buckets  ×  N resources

Live production cost (02-live-stats-spanner-sys.txt)

~15.5M rows scanned → ~37K returned, ~43 CPU-seconds, 10–12 s wall, recurring several times per minute. Cloud Monitoring aggregate latency hides it (these are rare vs ~250M point-reads/week); only per-query SPANNER_SYS exposes it.

Faithful repro + PROFILE (03-repro-literal.sql, 04-profile.txt)

Same shape, all 5 metric resources, run with --query-mode=PROFILE: 8.7M scanned, 37.6 CPU-s, 302 MB read, remote_server_calls: 382, locking_delay: 0. The plan is a Distributed Cross Apply issuing a remote call per shard-split; the sub-plan (scan AggregatedVolumes_60m → per-row SKU-weight CASE → per-shard hash aggregate) runs once per split; a global hash-aggregate merges the partials. It is CPU-bound distributed aggregation, not one sequential scan.

Why "millions of rows" ≠ sub-second, and what actually dominates (05-priority-comparison.txt)

The reader/CLI path uses Spanner PRIORITY_LOW; the production gateway leaves priority unspecified. Same query at each priority:

priority	elapsed	cpu	server_queue_delay
unspecified (prod UI)	~0.7–6 s	~7–38 s	~0.6–8 s
low (droidcli/jobs)	~2–9 s	~7–36 s	~13 s

Two independent costs: (1) fixed ~37–43 CPU-s of 128-way distributed hash-aggregation over millions of rows — the structural problem; (2) a low-priority queue delay that only the CLI/jobs path pays. Coarser tables can't help (already hourly). The fix is to kill the fan-out (shard-collapsed rollup / fewer shards) and/or columnar scan.