ClickHouse query cost attribution with log_comment and system.query_log

clickhouse-query-cost-attribution-gist.md

How to attribute ClickHouse query cost by endpoint, user/team and code path

This is the setup I would build if I wanted to answer these questions:

1. How many ClickHouse cores do we need?
2. What rate limit makes sense per endpoint, customer, or use case?
3. What is the max QPS ceiling we can safely give?
4. Which queries, users/teams, endpoints, and backend methods are using the most CPU?

Load tests against your own API are still useful, but this gives you the base database cost of each query shape in production.

1. Use CPU-seconds, not wall time

For query cost, use system.query_log and read:

ProfileEvents['OSCPUVirtualTimeMicroseconds'] / 1000000.0 AS cpu_seconds

This is CPU time summed across threads. It is not wall time.

A query can take 100ms wall time and still consume 1.6 CPU-seconds if it uses many threads. If you are capacity planning, CPU-seconds is the number you want.

Useful fields from system.query_log:

query_id,
query_start_time,
event_time,
query_duration_ms,
read_rows,
read_bytes,
memory_usage,
query_kind,
query,
normalized_query_hash,
ProfileEvents['OSCPUVirtualTimeMicroseconds'] AS cpu_us,
ProfileEvents['RealTimeMicroseconds'] AS real_us,
log_comment

2. Add structured context with log_comment

ClickHouse lets you pass a per-query setting called log_comment. It is then stored in system.query_log.

Put JSON in it:

{
  "request_id": "req_123",
  "user_id": 123,
  "team_id": 456,
  "endpoint": "/v1/jobs/search",
  "channel": "api",
  "caller": "src.domain.jobs.repository.JobRepository.search",
  "git_commit": "abc123"
}

Then extract it in SQL:

JSONExtractString(log_comment, 'endpoint') AS endpoint,
JSONExtractString(log_comment, 'channel') AS channel,
JSONExtractString(log_comment, 'caller') AS caller,
JSONExtractInt(log_comment, 'team_id') AS team_id,
JSONExtractString(log_comment, 'git_commit') AS git_commit

caller means the backend method that issued the ClickHouse query. In Python you can capture it from the stack at your ClickHouse executor boundary. In other languages, do the equivalent at the DB client wrapper layer.

3. Inject it once at the ClickHouse client boundary

Do not add log_comment manually in every repository method. Put it in the ClickHouse executor / client wrapper.

Pseudo-Python:

import inspect
import json
from contextvars import ContextVar

clickhouse_context = ContextVar("clickhouse_context", default={})


def set_clickhouse_context(**kwargs):
    clickhouse_context.set({k: v for k, v in kwargs.items() if v is not None})


def capture_caller():
    for frame in inspect.stack()[2:]:
        module = frame.frame.f_globals.get("__name__", "")
        if module not in {"myapp.clickhouse", "myapp.query_context"}:
            return f"{module}.{frame.function}"
    return "unknown"


def build_log_comment():
    ctx = dict(clickhouse_context.get() or {})
    ctx.setdefault("caller", capture_caller())
    return json.dumps(ctx, separators=(",", ":")) if ctx else None


def execute_clickhouse(session, query, params=None, settings=None):
    merged_settings = dict(settings or {})
    log_comment = build_log_comment()
    if log_comment:
        merged_settings["log_comment"] = log_comment
    return session.execute(query, params=params, execution_options={"settings": merged_settings})

Set the context at request/job entrypoints:

set_clickhouse_context(
    request_id=request.id,
    user_id=user.id,
    team_id=team.id,
    endpoint=request.url.path,
    channel="api",
)

For background jobs:

set_clickhouse_context(channel="task", team_id=team_id)

4. Archive the system tables you need

The live system tables are not a great long-term source of truth. They may have short retention and can disappear across operational events, depending on your setup.

Archive the tables you need into your own MergeTree tables with a TTL. For example, 90 days:

CREATE TABLE IF NOT EXISTS observability.system_query_log_archive
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, query_start_time, query_id, hostname)
TTL event_date + INTERVAL 90 DAY
SETTINGS ttl_only_drop_parts = 1
EMPTY AS
SELECT *, version() AS server_version
FROM system.query_log;

INSERT INTO observability.system_query_log_archive
SELECT *, version() AS server_version
FROM clusterAllReplicas(default, system.query_log)
WHERE (SELECT count() FROM observability.system_query_log_archive) = 0;

CREATE MATERIALIZED VIEW IF NOT EXISTS observability.system_query_log_archive_mv
TO observability.system_query_log_archive AS
SELECT *, version() AS server_version
FROM system.query_log;

Do the same for:

• system.metric_log: total CPU used by the cluster
• system.asynchronous_metric_log: available CPU capacity, for example CGroupMaxCPU
• optionally system.part_log: merges/mutations/background work

Storing server_version is useful. If a query changes performance after a server upgrade, you can group by version and separate app-code changes from engine changes.

5. Build the query-cost view

Create a view/model over the archived query log:

SELECT
  query_id,
  hostname AS host,
  query_start_time,
  event_time,
  event_date,
  query_duration_ms,
  read_bytes,
  read_rows,
  memory_usage,
  query_kind,
  query,
  toString(normalized_query_hash) AS query_shape,
  ProfileEvents['OSCPUVirtualTimeMicroseconds'] AS cpu_us,
  ProfileEvents['RealTimeMicroseconds'] AS real_us,
  JSONExtractString(log_comment, 'channel') AS channel,
  JSONExtractString(log_comment, 'caller') AS caller,
  JSONExtractString(log_comment, 'endpoint') AS endpoint,
  JSONExtractInt(log_comment, 'team_id') AS team_id,
  JSONExtractString(log_comment, 'git_commit') AS git_commit,
  server_version
FROM observability.system_query_log_archive
WHERE type = 'QueryFinish'

Measures:

sum(cpu_us) / 1000000.0 AS cpu_seconds,
avg(cpu_us) / 1000000.0 AS avg_cpu_seconds,
quantile(0.50)(cpu_us) / 1000000.0 AS p50_cpu_seconds,
quantile(0.95)(cpu_us) / 1000000.0 AS p95_cpu_seconds,
quantile(0.99)(cpu_us) / 1000000.0 AS p99_cpu_seconds,
sum(read_rows) AS read_rows,
sum(read_bytes) AS read_bytes,
count() AS query_count

Dimensions to make charts useful:

• caller
• endpoint
• team_id
• channel
• query_kind
• query_shape
• git_commit
• server_version

6. Chart CPU in cores

For a time bucket:

cores = sum(cpu_seconds) / bucket_seconds

Example by caller:

SELECT
  toStartOfInterval(query_start_time, INTERVAL 10 MINUTE) AS bucket,
  JSONExtractString(log_comment, 'caller') AS caller,
  sum(ProfileEvents['OSCPUVirtualTimeMicroseconds']) / 1000000.0 / 600 AS cores
FROM observability.system_query_log_archive
WHERE type = 'QueryFinish'
  AND query_start_time >= now() - INTERVAL 24 HOUR
GROUP BY bucket, caller
ORDER BY bucket

Tables below the chart should include:

• query count
• average CPU-seconds/query
• p50/p95/p99 CPU-seconds/query
• total CPU-seconds
• % of query CPU
• qps
• % of capacity
• max qps

7. Add capacity and max QPS

You need the total CPU capacity for the time window.

If you archive system.asynchronous_metric_log, you can use something like:

SELECT
  event_time,
  sum(value) AS available_cores
FROM observability.system_asynchronous_metric_log_archive
WHERE metric = 'CGroupMaxCPU'
  AND event_time >= now() - INTERVAL 24 HOUR
GROUP BY event_time

For a group over a selected window:

qps = query_count / window_seconds
pct_capacity = group_cpu_seconds / total_available_cpu_seconds * 100
max_qps = qps / (pct_capacity / 100)

Example:

56 qps using 4.2% of capacity
max_qps = 56 / 0.042 = 1333 qps

This is not a target. It is a quick estimate of the ceiling if this query shape used the full machine and per-query cost stayed roughly constant.

8. Views worth building

Start with these:

1. CPU over time grouped by caller
2. CPU over time grouped by endpoint
3. CPU over time grouped by team/customer
4. CPU over time grouped by channel: api, app/ui, export, webhook, task
5. CPU over time grouped by normalized query hash
6. CPU over time grouped by git commit
7. CPU over time grouped by server version
8. Query explorer for recent raw queries

For normalized queries, group by normalized_query_hash and show one sample normalized query text:

normalizeQuery(query)

9. Prompt to ask an AI to build this in your codebase

Use something like this:

I want to build ClickHouse query-cost attribution in this codebase.

Goal:
- attribute ClickHouse CPU usage by endpoint, customer/team/user, channel, backend caller method, normalized query shape, git commit, and ClickHouse server version
- estimate per-group max QPS from query CPU and available CPU capacity
- build charts/tables that help decide how many cores we need and what rate limits make sense per endpoint/customer

Important metric:
- use system.query_log ProfileEvents['OSCPUVirtualTimeMicroseconds'] / 1e6 as CPU-seconds
- do not use wall time as the cost metric

Please inspect the codebase and implement this end to end:

1. Find the central ClickHouse client/executor wrapper.
2. Add a request/job context mechanism that stores request_id, user_id, team_id/customer_id, endpoint, channel, and optional job/export/webhook IDs.
3. Capture the backend caller method that issues the ClickHouse query.
4. Inject a JSON log_comment setting into every ClickHouse query at the executor boundary.
5. Include the deployed git commit when available.
6. Add tests that verify log_comment is valid JSON, includes the expected fields, is safely quoted, and is attached to ClickHouse execution settings.
7. Create archived MergeTree tables for system.query_log, system.metric_log, and system.asynchronous_metric_log with a 90-day TTL. Include server_version with version().
8. Backfill each archive once and add materialized views so new system table rows are copied into the archive.
9. Build a semantic model or SQL views over the archived tables:
   - finished queries only: type = 'QueryFinish'
   - dimensions: caller, endpoint, team/customer, channel, query kind, normalized_query_hash, git_commit, server_version
   - measures: count, total CPU-seconds, avg/p50/p95/p99 CPU-seconds, read rows, read bytes, wall time, memory usage
10. Build dashboard views:
   - stacked CPU cores over time by each dimension
   - a stats table with count, avg/p95/p99 CPU, total CPU, qps, % capacity, max qps
   - a recent queries table with raw SQL, caller, team/customer, CPU, wall time, read rows, read bytes
11. Calculate max_qps as qps / (% capacity / 100).
12. Keep the implementation scoped and use existing patterns in the repo.

Before changing code, show me the relevant files and the plan.
After changing code, run the relevant tests and explain how to verify the dashboard in production.

10. Things to watch out for

• system.query_log can be per replica. If you read it live, use clusterAllReplicas(...) or you may miss rows.
• log_comment must be quoted safely as a ClickHouse string setting.
• If your query already has a SETTINGS clause, make sure your driver supports merging settings. Do not create invalid SQL with two settings clauses.
• Put log_comment last if your driver has settings-order quirks.
• Archive tables should be partitioned by date so the dashboard can prune partitions.
• Do not expose raw query text in a public dashboard. It can contain literal customer filters.
• Keep raw query drilldowns internal.