gxrdb — a Galaxy data store

A centrally hosted store of Galaxy operational data, contributed by usegalaxy.* servers. Its job is to capture the data reliably, keep it manageable, and let consumers extract from it quickly — nothing more.

Scope

In scope. Getting job-level data out of each Galaxy database cheaply and repeatedly; normalising it once so no consumer has to; storing it so a query over a month or a year returns in seconds; and operating all of that with a one-time setup per site.

Out of scope for now. Tool performance analysis, per-site configuration analysis, TPV default derivation, runtime prediction. These are consumers. They appear here only as a check that the store can serve them later, never as work to be done now. Where earlier investigation produced findings about tools or sites, those have been demoted to Appendix: evidence — they justify design decisions but are not the objective.

Requirements

From the operators of the servers whose data we need:

One-time setup per site. The KUI model: install a cron job once, never touch it again. No expectation of ongoing attention.
No service runs locally. A script that reads Postgres and pushes somewhere is acceptable; a local database, daemon or transformation layer is not.
Cheap and safe against production. Bounded, resumable, interruptible.
Privacy is a project-internal matter. Emails and free text stay home; numeric identifiers may travel. The store is not public.

And from the consumers:

Quick extraction. A question spanning a month, a year, or all servers should be one query returning in seconds — not a self-join over a hundred-million-row entity-attribute-value table.
No re-implementation. Unit conversion, cgroup version differences and tool-ID parsing are done once, at ingest, not by every consumer.

What is captured

Three tiers, in priority order. Only tier 1 is hard; the rest follow the same mechanics.

Tier 1 — jobs and their metrics. One row per terminal job, with metrics pivoted into columns. This is the data that is expensive to get and impossible to query usefully in place, so it is the whole point of the project.

Group	Fields
Identity	`server_id`, `job_id`, `create_time`, `update_time`, `month`
Outcome	`state`, `exit_code`
Tool	`tool_id_raw`, `tool_owner`, `tool_repo`, `tool_name`, `tool_version`, `tool_id_short`
Placement	`destination_id`, `handler`, `runner_name`
User	`user_id`
Allocated	`runtime_seconds`, `slots_allocated`, `memory_allocated_mb`
Consumed	`cpu_seconds`, `memory_peak_bytes`, `oom_events`
Node	`processor_count`, `memory_total_bytes`
Data	`input_count`, `input_bytes`, `input_bytes_max`, `output_count`, `output_bytes`
Quality	`has_core_metrics`, `has_cgroup_metrics`, `cgroup_version`, `cpu_exceeds_allocation`, `metrics_trust`

A note on slots_allocated: it is what Galaxy requested, and at least one contributing site does not enforce it as a CPU cap — 14% of jobs on its main destination consume more CPU than their declared slots allow. Treat it as a declaration, not a limit, and never assume cpu_seconds ≤ slots × runtime.

session_id was dropped. Its only real use is telling anonymous activity apart: a job run by a logged-out user has a null user_id but a session, so without it every anonymous job looks like the same null user. That is a question nobody has asked, it is a second pseudonymous identifier crossing the wire, and it is not needed by anything in scope. Add it back if counting anonymous usage becomes a goal — but as a deliberate decision, not by default.

How the variable-arity data is captured

A job may have one input or five hundred, so the shape of the input and output relationships is the one place this schema could go wrong. The wrong answer is a wide table — input1_bytes, input2_bytes and so on — which breaks on the first job that exceeds the column count. There are only two sane options, and the plan uses both, at different layers:

The landing layer keeps one row per input, exactly as Galaxy stores it. Variable arity is a solved problem in a normalised table.
fact_job carries scalars: input_count, input_bytes, input_bytes_max, output_count, output_bytes. One row per job, fixed width, no arity to reason about, no join for the common case.

The aggregation happens centrally, not at the site. That matters because the extraction read is identical either way — the query already walks job_to_input_dataset to compute a sum — so shipping the un-aggregated rows costs nothing extra at the site while leaving every future aggregate available without a re-extraction. Given that a backfill on the slowest site is 41–58 hours, "capture raw once, aggregate as often as you like" is worth the storage.

The storage is modest: measured on usegalaxy.org, jobs average 2.2–2.8 inputs and about three quarters of jobs have any input at all, so a per-input table runs roughly two to three times the job row count.

input_bytes_max is included because it is free — the same GROUP BY produces it — and because for many tools the largest single input predicts cost better than the total does.

A gap to close before phase 1. Galaxy records job inputs across three tables: job_to_input_dataset, job_to_input_dataset_collection, and job_to_input_dataset_collection_element. The query measured in phase 0 read only the first, so any tool consuming a whole collection would report an understated — possibly zero — input_bytes. The extractor has to cover all three, and the cost of the extra traversal needs measuring, since the single-table version already accounts for a third of chunk time on the slowest site. Until that is measured, treat the phase 0 timings for input sizes as a lower bound.

Tier 2 — users. user_id and create_time only. Cheap, and it is what lets registration and retention questions be answered from the store instead of from a live Galaxy query.

Tier 3 — entity counts. Monthly totals for histories, datasets, workflows and workflow invocations. Deliberately not row-level: usegalaxy.org has ~147M datasets, and capturing them would double the project to answer questions that a COUNT already answers. These are aggregated at the source, exactly as kui.sh does today.

Not captured at all: emails, usernames, history and dataset names, tool_state parameters, command lines, stdout/stderr, tracebacks, working directories.

Extraction

The measured constraint that shapes everything: the only fast access path in Galaxy's schema is by job_id. job_metric_numeric indexes job_id but not plugin or metric_name; job has no index on create_time. A query filtered by date and metric name sequentially scans hundreds of millions of rows — one such query was abandoned after five hours. The same data read in job_id ranges returns in seconds. (Appendix A.)

So the extractor asks only job-ID questions and never date questions:

watermark ← highest job id exported so far
ceiling   ← max(job.id) at start of run, minus a safety lag
for each ~50k-id chunk between watermark and ceiling:
      export jobs, metrics, and data associations for that range
      upload
      advance watermark

Every source table is reached by an indexed job_id range:

Landing table	Read from	Rows per job
`job`	`job`	1
`job_metric`	`job_metric_numeric`, `job_metric_text`	~7 wanted
`job_input`	`job_to_input_dataset` + the two collection tables	2.2–2.8
`job_output`	`job_to_output_dataset`	unmeasured

The data associations land un-aggregated, one row per input or output; the scalars consumers read are computed centrally. See How the variable-arity data is captured.

Three properties this buys, all of which serve "easy management":

No date filtering, so the missing create_time index never matters. The month is derived centrally from create_time, which travels with the row.
Resumable and idempotent. A run that dies resumes at the last committed chunk; re-exporting a chunk is harmless because the central side upserts on (server_id, job_id). An admin can rerun the script by hand without risk.
Late-completing jobs self-correct via a lagged ceiling plus a trailing overlap window, with the upsert absorbing the correction.

The queries live in gxadmin, not in this project

Every SQL statement the extractor runs is a gxadmin query. gxrdb-export.sh supplies orchestration — the watermark, the chunk loop, retry, upload — and nothing else. It owns no SQL.

This is the longer route and it is chosen deliberately. The queries stop being private to one pipeline: they are reviewed, maintained and improved by the community that understands Galaxy's schema, they become available to every admin whether or not their site contributes to gxrdb, and this project stops carrying schema knowledge it is not well placed to maintain.

The mechanics already fit. query_csv() in parts/03-query-utils.sh is a server-side COPY … TO STDOUT WITH CSV — streaming and unbuffered, which is exactly what bulk extraction needs — so gxadmin csvquery <name> --id-min=… --id-max=… | gzip is a first-class pipeline rather than a workaround. The ##? <arg> [--flag=<v>] convention carries the chunk bounds, and meta ADDED: gives a version to gate on.

Fixes to existing queries. These are worth filing on their own merits; gxrdb is not the reason they are wrong.

job-metrics joins dataset.id = job_to_input_dataset.dataset_id, but that column is a foreign key to history_dataset_association.id. Its total_filesize and num_files are therefore computed from unrelated datasets. The correct three-table path already exists in the same file, in tool-memory-per-inputs.
job-metrics, tool-memory-per-inputs and jobs-max-by-cpu-days read only cgroup v1 metric names. Of the three servers surveyed, two are wholly or mostly cgroup v2, so those queries return null memory or CPU on most modern deployments. A COALESCE across the v1 and v2 names fixes it everywhere.
monthly-cpu-stats has no plugin = 'core' filter, so a second plugin emitting runtime_seconds or galaxy_slots would multiply rows and inflate the total.
job-metrics is unbounded — its own help warns it scans every job and dataset row. Optional --id-min / --id-max make it usable on a large instance at all.

New queries. An export-* family of ID-bounded raw dumps mirroring the landing tables — jobs, job metrics, job inputs, job outputs — plus the metric trust diagnostic from Appendix B. "Get my job data out for offline analysis" is a real admin need, and the diagnostic found a destination reporting 2.3M× inflated CPU and another silently reporting none at all.

The version dependency, and why it is acceptable. gxrdb will require a minimum gxadmin version, pinned once the queries land (current release is 22). gxrdb-export.sh checks gxadmin version at startup and refuses to run with a clear message rather than producing wrong data. Ordinarily a version floor across six independently operated servers would violate the one-time-setup constraint, but gxadmin is a single file installed by curl … > /usr/bin/gxadmin && chmod +x, so upgrading is one line in the same session that installs the collector. parts/22-query.sh also shows queries landing every month or two, so the review path is active rather than dormant.

If a PR stalls, gxadmin supports local functions, so a site can carry a query locally as a stopgap. That reintroduces per-site setup and should be treated as a temporary measure, not a design.

What a site installs

The same shape as kui.sh, because that deployment model is already proven with these admins: a script in collector/, a config file, one cron line, sharing kui.conf.

Galaxy production Postgres
        │  gxadmin csvquery export-*  --id-min --id-max   (community-maintained)
        ▼
 collector/gxrdb-export.sh        ← orchestration only; owns no SQL
        │  bq load  (reuses the service account already set up for KUI)
        ▼
 BigQuery                         ← everything below is centrally operated

No local database, no DuckDB, no Parquet at the site. gxadmin streams CSV, gzip compresses, bq load uploads. BigQuery rather than a bucket because sites already have gcloud and bq configured for KUI, and the project already runs a dataset with per-server table-scoped IAM — so a contributing site needs no new credentials, no new CLI, no new permissions model.

Prerequisites at a site are therefore: gxadmin at or above the pinned version, the KUI service account key, and one cron line.

Backfill policy

Every site backfills to 2026-01-01, and no further. One date, fleet-wide, configurable in one place.

A uniform floor is worth more than the extra history a per-site policy would buy. Every cross-server query has the same window, so no consumer has to reason about which servers happen to reach back further, and no aggregate needs a footnote about ragged coverage. It also removes the per-site judgement call from onboarding, which matters when setup has to be a one-time effort.

The cost is modest even on the slowest site. Job counts are from the KUI snapshots; chunk times from Appendix A:

Site	Jobs since 2026-01	Chunks	Estimated backfill
usegalaxy.org	5.2M	104	~3.8 h
usegalaxy.eu	8.6M (4 months)	172	~9 min
usegalaxy.au	1.8M	35	~2 min
usegalaxy.fr	1.0M	20	~1 min
usegalaxy.ca	82k	2	seconds
usegalaxy.be	23k	<1	seconds

Roughly 16.7M jobs fleet-wide. usegalaxy.org is a single overnight run rather than the two-to-three-day job a full-history backfill would have been; everywhere else it finishes while the admin is still reading the instructions.

Finding the starting job ID. create_time is not indexed, so the floor cannot be applied as a WHERE clause without the sequential scan this design exists to avoid. Instead, binary-search the ID space using primary-key probes:

lo, hi ← min(job.id), max(job.id)
while hi - lo > 1000:
      mid ← (lo + hi) / 2
      if create_time of the first job with id >= mid < 2026-01-01:
            lo ← mid
      else: hi ← mid
start_id ← lo - 50_000        # safety margin

About 27 probes for a 100M-row ID space, each a single indexed lookup. Job IDs are only approximately monotonic in create_time, so the result is biased downward by a margin and the central side discards anything genuinely earlier than the floor. Slight over-extraction costs nothing; under-extraction would silently lose days.

What the floor costs. Year-over-year comparison and seasonality analysis need twelve months, so neither is possible from gxrdb until January 2027. That is a real limitation and worth stating to consumers rather than discovering later. KUI is unaffected: its monthly aggregates back to 2022-05 are already committed as JSON and are not being re-derived.

The floor can be moved earlier later, per site or fleet-wide, at the cost of reintroducing ragged coverage — so it should be a deliberate decision, not something that drifts.

Storage and management

Landing tables, one set per server, written by bq load in the shape extracted. Never queried directly by consumers; they exist so a bad transformation can be re-run without re-extracting from any site.

fact_job, the queryable table, built centrally from the landing tables:

Partitioned by month, so a query over one month scans one month.
Clustered by server_id, tool_id_short, destination_id — the three dimensions nearly every question filters on.
Wide and pre-pivoted. The entity-attribute-value layout is resolved once, here. No consumer ever writes a self-join to get two metrics.

Normalisation applied once, at transform time:

Pivot job_metric_numeric from EAV into columns.
Unify cgroup v1 and v2 into single fields — cpu_seconds from cpuacct.usage (ns) or cpu.stat.usage_usec (µs); memory_peak_bytes from memory.max_usage_in_bytes or memory.peak; oom_events from memory.failcnt or memory.events.oom_kill. Consumers never see which version a node ran. This is load-bearing, not defensive: one contributing server is 79% v2, another is 100% v1.
Parse toolshed paths into owner/repo/name/version plus a short ID.
Normalise units (bytes, seconds) and timezones (UTC).
Set quality flags, including metrics_trust — see below.

Quality flags rather than silent nulls. A period with metrics disabled must be distinguishable from a period of cheap jobs, and unreliable readings must be filterable without every consumer rediscovering why.

Quality is recorded at two levels, because one is not enough:

Per row, computed at transform time. has_cgroup_metrics because coverage varies within a site — one server has destinations with core metrics and no cgroup metrics at all. cpu_exceeds_allocation because the physical sanity check is a property of the row, not of the destination.
Per (destination, metric family, period), as metrics_trust, for the cases where a whole destination's accounting is wrong rather than a particular job being unusual.

The row level is load-bearing. A destination-only classification would have forced an all-or-nothing decision on one server's main destination, which is 86% sound and 14% anomalous — discarding it would have thrown away 139k good rows, and keeping it unflagged would have poisoned every aggregate. Appendix B records the three distinct failure modes this has to survive.

Management properties, stated as commitments:

Schema versioned, with schemaVersion on every landing file.
Transformations re-runnable centrally, without touching any site. This matters disproportionately: fixing a bug in site-side logic would take months to propagate across six independently operated servers, if it ever completed.
Per-server isolation, so one site's malformed export cannot corrupt another's data.
Staleness monitoring that alerts the project, not the site. A scheduled check on per-server watermarks raises an issue when a server stops contributing — the same pattern KUI uses. Requirement 1 means failures cannot depend on site attention.
A deletion path, per server and per month, for both landing and fact tables.

Query surface

What "quick extraction" means concretely — these should each be one query, in seconds, with no joins beyond the fact table:

Monthly job counts, by server, by state, by tool.
Allocated core-hours per month per server, from slots_allocated and runtime_seconds.
Consumed CPU where trusted (metrics_trust = 'ok').
Per-tool distributions of runtime, memory and input size.
Everything KUI currently derives from a dozen gxadmin invocations.

Materialised aggregates sit on top for the highest-traffic cuts, principally the KUI monthly numbers. KUI keeps its committed-JSON-to-git publishing model — reviewable diffs, no credentials for contributors — with gxrdb feeding it rather than replacing it.

Phasing

Phase 0 — measure the access path per site. gxrdb-probe.sql. Done for org, AU and EU; three servers remain. Confirms the extraction cost each site will carry.
Phase 1 — upstream the queries. Two pull requests against gxadmin: the fixes to existing queries, and the new export-* family plus the trust diagnostic. This gates everything downstream, so it starts first and its review cycle is the critical path. File the job-metrics join fix separately and immediately — it returns wrong numbers to anyone using it today and should not wait on the rest.
Phase 2 — extractor, one server. gxrdb-export.sh against usegalaxy.org, calling the released gxadmin queries. Exit: a month of jobs lands, with metric coverage matching monthly-jobs.
Phase 3 — transform and prove. Build fact_job; reproduce every KUI number from 2026-01 onward from gxrdb alone. Reproducing known-correct numbers is the cheapest proof the pipeline is sound; earlier months are out of reach by design and stay served by the committed snapshots.
Phase 4 — KUI reads from gxrdb. kui.sh becomes a thin query instead of a dozen gxadmin calls. This is where the project pays for itself, and it is reversible.
Phase 5 — roll out. Remaining servers; tier 2 and tier 3 data.
Later — consumers. Tool performance, TPV, prediction. Out of scope until the store exists and is trusted.

Phases 0–4 are worth doing on their own merits, and nothing central is required until phase 3. Phase 1 delivers value even if the rest of gxrdb is never built — the fixes correct queries the community is already relying on.

Risks

Extraction cost on the largest site. usegalaxy.org needs 20–30 minutes a month, with the job table already costing ~3.4 buffers per row. That is fine now and worth re-timing annually.
Schema drift in Galaxy. The extractor selects named columns and must fail loudly on a rename rather than silently emit nulls. Pin to a tested Galaxy release range. Routing the queries through gxadmin helps here: schema changes become the community's problem to track, which is a large part of why the long route is worth taking.
gxadmin is now on the critical path. Two dependencies follow. The review cycle gates phase 2, and a query whose output columns change upstream would break CSV parsing downstream — so the extractor must validate the header row and fail rather than mis-load. Mitigated by gxadmin being a single-file install, so a version floor is a one-line upgrade, and by pinning a minimum version that is checked at startup.
Metric coverage and meaning vary by site. Handled by normalisation and quality flags rather than by assuming uniformity; the flags are only as good as the classification behind them (Appendix B).
Text metrics differ by site and cannot be hardcoded — one server records scheduler and hostname fields another does not. The extractor needs a configurable list, and site-internal identifiers are a governance question even within one project.
Ownership. BigQuery at these volumes costs a few dollars a month, so cost is not the risk; a named operator is. Sites will not notice a broken central pipeline.
Something like this may already exist. Several usegalaxy sites run job telemetry into Grafana or similar. Worth asking before building. Open.

Sizing

Measured, not estimated. Roughly 7 wanted metric rows per job after filtering.

With the 2026-01-01 floor, the initial load is 16.7M jobs fleet-wide — roughly 120M metric rows and 40M input rows, on the order of 1.5 GB gzipped in total. Ongoing:

	usegalaxy.org	usegalaxy.au	usegalaxy.eu
Highest job ID	~79M	~15.7M	~111M
Monthly jobs	~750k	~250k	~2.1M
Monthly upload	50–70 MB gzipped	~20 MB	100–150 MB gzipped

Fleet-wide that is a few hundred megabytes a month and a few tens of gigabytes a year. Storage is not the constraint. Note that EU is the largest by job ID and by monthly volume yet the cheapest to extract from, which is why extraction cost and data volume must be estimated separately.

One incidental benefit of reading Galaxy directly: usegalaxy.eu's KUI snapshots are missing jobs data for 2026-03 through 2026-05, but those jobs are still in its database. A gxrdb backfill to the floor recovers them.

Appendix: evidence

Findings from investigation on usegalaxy.org and usegalaxy.au. Retained because they justify specific design decisions; not objectives in themselves.

A. Extraction cost

Measured with gxrdb-probe.sql, 50k-job windows, recent and mid-history.

Query	org recent	org cold	AU (200k)	EU (200k)
Jobs	104.6 s	59.8 s	2.0 s	1.8 s
Metrics	7.5 s	13.7 s	7.2 s	1.9 s
Input sizes	26.4 s	52.5 s	—	—

Plans are index-driven throughout on all three servers; no sequential scans. The query that previously ran five hours returns in 7.5 seconds when driven by job_id.

org's cost is site-specific, not a function of scale. EU's highest job ID is 110.7M against org's 79.1M — EU is not the smaller database — yet EU extracts 200k jobs about 200× faster. Whatever makes org expensive (table bloat, storage, contention) is particular to that deployment. The plan should therefore not present extraction cost as scaling with fleet size, and backfill policy stays a per-site measurement.

Two incidental findings: count(*) over recent job IDs timed out at 120 s while the same count over older IDs took 452 ms — recent pages lack visibility-map coverage — so the extractor must never call count(*); and recent, un-vacuumed pages are the expensive ones, so a lagged ceiling is cheaper as well as safer.

B. Why the store carries a trust classification

Metric meaning is not uniform, which is why quality is carried as data rather than left to consumers. Three servers produced three distinct failure modes.

Consumed CPU can be sanity-checked against cpu_seconds ≤ slots × runtime. Applied per destination over 200k jobs per site:

1. Inflated — the metric is wrong (usegalaxy.org). Every org destination but one sits below 1.0 with a maximum around 1.0. expanse reports a median efficiency of 2,260,439, on 100% of its jobs — consistent with a counter accumulating for the node since boot. Its memory metrics are fine, which is why trust is tracked per metric family. Excluding it costs 4.9% of consumed-CPU rows. Destination-level exclusion is the right remedy here.

2. Deflated — the metric misses the work (usegalaxy.au, usegalaxy.eu). Three AU GPU destinations report a median 0.45 CPU-seconds and 11.5 MB peak across AlphaFold2 jobs with a 59-minute median wallclock; EU's condor_container_gpu shows a median efficiency of 0.011 and embedded_pulsar_docker a memory ratio of 0.006. The cgroup is measuring a wrapper while the real work runs in a container or on a GPU outside it. GPU destinations under-report CPU at both sites — a systematic pattern, not a one-off.

Deflation resists automatic detection: a tool delegating to a remote web service legitimately reports near-zero and looks identical. The store records a classification from a flag-then-confirm process (gxrdb-metrics-probe.sql sections 4 and 7) rather than pretending the rule is mechanical.

3. Partial — the metric is right, the allocation is not (usegalaxy.eu). The newest and most consequential mode. EU's main destination condor_container (139k jobs) has a sane median efficiency of 0.556, but 13.8% of its rows are physically impossible, with a maximum of 1168.

The cause is not a broken counter. The impossible rows are overwhelmingly slots=1 jobs — 20% of all slots=1 jobs — and the median cores actually used on those rows is 1.8. These are single-slot jobs quietly running multithreaded, because the site does not enforce slots as a CPU cap. The cpu_seconds values are correct; it is slots_allocated that does not mean what a reader would assume. A small extreme tail remains (p99 of 27.5 cores, max 1168) that is genuinely anomalous.

Memory tells the complementary story on the same destination: 3.1% of jobs peak within 2% of their allocation and OOM kills occur on 0.01%, so a memory limit exists and occasionally binds. Memory is capped there; CPU is not.

This mode is why quality must be recorded per row. A destination-level verdict on condor_container is wrong either way — excluding discards 139k good rows, including unflagged poisons every aggregate. And it is why cpu_exceeds_allocation is stored as a flag rather than used as a filter at ingest: the underlying cpu_seconds is good data, and only the ratio is meaningless.

A limitation of the cross-destination calibration surfaced here too: it depends on a tool running on several destinations. EU puts 143k of 200k jobs on one destination, so that diagnostic returned only two usable rows. It caught expanse instantly and is near-useless on a homogeneous site.

C. Metric coverage and normalisation

	usegalaxy.org	usegalaxy.au	usegalaxy.eu
Jobs with any metric	84.4%	91.2%	88.8%
cgroup version	79% v2, 5.3% v1	100% v1	100% v2
OOM signal available	yes (v2)	no	yes (v2)
Text metrics	`uname`, container	+ `env/SLURM_*`, `env/HOSTNAME`	`hostname/hostname`
Destinations with no cgroup data	—	—	4 pulsar destinations

Three servers, three configurations. The unification step is required for the data to be comparable at all — one site is entirely v1, another entirely v2, the third mixed. Where memory.limit_in_bytes reads as 2^63−1 no limit is enforced, so memory.failcnt is uniformly zero and that site has no OOM signal at all; two of three sites do have one.

Text metrics are different at every site — three servers, three disjoint sets, none of which can be hardcoded. EU records hostname/hostname and no uname at all. Site-internal identifiers such as hostnames and scheduler job IDs are also a governance question even within one project.

Coverage varies within a site, not only between sites. EU has four pulsar destinations carrying core metrics but no cgroup metrics whatsoever, which is why has_cgroup_metrics has to be a per-row column.

Also present and worth capturing: cpuinfo/processor_count and meminfo/memtotal (~84–91% coverage) describe the node a job landed on, which is what makes it possible to interpret whether a cgroup figure is job-scoped.

D. Cross-server comparability

p95 peak memory for tools well represented on both servers, as a check that pooled queries are meaningful:

Tool	org GB	AU GB	AU/org
featurecounts	8.0	8.0	1.00
prokka	1.6	1.6	0.98
cutadapt	20.2	19.1	0.94
fastp	14.9	17.7	1.19
bowtie2	22.9	14.4	0.63
kraken2	327.0	497.9	1.52
upload1	3.7	0.3	0.08

Most tools agree within tens of percent. The outliers are explicable — differing user data for upload1 and fastqc, differing reference databases for kraken2 — rather than evidence of measurement error. Pooled queries are sound; consumers comparing servers should surface per-server figures rather than averaging silently.

E. Tooling produced

gxrdb-probe.sql — access-path and cost measurement. Run per site to confirm extraction cost.
gxrdb-metrics-probe.sql — metric inventory, coverage by destination, and the trust diagnostics behind metrics_trust.
Logs and 200k-job samples from usegalaxy.org, usegalaxy.au and usegalaxy.eu.

Both probes are prototypes for gxadmin queries rather than permanent tooling. Sections 4 and 7 of the metrics probe become the trust diagnostic proposed in phase 1; the phase 0 probe's three chunk queries become the export-* family. Once upstream, this project keeps neither.

afgane/gxrdb_plan.md

Select an option

No results found