snoby-demo-org observability platform design: Mimir + Loki + Tempo + Grafana + OTEL Collector (AKS, Flux GitOps, multitenant, Grafana v13 Git Sync self-service dashboards)

architecture.md

Observability Platform — snoby-demo-org

Stack: Mimir (metrics) · Loki (logs) · Tempo (traces) · Grafana (UI) · OTEL Collector (ingestion) Deploy mechanism: Flux GitOps (clusters/aks root, HelmRelease per component) Status: designed, repo scaffolded at snoby-demo-org/observability; AKS-only authoring, no live cluster yet.

---

1. Architecture

                    ┌────────────────────────────────────────────┐
 apps / infra       │           OTEL Collector gateway           │
 pods ──OTLP──▶     │  DaemonSet per node: OTLP receivers +      │
                    │  filelog (node/container logs), k8sattrs   │
                    └──────┬──────────────┬──────────────┬───────┘
                           │ prometheusrw │ OTLP/HTTP    │ OTLP/gRPC
                           ▼              ▼              ▼
                        ┌──────┐      ┌──────┐       ┌───────┐
                        │Mimir │      │ Loki │       │ Tempo │──generators──▶ Mimir
                        └──┬───┘      └──┬───┘       └───┬───┘   (service graph, spanmetrics)
                           └────────────┼───────────────┘
                                        ▼
                                 ┌────────────┐
                                 │  Grafana   │◀── Backstage (backstage-plugin-loki)
                                 └────────────┘

All three backends are stateless compute over object storage; everything durable lands in object storage. Latency is dominated by the local cache tiers, not the backend.

Component decisions

Concern	Choice	Why
Metrics store	Mimir, microservices mode (distributor/ingester/querier/store-gateway ×2)	Native Prometheus remote-write HA; horizontal scaling
Logs	Loki, SimpleScalable (write ×2 / read ×2 / backend ×2 + compactor)	Standard mid-scale shape; TSDB index, single-store config
Traces	Tempo, monolithic (single-binary), scale queriers vertically	Grafana's own guidance: scale query-frontends vertically; promote to microservices only when search volume demands
Ingestion front door	OTEL Collector gateway (DaemonSet)	One OTLP endpoint for apps; enrichment, tenant stamping, routing, retries; isolates apps from backend APIs
Node logs	OTEL filelog receiver in the collector DaemonSet	LOCKED: collector-only, single agent, no Promtail
Caching	Memcached tiers: Loki chunk+index cache, Mimir block cache, Tempo bloom/footer cache	Cache hit ratios (~90%) are what actually decide query latency
UI	Grafana in-stack (HelmRelease, provisioned datasources)	Datasource federation to any external Grafana later is trivial
Alerting	Mimir ruler + Alertmanager, metrics-only for now	LogQL alerting deferred
Certs/auth	Internal ingress only; basic auth/OIDC at ingress for push+query APIs	Backends don't do end-user auth; ingress is the choke point

2. Multitenancy (from day one)

Tenants: infra, backstage, crypto.

• OTEL Collector is the tenant-stamping authority. Apps cannot set X-Scope-OrgID; the collector derives the tenant from the namespace label observability.lyoshkatel/tenant=<tenant> via k8sattributes + transform processors, unlabelled namespaces default to infra.
• Ingress validates the header against the allowed tenant list per backend.
• Per-tenant ingester limits, retention overrides, and Grafana datasources (9 provisioned: mimir|loki|tempo × infra|backstage|crypto, pinned with X-Scope-OrgID).
• Object store layout partitions per tenant natively — adding tenants later is additive config, no migration.

3. Storage

Two profiles, same component configs — endpoint/auth swap only:

	AKS (production target)	Lab (k3s)
Backend	Native Azure Blob (hierarchical namespace disabled — required by Mimir)	MinIO (S3-compatible)
Auth	Workload identity (OIDC federated) for Loki + Tempo; account key for Mimir (Mimir does NOT support federated tokens)	MinIO access keys
Containers/buckets	mimir-blocks, mimir-alertmanager, mimir-ruler, loki-chunks, tempo-blocks	bucket prefixes per profile

S3 vs Blob: Blob is a production-safe but second-class client (historically e.g. grafana/mimir#2229 buffered whole TSDB files in memory on the Azure client). Deciding factor: performance is cache-dominated, so either backend works once the memcached tiers are sized. Azure Blob chosen for AKS because the cluster lives there; the config abstraction keeps the S3 path available.

Retention (object store lifecycle + native deletion)

Signal	Retention	Mechanism
Traces (Tempo)	30 days	Tempo retention
Logs (Loki)	90 days	Loki compactor retention_enabled (retention-driven deletion on object backends)
Metrics (Mimir)	13 months	Mimir compactor blocks_retention_period + downsampling

4. Repo layout

observability/
├── clusters/
│   ├── aks/                      ← Flux Kustomization root (watched)
│   │   ├── flux-system/          HelmRepository (grafana helm-charts)
│   │   ├── mimir/                HelmRelease + values-base.yaml (configMapGenerator)
│   │   ├── loki/                 HelmRelease + values-base.yaml
│   │   ├── tempo/                HelmRelease
│   │   ├── grafana/              HelmRelease (9 datasources + Git Sync repo resources)
│   │   ├── otel-collector/       gateway.yaml (DaemonSet, config, RBAC, svc)
│   │   └── tenants/
│   │       ├── infra/            namespace label (observability.lyoshkatel/tenant=infra)
│   │       ├── backstage/
│   │       └── crypto/
│   └── lab/                      k3s + MinIO profile (same values, endpoint swap)
├── dashboards/                   Git Sync targets: dashboards/<team>/ (bidirectional)
├── docs/                         architecture.md (this file), runbook.md
└── .github/workflows/            CI: lint, kubeconform, helm template diff

5. Ingestion & onboarding

App onboarding (3 steps):

1. Label the app's namespace: observability.lyoshkatel/tenant=<tenant>
2. Point OTLP at otel-collector-gateway.otel.svc:4317 (gRPC) or :4318 (HTTP)
3. Use your tenant's Grafana datasources (mimir-<tenant>, loki-<tenant>, tempo-<tenant>)

Collector pipelines:

Pipeline	Receivers	Key processors	Exporters
traces	otlp	k8sattributes → tenant-stamp → batch	otlp/tempo
metrics	otlp	k8sattributes → tenant-stamp → batch	prometheusremotewrite/mimir
logs	otlp + filelog (/var/log/pods/**)	k8sattributes → tenant-stamp → batch	otlphttp/loki (/otlp)

Tempo's service-graph and spanmetrics generators remote-write into Mimir under tenant infra — RED metrics derived from traces appear next to raw metrics.

Backstage integration: the existing backstage-plugin-loki points at the new Loki query API using the backstage tenant credentials.

6. Grafana self-service (LOCKED: Git Sync)

Dashboards are UI-first self-service with git as the source of record, using Grafana's built-in Git Sync (GA 2026-04-20, self-managed OSS included):

• Bidirectional sync between Grafana and this repo — dashboards edited in the UI are committed to git automatically; git changes sync back into Grafana (60s poll on OSS, or instant via webhooks). GA announcement (2026-04-20, self-managed Grafana v13.0.0+): https://grafana.com/whats-new/2026-04-20-git-sync-for-grafana-dashboards-and-folders-now-generally-available/ Also shipped in that release: folder permissions restored to the folder UI, clearer connection/commit error handling, and an enhanced connection status overview.
• One Git Sync connection per team, pointing at its own repo path (dashboards/infra/, dashboards/backstage/, dashboards/crypto/), with folder sync so each team's dashboards land in their own Grafana folder. Teams onboard by adding a subtree.
• PR-on-save enabled by default: UI edits open pull requests instead of pushing straight to main — dashboard changes stay reviewable without gating day-to-day editing.
• No export CronJob or sidecar needed: Git Sync replaces the homegrown "export dashboards to repo" pattern. The connection is a declarative Grafana "repository resource," provisioned through the HelmRelease values (fits Flux GitOps).
• Open decision: whether to also deploy the official Grafana MCP server (mcp-grafana) for agent-driven dashboard/query access — orthogonal to Git Sync; not yet decided.

7. Flux / GitOps flow

• Cluster root: Flux watches clusters/aks kustomization → per-component kustomizations.
• Components = HelmRelease (chart source: grafana helm repo) with values via configMapGenerator/valuesFrom — pinned chart versions (>=5.5.0 mimir-distributed, >=6.0.0 loki, >=1.10.0 tempo, >=13.0.0 grafana — Git Sync GA requires self-managed Grafana v13.0.0+).
• Per standing convention: no manual kubectl rollouts — all changes via git commit → Flux.
• Image/config automation (collector config bundle) follows the GHCR image-automation pattern used elsewhere in the org.
• CI (.github/workflows/validate.yaml): yamllint, kustomize render + kubeconform strict (datreeio CRD catalog, Git Sync repository resources included), and helm template render of each component's base values to catch chart-pin/value-schema drift.

8. Risks / tradeoffs

• Mimir microservices is operationally heavy (~8 deployments). Lab profile reuses the shared base values; scale-down on lab is a profile overlay, keeping drift minimal.
• Azure Blob second-class client — mitigate with cache sizing; watch for Azure-client regressions upstream.
• Mimir needs account-key auth (no federated tokens) — key must be mounted via KeyVault CSI, never in git.
• Collector-only node logs — filelog coverage gaps are our responsibility (no promtail fallback); validate containerd pod-log formats on AKS early.
• Tempo monolithic — fine at current volume; revisit if TraceQL search latency degrades.

9. Open items

• AKS cluster details (region, node pools, storage account) — needed before first deploy.
• KeyVault CSI wiring for the Mimir account key.
• Ingress controller + OIDC provider choice on AKS.
• Grafana MCP server (mcp-grafana) deploy decision.
• Lab profile (clusters/lab) values for MinIO endpoints — when lab deploy is wanted.