Skip to content

Instantly share code, notes, and snippets.

@d3v-null
Created October 9, 2026 04:08
Show Gist options
  • Select an option

  • Save d3v-null/fca98a7f5e1cd93988e2acb6c032ebe4 to your computer and use it in GitHub Desktop.

Select an option

Save d3v-null/fca98a7f5e1cd93988e2acb6c032ebe4 to your computer and use it in GitHub Desktop.
EF requirements gap fill

Execution Framework components: requirements gap fill

Generated 2026-10-09 (read-only; nothing was applied, synced or changed in any cluster or repo). Secret values are never printed; only Secret names, ExternalSecret Vault paths and key names.

Sources actually used

Ref Source Notes
dump Artifacts of GitLab job 17034588345 (project ska-telescope/src/src-api/ska-src-api-deployment-stack), out/deploy-ci-2927014939/cluster-dump/<namespace>/*.json, out/deploy-ci-2927014939/log, report.html Job 17048104007 had no artifacts at 2026-10-09 03:23 UTC: the API returned 404 Not Found for /jobs/17048104007/artifacts and /jobs/17048104007 reported "status":"running" (started 2026-10-09T02:36:00Z). The fallback job was used.
srv-cd ska-telescope/src/deployments/skaosrc/ska-src-skaosrc-services-cd @ 146b9312 (branch tiger2-poc) Extracted with git archive; this is also the exact commit pinned by the stack gitlink infra/ska-src-skaosrc-services-cd at ff66cd3e.
stack ska-telescope/src/src-api/ska-src-api-deployment-stack @ ff66cd3e (branch tiger2-poc) etc/env/dev/configuration.yaml, etc/env/dev/services/*.yaml, include/env/dev/services/*.mk.
gw svc/ska-src-ef-local-job-gateway @ 265cc254 (gitlink at stack ff66cd3e) chart charts/job-gateway (Chart.yaml version 0.4.1) and src/.
condor svc/ska-src-ef-htcondor-deployment @ c5b664c9 chart charts/htcondor, images/pilot/.
broker svc/ska-src-ef-computing-broker @ 5c650557 chart charts/computing-broker, etc/leader/, src/.
mba svc/ska-src-ef-mock-battle-api @ 55a964f1 chart charts/mock-battle-api.
cavern svc/ska-src-dm-cavern-deployment @ c6cfbefa docker-compose based.
kueue-chart, slinky-chart Charts cached on the local disk under infra/ska-src-skaosrc-services-cd/apps/kueue/overlays/dev/charts/kueue (Chart.yaml version 0.18.1) and apps/slurm-operator/overlays/dev/charts/slurm-operator-1.0.2 These directories are gitignored and are not in commit 146b9312; only used for RBAC rules of third-party operators.

What the dump contains. kubectl cluster-info dump format: per namespace only deployments.json, daemonsets.json, replicasets.json, pods.json, services.json, events.json, plus per-pod logs.txt, and nodes.json. It contains no StatefulSet, NodeSet, PVC, PV, ConfigMap, Secret, ExternalSecret, Role/ClusterRole, Kueue CR, Ingress or NetworkPolicy objects. StatefulSet/NodeSet membership below is taken from pod ownerReferences; everything else of those kinds comes from the repo sources and from the serverside-applied lines in the CI deploy log (out/deploy-ci-2927014939/log), which confirm the names of objects that were actually applied.

Nodes (cluster-dump/nodes.json): k3d-deploy-ci-2927014939-server-0 and -agent-0, each capacity cpu 64, memory 237266968Ki, pods 110.

Overlay/chart layering used by dev (srv-cd apps/<app>/overlays/dev/.../kustomization.yaml): job-gateway, htcondor, broker and mock-battle-api inflate the chart from the svc/ submodule (helmGlobals.chartHome: ../../../../../../svc/<repo>/charts) with base/values.yaml + overlay values.yaml; kueue inflates a locally cached OCI chart; harbor uses https://helm.goharbor.io chart 1.15.0; slurm / slurm-operator use oci://ghcr.io/slinkyproject/charts 1.0.2.


Part A: per component

Conventions: "none set" = field absent in the live pod spec in the dump. Every pod in all ten namespaces had hostNetwork=false, hostPID=false and only the default node.kubernetes.io/not-ready / node.kubernetes.io/unreachable (NoExecute, 300 s) tolerations unless stated otherwise (cluster-dump/<ns>/pods.json). No pod in these namespaces mounts a hostPath volume directly; all shared storage is reached through PVC "clones" whose PVs are hostPath (see each PVC table). Image digests are from status.containerStatuses[].imageID in pods.json.

A.1 job-gateway-1 (and job-gateway-2, differences listed at the end)

Workloads (cluster-dump/job-gateway-1/deployments.json, pods.json):

Workload Kind Replicas (ready) Container Image CPU req / lim Mem req / lim
job-gateway-1 Deployment 1 (1) job-gateway harbor.test/library/job-gateway:dev = harbor.test/library/job-gateway@sha256:a8793fab7729319176c5e127c1d46c9a276c1ec4bbc273ba444eb74a0fe00dc9 (built from gw images/job-gateway/Dockerfile, base python:3.13-slim) 100m / 500m 128Mi / 512Mi
pilot-dynamic-<id> (5 present at dump time, each a batch/v1 Job created by the gateway) Job → Pod 1 each init fix-storage-perms harbor.test/library/htcondor-pilot:dev = ...@sha256:69a7b489ee2bcf23e5b52ae58cd7e03be48687a6f89e7b72b7c6b8f7c493b477 none set none set
startd same image observed 1 / 1 (×3 pods), 2 / 2 (×1), 1 / 1 (×1) 512Mi / 512Mi (×4), 256Mi / 256Mi (×1)

Source of resource values: chart gw charts/job-gateway/values.yaml (resources) via srv-cd apps/job-gateway/base/values.yaml; pilot sizes come from gw src/ska_src_ef_job_gateway/clients/kubernetes_provisioner.py::build_job_manifest (requests == limits, sized per matched task, defaults PILOT_DEFAULT_CPUS=2, PILOT_DEFAULT_MEMORY_MB=4096 in the gateway env).

Security context:

Pod Pod-level Container-level hostNetwork / hostPID hostPath volumes
job-gateway-1 {} none set (no privileged, no runAsUser/runAsNonRoot, no allowPrivilegeEscalation, no capabilities) false / false none
pilot-dynamic-* {} startd: {"privileged": true}; init fix-storage-perms: none set (PVC mode; the code sets privileged: true on the init container only in hostPath mode, kubernetes_provisioner.py lines ~258-276) false / false none (PVC mode because STORAGE_PVCS is set)

Ports and Services (cluster-dump/job-gateway-1/services.json):

Object Detail
containerPort job-gateway: http 8081/TCP. Pilot pods: no containerPorts declared.
Service job-gateway-1 ClusterIP 10.43.132.95, port http 8081 → targetPort http (8081), TCP, selector app.kubernetes.io/instance=job-gateway-1, app.kubernetes.io/name=job-gateway
Ingress job-gateway-1.test, class nginx, TLS secret job-gateway-1-tls, issuer ska-local-ca-issuer (srv-cd apps/job-gateway/overlays/dev/job-gateway-1/values.yaml); Ingress object not in dump

PVCs mounted (gateway pod and every pilot pod; dump pods.json volumes):

PVC (namespace job-gateway-1) Mount Size / class / access PV behind it Where defined
site-scratch /srv/storage (rw) clone of storm1/site-scratch: 10Gi, storageClassName: hostpath, RWX storm1-site-scratch-pv-job-gateway-1 (name pattern <src-pv>-<dst-ns>), hostPath /srv/storage/site-scratch-1 DirectoryOrCreate, reclaim Retain (forced by the clone script) source: srv-cd apps/storm/overlays/dev/storm1/site-scratch-pv.yaml, site-scratch-pvc.yaml; clone job: srv-cd apps/job-gateway/overlays/dev/job-gateway-1/deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml running srv-cd scripts/common/provision-shared-storage/create-shared-pvc.py
storm1-sa /srv/storage/storm1 (ro) clone of storm1/storage-area: 10Gi, hostpath, RWX storm1-datasets-pv-job-gateway-1, hostPath /srv/storage/storm1, Retain srv-cd apps/storm/overlays/dev/storm1/storm1-datasets-pv.yaml, storage-area-pvc.yaml + same clone job

PVC/PV objects themselves are not in the dump; the CI log (log lines 4335-4338) confirms the clone job's ServiceAccount/ClusterRole/ConfigMap were applied.

Secrets (names only) and ExternalSecrets (srv-cd apps/job-gateway/overlays/dev/job-gateway-1/externalsecrets.yaml and deploy/post/bootstrap/externalsecrets.yaml; all use ClusterSecretStore vault-cluster-secretstore, refresh 1h):

Secret Used by ExternalSecret Vault path Vault property → secret key
htcondor-gateway-idtoken gateway volume /etc/condor/tokens.d htcondor-gateway-idtoken secret/data/htcondor READ_IDTOKEN → gateway.token
htcondor-pilot-idtoken gateway volume /etc/pilot; every pilot pod /root/secrets/idtoken (subPath idtoken, defaultMode 0400) htcondor-pilot-idtoken secret/data/htcondor/pilots/NODE1_SITE1 PILOT_IDTOKEN → templated key idtoken
slurm-jwt gateway volume /etc/slurm-jwt slurm-jwt secret/data/slurm JWT_HS256_KEY → jwt_hs256.key
ska-local-ca-secret post-deploy register job only ska-local-ca-secret secret/data/cert-manager CA_CRT → ca.crt
scapi-client post-deploy register job env scapi-client-secret secret/data/iam/clients SITE_CAPABILITIES_API_CLIENT_ID → clientId, SITE_CAPABILITIES_API_CLIENT_SECRET → clientSecret
job-gateway-1-tls Ingress cert-manager issued (not an ExternalSecret) – –

No container in this namespace reads a Secret through env.valueFrom (dump: envSecrets empty for gateway and pilots). The gateway has an env var SENTRY_DSN whose value is set inline in values.yaml (value not reproduced here).

Scheduling: nodeSelector none, affinity none, tolerations default only (chart values.yaml has nodeSelector: {}, tolerations: [], affinity: {}; pilot manifest sets none). ServiceAccount: gateway job-gateway-1; pilots default.

job-gateway-2 differences (dump env diff job-gateway-1 vs job-gateway-2 pods; srv-cd apps/job-gateway/overlays/dev/job-gateway-2/values.yaml): SITE_ID=NODE2_SITE1, PILOT_NAMESPACE=job-gateway-2, PVC storm2-sa (clone of storm2/storage-area, PV storm2-datasets-pv-job-gateway-2, hostPath /srv/storage/storm2) at /srv/storage/storm2, site-scratch clone of storm2/site-scratch (PV storm2-site-scratch-pv-job-gateway-2, hostPath /srv/storage/site-scratch-2), PILOT_SCRATCH_ROOT=/srv/storage/pilot-scratch-2, BATTLE_API_BASE=http://mock-battle-api-2.mock-battle-api-2.svc.cluster.local:8765, SLURM_PARTITION=compute2, SLURM_PARTITION_CAPACITY_CPUS=16, SLURM_PARTITION_CAPACITY_MEMORY_MB=16384, Kueue ClusterQueue pilot-pool-storm2, pilot IDTOKEN from secret/data/htcondor/pilots/NODE2_SITE1, Ingress job-gateway-2.test / job-gateway-2-tls. Service job-gateway-2 ClusterIP 10.43.190.187 8081→http. One pilot pod present at dump time (1 CPU / 512Mi, privileged). Everything else identical.

Common gateway env worth recording (dump): HTCONDOR_ENABLED=true, CONDOR_HOST=htcondor-cm.htcondor.svc.cluster.local, HTCONDOR_SCHEDD_NAME=htcondor-submit@htcondor-submit.htcondor.svc.cluster.local, DESIRED_PILOTS=1, SLURM_ENABLED=true, SLURM_ENDPOINT=http://slurm-restapi.slurm.svc.cluster.local:6820, SLURM_DESIRED_PILOTS=1, SLURM_PILOT_USER=ska-pilot, PAYLOAD_IDENTITY=pool, PAYLOAD_EXECUTION=in-pilot, KUEUE_LOCAL_QUEUE=pilots, OTEL_EXPORTER_OTLP_ENDPOINT=http://monitoring-opentelemetry-collector.monitoring.svc.cluster.local:4317.

A.2 kueue-system

Item Value Source
Deployment kueue-controller-manager replicas 1 (1 ready) cluster-dump/kueue-system/deployments.json
Container manager image registry.k8s.io/kueue/kueue:v0.18.1 = ...@sha256:50eb1d808ad6597599a21a1234ca624f8acdb364ab75a93d003d442341ebef0f pods.json
Resources requests 100m / 256Mi, limits 1 / 512Mi dump; srv-cd apps/kueue/overlays/dev/values.yaml
Pod securityContext runAsNonRoot: true, seccompProfile: RuntimeDefault dump
Container securityContext capabilities.drop: [ALL], readOnlyRootFilesystem: true, allowPrivilegeEscalation: false; privileged not set; runAsUser not set dump
containerPorts visibility 8082, webhook-server 9443, metrics 8443 (all TCP) dump
Services kueue-controller-manager-metrics-service ClusterIP https 8443→8443; kueue-visibility-server ClusterIP https 443→8082; kueue-webhook-service ClusterIP 443→9443 services.json
Volumes Secret kueue-webhook-server-cert (chart-generated, enableCertManager: false), ConfigMap kueue-manager-config, emptyDir visibility dump; overlays/dev/values.yaml
PVC / ExternalSecret none –
ServiceAccount kueue-controller-manager dump
nodeSelector / tolerations / affinity none / default / none dump
Chart version 0.18.1 (cached chart Chart.yaml); pulled from oci://registry.k8s.io/kueue/charts/kueue by stack include/env/dev/services/kueue.mk; KUEUE_CHART_VERSION comes from .env.dev, which is not in the sources kueue.mk
Extra RoleBinding kueue-visibility-server-auth-reader in kube-system binding SA kueue-controller-manager to Role extension-apiserver-authentication-reader (srv-cd apps/kueue/overlays/dev/deploy/pre/visibility-rbac/rolebinding.yaml)

A.3 mock-battle-api-1 (and -2)

Item Value Source
Deployment mock-battle-api-1 replicas 1 (1) cluster-dump/mock-battle-api-1/deployments.json
Image harbor.test/library/mock-battle-api:dev = ...@sha256:0b32110c70d9e35c14f2b017efa725163471167ed67a615c9f20f6a7da951466 pods.json
Resources requests 100m / 128Mi, limits 500m / 512Mi dump; srv-cd apps/mock-battle-api/overlays/dev/mock-battle-api-1/values.yaml
securityContext pod {}; container none set dump
containerPort http 8765/TCP dump
Service mock-battle-api-1 ClusterIP 10.43.42.17, http 8765 → http, TCP services.json
Ingress mock-battle-api-1.test (nginx, no TLS block) overlay values.yaml
PVCs site-scratch → /srv/storage (clone of storm1/site-scratch, PV storm1-site-scratch-pv-mock-battle-api-1, hostPath, Retain, 10Gi RWX hostpath); storm1-sa → /srv/storage/storm1 ro (clone of storm1/storage-area) overlay values.yaml extraVolumes; deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml
Secrets / ExternalSecrets none overlay has no externalsecrets.yaml; dump shows no Secret volumes or env refs
Env (plain) BATTLE_STORAGE=/srv/storage, BATTLE_DATA_ROOTS=/srv/storage/storm1/sa, BATTLE_SCRATCH_ROOT=/srv/storage/pilot-scratch-1, BATTLE_SCRATCH same, BATTLE_STORAGE_REGISTRY JSON with dav://storm1.test/sa/inputs, scratch://storm1.test/outputs, davs://storm1.test:443/sa/deterministic dump
ServiceAccount / scheduling default; nodeSelector none, affinity none, default tolerations dump

mock-battle-api-2: identical except namespace, image digest not separately verified (same tag), PVC storm2-sa → /srv/storage/storm2, scratch /srv/storage/pilot-scratch-2, Service ClusterIP 10.43.191.86 (cluster-dump/mock-battle-api-2/). A suspended test CronJob mock-battle-api-1-test-job (image harbor.test/library/mock-battle-api-test:dev, runAsNonRoot uid/gid 1000) is defined in the overlay.

A.4 harbor

Workloads (cluster-dump/harbor/deployments.json, pods.json; StatefulSets inferred from ownerReferences). Release name is release-name (no fullnameOverride in dev).

Workload Kind Replicas Containers (image, all goharbor/*:v2.11.0 unless stated) Resources
release-name-harbor-core Deployment 1 core harbor-core none set
release-name-harbor-jobservice Deployment 1 jobservice harbor-jobservice none set
release-name-harbor-portal Deployment 1 portal harbor-portal none set
release-name-harbor-registry Deployment 1 init registry-permission-fix busybox (resolved busybox:1.38.0); registry registry-photon; registryctl harbor-registryctl none set
release-name-harbor-database StatefulSet 1 pod init data-permissions-ensurer harbor-db v2.11.0 (req 100m/128Mi), data-migrator harbor-db v2.10.0, data-permissions-ensurer-1 harbor-db v2.10.0; database harbor-db none set (except the init request)
release-name-harbor-redis StatefulSet 1 pod init redis-permission-fix busybox; redis redis-photon none set
release-name-harbor-trivy StatefulSet 1 pod init trivy-permission-fix busybox; trivy trivy-adapter-photon requests 200m / 512Mi, limits 1 / 1Gi

Image digests (dump): harbor-core sha256:6092e603…, harbor-jobservice sha256:862d1120…, harbor-portal sha256:56c5e205…, registry-photon sha256:545abacd…, harbor-registryctl sha256:fc63bf62…, harbor-db v2.11.0 sha256:10382001…, harbor-db v2.10.0 sha256:bd3ed7a8…, redis-photon sha256:a7fad8c1…, trivy-adapter-photon sha256:03db1258…, busybox sha256:fd7dc986… (full values in cluster-dump/harbor/pods.json imageID).

Security context (dump): pod-level runAsUser: 10000, fsGroup: 10000 for core, jobservice, portal, registry (registry adds fsGroupChangePolicy: OnRootMismatch) and trivy; runAsUser: 999, fsGroup: 999 for database and redis. Every main container: privileged: false, runAsNonRoot: true, allowPrivilegeEscalation: false, capabilities.drop: [ALL], seccompProfile: RuntimeDefault (from srv-cd apps/harbor/base/values.yaml containerSecurityContext). The five permission-fix / migrator init containers run with runAsUser: 0. No hostNetwork/hostPID, no hostPath.

Ports and Services (cluster-dump/harbor/services.json):

containerPorts Service Type Port → targetPort / protocol
core 8080 release-name-harbor-core ClusterIP http-web 80 → 8080 TCP
database: none declared release-name-harbor-database ClusterIP 5432 → 5432 TCP
jobservice 8080 release-name-harbor-jobservice ClusterIP http-jobservice 80 → 8080 TCP
portal 8080 release-name-harbor-portal ClusterIP 80 → 8080 TCP
redis: none declared release-name-harbor-redis ClusterIP 6379 → 6379 TCP
registry 5000, 5001; registryctl 8080 release-name-harbor-registry ClusterIP http-registry 5000 → 5000, http-controller 8080 → 8080 TCP
trivy api-server 8080 release-name-harbor-trivy ClusterIP http-trivy 8080 → 8080 TCP

Ingress: harbor.test, externalURL: https://harbor.test, class nginx (srv-cd apps/harbor/overlays/dev/values.yaml).

PVCs (claim names from dump pod volumes; sizes/access from srv-cd apps/harbor/base/values.yaml persistence.persistentVolumeClaim; class from overlays/dev/values.yaml):

PVC Mount Size storageClassName accessModes PV
release-name-harbor-registry registry + registryctl /storage 100Gi local-path RWO dynamic (local-path provisioner); PV object and reclaim policy not in dump
release-name-harbor-jobservice /var/log/jobs 1Gi local-path RWO same
database-data-release-name-harbor-database-0 /var/lib/postgresql/data 1Gi local-path RWO same
data-release-name-harbor-redis-0 /var/lib/redis 1Gi local-path RWO same
data-release-name-harbor-trivy-0 /home/scanner/.cache 5Gi local-path RWO same

CI log confirms persistentvolumeclaim/release-name-harbor-jobservice and /release-name-harbor-registry applied (log lines 576-577); the three StatefulSet claims are created by volumeClaimTemplates.

Secrets and ExternalSecrets:

Secret Referenced by Origin
harbor core env secret, JOBSERVICE_SECRET, HARBOR_ADMIN_PASSWORD, REGISTRY_PASSWD; jobservice env; registryctl env; registry volume registry-htpasswd; core volume secret-key ExternalSecret harbor-secrets (srv-cd apps/harbor/overlays/dev/externalsecrets.yaml), Vault secret/data/harbor, properties → keys: HARBOR_ADMIN_PASSWORD→HARBOR_ADMIN_PASSWORD, JOBSERVICE_SECRET→JOBSERVICE_SECRET, REGISTRY_HTPASSWD→REGISTRY_HTPASSWD, REGISTRY_PASSWD→REGISTRY_PASSWD, CORE_SECRET→secret, SECRET_KEY→secretKey
release-name-harbor-core core envFrom + volume token-service-private-key chart-rendered Secret (CI log line 563)
release-name-harbor-database database envFrom chart Secret
release-name-harbor-jobservice jobservice envFrom chart Secret
release-name-harbor-registry registry/registryctl envFrom chart Secret
release-name-harbor-registryctl registryctl envFrom chart Secret
release-name-harbor-trivy trivy env gitHubToken, redisURL chart Secret

ServiceAccount default for all pods; nodeSelector/affinity none; default tolerations.

A.5 htcondor

Workloads (cluster-dump/htcondor/):

Deployment Replicas Containers Image CPU req / lim Mem req / lim
htcondor-cm 1 (1) cm docker.io/htcondor/cm:lts = ...@sha256:b025796d6ef2cad5bf2be83eaec520d92038692f17fcdd96849d7ba40c8a4b04 100m / 500m 256Mi / 1Gi
logfile (sidecar) alpine:3.20 (sha256:d9e853e8…) 10m / 50m 16Mi / 32Mi
htcondor-execute 2 (2) execute docker.io/htcondor/execute:lts = ...@sha256:73f4e5c805b86e867420b16a950894b0bd486e0782a8ebbc4bc6729f014bde51 250m / 2 512Mi / 2Gi
logfile alpine:3.20 10m / 50m 16Mi / 32Mi
htcondor-submit 1 (1) init init-shared-scratch busybox:1.36 (sha256:73aaf090…) none set none set
submit docker.io/htcondor/submit:lts = ...@sha256:d68be980711682a8a1defa4f00187a331d69418033a1d8d6b68a4b498f5ae70d 100m / 500m 256Mi / 1Gi
logfile alpine:3.20 10m / 50m 16Mi / 32Mi

Values: srv-cd apps/htcondor/base/values.yaml + overlays/dev/values.yaml (execute.replicaCount: 2, numCpus: 2, memoryMb: 2048). No StatefulSet (cm.ha.enabled is false, chart default).

Security: pod and container securityContext none set on all three Deployments (the chart templates cm-deployment.yaml, execute-deployment.yaml, submit-deployment.yaml carry no securityContext); hostNetwork/hostPID false; no hostPath (submit uses PVC mode because submit.sharedStorage.pvcs is set; the chart falls back to a hostPath /srv/storage only when that list is empty, condor charts/htcondor/templates/submit-deployment.yaml volumes block). ServiceAccount htcondor (chart serviceaccount.yaml; the chart defines no Role/ClusterRole for it).

Ports / Services (services.json):

containerPort Service Type Port → target
cm condor 9618/TCP htcondor-cm ClusterIP 10.43.184.196 condor 9618 → condor (9618) TCP
submit condor 9618/TCP htcondor-submit ClusterIP 10.43.65.217 condor 9618 → condor TCP
execute condor 9618/TCP (no Service) – –

Port 9618 / CCB answer. Yes, HTCondor uses 9618: containerPort 9618 on cm, execute and submit and Services htcondor-cm and htcondor-submit expose 9618/TCP (dump; chart values.yaml cm.service.port, submit.service.port). The central manager log shows DaemonCore: command socket at <10.42.1.87:9618?...&sock=collector> and Enabling CCB Server. (cluster-dump/htcondor/htcondor-cm-79688bf64-94zn9/logs.txt). The pilot config written by condor images/pilot/pilot-entrypoint.sh (/etc/condor/config.d/01-pilot.conf) sets CONDOR_HOST, DAEMON_LIST = MASTER STARTD, USE_SHARED_PORT = TRUE, IDTOKENS-only auth with SEC_DEFAULT_ENCRYPTION/INTEGRITY = REQUIRED, and does not set CCB_ADDRESS (no CCB, PRIVATE_NETWORK_NAME or TCP_FORWARDING_HOST setting anywhere in images/pilot/, the gateway chart's pilot-configmap.yaml, or the gateway source; the only CCB strings in the sources are comments). The pilot therefore does not register with the CM's CCB server and expects inbound TCP connections from the schedd/shadow to its own pod address on shared port 9618 (pilot pods declare no containerPort and have no Service; reachability relies on routable pod IPs inside the cluster). The stock htcondor/execute and htcondor/submit images additionally load a /etc/condor/config.d/01-ccb.conf from the image (htcondor-execute-*/logs.txt); its content is not in the sources. Pilot daemon logs are written to /srv/storage/pilot-logs/<hostname>/ and are not in the dump, so no runtime confirmation of pilot connection direction was possible.

Volumes / PVCs:

Volume Pods Type Detail
pool-password cm, execute, submit Secret htcondor-pool-password mounted /etc/condor/passwords-orig.d
condor-config all ConfigMap htcondor-config (/root/config); submit also mounts key condor_mapfile at /etc/condor/condor_mapfile chart configmap.yaml
condor-logs all emptyDir /var/log/condor
ska-local-ca-secret all Secret ska-local-ca-secret key ca.crt at /etc/ssl/certs/ska-local-ca.crt, env REQUESTS_CA_BUNDLE srv-cd apps/htcondor/overlays/dev/ca-trust-patch.yaml
condor-state submit PVC htcondor-submit → /var/lib/condor: 5Gi, storageClass: "" (cluster default), RWO (overlays/dev/values.yaml submit.persistence, chart submit-pvc.yaml); PV dynamic, not in dump CI log line 3873
shared-storage-0 submit PVC shared-scratch → /srv/storage: 10Gi, RWX, storageClassName: hostpath, bound to PV shared-scratch-pv (hostPath /srv/storage/shared-scratch, DirectoryOrCreate, reclaim Retain) overlays/dev/shared-scratch-pv.yaml, shared-scratch-pvc.yaml; CI log lines 3872, 3874

ExternalSecrets (srv-cd apps/htcondor/overlays/dev/externalsecrets.yaml):

ExternalSecret → Secret Vault path property → key
htcondor-pool-password secret/data/htcondor POOL_PASSWORD → POOL
ska-local-ca-secret secret/data/cert-manager CA_CRT → ca.crt

Vault keys written by the stack hooks for this component (CI log lines 3774-3910, paths only): secret/data/htcondor (pre-deploy and post-deploy writes; consumed as POOL_PASSWORD, READ_IDTOKEN, SUBMIT_IDTOKEN by the ExternalSecrets above and in broker/gateways), secret/data/htcondor/pilots/NODE1_SITE1 and secret/data/htcondor/pilots/NODE2_SITE1 (PILOT_IDTOKEN). Sites list: srv-cd apps/htcondor/overlays/dev/pilot-sites.txt (job-gateway-1 NODE1_SITE1, job-gateway-2 NODE2_SITE1); IDTOKEN subjects pilot-NODE1_SITE1@condor, pilot-NODE2_SITE1@condor (base/values.yaml security.pilotIdentities).

Scheduling: nodeSelector/affinity none, default tolerations (chart values {}/[]).

A.6 broker

Workloads (cluster-dump/broker/):

Deployment Replicas Container Image CPU req / lim Mem req / lim
broker-computing-broker 3 (3) computing-broker harbor.test/library/computing-broker:dev = ...@sha256:014033a29fa3db1fa29a26dec6bd4f3592372fc3bf44c156c20ccb14234ceb8d (base python:3.13-slim, broker images/computing-broker/Dockerfile) 200m / 1 256Mi / 1Gi
broker-postgresql 1 (1) postgresql postgres:16 = docker.io/library/postgres@sha256:ca0bd484cb98bf4b24eb1010e73fb3fcbd6714d240fbc1a10eea5b7dbecb641d 100m / 500m 256Mi / 1Gi
broker-rabbitmq 1 (1) rabbitmq rabbitmq:3.13-management = ...@sha256:e582c0bc7766f3342496d8485efb5a1df782b5ce3886ad017e2eaae442311f69 100m / 500m 256Mi / 512Mi
broker-smoke-test-job-1791475619 (Job, Succeeded) – broker-smoke-test-job harbor.test/library/test:dev (sha256:427dfc7b…) none set none set

replicaCount: 3 from srv-cd apps/broker/overlays/dev/values.yaml. Security: pod {} and container securityContext none set on all four pods (chart templates carry none); no hostNetwork/hostPID/hostPath (hostPath /srv/storage would be used only if broker.condorSubmit.storagePvcs were empty, broker charts/computing-broker/templates/deployment.yaml volumes block). ServiceAccount default (no Role bound).

Ports / Services (services.json):

containerPort Service Type Port → target
computing-broker http 8082/TCP broker-computing-broker ClusterIP 10.43.78.207 http 8082 → http TCP
postgresql pg 5432/TCP broker-postgresql ClusterIP 10.43.208.250 pg 5432 → pg TCP
rabbitmq amqp 5672/TCP broker-rabbitmq ClusterIP 10.43.92.5 amqp 5672 → amqp TCP

Ingress broker.test, TLS secret broker-tls, issuer ska-local-ca-issuer (overlay values.yaml).

PVCs:

PVC Pods / mount Spec PV
shared-scratch (broker ns) computing-broker /srv/storage clone of htcondor/shared-scratch: 10Gi RWX hostpath shared-scratch-pv-broker, hostPath /srv/storage/shared-scratch, Retain (srv-cd apps/broker/overlays/dev/deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml)
ingest-area (broker ns) smoke-test job /tmp/ingest clone of ingestor/ingest-area: 10Gi RWX hostpath ingestor-storm1-pv-broker, hostPath /srv/storage/storm1/sa/ingest, Retain (source srv-cd apps/ingestor/overlays/dev/ingestor-storm1-storage.yaml)
broker-postgresql postgresql /var/lib/postgresql/data 10Gi, RWO, storageClassName: local-path (overlays/dev/values.yaml postgresql.storage; chart postgresql.yaml) dynamic, not in dump
(rabbitmq) /var/lib/rabbitmq emptyDir –

Secrets referenced (dump env.valueFrom.secretKeyRef, volumes) and ExternalSecrets (srv-cd apps/broker/overlays/dev/externalsecrets.yaml):

Secret / key used Consumer ExternalSecret Vault path property → key
broker-rabbitmq-auth / username, password computing-broker env, rabbitmq env broker-rabbitmq-auth secret/data/broker RABBITMQ_USERNAME→username, RABBITMQ_PASSWORD→password
broker-postgresql-auth / database_url (broker), database, username, password (postgres) env broker-postgresql-auth secret/data/broker POSTGRESQL_USERNAME, POSTGRESQL_PASSWORD, POSTGRESQL_DATABASE → username, password, database, templated database_url
broker-oauth-exchange / SKA_OAUTH_EXCHANGE_CLIENT_ID, SKA_OAUTH_EXCHANGE_CLIENT_SECRET computing-broker env broker-oauth-exchange secret/data/iam/clients SKA_SRC_EF_CREDMON_CLIENT_ID, SKA_SRC_EF_CREDMON_CLIENT_SECRET (same key names)
htcondor-broker-idtoken volume /etc/condor/tokens.d htcondor-broker-idtoken secret/data/htcondor READ_IDTOKEN → broker.token
htcondor-broker-submit-idtoken volume /etc/condor/submit-tokens.d htcondor-broker-submit-idtoken secret/data/htcondor SUBMIT_IDTOKEN → broker-submit.token
ska-local-ca-secret / ca.crt broker volume (truststore-patch.yaml), smoke test volume ska-local-ca-secret secret/data/cert-manager CA_CRT → ca.crt
broker-test-token / token smoke-test job env not found: no ExternalSecret or manifest creates it in srv-cd apps/broker/overlays/dev, stack bin/, include/, lib/ (only consumed by broker-smoke-test-cronjob.yaml) – –

Env of note (dump): BROKER_COLLECTOR_HOST=htcondor-cm.htcondor.svc.cluster.local, BROKER_CONDOR_SCHEDD_NAME=htcondor-submit@htcondor-submit.htcondor.svc.cluster.local, BROKER_AAPI_URL=http://core.aapi.svc.cluster.local:8080, BROKER_DATA_MANAGEMENT_API_URL=http://core.dmapi.svc.cluster.local:8080/v1, BROKER_SITE_CAPABILITIES_URL=http://core.scapi.svc.cluster.local:8080/v1, BROKER_PERMISSIONS_API_URL=https://papi.test/api/v1, SKA_OAUTH_EXCHANGE_TOKEN_URL=https://iam.test/token, SENTRY_DSN (inline), authzEnabled: false in dev.

Scheduling: nodeSelector/affinity none, default tolerations.

A.7 slurm

Workloads (cluster-dump/slurm/; StatefulSets and NodeSets from pod ownerReferences; chart oci://ghcr.io/slinkyproject/charts/slurm 1.0.2 via srv-cd apps/slurm/overlays/dev/kustomization.yaml):

Workload Kind Replicas (pods) Containers Image CPU req / lim Mem req / lim
slurm-controller StatefulSet 1 init reconfigure (slurmctld image), init logfile (alpine:latest); slurmctld ghcr.io/slinkyproject/slurmctld:25.11-ubuntu24.04 (sha256:d97e4652…) 1 / 2 1Gi / 2Gi
slurm-accounting StatefulSet 1 slurmdbd ghcr.io/slinkyproject/slurmdbd:25.11-ubuntu24.04 (sha256:2e0107d2…) none set none set
slurm-restapi Deployment 1 (1) slurmrestd ghcr.io/slinkyproject/slurmrestd:25.11-ubuntu24.04 (sha256:0b56b88b…) 250m / 500m 256Mi / 512Mi
slurm-login-slinky Deployment 1 (1) init initconf (alpine:latest); login ghcr.io/slinkyproject/login:25.11-ubuntu24.04 (sha256:703aafd0…) none set none set
mariadb Deployment 1 (1) mariadb mariadb:10.11 (sha256:7db29378…) none set none set
slurm-worker-compute NodeSet (slinky.slurm.net) 2 (-0, -1) init logfile alpine:latest, init install-apptainer ubuntu:24.04 (runAsUser: 0); slurmd ghcr.io/slinkyproject/slurmd:25.11-ubuntu24.04 (sha256:64aedef5…) 2 / 16 4Gi / 16Gi
slurm-worker-compute2 NodeSet 1 (-0) same same 2 / 16 4Gi / 16Gi

Values: srv-cd apps/slurm/overlays/dev/values.yaml (controller 1/1Gi–2/2Gi, restapi 250m/256Mi–500m/512Mi, nodesets.compute.replicas: 2, compute2.replicas: 1, slurmd 2/4Gi–16/16Gi, useResourceLimits: true), base/values.yaml, overlays/dev/mariadb.yaml.

Security context (dump):

Pod Pod-level Container-level
slurm-controller-0 runAsUser/runAsGroup/fsGroup 401, runAsNonRoot: true slurmctld: runAsUser 401, runAsGroup 401, runAsNonRoot true
slurm-accounting-0 same (401) slurmdbd: same
slurm-restapi runAsUser/runAsGroup/fsGroup 65534, runAsNonRoot: true same (65534)
slurm-login-slinky {} login: privileged: false
mariadb {} none set
slurm-worker-compute*-N {} slurmd: privileged: true, capabilities.add: [BPF, NET_ADMIN, SYS_ADMIN, SYS_NICE]; init install-apptainer runAsUser: 0

hostNetwork/hostPID false everywhere; no hostPath volumes. nodeSelector kubernetes.io/os: linux on all chart pods (mariadb none); affinity {}; workers additionally tolerate nodeset.slinky.slurm.net/worker (Equal, NoExecute). ServiceAccount default for all.

Ports / Services (services.json):

containerPort Service Type Port → target
slurmctld 6817 slurm-controller ClusterIP slurmctld 6817 → slurmctld TCP
slurmdbd 6819 slurm-accounting ClusterIP slurmdbd 6819 → slurmdbd TCP
slurmrestd 6820 slurm-restapi ClusterIP slurmrestd 6820 → slurmrestd TCP; Ingress slurmrestd.test → 6820 (overlays/dev/ingress.yaml)
login 22 slurm-login-slinky LoadBalancer (nodePort 31616) login 22 → login TCP
slurmd 6818 slurm-workers-slurm ClusterIP headless (None) slurmd 6818 → slurmd TCP
mysql 3306 mariadb ClusterIP 3306 → 3306 TCP

PVCs (dump pod claims; specs from overlay):

PVC Pod / mount Spec PV
statesave-slurm-controller-0 slurmctld /var/spool/slurmctld/ska-src-dev 5Gi, RWO, storageClassName: local-path (overlays/dev/values.yaml controller.persistence) dynamic, not in dump
mariadb-data mariadb /var/lib/mysql 5Gi, RWO, no storageClassName (cluster default) (overlays/dev/mariadb.yaml) dynamic, not in dump
site-scratch-1, site-scratch-2 compute / compute2 /srv/storage clones of storm1/site-scratch, storm2/site-scratch (10Gi RWX hostpath) storm1-site-scratch-pv-slurm, storm2-site-scratch-pv-slurm, hostPath /srv/storage/site-scratch-{1,2}, Retain (overlays/dev/deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml)
storm1-sa, storm2-sa compute /srv/storage/storm1 ro, compute2 /srv/storage/storm2 ro clones of storm{1,2}/storage-area storm{1,2}-datasets-pv-slurm, hostPath /srv/storage/storm{1,2}, Retain

Secrets / ExternalSecrets (srv-cd apps/slurm/overlays/dev/externalsecrets.yaml, base/values.yaml):

Secret Use ExternalSecret Vault path property → key
slurm-munge slurmKeyRef (auth key) slurm-munge secret/data/slurm MUNGE_KEY → munge.key
slurm-jwt-vault jwtHs256KeyRef (controller JWT key) slurm-jwt-vault secret/data/slurm JWT_HS256_KEY → jwt_hs256.key
mariadb-password / password, rootPassword mariadb env, slurmdbd storageConfig.passwordKeyRef mariadb-password secret/data/slurm MARIADB_PASSWORD → password, MARIADB_ROOT_PASSWORD → rootPassword
slurm-auth-jwths256, slurm-auth-slurm, slurm-login-slinky-sssd-conf chart-rendered – (CI log lines 2641-2643) – –

Other mounts (chart-managed projected/config volumes): slurm-etc, ssh-host-keys, ssh-config, sssd-conf; ConfigMaps install-apptainer-script (from overlays/dev/scripts/install-apptainer.sh), slurm-subuid.

A.8 slinky (slurm-operator)

Item Value Source
Deployments slurm-operator 1 (1), slurm-operator-webhook 1 (1) cluster-dump/slinky/deployments.json
Images ghcr.io/slinkyproject/slurm-operator:1.0.2 (sha256:7db52099…), ghcr.io/slinkyproject/slurm-operator-webhook:1.0.2 (sha256:d33c3234…) pods.json
Resources none set (both) dump
securityContext pod {}; containers none set dump
containerPorts none declared dump
Services slurm-operator ClusterIP headless: metrics 8080→8080, health 8081→8081 TCP; slurm-operator-webhook ClusterIP: https 443→9443, health 8081→8081 TCP services.json
Volumes webhook: Secret slurm-operator-webhook-ca (cert-manager, certManager.enabled: true in srv-cd apps/slurm-operator/base/values.yaml) dump
PVC / ExternalSecret none –
ServiceAccounts slurm-operator, slurm-operator-webhook dump
Scheduling nodeSelector/affinity none, default tolerations dump

Part B: Cavern and BATTLE

Cavern

  • Not a service in etc/env/dev/configuration.yaml at ff66cd3e: it is in neither the 29 enabled nor the 19 disabled entries (list in Part D). The only Cavern-related items in the stack are the dpapi pre-deploy hook writing Vault key CAVERN_API_KEY under secret/cavern (stack include/env/dev/services/dpapi.mk) and the dpapi dev overlay's dpapi-cavern-pv / dpapi-cavern-pvc + dpapi-cavern-bootstrap-job (srv-cd apps/dpapi/overlays/dev/dpapi-cavern-storage.yaml, dpapi-cavern-bootstrap-job.yaml). No skaha*/canfar/cavern namespace exists in the dump.
  • srv-cd has an unrelated apps/canfar app (prod/dev/staging overlays, ArgoCD application only under argocd-apps/canfar/prod/application.yaml) that deploys Cavern as part of the CANFAR/skaha platform into namespace skaha-system. It is not referenced by the stack configuration and was not deployed in the CI run.

From the cavern-deployment repo (svc/ska-src-dm-cavern-deployment @ c6cfbefa): it is a docker-compose deployment. There is no Helm chart (etc/helm/values.yaml is a 0-byte file; no Chart.yaml, no templates/*.yaml; templates/ holds VOSpace XML samples; README offers "Kubernetes Deployment" prerequisites but no manifests). From docker-compose.yml and infra/haproxy/config/haproxy.cfg:

Service Image Ports User / securityContext Storage Secrets Resources / replicas
haproxy haproxytech/haproxy-debian:latest host 8442 → 443 (bind *:443 ssl); routes /reg, /src/cavern, /src/posix-mapper root:root ./certificates/haproxy-host.pem (TLS key, bind-mounted) TLS cert file none set (compose has no limits/replicas)
reg images.opencadc.org/core/reg:1.2.0 8080 (internal) tomcat:tomcat config bind mounts – none set
posix-mapper-db postgres:14 5432 (internal) default named volume posix-mapper-db-data .env: DB_USERNAME, DB_PASSWORD, DB_NAME none set
posix-mapper images.opencadc.org/platform/posix-mapper:0.3.3 host 8098 → 8080 root:root config bind mounts – none set
cavern-db postgres:14 5432 (internal) default named volume cavern-db-data .env: NODE_DB_USERNAME, NODE_DB_PASSWORD, NODE_DB_NAME none set
cavern-core images.opencadc.org/platform/cavern:0.9.2 8080 (internal, via haproxy) root:root named volume cavern-storage at /data; config bind mounts – none set

No CPU/memory limits, no replica counts, no Kubernetes securityContext, no PVC/PV exist in that repo (compose only). For completeness, the Kubernetes values in srv-cd apps/canfar/base/values-cavern.yaml (chart cavern 0.7.1 from https://images.opencadc.org/chartrepo/platform) are: image images.opencadc.org/platform/cavern:0.9.1, replicaCount: 1, resources requests = limits 500m / 4Gi, UWS PostgreSQL install: true (runUID 999), storage PVC skaha-pvc (1000Gi RWX csi-manila-cephfs-sc-retain, base/skaha-pvc.yaml), postgresdb-pvc 4Gi RWO; posix-mapper 0.3.1 at 500m/1Gi (values-posixmapper.yaml); dev overlay only changes hostnames to canfar.test. No securityContext or ports are set in those values files.

BATTLE

Only the mock exists. Across all sources the only BATTLE implementation is ska-telescope/src/src-ef/ska-src-ef-mock-battle-api (submodule svc/ska-src-ef-mock-battle-api, chart charts/mock-battle-api, overlays srv-cd apps/mock-battle-api/overlays/{dev,staging/stfc}). The stack configuration lists only mock-battle-api-1 and mock-battle-api-2; gateways point pilots at BATTLE_API_BASE=http://mock-battle-api-N.mock-battle-api-N.svc.cluster.local:8765; .gitmodules has no other battle repo. A grep for battle excluding mock over srv-cd, stack etc/, stack .gitmodules and the component READMEs returns only comments/docs describing the BATTLE API contract (e.g. gw README.md lines 62, 102; broker overlays/dev/values.yaml line 88). No non-mock BATTLE chart, image or repo URL was found.


Part C: compute workloads

C.1 Pilot pod template

Aspect Value Source
Where the pod template is built In code: gw src/ska_src_ef_job_gateway/clients/kubernetes_provisioner.py::build_job_manifest (returns a batch/v1 Job: suspend: true when a Kueue queue is set, backoffLimit: 0, ttlSecondsAfterFinished: 300, restartPolicy: Never, labels app.kubernetes.io/name=htcondor-pilot, app.kubernetes.io/component=pilot-dynamic, kueue.x-k8s.io/queue-name=pilots). Not in a ConfigMap. code
ConfigMap htcondor-pilot-config Chart template gw charts/job-gateway/templates/pilot-configmap.yaml, rendered into each gateway namespace (CI log line 4346 configmap/htcondor-pilot-config), mounted at /root/config/99-pilot.conf. Content: PILOT_SITE = "<siteId>", PILOT_RESOURCE = "kubernetes", STARTD_ATTRS = $(STARTD_ATTRS) PILOT_SITE PILOT_RESOURCE, STARTD_NOCLAIM_SHUTDOWN = 300, SEC_CREDENTIAL_REFRESH = 60, STARTER_UPDATE_INTERVAL = 60, one static slot (NUM_SLOTS = 1, SLOT_TYPE_1 = cpus=100%,mem=100%,disk=100%), PILOT_USER = ska-pilot, SLOT1_USER = $(PILOT_USER), STARTER_ALLOW_RUNAS_OWNER = FALSE. chart
Image harbor.test/library/htcondor-pilot:dev (PILOT_IMAGE env; built by the stack from condor images/pilot/ per stack etc/env/dev/services/htcondor.yaml; Dockerfile: FROM ubuntu:22.04, condor from htcss-downloads.chtc.wisc.edu 24.x repo, singularity binary copied from quay.io/ucsc_cgl/toil:9.3.0); imagePullPolicy: Always. Same image for Slurm pilots (SLURM_PILOT_IMAGE). dump env; Dockerfile
CPU / memory requests == limits: cpu: <cpus>, memory: <MiB>Mi, sized from the matched task's Request ads, else defaults PILOT_DEFAULT_CPUS=2 / PILOT_DEFAULT_MEMORY_MB=4096 (dump env; chart values pilotDefaultCpus, pilotDefaultMemoryMb). Observed at dump time: 1 CPU/512Mi, 2 CPU/512Mi, 1 CPU/256Mi. Init container: no resources. code lines 162-173; dump
GPU nvidia.com/gpu is added to both requests and limits only when the planned size has gpus > 0 (GPU_RESOURCE = "nvidia.com/gpu", lines 50, 171-173). No pilot in the dump requests it. code
securityContext startd: {"privileged": true} (unconditional). Init fix-storage-perms: privileged: true only in hostPath mode; none in PVC mode (dev). No pod-level securityContext. code lines 258-286; dump
Volumes Secret htcondor-pilot-idtoken (defaultMode 0400, subPath idtoken → /root/secrets/idtoken), ConfigMap htcondor-pilot-config (subPath 99-pilot.conf), PVCs from STORAGE_PVCS=site-scratch:/srv/storage,stormN-sa:/srv/storage/stormN:ro (sorted by mountPath). hostPath /srv/storage only when STORAGE_PVCS is empty (not the case in dev). code; dump
nodeSelector / tolerations / affinity none set (manifest has none; only kubelet default tolerations in the live pod) code; dump
ServiceAccount default in the pilot namespace (none set in manifest) dump
Env CONDOR_HOST, CONDOR_SERVICE_HOST, _CONDOR_TRUST_DOMAIN=condor, _CONDOR_STARTD_NOCLAIM_SHUTDOWN=300, SKA_PAYLOAD_IDENTITY=pool, SKA_PVC_ROOT=/srv/storage/pilot-scratch-N, NUM_CPUS, MEMORY, POD_NAMESPACE (downward API), plus PILOT_EXTRA_ENV (BATTLE_API_BASE=…, SKA_STAGEIN_TIMEOUT_S=60) dump
Condor config inside the pilot condor images/pilot/pilot-entrypoint.sh → /etc/condor/config.d/01-pilot.conf: DAEMON_LIST = MASTER STARTD, USE_SHARED_PORT = TRUE, IDTOKENS only, encryption+integrity REQUIRED, START = TRUE (in-pilot model), job hooks start_payload.sh / finish_payload.sh, no CCB settings (see A.5). entrypoint
Slurm pilots Same image run through sbatch + apptainer by gw src/.../clients/slurm_provisioner.py as user ska-pilot (SLURM_PILOT_USER), scratch /srv/storage/pilot-scratch-N, partition compute / compute2; capacity fallback 32 CPU/32768 MB (site 1) and 16 CPU/16384 MB (site 2). dump env

C.2 Kueue objects

ClusterQueues (srv-cd apps/kueue/overlays/dev/deploy/post/configure/clusterqueue.yaml, applied by the kueue deploy.post step; identical spec for both):

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: pilot-pool-storm1      # and: pilot-pool-storm2
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default
          resources:
            - name: cpu
              nominalQuota: "20"
            - name: memory
              nominalQuota: "40Gi"

No cohort (no borrowing between sites). The gateway logs the quota at startup: capacity[kubernetes]: 20 cpus / 40960 MiB (kueue-quota) (cluster-dump/job-gateway-1/job-gateway-1-788dddbb56-p4mr9/logs.txt).

ResourceFlavor (.../configure/resourceflavor.yaml; CI log line 1736):

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: default
# no nodeLabels / taints / tolerations: matches any node

LocalQueues (chart template gw charts/job-gateway/templates/localqueue.yaml, rendered because gateway.kueueLocalQueue: pilots; CI log line 4352 localqueue.kueue.x-k8s.io/pilots for job-gateway-1 and the equivalent for job-gateway-2):

Namespace LocalQueue spec.clusterQueue Evidence in dump
job-gateway-1 pilots pilot-pool-storm1 pilot pods labelled kueue.x-k8s.io/local-queue-name=pilots, kueue.x-k8s.io/cluster-queue-name=pilot-pool-storm1, annotation kueue.x-k8s.io/workload=job-pilot-dynamic-…
job-gateway-2 pilots pilot-pool-storm2 same labels with pilot-pool-storm2

Kueue operator config (cached chart values.yaml managerConfig): manageJobsWithoutQueueName left at default false; integrations include batch/job and pod; waitForPodsReady not set.

C.3 RBAC

RBAC objects are not in the dump; rules below are from the chart/overlay templates, and the CI log confirms each named object was applied (log lines 4341-4345, 4355-4357, 4335-4337, 4573-4575, 2634-2635, 4209-4211). job-gateway-N is the Helm fullname (release name contains the chart name, _helpers.tpl).

Role job-gateway-1-pilot / job-gateway-2-pilot (namespace job-gateway-N; gw charts/job-gateway/templates/pilot-role.yaml), bound by RoleBinding job-gateway-N-pilot to ServiceAccount job-gateway-N (pilot-rolebinding.yaml):

apiGroups resources verbs
batch jobs create, delete, get, list, watch
"" pods create, delete, get, list
"" configmaps get
kueue.x-k8s.io workloads create, delete, get, list, watch
kueue.x-k8s.io localqueues get

ClusterRole job-gateway-N-kueue-quota (kueue-quota-clusterrole.yaml), bound by ClusterRoleBinding job-gateway-N-kueue-quota to SA job-gateway-N (namespace job-gateway-N):

apiGroups resources verbs
kueue.x-k8s.io clusterqueues get

Other Roles/ClusterRoles that allow creating batch/jobs or pods, or acting on them:

Object Rules (jobs/pods only) Bound to Source
ClusterRole slurm-operator core configmaps, events, persistentvolumeclaims, pods, secrets, services: create, delete, get, list, patch, update, watch; pods/status: get, patch, update SA slurm-operator (ns slinky) via ClusterRoleBinding slinky-chart templates/operator/rbac.yaml; CI log line 1338
ClusterRole kueue-manager-role core pods: delete, get, list, patch, update, watch (no create); pods/finalizers get, update; pods/status get, patch; batch jobs: delete, get, list, patch, update, watch (no create); jobs/finalizers, jobs/status: get, patch, update SA kueue-controller-manager (ns kueue-system) via ClusterRoleBinding kueue-manager-rolebinding kueue-chart templates/rbac/role.yaml, role_binding.yaml; CI log lines 1687, 1721
Role job-gateway-N-payload-pod-pilot (+ SA pilot-payload-pod) pods, pods/log: get, list, create, patch, delete; workloads get, list not rendered in dev: only when payloadExecution=payload-pod; dev runs PAYLOAD_EXECUTION=in-pilot (dump env) and the CI log shows no such object gw charts/job-gateway/templates/payload-pod-pilot-role.yaml
ClusterRoles job-gateway-N-shared-pvc-creator, mock-battle-api-N-shared-pvc-creator, broker-shared-pvc-creator, slurm-shared-pvc-creator persistentvolumes: get, list, create, patch; persistentvolumeclaims: get, list, create (no pods/jobs) SAs of the same name in their namespace (pre-deploy clone jobs) .../deploy/pre/provision-shared-storage/rbac.yaml in each overlay
Roles job-gateway-N-wait-for-secrets, broker-wait-for-idtoken, slurm-wait-for-cluster externalsecrets.external-secrets.io: get, list, watch (wait-for-cluster not read) SAs of the same name overlay deploy/post/.../rbac.yaml
htcondor SA htcondor, broker/mock-battle/slurm pods SA default no Role/ClusterRole in the charts or overlays – condor charts/htcondor/templates/ (only serviceaccount.yaml), broker charts/computing-broker/templates/, mba charts/mock-battle-api/templates/

C.4 Dask / distributed jobs

  • There is no Dask operator, DaskCluster CRD or Kubernetes-native Dask object in any source. Dask runs entirely inside HTCondor.
  • The broker validates --dask requests (broker src/ska_src_ef_computing_broker/services/payload.py::validate_dask_parameters: requires --image and --cmd, --dask-workers positive integer) and submits a run leader to the schedd in the HTCondor local universe (broker src/.../clients/condor_submit_client.py: universe = local, request_cpus = 1, request_memory = 128, runs on the htcondor-submit pod).
  • The leader (broker etc/leader/ska_run_leader.py, staged to the schedd by the broker) creates the Dask roles as separate HTCondor vanilla jobs on pilots: first the scheduler (_dask_scheduler_spec: exec dask scheduler --scheduler-file /srcnet/work/scheduler.json, ClassAd +SkaChirpScheduler = true, address published back via condor_chirp from condor images/pilot/hooks/handoff.sh / etc/leader/ska_task_exec.py), then N workers (_dask_role_specs: exec dask worker <address> --nthreads <--dask-worker-cpu, default 1> --memory-limit <--dask-worker-memory, default 2048>MiB) and the client (user --cmd). All roles use the one job --image.
  • Resources per role job: request_cpus = {cpu}, request_memory = {memory}, optional request_gpus = {gpu} when --gpu > 0, request_disk when set; requirements = (TARGET.PILOT_RESOURCE =?= "<kubernetes|slurm>") && site clause (ska_run_leader.py lines ~405-430). Each matched job runs in a pilot pod the gateway sizes from those ads (C.1), so the Kubernetes-side resources are the pilot pod's requests == limits.
  • ServiceAccount: the pilot pods executing scheduler/workers/client use the default ServiceAccount of job-gateway-1 / job-gateway-2 (dump); on Slurm they run as OS user ska-pilot. The leader itself runs as the schedd's submituser on htcondor-submit (chart condor_mapfile), SA htcondor.
  • Capacity note from srv-cd apps/kueue/overlays/dev/deploy/post/configure/clusterqueue.yaml: the 20 CPU / 40Gi quota is sized so "scheduler + N workers + client pilots" can be admitted concurrently. The broker smoke test uses image harbor.test/library/dask-base:dev (built from broker images/dask-base/Dockerfile, stack etc/env/dev/services/broker.yaml).

C.5 Slurm NodeSets

NodeSet Replicas slurmd requests / limits Advertised to Slurm Partition Source
slurm-worker-compute 2 (pods slurm-worker-compute-0, -1 in dump) 2 CPU / 4Gi → 16 CPU / 16Gi 16 CPU / 16Gi per node (useResourceLimits: true) compute (Default YES) srv-cd apps/slurm/overlays/dev/values.yaml nodesets.compute; CI log line 2654
slurm-worker-compute2 1 (pod slurm-worker-compute2-0) 2 CPU / 4Gi → 16 CPU / 16Gi 16 CPU / 16Gi compute2 same file nodesets.compute2; CI log line 2655

Both: scalingMode: StatefulSet, workloadDisruptionProtection: false, init container installs Apptainer into an emptyDir, env APPTAINER_CACHEDIR=/srv/storage/apptainer-cache, APPTAINER_NOHTTPS=1; shared batch partition disabled.

C.6 GPU / extended resources

  • Dump: none. grep -rE 'nvidia\.com|amd\.com/gpu|habana' over cluster-dump/ returns no hits; no pod requests any extended resource; the only non-standard *.io/* keys are Elastic annotations in elastic-system/monitoring. Node capacity lists only cpu, memory, ephemeral-storage, hugepages, pods.
  • Sources: nvidia.com/gpu appears only as a conditional in code: gw kubernetes_provisioner.py (GPU_RESOURCE, added when a task requests GPUs), condor images/pilot/hooks/handoff.sh (PAYLOAD_POD_GPUS, payload-pod model only), broker etc/leader/ska_run_leader.py (request_gpus when --gpu > 0). srv-cd apps/kueue/overlays/dev/deploy/post/configure/resourceflavor.yaml comments that a GPU flavor "can be added when GPU pilots land"; no ClusterQueue covers a GPU resource.

Part D: configuration

D.1 Services in etc/env/dev/configuration.yaml (stack @ ff66cd3e)

enabled is deploy.enabled in each etc/env/dev/services/<name>.yaml (the list file only names them).

Enabled (29): cert-manager, vault, external-secrets, ingress-nginx, harbor, eck-operator, monitoring, indigo-iam, papi, aapi, scapi, storm1, storm2, fts, rucio, ingestor, gatekeeper, dpapi, dmapi, caom, slurm-operator, slurm, broker, job-gateway-1, job-gateway-2, mock-battle-api-1, mock-battle-api-2, kueue, htcondor.

Disabled (19): headlamp, ums, rucio-task-manager, datalink, product-streamer, soda, sdapi, sreg, jupyterhub, oper-cli, oper-gui, geapi, panda, harvester, lur, aqs, waldur, iam-waldur-sync, panda-usage-reporter.

Group ef ("Execution Framework: broker, HTCondor pool, pilot gateways, and the Slurm and Kueue pilot pools") is declared on broker, htcondor, job-gateway-1/2, kueue, slurm, slurm-operator, mock-battle-api-1/2.

D.2 NetworkPolicy

None. The dump contains no NetworkPolicy text (and cluster-info dump would not include that kind anyway). The CI deploy log has no networkpolicy apply line. No kind: NetworkPolicy exists anywhere under srv-cd apps/, argocd-apps/, flux-apps/, resources/ at 146b9312.

D.3 External hostnames (not *.test, not *.svc) in ConfigMaps or container env of the Part A namespaces

Container env (from cluster-dump/<ns>/pods.json; ConfigMaps are not in the dump):

Hostname Component Where
o4507377535287296.ingest.de.sentry.io broker (broker-computing-broker, env SENTRY_DSN), job-gateway-1 and job-gateway-2 (env SENTRY_DSN) values: srv-cd apps/broker/overlays/dev/values.yaml broker.sentryDsn, apps/job-gateway/overlays/dev/job-gateway-{1,2}/values.yaml gateway.sentryDsn (DSN key not reproduced)

ConfigMap-sourced content (from the overlay files that generate the ConfigMaps):

Hostname ConfigMap / component Where
github.com (https://github.com/apptainer/apptainer/releases/download/...) install-apptainer-script in namespace slurm (init container of both NodeSets) srv-cd apps/slurm/overlays/dev/scripts/install-apptainer.sh line 14
projects.pawsey.org.au (https://projects.pawsey.org.au/srcnet/...) broker-test-files (smoke-test suite) in namespace broker broker tests/integration/ (test data URLs)

Not env/ConfigMap but external pull sources of the same namespaces (image and chart references): docker.io (htcondor/*, postgres, rabbitmq, mariadb, alpine, busybox, ubuntu), ghcr.io (slinkyproject images and charts), registry.k8s.io (kueue image and OCI chart), helm.goharbor.io (harbor chart), quay.io and htcss-downloads.chtc.wisc.edu (pilot image build only), registry.gitlab.com (base-values image repos, overridden to harbor.test in dev), harbor.srcnet.skao.int (harbor base values, overridden to harbor.test in dev). No other external hostname was found in the dev overlays or chart values of job-gateway, kueue, harbor, htcondor, broker, mock-battle-api, slurm, slurm-operator.

D.4 Other overlays beside dev

Overlay directories at 146b9312 (srv-cd apps/<app>/overlays/): job-gateway staging/stfc/{job-gateway-1,job-gateway-2}; kueue staging/stfc; harbor staging/stfc and prod; htcondor staging/stfc; broker staging/stfc; (also mock-battle-api staging/stfc/{1,2}, slurm staging/stfc, slurm-operator staging/stfc). Staging overlays do not inflate the chart themselves ("published chart is a separate Argo source").

App / overlay Replicas Resources Storage Other differences from dev
job-gateway staging/stfc (1 and 2) gateway 1 (unchanged); desiredPilots: 2 (dev 1); slurm desiredPilots: 1 unchanged (chart defaults 100m/128Mi–500m/512Mi) same claims site-scratch, storm{1,2}-sa cloned from the storm staging overlays (not read); slurm.storageHostPath: /srv/storage images registry.gitlab.com/ska-telescope/src/src-ef/ska-src-ef-local-job-gateway/job-gateway:0.4.1 and pilot .../ska-src-ef-htcondor-deployment/pilot:0.4.1; payloadIdentity: matched on gateway-1 (pool on gateway-2); pilotExtraEnv adds _CONDOR_STARTER_UPDATE_INTERVAL=60,_CONDOR_SEC_CREDENTIAL_REFRESH=60; ingress disabled; CA secret ca-secret (ExternalSecret cert-ca); register job image harbor.staging.srcnet.skao.int/library/integration:latest, URLs iam.staging.srcnet.skao.int, site-capabilities.staging.srcnet.skao.int
kueue staging/stfc 1 manager 100m/256Mi–1/512Mi (same) none ClusterQueues pilot-pool-storm1/2 identical (20 CPU / 40Gi each, no cohort), ResourceFlavor default identical; no kustomization.yaml and no deploy/pre/visibility-rbac
harbor staging/stfc 1 each (same) same (only trivy set) same sizes (registry 100Gi, jobservice 1Gi, database 1Gi, redis 1Gi, trivy 5Gi) but storageClass: csi-manila-cephfs-sc-retain for all five (dev local-path) host harbor.staging.srcnet.skao.int; same harbor ExternalSecret
harbor prod 1 each same base values: csi-cinder-sc-retain, same sizes host harbor.srcnet.skao.int; fullnameOverride: harbor; deletes chart Secret harbor-database; core envFrom Secret harbor; no ExternalSecret file in the overlay
htcondor staging/stfc cm 1, submit 1, execute 2 (same) same (cm 100m/256Mi–500m/1Gi, execute 250m/512Mi–2/2Gi, submit 100m/256Mi–500m/1Gi) shared-scratch created by the chart (scratchPvc.create: true) 100Gi RWX csi-manila-cephfs-sc-retain (dev: 10Gi hostPath PV); submit state PVC 5Gi csi-manila-cephfs-sc-retain (dev: default class) no ca-trust-patch.yaml, no ska-local-ca-secret ExternalSecret; same pilot-sites.txt
broker staging/stfc 3 (same) same postgres 10Gi csi-manila-cephfs-sc-retain (dev local-path); clones shared-scratch, ingest-area as in dev image .../computing-broker:0.4.1; authzEnabled: true; staging URLs (permissions., authn., data-management., site-capabilities.staging.srcnet.skao.int, iam.staging.srcnet.skao.int); host broker.staging.srcnet.skao.int with shared ingress TLS; extra ExternalSecrets cert-ca → ca-secret (secret/data/cert-manager CA_CRT) and broker-test-user (secret/data/iam TEST_USER1_PASSWORD); no truststore patch; smoke test serial (-n 0)
mock-battle-api staging/stfc 1 same same clones image harbor.staging.srcnet.skao.int/library/mock-battle-api:latest; ingress disabled; RSE names NODE1_SITE1_STORM1, NODE1_SITE1_SCRATCH, storm URLs storm1.staging.srcnet.skao.int
slurm staging/stfc NodeSets compute 2, compute2 1 (same); restapi 1 same controller statesave 10Gi csi-manila-cephfs-sc-retain (dev 5Gi local-path); mariadb-data unchanged clusterName: ska-src-staging-stfc; no APPTAINER_NOHTTPS; no slurmrestd Ingress; wait-for-cluster timeouts 600s
slurm-operator staging/stfc 1 / 1 none set none operator.logLevel: info (dev debug)

Items not found (and where I looked)

Item Status
Artifacts of job 17048104007 job still running at 2026-10-09 03:23 UTC; /jobs/17048104007/artifacts returned 404. Used job 17034588345.
StatefulSet / NodeSet / PVC / PV / ConfigMap / Secret / ExternalSecret / RBAC / Kueue / Ingress objects in the dump not present in cluster-info dump output; reconstructed from repo sources and the CI apply log as cited.
PV reclaim policy for dynamically provisioned local-path claims (harbor ×5, htcondor-submit, broker-postgresql, statesave-slurm-controller-0, mariadb-data) not found: no PV objects in the dump and the provisioner's policy is not declared in the sources.
Creator of Secret broker-test-token not found in srv-cd apps/broker/overlays/dev, stack bin/, include/, lib/.
KUEUE_CHART_VERSION value lives in .env.dev (not in the repo); the cached chart on disk is 0.18.1 and the running image is v0.18.1.
Content of the stock HTCondor images' 01-ccb.conf only its load is visible in htcondor-execute-*/logs.txt; the file is inside docker.io/htcondor/*:lts, not in any source.
Pilot daemon logs (to confirm CCB vs inbound at runtime) written to /srv/storage/pilot-logs/, not captured in the dump; conclusion in A.5 is from the pilot config in pilot-entrypoint.sh.
Kubernetes manifests in the cavern-deployment repo none exist at c6cfbefa (docker-compose only; etc/helm/values.yaml is empty).
A real (non-mock) BATTLE service none referenced in any source.
Staging storm overlays (source PVCs for staging clones) not read; listed as "not read" in D.4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment