Generated 2026-10-09 (read-only; nothing was applied, synced or changed in any cluster or repo). Secret values are never printed; only Secret names, ExternalSecret Vault paths and key names.
| Ref | Source | Notes |
|---|---|---|
dump |
Artifacts of GitLab job 17034588345 (project ska-telescope/src/src-api/ska-src-api-deployment-stack), out/deploy-ci-2927014939/cluster-dump/<namespace>/*.json, out/deploy-ci-2927014939/log, report.html |
Job 17048104007 had no artifacts at 2026-10-09 03:23 UTC: the API returned 404 Not Found for /jobs/17048104007/artifacts and /jobs/17048104007 reported "status":"running" (started 2026-10-09T02:36:00Z). The fallback job was used. |
srv-cd |
ska-telescope/src/deployments/skaosrc/ska-src-skaosrc-services-cd @ 146b9312 (branch tiger2-poc) |
Extracted with git archive; this is also the exact commit pinned by the stack gitlink infra/ska-src-skaosrc-services-cd at ff66cd3e. |
stack |
ska-telescope/src/src-api/ska-src-api-deployment-stack @ ff66cd3e (branch tiger2-poc) |
etc/env/dev/configuration.yaml, etc/env/dev/services/*.yaml, include/env/dev/services/*.mk. |
gw |
svc/ska-src-ef-local-job-gateway @ 265cc254 (gitlink at stack ff66cd3e) |
chart charts/job-gateway (Chart.yaml version 0.4.1) and src/. |
condor |
svc/ska-src-ef-htcondor-deployment @ c5b664c9 |
chart charts/htcondor, images/pilot/. |
broker |
svc/ska-src-ef-computing-broker @ 5c650557 |
chart charts/computing-broker, etc/leader/, src/. |
mba |
svc/ska-src-ef-mock-battle-api @ 55a964f1 |
chart charts/mock-battle-api. |
cavern |
svc/ska-src-dm-cavern-deployment @ c6cfbefa |
docker-compose based. |
kueue-chart, slinky-chart |
Charts cached on the local disk under infra/ska-src-skaosrc-services-cd/apps/kueue/overlays/dev/charts/kueue (Chart.yaml version 0.18.1) and apps/slurm-operator/overlays/dev/charts/slurm-operator-1.0.2 |
These directories are gitignored and are not in commit 146b9312; only used for RBAC rules of third-party operators. |
What the dump contains. kubectl cluster-info dump format: per namespace only deployments.json, daemonsets.json, replicasets.json, pods.json, services.json, events.json, plus per-pod logs.txt, and nodes.json. It contains no StatefulSet, NodeSet, PVC, PV, ConfigMap, Secret, ExternalSecret, Role/ClusterRole, Kueue CR, Ingress or NetworkPolicy objects. StatefulSet/NodeSet membership below is taken from pod ownerReferences; everything else of those kinds comes from the repo sources and from the serverside-applied lines in the CI deploy log (out/deploy-ci-2927014939/log), which confirm the names of objects that were actually applied.
Nodes (cluster-dump/nodes.json): k3d-deploy-ci-2927014939-server-0 and -agent-0, each capacity cpu 64, memory 237266968Ki, pods 110.
Overlay/chart layering used by dev (srv-cd apps/<app>/overlays/dev/.../kustomization.yaml): job-gateway, htcondor, broker and mock-battle-api inflate the chart from the svc/ submodule (helmGlobals.chartHome: ../../../../../../svc/<repo>/charts) with base/values.yaml + overlay values.yaml; kueue inflates a locally cached OCI chart; harbor uses https://helm.goharbor.io chart 1.15.0; slurm / slurm-operator use oci://ghcr.io/slinkyproject/charts 1.0.2.
Conventions: "none set" = field absent in the live pod spec in the dump. Every pod in all ten namespaces had hostNetwork=false, hostPID=false and only the default node.kubernetes.io/not-ready / node.kubernetes.io/unreachable (NoExecute, 300 s) tolerations unless stated otherwise (cluster-dump/<ns>/pods.json). No pod in these namespaces mounts a hostPath volume directly; all shared storage is reached through PVC "clones" whose PVs are hostPath (see each PVC table). Image digests are from status.containerStatuses[].imageID in pods.json.
Workloads (cluster-dump/job-gateway-1/deployments.json, pods.json):
| Workload | Kind | Replicas (ready) | Container | Image | CPU req / lim | Mem req / lim |
|---|---|---|---|---|---|---|
| job-gateway-1 | Deployment | 1 (1) | job-gateway | harbor.test/library/job-gateway:dev = harbor.test/library/job-gateway@sha256:a8793fab7729319176c5e127c1d46c9a276c1ec4bbc273ba444eb74a0fe00dc9 (built from gw images/job-gateway/Dockerfile, base python:3.13-slim) |
100m / 500m | 128Mi / 512Mi |
| pilot-dynamic-<id> (5 present at dump time, each a batch/v1 Job created by the gateway) | Job → Pod | 1 each | init fix-storage-perms |
harbor.test/library/htcondor-pilot:dev = ...@sha256:69a7b489ee2bcf23e5b52ae58cd7e03be48687a6f89e7b72b7c6b8f7c493b477 |
none set | none set |
startd |
same image | observed 1 / 1 (×3 pods), 2 / 2 (×1), 1 / 1 (×1) | 512Mi / 512Mi (×4), 256Mi / 256Mi (×1) |
Source of resource values: chart gw charts/job-gateway/values.yaml (resources) via srv-cd apps/job-gateway/base/values.yaml; pilot sizes come from gw src/ska_src_ef_job_gateway/clients/kubernetes_provisioner.py::build_job_manifest (requests == limits, sized per matched task, defaults PILOT_DEFAULT_CPUS=2, PILOT_DEFAULT_MEMORY_MB=4096 in the gateway env).
Security context:
| Pod | Pod-level | Container-level | hostNetwork / hostPID | hostPath volumes |
|---|---|---|---|---|
| job-gateway-1 | {} |
none set (no privileged, no runAsUser/runAsNonRoot, no allowPrivilegeEscalation, no capabilities) | false / false | none |
| pilot-dynamic-* | {} |
startd: {"privileged": true}; init fix-storage-perms: none set (PVC mode; the code sets privileged: true on the init container only in hostPath mode, kubernetes_provisioner.py lines ~258-276) |
false / false | none (PVC mode because STORAGE_PVCS is set) |
Ports and Services (cluster-dump/job-gateway-1/services.json):
| Object | Detail |
|---|---|
| containerPort | job-gateway: http 8081/TCP. Pilot pods: no containerPorts declared. |
Service job-gateway-1 |
ClusterIP 10.43.132.95, port http 8081 → targetPort http (8081), TCP, selector app.kubernetes.io/instance=job-gateway-1, app.kubernetes.io/name=job-gateway |
| Ingress | job-gateway-1.test, class nginx, TLS secret job-gateway-1-tls, issuer ska-local-ca-issuer (srv-cd apps/job-gateway/overlays/dev/job-gateway-1/values.yaml); Ingress object not in dump |
PVCs mounted (gateway pod and every pilot pod; dump pods.json volumes):
| PVC (namespace job-gateway-1) | Mount | Size / class / access | PV behind it | Where defined |
|---|---|---|---|---|
site-scratch |
/srv/storage (rw) |
clone of storm1/site-scratch: 10Gi, storageClassName: hostpath, RWX |
storm1-site-scratch-pv-job-gateway-1 (name pattern <src-pv>-<dst-ns>), hostPath /srv/storage/site-scratch-1 DirectoryOrCreate, reclaim Retain (forced by the clone script) |
source: srv-cd apps/storm/overlays/dev/storm1/site-scratch-pv.yaml, site-scratch-pvc.yaml; clone job: srv-cd apps/job-gateway/overlays/dev/job-gateway-1/deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml running srv-cd scripts/common/provision-shared-storage/create-shared-pvc.py |
storm1-sa |
/srv/storage/storm1 (ro) |
clone of storm1/storage-area: 10Gi, hostpath, RWX |
storm1-datasets-pv-job-gateway-1, hostPath /srv/storage/storm1, Retain |
srv-cd apps/storm/overlays/dev/storm1/storm1-datasets-pv.yaml, storage-area-pvc.yaml + same clone job |
PVC/PV objects themselves are not in the dump; the CI log (log lines 4335-4338) confirms the clone job's ServiceAccount/ClusterRole/ConfigMap were applied.
Secrets (names only) and ExternalSecrets (srv-cd apps/job-gateway/overlays/dev/job-gateway-1/externalsecrets.yaml and deploy/post/bootstrap/externalsecrets.yaml; all use ClusterSecretStore vault-cluster-secretstore, refresh 1h):
| Secret | Used by | ExternalSecret | Vault path | Vault property → secret key |
|---|---|---|---|---|
htcondor-gateway-idtoken |
gateway volume /etc/condor/tokens.d |
htcondor-gateway-idtoken |
secret/data/htcondor |
READ_IDTOKEN → gateway.token |
htcondor-pilot-idtoken |
gateway volume /etc/pilot; every pilot pod /root/secrets/idtoken (subPath idtoken, defaultMode 0400) |
htcondor-pilot-idtoken |
secret/data/htcondor/pilots/NODE1_SITE1 |
PILOT_IDTOKEN → templated key idtoken |
slurm-jwt |
gateway volume /etc/slurm-jwt |
slurm-jwt |
secret/data/slurm |
JWT_HS256_KEY → jwt_hs256.key |
ska-local-ca-secret |
post-deploy register job only | ska-local-ca-secret |
secret/data/cert-manager |
CA_CRT → ca.crt |
scapi-client |
post-deploy register job env | scapi-client-secret |
secret/data/iam/clients |
SITE_CAPABILITIES_API_CLIENT_ID → clientId, SITE_CAPABILITIES_API_CLIENT_SECRET → clientSecret |
job-gateway-1-tls |
Ingress | cert-manager issued (not an ExternalSecret) | – | – |
No container in this namespace reads a Secret through env.valueFrom (dump: envSecrets empty for gateway and pilots). The gateway has an env var SENTRY_DSN whose value is set inline in values.yaml (value not reproduced here).
Scheduling: nodeSelector none, affinity none, tolerations default only (chart values.yaml has nodeSelector: {}, tolerations: [], affinity: {}; pilot manifest sets none). ServiceAccount: gateway job-gateway-1; pilots default.
job-gateway-2 differences (dump env diff job-gateway-1 vs job-gateway-2 pods; srv-cd apps/job-gateway/overlays/dev/job-gateway-2/values.yaml): SITE_ID=NODE2_SITE1, PILOT_NAMESPACE=job-gateway-2, PVC storm2-sa (clone of storm2/storage-area, PV storm2-datasets-pv-job-gateway-2, hostPath /srv/storage/storm2) at /srv/storage/storm2, site-scratch clone of storm2/site-scratch (PV storm2-site-scratch-pv-job-gateway-2, hostPath /srv/storage/site-scratch-2), PILOT_SCRATCH_ROOT=/srv/storage/pilot-scratch-2, BATTLE_API_BASE=http://mock-battle-api-2.mock-battle-api-2.svc.cluster.local:8765, SLURM_PARTITION=compute2, SLURM_PARTITION_CAPACITY_CPUS=16, SLURM_PARTITION_CAPACITY_MEMORY_MB=16384, Kueue ClusterQueue pilot-pool-storm2, pilot IDTOKEN from secret/data/htcondor/pilots/NODE2_SITE1, Ingress job-gateway-2.test / job-gateway-2-tls. Service job-gateway-2 ClusterIP 10.43.190.187 8081→http. One pilot pod present at dump time (1 CPU / 512Mi, privileged). Everything else identical.
Common gateway env worth recording (dump): HTCONDOR_ENABLED=true, CONDOR_HOST=htcondor-cm.htcondor.svc.cluster.local, HTCONDOR_SCHEDD_NAME=htcondor-submit@htcondor-submit.htcondor.svc.cluster.local, DESIRED_PILOTS=1, SLURM_ENABLED=true, SLURM_ENDPOINT=http://slurm-restapi.slurm.svc.cluster.local:6820, SLURM_DESIRED_PILOTS=1, SLURM_PILOT_USER=ska-pilot, PAYLOAD_IDENTITY=pool, PAYLOAD_EXECUTION=in-pilot, KUEUE_LOCAL_QUEUE=pilots, OTEL_EXPORTER_OTLP_ENDPOINT=http://monitoring-opentelemetry-collector.monitoring.svc.cluster.local:4317.
| Item | Value | Source |
|---|---|---|
Deployment kueue-controller-manager |
replicas 1 (1 ready) | cluster-dump/kueue-system/deployments.json |
Container manager image |
registry.k8s.io/kueue/kueue:v0.18.1 = ...@sha256:50eb1d808ad6597599a21a1234ca624f8acdb364ab75a93d003d442341ebef0f |
pods.json |
| Resources | requests 100m / 256Mi, limits 1 / 512Mi | dump; srv-cd apps/kueue/overlays/dev/values.yaml |
| Pod securityContext | runAsNonRoot: true, seccompProfile: RuntimeDefault |
dump |
| Container securityContext | capabilities.drop: [ALL], readOnlyRootFilesystem: true, allowPrivilegeEscalation: false; privileged not set; runAsUser not set |
dump |
| containerPorts | visibility 8082, webhook-server 9443, metrics 8443 (all TCP) |
dump |
| Services | kueue-controller-manager-metrics-service ClusterIP https 8443→8443; kueue-visibility-server ClusterIP https 443→8082; kueue-webhook-service ClusterIP 443→9443 |
services.json |
| Volumes | Secret kueue-webhook-server-cert (chart-generated, enableCertManager: false), ConfigMap kueue-manager-config, emptyDir visibility |
dump; overlays/dev/values.yaml |
| PVC / ExternalSecret | none | – |
| ServiceAccount | kueue-controller-manager |
dump |
| nodeSelector / tolerations / affinity | none / default / none | dump |
| Chart | version 0.18.1 (cached chart Chart.yaml); pulled from oci://registry.k8s.io/kueue/charts/kueue by stack include/env/dev/services/kueue.mk; KUEUE_CHART_VERSION comes from .env.dev, which is not in the sources |
kueue.mk |
| Extra | RoleBinding kueue-visibility-server-auth-reader in kube-system binding SA kueue-controller-manager to Role extension-apiserver-authentication-reader (srv-cd apps/kueue/overlays/dev/deploy/pre/visibility-rbac/rolebinding.yaml) |
| Item | Value | Source |
|---|---|---|
Deployment mock-battle-api-1 |
replicas 1 (1) | cluster-dump/mock-battle-api-1/deployments.json |
| Image | harbor.test/library/mock-battle-api:dev = ...@sha256:0b32110c70d9e35c14f2b017efa725163471167ed67a615c9f20f6a7da951466 |
pods.json |
| Resources | requests 100m / 128Mi, limits 500m / 512Mi | dump; srv-cd apps/mock-battle-api/overlays/dev/mock-battle-api-1/values.yaml |
| securityContext | pod {}; container none set |
dump |
| containerPort | http 8765/TCP |
dump |
Service mock-battle-api-1 |
ClusterIP 10.43.42.17, http 8765 → http, TCP |
services.json |
| Ingress | mock-battle-api-1.test (nginx, no TLS block) |
overlay values.yaml |
| PVCs | site-scratch → /srv/storage (clone of storm1/site-scratch, PV storm1-site-scratch-pv-mock-battle-api-1, hostPath, Retain, 10Gi RWX hostpath); storm1-sa → /srv/storage/storm1 ro (clone of storm1/storage-area) |
overlay values.yaml extraVolumes; deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml |
| Secrets / ExternalSecrets | none | overlay has no externalsecrets.yaml; dump shows no Secret volumes or env refs |
| Env (plain) | BATTLE_STORAGE=/srv/storage, BATTLE_DATA_ROOTS=/srv/storage/storm1/sa, BATTLE_SCRATCH_ROOT=/srv/storage/pilot-scratch-1, BATTLE_SCRATCH same, BATTLE_STORAGE_REGISTRY JSON with dav://storm1.test/sa/inputs, scratch://storm1.test/outputs, davs://storm1.test:443/sa/deterministic |
dump |
| ServiceAccount / scheduling | default; nodeSelector none, affinity none, default tolerations |
dump |
mock-battle-api-2: identical except namespace, image digest not separately verified (same tag), PVC storm2-sa → /srv/storage/storm2, scratch /srv/storage/pilot-scratch-2, Service ClusterIP 10.43.191.86 (cluster-dump/mock-battle-api-2/). A suspended test CronJob mock-battle-api-1-test-job (image harbor.test/library/mock-battle-api-test:dev, runAsNonRoot uid/gid 1000) is defined in the overlay.
Workloads (cluster-dump/harbor/deployments.json, pods.json; StatefulSets inferred from ownerReferences). Release name is release-name (no fullnameOverride in dev).
| Workload | Kind | Replicas | Containers (image, all goharbor/*:v2.11.0 unless stated) |
Resources |
|---|---|---|---|---|
| release-name-harbor-core | Deployment | 1 | core harbor-core |
none set |
| release-name-harbor-jobservice | Deployment | 1 | jobservice harbor-jobservice |
none set |
| release-name-harbor-portal | Deployment | 1 | portal harbor-portal |
none set |
| release-name-harbor-registry | Deployment | 1 | init registry-permission-fix busybox (resolved busybox:1.38.0); registry registry-photon; registryctl harbor-registryctl |
none set |
| release-name-harbor-database | StatefulSet | 1 pod | init data-permissions-ensurer harbor-db v2.11.0 (req 100m/128Mi), data-migrator harbor-db v2.10.0, data-permissions-ensurer-1 harbor-db v2.10.0; database harbor-db |
none set (except the init request) |
| release-name-harbor-redis | StatefulSet | 1 pod | init redis-permission-fix busybox; redis redis-photon |
none set |
| release-name-harbor-trivy | StatefulSet | 1 pod | init trivy-permission-fix busybox; trivy trivy-adapter-photon |
requests 200m / 512Mi, limits 1 / 1Gi |
Image digests (dump): harbor-core sha256:6092e603…, harbor-jobservice sha256:862d1120…, harbor-portal sha256:56c5e205…, registry-photon sha256:545abacd…, harbor-registryctl sha256:fc63bf62…, harbor-db v2.11.0 sha256:10382001…, harbor-db v2.10.0 sha256:bd3ed7a8…, redis-photon sha256:a7fad8c1…, trivy-adapter-photon sha256:03db1258…, busybox sha256:fd7dc986… (full values in cluster-dump/harbor/pods.json imageID).
Security context (dump): pod-level runAsUser: 10000, fsGroup: 10000 for core, jobservice, portal, registry (registry adds fsGroupChangePolicy: OnRootMismatch) and trivy; runAsUser: 999, fsGroup: 999 for database and redis. Every main container: privileged: false, runAsNonRoot: true, allowPrivilegeEscalation: false, capabilities.drop: [ALL], seccompProfile: RuntimeDefault (from srv-cd apps/harbor/base/values.yaml containerSecurityContext). The five permission-fix / migrator init containers run with runAsUser: 0. No hostNetwork/hostPID, no hostPath.
Ports and Services (cluster-dump/harbor/services.json):
| containerPorts | Service | Type | Port → targetPort / protocol |
|---|---|---|---|
| core 8080 | release-name-harbor-core | ClusterIP | http-web 80 → 8080 TCP |
| database: none declared | release-name-harbor-database | ClusterIP | 5432 → 5432 TCP |
| jobservice 8080 | release-name-harbor-jobservice | ClusterIP | http-jobservice 80 → 8080 TCP |
| portal 8080 | release-name-harbor-portal | ClusterIP | 80 → 8080 TCP |
| redis: none declared | release-name-harbor-redis | ClusterIP | 6379 → 6379 TCP |
| registry 5000, 5001; registryctl 8080 | release-name-harbor-registry | ClusterIP | http-registry 5000 → 5000, http-controller 8080 → 8080 TCP |
trivy api-server 8080 |
release-name-harbor-trivy | ClusterIP | http-trivy 8080 → 8080 TCP |
Ingress: harbor.test, externalURL: https://harbor.test, class nginx (srv-cd apps/harbor/overlays/dev/values.yaml).
PVCs (claim names from dump pod volumes; sizes/access from srv-cd apps/harbor/base/values.yaml persistence.persistentVolumeClaim; class from overlays/dev/values.yaml):
| PVC | Mount | Size | storageClassName | accessModes | PV |
|---|---|---|---|---|---|
release-name-harbor-registry |
registry + registryctl /storage |
100Gi | local-path |
RWO | dynamic (local-path provisioner); PV object and reclaim policy not in dump |
release-name-harbor-jobservice |
/var/log/jobs |
1Gi | local-path |
RWO | same |
database-data-release-name-harbor-database-0 |
/var/lib/postgresql/data |
1Gi | local-path |
RWO | same |
data-release-name-harbor-redis-0 |
/var/lib/redis |
1Gi | local-path |
RWO | same |
data-release-name-harbor-trivy-0 |
/home/scanner/.cache |
5Gi | local-path |
RWO | same |
CI log confirms persistentvolumeclaim/release-name-harbor-jobservice and /release-name-harbor-registry applied (log lines 576-577); the three StatefulSet claims are created by volumeClaimTemplates.
Secrets and ExternalSecrets:
| Secret | Referenced by | Origin |
|---|---|---|
harbor |
core env secret, JOBSERVICE_SECRET, HARBOR_ADMIN_PASSWORD, REGISTRY_PASSWD; jobservice env; registryctl env; registry volume registry-htpasswd; core volume secret-key |
ExternalSecret harbor-secrets (srv-cd apps/harbor/overlays/dev/externalsecrets.yaml), Vault secret/data/harbor, properties → keys: HARBOR_ADMIN_PASSWORD→HARBOR_ADMIN_PASSWORD, JOBSERVICE_SECRET→JOBSERVICE_SECRET, REGISTRY_HTPASSWD→REGISTRY_HTPASSWD, REGISTRY_PASSWD→REGISTRY_PASSWD, CORE_SECRET→secret, SECRET_KEY→secretKey |
release-name-harbor-core |
core envFrom + volume token-service-private-key |
chart-rendered Secret (CI log line 563) |
release-name-harbor-database |
database envFrom | chart Secret |
release-name-harbor-jobservice |
jobservice envFrom | chart Secret |
release-name-harbor-registry |
registry/registryctl envFrom | chart Secret |
release-name-harbor-registryctl |
registryctl envFrom | chart Secret |
release-name-harbor-trivy |
trivy env gitHubToken, redisURL |
chart Secret |
ServiceAccount default for all pods; nodeSelector/affinity none; default tolerations.
Workloads (cluster-dump/htcondor/):
| Deployment | Replicas | Containers | Image | CPU req / lim | Mem req / lim |
|---|---|---|---|---|---|
| htcondor-cm | 1 (1) | cm |
docker.io/htcondor/cm:lts = ...@sha256:b025796d6ef2cad5bf2be83eaec520d92038692f17fcdd96849d7ba40c8a4b04 |
100m / 500m | 256Mi / 1Gi |
logfile (sidecar) |
alpine:3.20 (sha256:d9e853e8…) |
10m / 50m | 16Mi / 32Mi | ||
| htcondor-execute | 2 (2) | execute |
docker.io/htcondor/execute:lts = ...@sha256:73f4e5c805b86e867420b16a950894b0bd486e0782a8ebbc4bc6729f014bde51 |
250m / 2 | 512Mi / 2Gi |
logfile |
alpine:3.20 | 10m / 50m | 16Mi / 32Mi | ||
| htcondor-submit | 1 (1) | init init-shared-scratch |
busybox:1.36 (sha256:73aaf090…) |
none set | none set |
submit |
docker.io/htcondor/submit:lts = ...@sha256:d68be980711682a8a1defa4f00187a331d69418033a1d8d6b68a4b498f5ae70d |
100m / 500m | 256Mi / 1Gi | ||
logfile |
alpine:3.20 | 10m / 50m | 16Mi / 32Mi |
Values: srv-cd apps/htcondor/base/values.yaml + overlays/dev/values.yaml (execute.replicaCount: 2, numCpus: 2, memoryMb: 2048). No StatefulSet (cm.ha.enabled is false, chart default).
Security: pod and container securityContext none set on all three Deployments (the chart templates cm-deployment.yaml, execute-deployment.yaml, submit-deployment.yaml carry no securityContext); hostNetwork/hostPID false; no hostPath (submit uses PVC mode because submit.sharedStorage.pvcs is set; the chart falls back to a hostPath /srv/storage only when that list is empty, condor charts/htcondor/templates/submit-deployment.yaml volumes block). ServiceAccount htcondor (chart serviceaccount.yaml; the chart defines no Role/ClusterRole for it).
Ports / Services (services.json):
| containerPort | Service | Type | Port → target |
|---|---|---|---|
cm condor 9618/TCP |
htcondor-cm |
ClusterIP 10.43.184.196 | condor 9618 → condor (9618) TCP |
submit condor 9618/TCP |
htcondor-submit |
ClusterIP 10.43.65.217 | condor 9618 → condor TCP |
execute condor 9618/TCP |
(no Service) | – | – |
Port 9618 / CCB answer. Yes, HTCondor uses 9618: containerPort 9618 on cm, execute and submit and Services htcondor-cm and htcondor-submit expose 9618/TCP (dump; chart values.yaml cm.service.port, submit.service.port). The central manager log shows DaemonCore: command socket at <10.42.1.87:9618?...&sock=collector> and Enabling CCB Server. (cluster-dump/htcondor/htcondor-cm-79688bf64-94zn9/logs.txt). The pilot config written by condor images/pilot/pilot-entrypoint.sh (/etc/condor/config.d/01-pilot.conf) sets CONDOR_HOST, DAEMON_LIST = MASTER STARTD, USE_SHARED_PORT = TRUE, IDTOKENS-only auth with SEC_DEFAULT_ENCRYPTION/INTEGRITY = REQUIRED, and does not set CCB_ADDRESS (no CCB, PRIVATE_NETWORK_NAME or TCP_FORWARDING_HOST setting anywhere in images/pilot/, the gateway chart's pilot-configmap.yaml, or the gateway source; the only CCB strings in the sources are comments). The pilot therefore does not register with the CM's CCB server and expects inbound TCP connections from the schedd/shadow to its own pod address on shared port 9618 (pilot pods declare no containerPort and have no Service; reachability relies on routable pod IPs inside the cluster). The stock htcondor/execute and htcondor/submit images additionally load a /etc/condor/config.d/01-ccb.conf from the image (htcondor-execute-*/logs.txt); its content is not in the sources. Pilot daemon logs are written to /srv/storage/pilot-logs/<hostname>/ and are not in the dump, so no runtime confirmation of pilot connection direction was possible.
Volumes / PVCs:
| Volume | Pods | Type | Detail |
|---|---|---|---|
pool-password |
cm, execute, submit | Secret htcondor-pool-password |
mounted /etc/condor/passwords-orig.d |
condor-config |
all | ConfigMap htcondor-config (/root/config); submit also mounts key condor_mapfile at /etc/condor/condor_mapfile |
chart configmap.yaml |
condor-logs |
all | emptyDir /var/log/condor |
|
ska-local-ca-secret |
all | Secret ska-local-ca-secret key ca.crt at /etc/ssl/certs/ska-local-ca.crt, env REQUESTS_CA_BUNDLE |
srv-cd apps/htcondor/overlays/dev/ca-trust-patch.yaml |
condor-state |
submit | PVC htcondor-submit → /var/lib/condor: 5Gi, storageClass: "" (cluster default), RWO (overlays/dev/values.yaml submit.persistence, chart submit-pvc.yaml); PV dynamic, not in dump |
CI log line 3873 |
shared-storage-0 |
submit | PVC shared-scratch → /srv/storage: 10Gi, RWX, storageClassName: hostpath, bound to PV shared-scratch-pv (hostPath /srv/storage/shared-scratch, DirectoryOrCreate, reclaim Retain) |
overlays/dev/shared-scratch-pv.yaml, shared-scratch-pvc.yaml; CI log lines 3872, 3874 |
ExternalSecrets (srv-cd apps/htcondor/overlays/dev/externalsecrets.yaml):
| ExternalSecret → Secret | Vault path | property → key |
|---|---|---|
htcondor-pool-password |
secret/data/htcondor |
POOL_PASSWORD → POOL |
ska-local-ca-secret |
secret/data/cert-manager |
CA_CRT → ca.crt |
Vault keys written by the stack hooks for this component (CI log lines 3774-3910, paths only): secret/data/htcondor (pre-deploy and post-deploy writes; consumed as POOL_PASSWORD, READ_IDTOKEN, SUBMIT_IDTOKEN by the ExternalSecrets above and in broker/gateways), secret/data/htcondor/pilots/NODE1_SITE1 and secret/data/htcondor/pilots/NODE2_SITE1 (PILOT_IDTOKEN). Sites list: srv-cd apps/htcondor/overlays/dev/pilot-sites.txt (job-gateway-1 NODE1_SITE1, job-gateway-2 NODE2_SITE1); IDTOKEN subjects pilot-NODE1_SITE1@condor, pilot-NODE2_SITE1@condor (base/values.yaml security.pilotIdentities).
Scheduling: nodeSelector/affinity none, default tolerations (chart values {}/[]).
Workloads (cluster-dump/broker/):
| Deployment | Replicas | Container | Image | CPU req / lim | Mem req / lim |
|---|---|---|---|---|---|
| broker-computing-broker | 3 (3) | computing-broker |
harbor.test/library/computing-broker:dev = ...@sha256:014033a29fa3db1fa29a26dec6bd4f3592372fc3bf44c156c20ccb14234ceb8d (base python:3.13-slim, broker images/computing-broker/Dockerfile) |
200m / 1 | 256Mi / 1Gi |
| broker-postgresql | 1 (1) | postgresql |
postgres:16 = docker.io/library/postgres@sha256:ca0bd484cb98bf4b24eb1010e73fb3fcbd6714d240fbc1a10eea5b7dbecb641d |
100m / 500m | 256Mi / 1Gi |
| broker-rabbitmq | 1 (1) | rabbitmq |
rabbitmq:3.13-management = ...@sha256:e582c0bc7766f3342496d8485efb5a1df782b5ce3886ad017e2eaae442311f69 |
100m / 500m | 256Mi / 512Mi |
| broker-smoke-test-job-1791475619 (Job, Succeeded) | – | broker-smoke-test-job |
harbor.test/library/test:dev (sha256:427dfc7b…) |
none set | none set |
replicaCount: 3 from srv-cd apps/broker/overlays/dev/values.yaml. Security: pod {} and container securityContext none set on all four pods (chart templates carry none); no hostNetwork/hostPID/hostPath (hostPath /srv/storage would be used only if broker.condorSubmit.storagePvcs were empty, broker charts/computing-broker/templates/deployment.yaml volumes block). ServiceAccount default (no Role bound).
Ports / Services (services.json):
| containerPort | Service | Type | Port → target |
|---|---|---|---|
computing-broker http 8082/TCP |
broker-computing-broker |
ClusterIP 10.43.78.207 | http 8082 → http TCP |
postgresql pg 5432/TCP |
broker-postgresql |
ClusterIP 10.43.208.250 | pg 5432 → pg TCP |
rabbitmq amqp 5672/TCP |
broker-rabbitmq |
ClusterIP 10.43.92.5 | amqp 5672 → amqp TCP |
Ingress broker.test, TLS secret broker-tls, issuer ska-local-ca-issuer (overlay values.yaml).
PVCs:
| PVC | Pods / mount | Spec | PV |
|---|---|---|---|
shared-scratch (broker ns) |
computing-broker /srv/storage |
clone of htcondor/shared-scratch: 10Gi RWX hostpath |
shared-scratch-pv-broker, hostPath /srv/storage/shared-scratch, Retain (srv-cd apps/broker/overlays/dev/deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml) |
ingest-area (broker ns) |
smoke-test job /tmp/ingest |
clone of ingestor/ingest-area: 10Gi RWX hostpath |
ingestor-storm1-pv-broker, hostPath /srv/storage/storm1/sa/ingest, Retain (source srv-cd apps/ingestor/overlays/dev/ingestor-storm1-storage.yaml) |
broker-postgresql |
postgresql /var/lib/postgresql/data |
10Gi, RWO, storageClassName: local-path (overlays/dev/values.yaml postgresql.storage; chart postgresql.yaml) |
dynamic, not in dump |
| (rabbitmq) | /var/lib/rabbitmq |
emptyDir | – |
Secrets referenced (dump env.valueFrom.secretKeyRef, volumes) and ExternalSecrets (srv-cd apps/broker/overlays/dev/externalsecrets.yaml):
| Secret / key used | Consumer | ExternalSecret | Vault path | property → key |
|---|---|---|---|---|
broker-rabbitmq-auth / username, password |
computing-broker env, rabbitmq env | broker-rabbitmq-auth |
secret/data/broker |
RABBITMQ_USERNAME→username, RABBITMQ_PASSWORD→password |
broker-postgresql-auth / database_url (broker), database, username, password (postgres) |
env | broker-postgresql-auth |
secret/data/broker |
POSTGRESQL_USERNAME, POSTGRESQL_PASSWORD, POSTGRESQL_DATABASE → username, password, database, templated database_url |
broker-oauth-exchange / SKA_OAUTH_EXCHANGE_CLIENT_ID, SKA_OAUTH_EXCHANGE_CLIENT_SECRET |
computing-broker env | broker-oauth-exchange |
secret/data/iam/clients |
SKA_SRC_EF_CREDMON_CLIENT_ID, SKA_SRC_EF_CREDMON_CLIENT_SECRET (same key names) |
htcondor-broker-idtoken |
volume /etc/condor/tokens.d |
htcondor-broker-idtoken |
secret/data/htcondor |
READ_IDTOKEN → broker.token |
htcondor-broker-submit-idtoken |
volume /etc/condor/submit-tokens.d |
htcondor-broker-submit-idtoken |
secret/data/htcondor |
SUBMIT_IDTOKEN → broker-submit.token |
ska-local-ca-secret / ca.crt |
broker volume (truststore-patch.yaml), smoke test volume |
ska-local-ca-secret |
secret/data/cert-manager |
CA_CRT → ca.crt |
broker-test-token / token |
smoke-test job env | not found: no ExternalSecret or manifest creates it in srv-cd apps/broker/overlays/dev, stack bin/, include/, lib/ (only consumed by broker-smoke-test-cronjob.yaml) |
– | – |
Env of note (dump): BROKER_COLLECTOR_HOST=htcondor-cm.htcondor.svc.cluster.local, BROKER_CONDOR_SCHEDD_NAME=htcondor-submit@htcondor-submit.htcondor.svc.cluster.local, BROKER_AAPI_URL=http://core.aapi.svc.cluster.local:8080, BROKER_DATA_MANAGEMENT_API_URL=http://core.dmapi.svc.cluster.local:8080/v1, BROKER_SITE_CAPABILITIES_URL=http://core.scapi.svc.cluster.local:8080/v1, BROKER_PERMISSIONS_API_URL=https://papi.test/api/v1, SKA_OAUTH_EXCHANGE_TOKEN_URL=https://iam.test/token, SENTRY_DSN (inline), authzEnabled: false in dev.
Scheduling: nodeSelector/affinity none, default tolerations.
Workloads (cluster-dump/slurm/; StatefulSets and NodeSets from pod ownerReferences; chart oci://ghcr.io/slinkyproject/charts/slurm 1.0.2 via srv-cd apps/slurm/overlays/dev/kustomization.yaml):
| Workload | Kind | Replicas (pods) | Containers | Image | CPU req / lim | Mem req / lim |
|---|---|---|---|---|---|---|
| slurm-controller | StatefulSet | 1 | init reconfigure (slurmctld image), init logfile (alpine:latest); slurmctld |
ghcr.io/slinkyproject/slurmctld:25.11-ubuntu24.04 (sha256:d97e4652…) |
1 / 2 | 1Gi / 2Gi |
| slurm-accounting | StatefulSet | 1 | slurmdbd |
ghcr.io/slinkyproject/slurmdbd:25.11-ubuntu24.04 (sha256:2e0107d2…) |
none set | none set |
| slurm-restapi | Deployment | 1 (1) | slurmrestd |
ghcr.io/slinkyproject/slurmrestd:25.11-ubuntu24.04 (sha256:0b56b88b…) |
250m / 500m | 256Mi / 512Mi |
| slurm-login-slinky | Deployment | 1 (1) | init initconf (alpine:latest); login |
ghcr.io/slinkyproject/login:25.11-ubuntu24.04 (sha256:703aafd0…) |
none set | none set |
| mariadb | Deployment | 1 (1) | mariadb |
mariadb:10.11 (sha256:7db29378…) |
none set | none set |
| slurm-worker-compute | NodeSet (slinky.slurm.net) | 2 (-0, -1) |
init logfile alpine:latest, init install-apptainer ubuntu:24.04 (runAsUser: 0); slurmd |
ghcr.io/slinkyproject/slurmd:25.11-ubuntu24.04 (sha256:64aedef5…) |
2 / 16 | 4Gi / 16Gi |
| slurm-worker-compute2 | NodeSet | 1 (-0) |
same | same | 2 / 16 | 4Gi / 16Gi |
Values: srv-cd apps/slurm/overlays/dev/values.yaml (controller 1/1Gi–2/2Gi, restapi 250m/256Mi–500m/512Mi, nodesets.compute.replicas: 2, compute2.replicas: 1, slurmd 2/4Gi–16/16Gi, useResourceLimits: true), base/values.yaml, overlays/dev/mariadb.yaml.
Security context (dump):
| Pod | Pod-level | Container-level |
|---|---|---|
| slurm-controller-0 | runAsUser/runAsGroup/fsGroup 401, runAsNonRoot: true |
slurmctld: runAsUser 401, runAsGroup 401, runAsNonRoot true |
| slurm-accounting-0 | same (401) | slurmdbd: same |
| slurm-restapi | runAsUser/runAsGroup/fsGroup 65534, runAsNonRoot: true |
same (65534) |
| slurm-login-slinky | {} |
login: privileged: false |
| mariadb | {} |
none set |
| slurm-worker-compute*-N | {} |
slurmd: privileged: true, capabilities.add: [BPF, NET_ADMIN, SYS_ADMIN, SYS_NICE]; init install-apptainer runAsUser: 0 |
hostNetwork/hostPID false everywhere; no hostPath volumes. nodeSelector kubernetes.io/os: linux on all chart pods (mariadb none); affinity {}; workers additionally tolerate nodeset.slinky.slurm.net/worker (Equal, NoExecute). ServiceAccount default for all.
Ports / Services (services.json):
| containerPort | Service | Type | Port → target |
|---|---|---|---|
| slurmctld 6817 | slurm-controller |
ClusterIP | slurmctld 6817 → slurmctld TCP |
| slurmdbd 6819 | slurm-accounting |
ClusterIP | slurmdbd 6819 → slurmdbd TCP |
| slurmrestd 6820 | slurm-restapi |
ClusterIP | slurmrestd 6820 → slurmrestd TCP; Ingress slurmrestd.test → 6820 (overlays/dev/ingress.yaml) |
| login 22 | slurm-login-slinky |
LoadBalancer (nodePort 31616) | login 22 → login TCP |
| slurmd 6818 | slurm-workers-slurm |
ClusterIP headless (None) | slurmd 6818 → slurmd TCP |
| mysql 3306 | mariadb |
ClusterIP | 3306 → 3306 TCP |
PVCs (dump pod claims; specs from overlay):
| PVC | Pod / mount | Spec | PV |
|---|---|---|---|
statesave-slurm-controller-0 |
slurmctld /var/spool/slurmctld/ska-src-dev |
5Gi, RWO, storageClassName: local-path (overlays/dev/values.yaml controller.persistence) |
dynamic, not in dump |
mariadb-data |
mariadb /var/lib/mysql |
5Gi, RWO, no storageClassName (cluster default) (overlays/dev/mariadb.yaml) |
dynamic, not in dump |
site-scratch-1, site-scratch-2 |
compute / compute2 /srv/storage |
clones of storm1/site-scratch, storm2/site-scratch (10Gi RWX hostpath) |
storm1-site-scratch-pv-slurm, storm2-site-scratch-pv-slurm, hostPath /srv/storage/site-scratch-{1,2}, Retain (overlays/dev/deploy/pre/provision-shared-storage/create-shared-pvc-job.yaml) |
storm1-sa, storm2-sa |
compute /srv/storage/storm1 ro, compute2 /srv/storage/storm2 ro |
clones of storm{1,2}/storage-area |
storm{1,2}-datasets-pv-slurm, hostPath /srv/storage/storm{1,2}, Retain |
Secrets / ExternalSecrets (srv-cd apps/slurm/overlays/dev/externalsecrets.yaml, base/values.yaml):
| Secret | Use | ExternalSecret | Vault path | property → key |
|---|---|---|---|---|
slurm-munge |
slurmKeyRef (auth key) |
slurm-munge |
secret/data/slurm |
MUNGE_KEY → munge.key |
slurm-jwt-vault |
jwtHs256KeyRef (controller JWT key) |
slurm-jwt-vault |
secret/data/slurm |
JWT_HS256_KEY → jwt_hs256.key |
mariadb-password / password, rootPassword |
mariadb env, slurmdbd storageConfig.passwordKeyRef |
mariadb-password |
secret/data/slurm |
MARIADB_PASSWORD → password, MARIADB_ROOT_PASSWORD → rootPassword |
slurm-auth-jwths256, slurm-auth-slurm, slurm-login-slinky-sssd-conf |
chart-rendered | – (CI log lines 2641-2643) | – | – |
Other mounts (chart-managed projected/config volumes): slurm-etc, ssh-host-keys, ssh-config, sssd-conf; ConfigMaps install-apptainer-script (from overlays/dev/scripts/install-apptainer.sh), slurm-subuid.
| Item | Value | Source |
|---|---|---|
| Deployments | slurm-operator 1 (1), slurm-operator-webhook 1 (1) |
cluster-dump/slinky/deployments.json |
| Images | ghcr.io/slinkyproject/slurm-operator:1.0.2 (sha256:7db52099…), ghcr.io/slinkyproject/slurm-operator-webhook:1.0.2 (sha256:d33c3234…) |
pods.json |
| Resources | none set (both) | dump |
| securityContext | pod {}; containers none set |
dump |
| containerPorts | none declared | dump |
| Services | slurm-operator ClusterIP headless: metrics 8080→8080, health 8081→8081 TCP; slurm-operator-webhook ClusterIP: https 443→9443, health 8081→8081 TCP |
services.json |
| Volumes | webhook: Secret slurm-operator-webhook-ca (cert-manager, certManager.enabled: true in srv-cd apps/slurm-operator/base/values.yaml) |
dump |
| PVC / ExternalSecret | none | – |
| ServiceAccounts | slurm-operator, slurm-operator-webhook |
dump |
| Scheduling | nodeSelector/affinity none, default tolerations | dump |
- Not a service in
etc/env/dev/configuration.yamlatff66cd3e: it is in neither the 29 enabled nor the 19 disabled entries (list in Part D). The only Cavern-related items in the stack are the dpapi pre-deploy hook writing Vault keyCAVERN_API_KEYundersecret/cavern(stack include/env/dev/services/dpapi.mk) and the dpapi dev overlay'sdpapi-cavern-pv/dpapi-cavern-pvc+dpapi-cavern-bootstrap-job(srv-cd apps/dpapi/overlays/dev/dpapi-cavern-storage.yaml,dpapi-cavern-bootstrap-job.yaml). Noskaha*/canfar/cavernnamespace exists in the dump. srv-cdhas an unrelatedapps/canfarapp (prod/dev/staging overlays, ArgoCD application only underargocd-apps/canfar/prod/application.yaml) that deploys Cavern as part of the CANFAR/skaha platform into namespaceskaha-system. It is not referenced by the stack configuration and was not deployed in the CI run.
From the cavern-deployment repo (svc/ska-src-dm-cavern-deployment @ c6cfbefa): it is a docker-compose deployment. There is no Helm chart (etc/helm/values.yaml is a 0-byte file; no Chart.yaml, no templates/*.yaml; templates/ holds VOSpace XML samples; README offers "Kubernetes Deployment" prerequisites but no manifests). From docker-compose.yml and infra/haproxy/config/haproxy.cfg:
| Service | Image | Ports | User / securityContext | Storage | Secrets | Resources / replicas |
|---|---|---|---|---|---|---|
| haproxy | haproxytech/haproxy-debian:latest |
host 8442 → 443 (bind *:443 ssl); routes /reg, /src/cavern, /src/posix-mapper |
root:root |
./certificates/haproxy-host.pem (TLS key, bind-mounted) |
TLS cert file | none set (compose has no limits/replicas) |
| reg | images.opencadc.org/core/reg:1.2.0 |
8080 (internal) | tomcat:tomcat |
config bind mounts | – | none set |
| posix-mapper-db | postgres:14 |
5432 (internal) | default | named volume posix-mapper-db-data |
.env: DB_USERNAME, DB_PASSWORD, DB_NAME |
none set |
| posix-mapper | images.opencadc.org/platform/posix-mapper:0.3.3 |
host 8098 → 8080 | root:root |
config bind mounts | – | none set |
| cavern-db | postgres:14 |
5432 (internal) | default | named volume cavern-db-data |
.env: NODE_DB_USERNAME, NODE_DB_PASSWORD, NODE_DB_NAME |
none set |
| cavern-core | images.opencadc.org/platform/cavern:0.9.2 |
8080 (internal, via haproxy) | root:root |
named volume cavern-storage at /data; config bind mounts |
– | none set |
No CPU/memory limits, no replica counts, no Kubernetes securityContext, no PVC/PV exist in that repo (compose only). For completeness, the Kubernetes values in srv-cd apps/canfar/base/values-cavern.yaml (chart cavern 0.7.1 from https://images.opencadc.org/chartrepo/platform) are: image images.opencadc.org/platform/cavern:0.9.1, replicaCount: 1, resources requests = limits 500m / 4Gi, UWS PostgreSQL install: true (runUID 999), storage PVC skaha-pvc (1000Gi RWX csi-manila-cephfs-sc-retain, base/skaha-pvc.yaml), postgresdb-pvc 4Gi RWO; posix-mapper 0.3.1 at 500m/1Gi (values-posixmapper.yaml); dev overlay only changes hostnames to canfar.test. No securityContext or ports are set in those values files.
Only the mock exists. Across all sources the only BATTLE implementation is ska-telescope/src/src-ef/ska-src-ef-mock-battle-api (submodule svc/ska-src-ef-mock-battle-api, chart charts/mock-battle-api, overlays srv-cd apps/mock-battle-api/overlays/{dev,staging/stfc}). The stack configuration lists only mock-battle-api-1 and mock-battle-api-2; gateways point pilots at BATTLE_API_BASE=http://mock-battle-api-N.mock-battle-api-N.svc.cluster.local:8765; .gitmodules has no other battle repo. A grep for battle excluding mock over srv-cd, stack etc/, stack .gitmodules and the component READMEs returns only comments/docs describing the BATTLE API contract (e.g. gw README.md lines 62, 102; broker overlays/dev/values.yaml line 88). No non-mock BATTLE chart, image or repo URL was found.
| Aspect | Value | Source |
|---|---|---|
| Where the pod template is built | In code: gw src/ska_src_ef_job_gateway/clients/kubernetes_provisioner.py::build_job_manifest (returns a batch/v1 Job: suspend: true when a Kueue queue is set, backoffLimit: 0, ttlSecondsAfterFinished: 300, restartPolicy: Never, labels app.kubernetes.io/name=htcondor-pilot, app.kubernetes.io/component=pilot-dynamic, kueue.x-k8s.io/queue-name=pilots). Not in a ConfigMap. |
code |
ConfigMap htcondor-pilot-config |
Chart template gw charts/job-gateway/templates/pilot-configmap.yaml, rendered into each gateway namespace (CI log line 4346 configmap/htcondor-pilot-config), mounted at /root/config/99-pilot.conf. Content: PILOT_SITE = "<siteId>", PILOT_RESOURCE = "kubernetes", STARTD_ATTRS = $(STARTD_ATTRS) PILOT_SITE PILOT_RESOURCE, STARTD_NOCLAIM_SHUTDOWN = 300, SEC_CREDENTIAL_REFRESH = 60, STARTER_UPDATE_INTERVAL = 60, one static slot (NUM_SLOTS = 1, SLOT_TYPE_1 = cpus=100%,mem=100%,disk=100%), PILOT_USER = ska-pilot, SLOT1_USER = $(PILOT_USER), STARTER_ALLOW_RUNAS_OWNER = FALSE. |
chart |
| Image | harbor.test/library/htcondor-pilot:dev (PILOT_IMAGE env; built by the stack from condor images/pilot/ per stack etc/env/dev/services/htcondor.yaml; Dockerfile: FROM ubuntu:22.04, condor from htcss-downloads.chtc.wisc.edu 24.x repo, singularity binary copied from quay.io/ucsc_cgl/toil:9.3.0); imagePullPolicy: Always. Same image for Slurm pilots (SLURM_PILOT_IMAGE). |
dump env; Dockerfile |
| CPU / memory | requests == limits: cpu: <cpus>, memory: <MiB>Mi, sized from the matched task's Request ads, else defaults PILOT_DEFAULT_CPUS=2 / PILOT_DEFAULT_MEMORY_MB=4096 (dump env; chart values pilotDefaultCpus, pilotDefaultMemoryMb). Observed at dump time: 1 CPU/512Mi, 2 CPU/512Mi, 1 CPU/256Mi. Init container: no resources. |
code lines 162-173; dump |
| GPU | nvidia.com/gpu is added to both requests and limits only when the planned size has gpus > 0 (GPU_RESOURCE = "nvidia.com/gpu", lines 50, 171-173). No pilot in the dump requests it. |
code |
| securityContext | startd: {"privileged": true} (unconditional). Init fix-storage-perms: privileged: true only in hostPath mode; none in PVC mode (dev). No pod-level securityContext. |
code lines 258-286; dump |
| Volumes | Secret htcondor-pilot-idtoken (defaultMode 0400, subPath idtoken → /root/secrets/idtoken), ConfigMap htcondor-pilot-config (subPath 99-pilot.conf), PVCs from STORAGE_PVCS=site-scratch:/srv/storage,stormN-sa:/srv/storage/stormN:ro (sorted by mountPath). hostPath /srv/storage only when STORAGE_PVCS is empty (not the case in dev). |
code; dump |
| nodeSelector / tolerations / affinity | none set (manifest has none; only kubelet default tolerations in the live pod) | code; dump |
| ServiceAccount | default in the pilot namespace (none set in manifest) |
dump |
| Env | CONDOR_HOST, CONDOR_SERVICE_HOST, _CONDOR_TRUST_DOMAIN=condor, _CONDOR_STARTD_NOCLAIM_SHUTDOWN=300, SKA_PAYLOAD_IDENTITY=pool, SKA_PVC_ROOT=/srv/storage/pilot-scratch-N, NUM_CPUS, MEMORY, POD_NAMESPACE (downward API), plus PILOT_EXTRA_ENV (BATTLE_API_BASE=…, SKA_STAGEIN_TIMEOUT_S=60) |
dump |
| Condor config inside the pilot | condor images/pilot/pilot-entrypoint.sh → /etc/condor/config.d/01-pilot.conf: DAEMON_LIST = MASTER STARTD, USE_SHARED_PORT = TRUE, IDTOKENS only, encryption+integrity REQUIRED, START = TRUE (in-pilot model), job hooks start_payload.sh / finish_payload.sh, no CCB settings (see A.5). |
entrypoint |
| Slurm pilots | Same image run through sbatch + apptainer by gw src/.../clients/slurm_provisioner.py as user ska-pilot (SLURM_PILOT_USER), scratch /srv/storage/pilot-scratch-N, partition compute / compute2; capacity fallback 32 CPU/32768 MB (site 1) and 16 CPU/16384 MB (site 2). |
dump env |
ClusterQueues (srv-cd apps/kueue/overlays/dev/deploy/post/configure/clusterqueue.yaml, applied by the kueue deploy.post step; identical spec for both):
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: pilot-pool-storm1 # and: pilot-pool-storm2
spec:
namespaceSelector: {}
resourceGroups:
- coveredResources: ["cpu", "memory"]
flavors:
- name: default
resources:
- name: cpu
nominalQuota: "20"
- name: memory
nominalQuota: "40Gi"No cohort (no borrowing between sites). The gateway logs the quota at startup: capacity[kubernetes]: 20 cpus / 40960 MiB (kueue-quota) (cluster-dump/job-gateway-1/job-gateway-1-788dddbb56-p4mr9/logs.txt).
ResourceFlavor (.../configure/resourceflavor.yaml; CI log line 1736):
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: default
# no nodeLabels / taints / tolerations: matches any nodeLocalQueues (chart template gw charts/job-gateway/templates/localqueue.yaml, rendered because gateway.kueueLocalQueue: pilots; CI log line 4352 localqueue.kueue.x-k8s.io/pilots for job-gateway-1 and the equivalent for job-gateway-2):
| Namespace | LocalQueue | spec.clusterQueue |
Evidence in dump |
|---|---|---|---|
| job-gateway-1 | pilots |
pilot-pool-storm1 |
pilot pods labelled kueue.x-k8s.io/local-queue-name=pilots, kueue.x-k8s.io/cluster-queue-name=pilot-pool-storm1, annotation kueue.x-k8s.io/workload=job-pilot-dynamic-… |
| job-gateway-2 | pilots |
pilot-pool-storm2 |
same labels with pilot-pool-storm2 |
Kueue operator config (cached chart values.yaml managerConfig): manageJobsWithoutQueueName left at default false; integrations include batch/job and pod; waitForPodsReady not set.
RBAC objects are not in the dump; rules below are from the chart/overlay templates, and the CI log confirms each named object was applied (log lines 4341-4345, 4355-4357, 4335-4337, 4573-4575, 2634-2635, 4209-4211). job-gateway-N is the Helm fullname (release name contains the chart name, _helpers.tpl).
Role job-gateway-1-pilot / job-gateway-2-pilot (namespace job-gateway-N; gw charts/job-gateway/templates/pilot-role.yaml), bound by RoleBinding job-gateway-N-pilot to ServiceAccount job-gateway-N (pilot-rolebinding.yaml):
| apiGroups | resources | verbs |
|---|---|---|
batch |
jobs |
create, delete, get, list, watch |
"" |
pods |
create, delete, get, list |
"" |
configmaps |
get |
kueue.x-k8s.io |
workloads |
create, delete, get, list, watch |
kueue.x-k8s.io |
localqueues |
get |
ClusterRole job-gateway-N-kueue-quota (kueue-quota-clusterrole.yaml), bound by ClusterRoleBinding job-gateway-N-kueue-quota to SA job-gateway-N (namespace job-gateway-N):
| apiGroups | resources | verbs |
|---|---|---|
kueue.x-k8s.io |
clusterqueues |
get |
Other Roles/ClusterRoles that allow creating batch/jobs or pods, or acting on them:
| Object | Rules (jobs/pods only) | Bound to | Source |
|---|---|---|---|
ClusterRole slurm-operator |
core configmaps, events, persistentvolumeclaims, pods, secrets, services: create, delete, get, list, patch, update, watch; pods/status: get, patch, update |
SA slurm-operator (ns slinky) via ClusterRoleBinding |
slinky-chart templates/operator/rbac.yaml; CI log line 1338 |
ClusterRole kueue-manager-role |
core pods: delete, get, list, patch, update, watch (no create); pods/finalizers get, update; pods/status get, patch; batch jobs: delete, get, list, patch, update, watch (no create); jobs/finalizers, jobs/status: get, patch, update |
SA kueue-controller-manager (ns kueue-system) via ClusterRoleBinding kueue-manager-rolebinding |
kueue-chart templates/rbac/role.yaml, role_binding.yaml; CI log lines 1687, 1721 |
Role job-gateway-N-payload-pod-pilot (+ SA pilot-payload-pod) |
pods, pods/log: get, list, create, patch, delete; workloads get, list | not rendered in dev: only when payloadExecution=payload-pod; dev runs PAYLOAD_EXECUTION=in-pilot (dump env) and the CI log shows no such object |
gw charts/job-gateway/templates/payload-pod-pilot-role.yaml |
ClusterRoles job-gateway-N-shared-pvc-creator, mock-battle-api-N-shared-pvc-creator, broker-shared-pvc-creator, slurm-shared-pvc-creator |
persistentvolumes: get, list, create, patch; persistentvolumeclaims: get, list, create (no pods/jobs) |
SAs of the same name in their namespace (pre-deploy clone jobs) | .../deploy/pre/provision-shared-storage/rbac.yaml in each overlay |
Roles job-gateway-N-wait-for-secrets, broker-wait-for-idtoken, slurm-wait-for-cluster |
externalsecrets.external-secrets.io: get, list, watch (wait-for-cluster not read) |
SAs of the same name | overlay deploy/post/.../rbac.yaml |
htcondor SA htcondor, broker/mock-battle/slurm pods SA default |
no Role/ClusterRole in the charts or overlays | – | condor charts/htcondor/templates/ (only serviceaccount.yaml), broker charts/computing-broker/templates/, mba charts/mock-battle-api/templates/ |
- There is no Dask operator, DaskCluster CRD or Kubernetes-native Dask object in any source. Dask runs entirely inside HTCondor.
- The broker validates
--daskrequests (broker src/ska_src_ef_computing_broker/services/payload.py::validate_dask_parameters: requires--imageand--cmd,--dask-workerspositive integer) and submits a run leader to the schedd in the HTCondor local universe (broker src/.../clients/condor_submit_client.py:universe = local,request_cpus = 1,request_memory = 128, runs on thehtcondor-submitpod). - The leader (
broker etc/leader/ska_run_leader.py, staged to the schedd by the broker) creates the Dask roles as separate HTCondor vanilla jobs on pilots: first the scheduler (_dask_scheduler_spec:exec dask scheduler --scheduler-file /srcnet/work/scheduler.json, ClassAd+SkaChirpScheduler = true, address published back viacondor_chirpfromcondor images/pilot/hooks/handoff.sh/etc/leader/ska_task_exec.py), then N workers (_dask_role_specs:exec dask worker <address> --nthreads <--dask-worker-cpu, default 1> --memory-limit <--dask-worker-memory, default 2048>MiB) and the client (user--cmd). All roles use the one job--image. - Resources per role job:
request_cpus = {cpu},request_memory = {memory}, optionalrequest_gpus = {gpu}when--gpu > 0,request_diskwhen set;requirements = (TARGET.PILOT_RESOURCE =?= "<kubernetes|slurm>") && site clause(ska_run_leader.pylines ~405-430). Each matched job runs in a pilot pod the gateway sizes from those ads (C.1), so the Kubernetes-side resources are the pilot pod'srequests == limits. - ServiceAccount: the pilot pods executing scheduler/workers/client use the
defaultServiceAccount ofjob-gateway-1/job-gateway-2(dump); on Slurm they run as OS userska-pilot. The leader itself runs as the schedd'ssubmituseronhtcondor-submit(chartcondor_mapfile), SAhtcondor. - Capacity note from
srv-cd apps/kueue/overlays/dev/deploy/post/configure/clusterqueue.yaml: the 20 CPU / 40Gi quota is sized so "scheduler + N workers + client pilots" can be admitted concurrently. The broker smoke test uses imageharbor.test/library/dask-base:dev(built frombroker images/dask-base/Dockerfile,stack etc/env/dev/services/broker.yaml).
| NodeSet | Replicas | slurmd requests / limits | Advertised to Slurm | Partition | Source |
|---|---|---|---|---|---|
slurm-worker-compute |
2 (pods slurm-worker-compute-0, -1 in dump) |
2 CPU / 4Gi → 16 CPU / 16Gi | 16 CPU / 16Gi per node (useResourceLimits: true) |
compute (Default YES) |
srv-cd apps/slurm/overlays/dev/values.yaml nodesets.compute; CI log line 2654 |
slurm-worker-compute2 |
1 (pod slurm-worker-compute2-0) |
2 CPU / 4Gi → 16 CPU / 16Gi | 16 CPU / 16Gi | compute2 |
same file nodesets.compute2; CI log line 2655 |
Both: scalingMode: StatefulSet, workloadDisruptionProtection: false, init container installs Apptainer into an emptyDir, env APPTAINER_CACHEDIR=/srv/storage/apptainer-cache, APPTAINER_NOHTTPS=1; shared batch partition disabled.
- Dump: none.
grep -rE 'nvidia\.com|amd\.com/gpu|habana'overcluster-dump/returns no hits; no pod requests any extended resource; the only non-standard*.io/*keys are Elastic annotations inelastic-system/monitoring. Node capacity lists only cpu, memory, ephemeral-storage, hugepages, pods. - Sources:
nvidia.com/gpuappears only as a conditional in code:gw kubernetes_provisioner.py(GPU_RESOURCE, added when a task requests GPUs),condor images/pilot/hooks/handoff.sh(PAYLOAD_POD_GPUS, payload-pod model only),broker etc/leader/ska_run_leader.py(request_gpuswhen--gpu > 0).srv-cd apps/kueue/overlays/dev/deploy/post/configure/resourceflavor.yamlcomments that a GPU flavor "can be added when GPU pilots land"; no ClusterQueue covers a GPU resource.
enabled is deploy.enabled in each etc/env/dev/services/<name>.yaml (the list file only names them).
Enabled (29): cert-manager, vault, external-secrets, ingress-nginx, harbor, eck-operator, monitoring, indigo-iam, papi, aapi, scapi, storm1, storm2, fts, rucio, ingestor, gatekeeper, dpapi, dmapi, caom, slurm-operator, slurm, broker, job-gateway-1, job-gateway-2, mock-battle-api-1, mock-battle-api-2, kueue, htcondor.
Disabled (19): headlamp, ums, rucio-task-manager, datalink, product-streamer, soda, sdapi, sreg, jupyterhub, oper-cli, oper-gui, geapi, panda, harvester, lur, aqs, waldur, iam-waldur-sync, panda-usage-reporter.
Group ef ("Execution Framework: broker, HTCondor pool, pilot gateways, and the Slurm and Kueue pilot pools") is declared on broker, htcondor, job-gateway-1/2, kueue, slurm, slurm-operator, mock-battle-api-1/2.
None. The dump contains no NetworkPolicy text (and cluster-info dump would not include that kind anyway). The CI deploy log has no networkpolicy apply line. No kind: NetworkPolicy exists anywhere under srv-cd apps/, argocd-apps/, flux-apps/, resources/ at 146b9312.
D.3 External hostnames (not *.test, not *.svc) in ConfigMaps or container env of the Part A namespaces
Container env (from cluster-dump/<ns>/pods.json; ConfigMaps are not in the dump):
| Hostname | Component | Where |
|---|---|---|
o4507377535287296.ingest.de.sentry.io |
broker (broker-computing-broker, env SENTRY_DSN), job-gateway-1 and job-gateway-2 (env SENTRY_DSN) |
values: srv-cd apps/broker/overlays/dev/values.yaml broker.sentryDsn, apps/job-gateway/overlays/dev/job-gateway-{1,2}/values.yaml gateway.sentryDsn (DSN key not reproduced) |
ConfigMap-sourced content (from the overlay files that generate the ConfigMaps):
| Hostname | ConfigMap / component | Where |
|---|---|---|
github.com (https://github.com/apptainer/apptainer/releases/download/...) |
install-apptainer-script in namespace slurm (init container of both NodeSets) |
srv-cd apps/slurm/overlays/dev/scripts/install-apptainer.sh line 14 |
projects.pawsey.org.au (https://projects.pawsey.org.au/srcnet/...) |
broker-test-files (smoke-test suite) in namespace broker |
broker tests/integration/ (test data URLs) |
Not env/ConfigMap but external pull sources of the same namespaces (image and chart references): docker.io (htcondor/*, postgres, rabbitmq, mariadb, alpine, busybox, ubuntu), ghcr.io (slinkyproject images and charts), registry.k8s.io (kueue image and OCI chart), helm.goharbor.io (harbor chart), quay.io and htcss-downloads.chtc.wisc.edu (pilot image build only), registry.gitlab.com (base-values image repos, overridden to harbor.test in dev), harbor.srcnet.skao.int (harbor base values, overridden to harbor.test in dev). No other external hostname was found in the dev overlays or chart values of job-gateway, kueue, harbor, htcondor, broker, mock-battle-api, slurm, slurm-operator.
Overlay directories at 146b9312 (srv-cd apps/<app>/overlays/): job-gateway staging/stfc/{job-gateway-1,job-gateway-2}; kueue staging/stfc; harbor staging/stfc and prod; htcondor staging/stfc; broker staging/stfc; (also mock-battle-api staging/stfc/{1,2}, slurm staging/stfc, slurm-operator staging/stfc). Staging overlays do not inflate the chart themselves ("published chart is a separate Argo source").
| App / overlay | Replicas | Resources | Storage | Other differences from dev |
|---|---|---|---|---|
| job-gateway staging/stfc (1 and 2) | gateway 1 (unchanged); desiredPilots: 2 (dev 1); slurm desiredPilots: 1 |
unchanged (chart defaults 100m/128Mi–500m/512Mi) | same claims site-scratch, storm{1,2}-sa cloned from the storm staging overlays (not read); slurm.storageHostPath: /srv/storage |
images registry.gitlab.com/ska-telescope/src/src-ef/ska-src-ef-local-job-gateway/job-gateway:0.4.1 and pilot .../ska-src-ef-htcondor-deployment/pilot:0.4.1; payloadIdentity: matched on gateway-1 (pool on gateway-2); pilotExtraEnv adds _CONDOR_STARTER_UPDATE_INTERVAL=60,_CONDOR_SEC_CREDENTIAL_REFRESH=60; ingress disabled; CA secret ca-secret (ExternalSecret cert-ca); register job image harbor.staging.srcnet.skao.int/library/integration:latest, URLs iam.staging.srcnet.skao.int, site-capabilities.staging.srcnet.skao.int |
| kueue staging/stfc | 1 | manager 100m/256Mi–1/512Mi (same) | none | ClusterQueues pilot-pool-storm1/2 identical (20 CPU / 40Gi each, no cohort), ResourceFlavor default identical; no kustomization.yaml and no deploy/pre/visibility-rbac |
| harbor staging/stfc | 1 each (same) | same (only trivy set) | same sizes (registry 100Gi, jobservice 1Gi, database 1Gi, redis 1Gi, trivy 5Gi) but storageClass: csi-manila-cephfs-sc-retain for all five (dev local-path) |
host harbor.staging.srcnet.skao.int; same harbor ExternalSecret |
| harbor prod | 1 each | same | base values: csi-cinder-sc-retain, same sizes |
host harbor.srcnet.skao.int; fullnameOverride: harbor; deletes chart Secret harbor-database; core envFrom Secret harbor; no ExternalSecret file in the overlay |
| htcondor staging/stfc | cm 1, submit 1, execute 2 (same) | same (cm 100m/256Mi–500m/1Gi, execute 250m/512Mi–2/2Gi, submit 100m/256Mi–500m/1Gi) | shared-scratch created by the chart (scratchPvc.create: true) 100Gi RWX csi-manila-cephfs-sc-retain (dev: 10Gi hostPath PV); submit state PVC 5Gi csi-manila-cephfs-sc-retain (dev: default class) |
no ca-trust-patch.yaml, no ska-local-ca-secret ExternalSecret; same pilot-sites.txt |
| broker staging/stfc | 3 (same) | same | postgres 10Gi csi-manila-cephfs-sc-retain (dev local-path); clones shared-scratch, ingest-area as in dev |
image .../computing-broker:0.4.1; authzEnabled: true; staging URLs (permissions., authn., data-management., site-capabilities.staging.srcnet.skao.int, iam.staging.srcnet.skao.int); host broker.staging.srcnet.skao.int with shared ingress TLS; extra ExternalSecrets cert-ca → ca-secret (secret/data/cert-manager CA_CRT) and broker-test-user (secret/data/iam TEST_USER1_PASSWORD); no truststore patch; smoke test serial (-n 0) |
| mock-battle-api staging/stfc | 1 | same | same clones | image harbor.staging.srcnet.skao.int/library/mock-battle-api:latest; ingress disabled; RSE names NODE1_SITE1_STORM1, NODE1_SITE1_SCRATCH, storm URLs storm1.staging.srcnet.skao.int |
| slurm staging/stfc | NodeSets compute 2, compute2 1 (same); restapi 1 | same | controller statesave 10Gi csi-manila-cephfs-sc-retain (dev 5Gi local-path); mariadb-data unchanged |
clusterName: ska-src-staging-stfc; no APPTAINER_NOHTTPS; no slurmrestd Ingress; wait-for-cluster timeouts 600s |
| slurm-operator staging/stfc | 1 / 1 | none set | none | operator.logLevel: info (dev debug) |
| Item | Status |
|---|---|
| Artifacts of job 17048104007 | job still running at 2026-10-09 03:23 UTC; /jobs/17048104007/artifacts returned 404. Used job 17034588345. |
| StatefulSet / NodeSet / PVC / PV / ConfigMap / Secret / ExternalSecret / RBAC / Kueue / Ingress objects in the dump | not present in cluster-info dump output; reconstructed from repo sources and the CI apply log as cited. |
PV reclaim policy for dynamically provisioned local-path claims (harbor ×5, htcondor-submit, broker-postgresql, statesave-slurm-controller-0, mariadb-data) |
not found: no PV objects in the dump and the provisioner's policy is not declared in the sources. |
Creator of Secret broker-test-token |
not found in srv-cd apps/broker/overlays/dev, stack bin/, include/, lib/. |
KUEUE_CHART_VERSION value |
lives in .env.dev (not in the repo); the cached chart on disk is 0.18.1 and the running image is v0.18.1. |
Content of the stock HTCondor images' 01-ccb.conf |
only its load is visible in htcondor-execute-*/logs.txt; the file is inside docker.io/htcondor/*:lts, not in any source. |
| Pilot daemon logs (to confirm CCB vs inbound at runtime) | written to /srv/storage/pilot-logs/, not captured in the dump; conclusion in A.5 is from the pilot config in pilot-entrypoint.sh. |
| Kubernetes manifests in the cavern-deployment repo | none exist at c6cfbefa (docker-compose only; etc/helm/values.yaml is empty). |
| A real (non-mock) BATTLE service | none referenced in any source. |
| Staging storm overlays (source PVCs for staging clones) | not read; listed as "not read" in D.4. |