Skip to content

Instantly share code, notes, and snippets.

@xianlubird

xianlubird/README.md

Created October 17, 2026 02:44
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save xianlubird/dfd872c257113917774935b80022ac42 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save xianlubird/dfd872c257113917774935b80022ac42 to your computer and use it in GitHub Desktop.
Download ZIP
Dynamo DEP #16623: adjacent operator 1.4.0 to 1.5.0 hash drift evidence (2026-10-17)
Raw
README.md

Adjacent operator 1.4.0 -> 1.5.0 hash-drift evidence

Experiment date: 2026-10-17. Related: ai-dynamo/dynamo#16623 and #16628.

An unchanged DGD with a fixed runtime 1.5 image changes worker hash on the adjacent operator 1.4.0 -> 1.5.0 transition. Both the custom parseable 1.5 tag and the official 1.5.0 image reproduce it; the official 1.4.0 runtime is stable.

Identical-input hash matrix

Fixed runtime Operator 1.2.0 Operator 1.3.0 Operator 1.4.0 Operator 1.5.0
v1.5.0-9aa3-repro 309e9927 309e9927 309e9927 142dbdff
Official 1.5.0 53549949 53549949 53549949 e332eabb
Official 1.4.0 7be83b1f 7be83b1f 7be83b1f 7be83b1f

The only hash-input differences for the two 1.5 cases are the addition of runtimeVersion = "1.5.0" to SglangPrefill and SglangDecode. Normalized inputs independently reproduce each production hash. YAML inputs and decoded DGD objects are equal across the four implementations within each case.

This matrix locates the code change. It does not declare every operator/runtime pair in the diagnostic matrix to be supported.

Same-API-server controller handoff

The experiment uses a Kubernetes 1.34.0 envtest API server and etcd, the released production admission/conversion handlers, and DGD/DCD controller managers.

  1. Install 1.4 CRDs/admission and create the cases normally.
  2. Let 1.4 controllers persist the hash, PCS and observedGeneration.
  3. Stop the 1.4 controller managers, keeping the same API server/etcd alive.
  4. Install 1.5 CRDs/admission against that server and inspect the stored objects.
  5. Record the real 1.5 transition planner decision, start its controllers, and capture their persisted output.

The objects are not reconstructed or loaded into a fake client. The 1.4 webhook server remains alive until parent cleanup; its registrations are replaced by 1.5. OperatorVersion matches the respective operator chart appVersion.

Fixed runtime Stored 1.4 hash Persisted 1.5 hash hashChanged PCS generation
Custom 1.5 e2fff36a 39ae5e0c true 1 -> 2
Official 1.5 90f01928 37070ad2 true 1 -> 2
Official 1.4 3cf136a2 3cf136a2 false 1 -> 1

DGD spec, runtime images, DGD UID and PCS UID stay unchanged in every case. Origin remains 1.4.0. Both worker templates in each 1.5 runtime case gain the new hash label and DYN_NAMESPACE_WORKER_SUFFIX. Their health-check setting changes from false to true and liveness failureThreshold from 1 to 3. The fixed 1.4 runtime preserves the entire PCS spec.

The handoff uses case-specific DGD names and an isolated namespace, hence its hash values differ from the identical-input matrix. Hashing the actual stored DGD with both releases reproduces these values and confirms the same two runtimeVersion input additions. Fresh handoff runs allocate a new namespace.

The observed endpoint is the immediate workload resource persisted by Dynamo. The environment runs API server/etcd and Dynamo admission/controller processes. Absent downstream PodCliques produce scale NotFound messages, handled by the production code as pending. Transient status conflicts use normal retries.

Evidence inventory

  • revisions.json: official GitHub release-tag commits.
  • Three runtime YAML files: exact inputs for the hash matrix.
  • hash-summary.json and normalized-hash-inputs.json: hashes and normalized inputs.
  • api-before.json and api-after.json: complete captured DGD/PCS objects.
  • api-decisions.json and api-summary.json: planner decisions and field differences.
  • hash-logs.txt, api-1.4.log, api-1.5.log: successful experiment execution logs.
  • grove-crd-provenance.json: release-pinned Grove CRD versions and checksums.
  • Go templates and Python runners: the audit harness.
  • SHA256SUMS: checksums of the published files.

Successful reproduction assertions mean the regression was reproduced. The compatibility observation for the two fixed runtime 1.5 cases is a failure.

Local absolute paths in API logs are replaced by REPRO_ROOT, GOMODCACHE and TMP_PATH placeholders. Captured Kubernetes object data and numeric results are unchanged. The portable API runner reads KUBEBUILDER_ASSETS instead of embedding the original machine's binary path. No kubeconfig or credentials are included.

Re-run

Prerequisites: a Dynamo Git clone, Python with tarfile data filtering, Go toolchain go1.26.8, Helm, and envtest Kubernetes 1.34.0 binaries for your platform.

From a directory containing all gist files:

python3 prepare_sources.py --repo /path/to/dynamo
python3 run_hashes.py
export KUBEBUILDER_ASSETS=/path/to/1.34.0-envtest-binaries
python3 run_api.py

The harness adds audit-only test files to exported release source directories. No production hashing, rendering or reconciliation code is modified.

Raw
api-1.4.log
=== RUN TestAdjacentAPI
2026-10-17T10:33:15+08:00 DEBUG controller-runtime.test-env starting control plane
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRDs
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.4.0/deploy/operator/config/crd/bases"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamocheckpoints.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamocomponentdeployments.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamographdeploymentrequests.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamographdeployments.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamographdeploymentscalingadapters.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamomodels.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamoworkermetadatas.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_podsnapshotcontents.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_podsnapshots.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/testing/prometheus"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "prometheusrule.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "servicemonitor.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/testing/inference.networking.k8s.io"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "inferencepools.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/testing/grove.io"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "clustertopologybindings.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_clustertopologybindings.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podcliques.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podcliquescalinggroups.yaml"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podcliquesets.yaml"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.test-env there are more than one CRD definitions with the same <Group, Version, Kind, Name> {"GVKN": {"GVK":{"Group":"apiextensions.k8s.io","Version":"v1","Kind":"CustomResourceDefinition"},"Name":"clustertopologybindings.grove.io"}}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamomodels.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podsnapshotcontents.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podsnapshots.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "prometheusrules.monitoring.coreos.com"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "clustertopologybindings.grove.io"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamocheckpoints.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamocomponentdeployments.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podcliquescalinggroups.grove.io"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "servicemonitors.monitoring.coreos.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamographdeployments.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamoworkermetadatas.nvidia.com"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "inferencepools.inference.networking.k8s.io"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podcliques.grove.io"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podcliquesets.grove.io"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamographdeploymentrequests.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int64"
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamographdeploymentscalingadapters.nvidia.com"}
2026-10-17T10:33:18+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "monitoring.coreos.com/v1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "monitoring.coreos.com/v1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "inference.networking.k8s.io/v1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing webhooks
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing mutating webhook {"webhook": "operatorenv-dynamo-operator-mutating"}
2026-10-17T10:33:18+08:00 DEBUG controller-runtime.test-env installing validating webhook {"webhook": "operatorenv-dynamo-operator-validating"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate/nvidia.com/v1beta1/dynamocomponentdeployments"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1alpha1-dynamocomponentdeployment"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate/nvidia.com/v1beta1/dynamographdeployments"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1alpha1-dynamographdeployment"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1alpha1-dynamocheckpoint"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1alpha1-dynamomodel"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1beta1-dynamographdeploymentrequest"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/convert"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoGraphDeploymentRequest"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoGraphDeployment"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoComponentDeployment"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoGraphDeploymentScalingAdapter"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-nvidia-com-v1beta1-dynamocomponentdeployment"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate/nvidia.com/v1beta1/dynamographdeployments"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-nvidia-com-v1alpha1-dynamographdeployment"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-nvidia-com-v1beta1-dynamographdeploymentrequest"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-core-v1-pod-checkpoint-restore"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Starting webhook server
2026-10-17T10:33:18+08:00 INFO controller-runtime.certwatcher Updated current TLS certificate {"cert": "<TMP_PATH>", "key": "<TMP_PATH>"}
2026-10-17T10:33:18+08:00 INFO controller-runtime.webhook Serving webhook server {"host": "127.0.0.1", "port": 64644}
2026-10-17T10:33:18+08:00 INFO controller-runtime.certwatcher Starting certificate poll+watcher {"cert": "<TMP_PATH>", "key": "<TMP_PATH>", "interval": "10s"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.PodCliqueScalingGroup"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.PodCliqueSet"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1beta1.DynamoComponentDeployment"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.DynamoGraphDeploymentScalingAdapter"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1beta1.DynamoComponentDeployment"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.PodClique"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1.Pod"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1.PersistentVolumeClaim"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.DynamoCheckpoint"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1.Service"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1beta1.DynamoGraphDeployment"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1.Ingress"}
2026-10-17T10:33:19+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1.Deployment"}
zz_adjacent_api_test.go:122: Create custom-1.5 through production 1.4.0 admission
2026-10-17T10:33:19+08:00 INFO controller-runtime.cache Warning: nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter
2026-10-17T10:33:20+08:00 INFO Starting Controller {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment"}
2026-10-17T10:33:20+08:00 INFO Starting workers {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "worker count": 1}
2026-10-17T10:33:20+08:00 INFO Starting Controller {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment"}
2026-10-17T10:33:20+08:00 INFO Starting workers {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "worker count": 1}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-defaulting-webhook stamped operator origin version on DGD {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "e1269183-277e-4ece-82ed-bd3bee8bf525", "name": "adjacent-custom-1-5", "namespace": "operatorenv-zkk2r5v9", "version": "1.4.0"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate create {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "291ca640-53c8-488f-a665-e97247275a73", "name": "adjacent-custom-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Adding finalizer to object {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "resourceVersion": "259"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "e595f5d9-ff70-4a32-bc69-e74695efd3cb", "name": "adjacent-custom-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Finalizer added to object {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "resourceVersion": "260"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "09e4e991-e5e9-46b1-b857-93f19ba7b18c", "name": "adjacent-custom-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new ServiceAccount operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"ServiceAccount","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-k8s-service-discovery","apiVersion":"v1"}, "reason": "CreateServiceAccount"}
2026-10-17T10:33:21+08:00 INFO ServiceAccount created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:21+08:00 DEBUG events Created ServiceAccount operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"ServiceAccount","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-k8s-service-discovery","uid":"52340b8a-8492-4e21-bf14-746710c08183","apiVersion":"v1","resourceVersion":"262"}, "reason": "CreateServiceAccount"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Role operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Role","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-k8s-service-discovery-role","apiVersion":"rbac.authorization.k8s.io/v1"}, "reason": "CreateRole"}
2026-10-17T10:33:21+08:00 INFO Role created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:21+08:00 DEBUG events Created Role operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Role","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-k8s-service-discovery-role","uid":"e04832ba-7e48-4ed5-9264-b2d93862833c","apiVersion":"rbac.authorization.k8s.io/v1","resourceVersion":"265"}, "reason": "CreateRole"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new RoleBinding operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"RoleBinding","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-k8s-service-discovery-binding","apiVersion":"rbac.authorization.k8s.io/v1"}, "reason": "CreateRoleBinding"}
2026-10-17T10:33:21+08:00 INFO RoleBinding created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:21+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
2026-10-17T10:33:21+08:00 DEBUG events Created RoleBinding operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"RoleBinding","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-k8s-service-discovery-binding","uid":"c7b29085-64c9-4380-be95-554a8fc211fb","apiVersion":"rbac.authorization.k8s.io/v1","resourceVersion":"268"}, "reason": "CreateRoleBinding"}
2026-10-17T10:33:21+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "hasMultinode": false, "lwsEnabled": false}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5", "resourceType": "PodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new PodCliqueSet operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"PodCliqueSet","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5","apiVersion":"grove.io/v1alpha1"}, "reason": "CreatePodCliqueSet"}
2026-10-17T10:33:21+08:00 INFO PodCliqueSet created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5", "resourceType": "PodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG events Created PodCliqueSet operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"PodCliqueSet","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5","uid":"40164c23-88ac-41d6-8e74-e9d57798c2bf","apiVersion":"grove.io/v1alpha1","resourceVersion":"272"}, "reason": "CreatePodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
2026-10-17T10:33:21+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:21+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-custom-1-5-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller_common/scale.go:43
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).scaleGroveResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:640
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveScaling
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1018
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1114
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:443
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:289
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:21+08:00 DEBUG Grove resource not found yet, skipping scaling for now - will retry on next reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:21+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-custom-1-5-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller_common/scale.go:43
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).scaleGroveResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:640
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveScaling
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1018
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1114
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:443
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:289
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:21+08:00 DEBUG Grove resource not found yet, skipping scaling for now - will retry on next reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-sglangprefill","apiVersion":"v1"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Service created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Created Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-sglangprefill","uid":"b1412758-c62d-4314-b084-ce5cd38b7591","apiVersion":"v1","resourceVersion":"276"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-sglangdecode","apiVersion":"v1"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Service created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Created Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-custom-1-5-sglangdecode","uid":"ce39cae8-9fc4-4490-87c1-94d52e6d9f3b","apiVersion":"v1","resourceVersion":"280"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
2026-10-17T10:33:21+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
2026-10-17T10:33:21+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "9cb7cc70-91d7-49c7-b887-a3c3817f3801"}
zz_adjacent_api_test.go:169: release=1.4.0 case=custom-1.5 persistedHash=e2fff36a PCS-generation=1
zz_adjacent_api_test.go:122: Create official-1.5 through production 1.4.0 admission
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-defaulting-webhook stamped operator origin version on DGD {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "3299116a-cf1a-42fb-8d88-c87a8fd727d2", "name": "adjacent-official-1-5", "namespace": "operatorenv-zkk2r5v9", "version": "1.4.0"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate create {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "daabe7ff-168f-481c-9eaf-c89b901ed1b9", "name": "adjacent-official-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Adding finalizer to object {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "resourceVersion": "283"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "8fd066db-ee0e-457b-894b-ccaabeb210ca", "name": "adjacent-official-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Finalizer added to object {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "resourceVersion": "284"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "027e8232-0ac3-4dda-ab48-d4cf4e2652e0", "name": "adjacent-official-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new ServiceAccount operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"ServiceAccount","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-k8s-service-discovery","apiVersion":"v1"}, "reason": "CreateServiceAccount"}
2026-10-17T10:33:21+08:00 INFO ServiceAccount created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:21+08:00 DEBUG events Created ServiceAccount operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"ServiceAccount","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-k8s-service-discovery","uid":"1bd6b4d0-e891-46d0-bf08-4bef2abd08d1","apiVersion":"v1","resourceVersion":"287"}, "reason": "CreateServiceAccount"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Role operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Role","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-k8s-service-discovery-role","apiVersion":"rbac.authorization.k8s.io/v1"}, "reason": "CreateRole"}
2026-10-17T10:33:21+08:00 INFO Role created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:21+08:00 DEBUG events Created Role operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Role","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-k8s-service-discovery-role","uid":"1c2460c1-c6e8-4fff-a032-e28ec746193a","apiVersion":"rbac.authorization.k8s.io/v1","resourceVersion":"289"}, "reason": "CreateRole"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new RoleBinding operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"RoleBinding","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-k8s-service-discovery-binding","apiVersion":"rbac.authorization.k8s.io/v1"}, "reason": "CreateRoleBinding"}
2026-10-17T10:33:21+08:00 INFO RoleBinding created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:21+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
2026-10-17T10:33:21+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "hasMultinode": false, "lwsEnabled": false}
2026-10-17T10:33:21+08:00 DEBUG events Created RoleBinding operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"RoleBinding","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-k8s-service-discovery-binding","uid":"a74b9e13-4862-4652-a03d-ad956c4f9286","apiVersion":"rbac.authorization.k8s.io/v1","resourceVersion":"291"}, "reason": "CreateRoleBinding"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5", "resourceType": "PodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new PodCliqueSet operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"PodCliqueSet","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5","apiVersion":"grove.io/v1alpha1"}, "reason": "CreatePodCliqueSet"}
2026-10-17T10:33:21+08:00 INFO PodCliqueSet created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5", "resourceType": "PodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
2026-10-17T10:33:21+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:21+08:00 DEBUG events Created PodCliqueSet operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"PodCliqueSet","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5","uid":"7deff7d4-4dfb-4bf3-b104-3b3be3853584","apiVersion":"grove.io/v1alpha1","resourceVersion":"296"}, "reason": "CreatePodCliqueSet"}
2026-10-17T10:33:21+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-5-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller_common/scale.go:43
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).scaleGroveResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:640
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveScaling
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1018
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1114
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:443
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:289
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:21+08:00 DEBUG Grove resource not found yet, skipping scaling for now - will retry on next reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:21+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-5-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller_common/scale.go:43
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).scaleGroveResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:640
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveScaling
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1018
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1114
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:443
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:289
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:21+08:00 DEBUG Grove resource not found yet, skipping scaling for now - will retry on next reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-sglangprefill","apiVersion":"v1"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Service created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Created Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-sglangprefill","uid":"0112d434-ad43-4f2a-b5d0-5907e2220f67","apiVersion":"v1","resourceVersion":"300"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-sglangdecode","apiVersion":"v1"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Service created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Created Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-5-sglangdecode","uid":"f38b5682-264e-4385-98fb-36ab07a0c29c","apiVersion":"v1","resourceVersion":"304"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
2026-10-17T10:33:21+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
2026-10-17T10:33:21+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "303e5686-a910-4550-bf27-00e949380f2b"}
zz_adjacent_api_test.go:169: release=1.4.0 case=official-1.5 persistedHash=90f01928 PCS-generation=1
zz_adjacent_api_test.go:122: Create official-1.4 through production 1.4.0 admission
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-defaulting-webhook stamped operator origin version on DGD {"object": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "c7a496d0-8ae5-47dc-92bd-eec8cc512236", "name": "adjacent-official-1-4", "namespace": "operatorenv-zkk2r5v9", "version": "1.4.0"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate create {"object": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "d7bc11dc-336a-4042-8dea-f324ff62190f", "name": "adjacent-official-1-4", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Adding finalizer to object {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "resourceVersion": "307"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "27981639-31c8-4f4d-a5a2-6149469d9609", "name": "adjacent-official-1-4", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Finalizer added to object {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "resourceVersion": "308"}
2026-10-17T10:33:21+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "92828afa-f731-4e45-a1ba-1b7818d95479", "name": "adjacent-official-1-4", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new ServiceAccount operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"ServiceAccount","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-k8s-service-discovery","apiVersion":"v1"}, "reason": "CreateServiceAccount"}
2026-10-17T10:33:21+08:00 INFO ServiceAccount created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:21+08:00 DEBUG events Created ServiceAccount operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"ServiceAccount","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-k8s-service-discovery","uid":"bf6fa15e-e3a2-43d2-8ba2-707c5a1717c8","apiVersion":"v1","resourceVersion":"311"}, "reason": "CreateServiceAccount"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Role operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Role","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-k8s-service-discovery-role","apiVersion":"rbac.authorization.k8s.io/v1"}, "reason": "CreateRole"}
2026-10-17T10:33:21+08:00 INFO Role created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:21+08:00 DEBUG events Created Role operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Role","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-k8s-service-discovery-role","uid":"c5bd245e-809d-44a7-abd9-b2c0c94721dc","apiVersion":"rbac.authorization.k8s.io/v1","resourceVersion":"313"}, "reason": "CreateRole"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new RoleBinding operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"RoleBinding","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-k8s-service-discovery-binding","apiVersion":"rbac.authorization.k8s.io/v1"}, "reason": "CreateRoleBinding"}
2026-10-17T10:33:21+08:00 INFO RoleBinding created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:21+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
2026-10-17T10:33:21+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "hasMultinode": false, "lwsEnabled": false}
2026-10-17T10:33:21+08:00 DEBUG events Created RoleBinding operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"RoleBinding","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-k8s-service-discovery-binding","uid":"4e3192b8-73ed-4a0f-a971-2efb8fe49feb","apiVersion":"rbac.authorization.k8s.io/v1","resourceVersion":"314"}, "reason": "CreateRoleBinding"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4", "resourceType": "PodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new PodCliqueSet operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"PodCliqueSet","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4","apiVersion":"grove.io/v1alpha1"}, "reason": "CreatePodCliqueSet"}
2026-10-17T10:33:21+08:00 INFO PodCliqueSet created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4", "resourceType": "PodCliqueSet"}
2026-10-17T10:33:21+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
2026-10-17T10:33:21+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:21+08:00 DEBUG events Created PodCliqueSet operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"PodCliqueSet","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4","uid":"8a4586e1-e198-4c1d-8c3a-6704f19a6828","apiVersion":"grove.io/v1alpha1","resourceVersion":"320"}, "reason": "CreatePodCliqueSet"}
2026-10-17T10:33:21+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-4-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller_common/scale.go:43
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).scaleGroveResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:640
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveScaling
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1018
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1114
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:443
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:289
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:21+08:00 DEBUG Grove resource not found yet, skipping scaling for now - will retry on next reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:21+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-4-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller_common/scale.go:43
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).scaleGroveResource
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:640
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveScaling
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1018
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileGroveResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:1114
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).reconcileResources
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:443
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:289
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.4.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:21+08:00 DEBUG Grove resource not found yet, skipping scaling for now - will retry on next reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:21+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-sglangprefill","apiVersion":"v1"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Service created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Created Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-sglangprefill","uid":"2373b9f2-97b5-4a9b-be54-a32b5cb903b7","apiVersion":"v1","resourceVersion":"324"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Resource not found. Creating a new one. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Creating a new Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-sglangdecode","apiVersion":"v1"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO Service created. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:21+08:00 DEBUG events Created Service operatorenv-zkk2r5v9 {"type": "Normal", "object": {"kind":"Service","namespace":"operatorenv-zkk2r5v9","name":"adjacent-official-1-4-sglangdecode","uid":"dff762c8-02b1-478d-a122-d2d74d25bf17","apiVersion":"v1","resourceVersion":"328"}, "reason": "CreateService"}
2026-10-17T10:33:21+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
2026-10-17T10:33:21+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
2026-10-17T10:33:21+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "50915dec-d633-4678-a0a6-2e141f57f124"}
zz_adjacent_api_test.go:169: release=1.4.0 case=official-1.4 persistedHash=3cf136a2 PCS-generation=1
2026-10-17T10:33:21+08:00 INFO Stopping and waiting for non leader election runnables
2026-10-17T10:33:21+08:00 INFO Stopping and waiting for leader election runnables
2026-10-17T10:33:21+08:00 INFO Stopping and waiting for warmup runnables
2026-10-17T10:33:21+08:00 INFO Shutdown signal received, waiting for all workers to finish {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment"}
2026-10-17T10:33:21+08:00 INFO Shutdown signal received, waiting for all workers to finish {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment"}
2026-10-17T10:33:21+08:00 INFO All workers finished {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment"}
2026-10-17T10:33:21+08:00 INFO All workers finished {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment"}
2026-10-17T10:33:21+08:00 INFO Stopping and waiting for caches
2026-10-17T10:33:21+08:00 INFO Stopping and waiting for webhooks
2026-10-17T10:33:21+08:00 INFO Stopping and waiting for HTTP servers
2026-10-17T10:33:21+08:00 INFO Wait completed, proceeding to shutdown the manager
zz_adjacent_api_test.go:174: Old controller managers stopped; keep etcd/API server alive while candidate takes over
2026-10-17T10:33:30+08:00 INFO Stopping and waiting for warmup runnables
2026-10-17T10:33:30+08:00 INFO Stopping and waiting for non leader election runnables
2026-10-17T10:33:30+08:00 INFO Stopping and waiting for leader election runnables
2026-10-17T10:33:30+08:00 INFO Stopping and waiting for caches
2026-10-17T10:33:30+08:00 INFO Stopping and waiting for webhooks
2026-10-17T10:33:30+08:00 INFO controller-runtime.webhook Shutting down webhook server with timeout of 1 minute
2026-10-17T10:33:31+08:00 INFO Stopping and waiting for HTTP servers
2026-10-17T10:33:31+08:00 INFO Wait completed, proceeding to shutdown the manager
--- PASS: TestAdjacentAPI (16.79s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/controller 17.649s
Raw
api-1.5.log
=== RUN TestAdjacentAPI
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env using existing cluster
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env automatically acquiring client configuration
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env installing CRDs
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.5.0/deploy/operator/config/crd/bases"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamocomponentdeployments.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamographdeploymentrequests.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamographdeployments.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamographdeploymentscalingadapters.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamomodels.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "nvidia.com_dynamoworkermetadatas.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/testing/prometheus"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "prometheusrule.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "servicemonitor.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/testing/inference.networking.k8s.io"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "inferencepools.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env reading CRDs from path {"path": "<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/testing/grove.io"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "clustertopologybindings.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_clustertopologybindings.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podcliques.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podcliquescalinggroups.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podcliquesets.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "grove.io_podgangmaps.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "podcliques.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "podcliquescalinggroups.yaml"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env read CRDs from file {"file": "podcliquesets.yaml"}
2026-10-17T10:33:26+08:00 INFO controller-runtime.test-env there are more than one CRD definitions with the same <Group, Version, Kind, Name> {"GVKN": {"GVK":{"Group":"apiextensions.k8s.io","Version":"v1","Kind":"CustomResourceDefinition"},"Name":"clustertopologybindings.grove.io"}}
2026-10-17T10:33:26+08:00 INFO controller-runtime.test-env there are more than one CRD definitions with the same <Group, Version, Kind, Name> {"GVKN": {"GVK":{"Group":"apiextensions.k8s.io","Version":"v1","Kind":"CustomResourceDefinition"},"Name":"podcliques.grove.io"}}
2026-10-17T10:33:26+08:00 INFO controller-runtime.test-env there are more than one CRD definitions with the same <Group, Version, Kind, Name> {"GVKN": {"GVK":{"Group":"apiextensions.k8s.io","Version":"v1","Kind":"CustomResourceDefinition"},"Name":"podcliquescalinggroups.grove.io"}}
2026-10-17T10:33:26+08:00 INFO controller-runtime.test-env there are more than one CRD definitions with the same <Group, Version, Kind, Name> {"GVKN": {"GVK":{"Group":"apiextensions.k8s.io","Version":"v1","Kind":"CustomResourceDefinition"},"Name":"podcliquesets.grove.io"}}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "inferencepools.inference.networking.k8s.io"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "inferencepools.inference.networking.k8s.io"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podcliques.grove.io"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "podcliques.grove.io"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamocomponentdeployments.nvidia.com"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "dynamocomponentdeployments.nvidia.com"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamographdeployments.nvidia.com"}
2026-10-17T10:33:26+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "dynamographdeployments.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamoworkermetadatas.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "dynamoworkermetadatas.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "prometheusrules.monitoring.coreos.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "prometheusrules.monitoring.coreos.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "clustertopologybindings.grove.io"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "clustertopologybindings.grove.io"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podcliquescalinggroups.grove.io"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "podcliquescalinggroups.grove.io"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podcliquesets.grove.io"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "podcliquesets.grove.io"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "podgangmaps.grove.io"}
2026-10-17T10:33:27+08:00 INFO unrecognized format "int32"
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamographdeploymentrequests.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "dynamographdeploymentrequests.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamographdeploymentscalingadapters.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "dynamographdeploymentscalingadapters.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "dynamomodels.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "dynamomodels.nvidia.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing CRD {"crd": "servicemonitors.monitoring.coreos.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env CRD already exists, updating {"crd": "servicemonitors.monitoring.coreos.com"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "inference.networking.k8s.io/v1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "monitoring.coreos.com/v1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "grove.io/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1beta1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "nvidia.com/v1alpha1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env adding API in waitlist {"GV": "monitoring.coreos.com/v1"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing webhooks
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing mutating webhook {"webhook": "operatorenv-dynamo-operator-mutating"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env Webhook configuration already exists, updating {"webhook": "operatorenv-dynamo-operator-mutating"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env installing validating webhook {"webhook": "operatorenv-dynamo-operator-validating"}
2026-10-17T10:33:27+08:00 DEBUG controller-runtime.test-env Webhook configuration already exists, updating {"webhook": "operatorenv-dynamo-operator-validating"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate/nvidia.com/v1beta1/dynamocomponentdeployments"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1alpha1-dynamocomponentdeployment"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate/nvidia.com/v1beta1/dynamographdeployments"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1alpha1-dynamomodel"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/validate-nvidia-com-v1beta1-dynamographdeploymentrequest"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/convert"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoGraphDeploymentRequest"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoGraphDeployment"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoComponentDeployment"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "nvidia.com/v1beta1, Kind=DynamoGraphDeploymentScalingAdapter"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-nvidia-com-v1beta1-dynamocomponentdeployment"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate/nvidia.com/v1beta1/dynamographdeployments"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-nvidia-com-v1beta1-dynamographdeploymentrequest"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Registering webhook {"path": "/mutate-core-v1-pod-checkpoint-restore"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Starting webhook server
2026-10-17T10:33:27+08:00 INFO controller-runtime.certwatcher Updated current TLS certificate {"cert": "<TMP_PATH>", "key": "<TMP_PATH>"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.webhook Serving webhook server {"host": "127.0.0.1", "port": 64682}
2026-10-17T10:33:27+08:00 INFO controller-runtime.certwatcher Starting certificate poll+watcher {"cert": "<TMP_PATH>", "key": "<TMP_PATH>", "interval": "10s"}
zz_adjacent_api_test.go:95: Candidate uses the SAME API server and existing objects; no historical state is recreated
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1beta1.DynamoComponentDeployment"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1beta1.DynamoGraphDeployment"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1.Deployment"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.DynamoGraphDeploymentScalingAdapter"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1.Service"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "source": "kind source: *v1.Ingress"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1beta1.DynamoComponentDeployment"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1.Pod"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.PodCliqueSet"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1.PersistentVolumeClaim"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.PodClique"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1.Service"}
2026-10-17T10:33:27+08:00 INFO Starting EventSource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "source": "kind source: *v1alpha1.PodCliqueScalingGroup"}
2026-10-17T10:33:27+08:00 INFO controller-runtime.cache Warning: nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter
2026-10-17T10:33:28+08:00 INFO Starting Controller {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment"}
2026-10-17T10:33:28+08:00 INFO Starting workers {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment", "worker count": 1}
2026-10-17T10:33:28+08:00 INFO Starting Controller {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment"}
2026-10-17T10:33:28+08:00 INFO Starting workers {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "worker count": 1}
2026-10-17T10:33:28+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "72280b3a-b5fc-4ac7-a1c1-25daca691502", "name": "adjacent-custom-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec changes detected {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5", "resourceType": "Unstructured", "diff": " map[string]any{\n \t\"replicas\": int64(1),\n \t\"template\": map[string]any{\n \t\t\"cliqueStartupType\": string(\"CliqueStartupTypeAnyOrder\"),\n \t\t\"cliques\": []any{\n \t\t\tmap[string]any{\n \t\t\t\t\"annotations\": map[string]any{\"nvidia.com/dynamo-discovery-backend\": string(\"kubernetes\"), \"nvidia.com/dynamo-operator-origin-version\": string(\"1.4.0\")},\n \t\t\t\t\"labels\": map[string]any{\n \t\t\t\t\t... // 5 identical entries\n \t\t\t\t\t\"nvidia.com/dynamo-namespace\": string(\"operatorenv-zkk2r5v9-adjacent-custom-1-5\"),\n \t\t\t\t\t\"nvidia.com/dynamo-sub-component-type\": string(\"prefill\"),\n+ \t\t\t\t\t\"nvidia.com/dynamo-worker-hash\": string(\"39ae5e0c\"),\n \t\t\t\t\t\"nvidia.com/metrics-enabled\": string(\"true\"),\n \t\t\t\t\t\"nvidia.com/selector\": string(\"adjacent-custom-1-5-sglangprefill\"),\n \t\t\t\t},\n \t\t\t\t\"name\": string(\"sglangprefill\"),\n \t\t\t\t\"spec\": map[string]any{\n \t\t\t\t\t\"minAvailable\": int64(1),\n \t\t\t\t\t\"podSpec\": map[string]any{\n \t\t\t\t\t\t\"containers\": []any{\n \t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\"args\": []any{string(\"--model-path\"), string(\"Qwen/Qwen3-0.6B\"), string(\"--disaggregation-mode\"), string(\"prefill\")},\n \t\t\t\t\t\t\t\t\"command\": []any{string(\"python3\"), string(\"-m\"), string(\"dynamo.sglang\")},\n \t\t\t\t\t\t\t\t\"env\": []any{\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_COMPONENT\"), \"value\": string(\"prefill\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_DISCOVERY_BACKEND\"), \"value\": string(\"kubernetes\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_FORWARDPASS_METRIC_PORT\"), \"value\": string(\"20380\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\t\t\"name\": string(\"DYN_HEALTH_CHECK_ENABLED\"),\n- \t\t\t\t\t\t\t\t\t\t\"value\": string(\"false\"),\n+ \t\t\t\t\t\t\t\t\t\t\"value\": string(\"true\"),\n \t\t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9-adjacent-custom-1-5\")},\n+ \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE_WORKER_SUFFIX\"), \"value\": string(\"39ae5e0c\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAME\"), \"value\": string(\"adjacent-custom-1-5\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9\")},\n \t\t\t\t\t\t\t\t\t... // 9 identical elements\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"image\": string(\"registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro\"),\n \t\t\t\t\t\t\t\t\"livenessProbe\": map[string]any{\n- \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(1),\n+ \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(3),\n \t\t\t\t\t\t\t\t\t\"httpGet\": map[string]any{\"path\": string(\"/live\"), \"port\": string(\"system\")},\n \t\t\t\t\t\t\t\t\t\"periodSeconds\": int64(5),\n \t\t\t\t\t\t\t\t\t\"timeoutSeconds\": int64(4),\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"name\": string(\"main\"),\n \t\t\t\t\t\t\t\t\"ports\": []any{map[string]any{\"containerPort\": int64(9090), \"name\": string(\"system\"), \"protocol\": string(\"TCP\")}, map[string]any{\"containerPort\": int64(19090), \"name\": string(\"nixl\"), \"protocol\": string(\"TCP\")}},\n \t\t\t\t\t\t\t\t... // 4 identical entries\n \t\t\t\t\t\t\t},\n \t\t\t\t\t\t},\n \t\t\t\t\t\t\"restartPolicy\": string(\"Always\"),\n \t\t\t\t\t\t\"securityContext\": map[string]any{\"fsGroup\": int64(1000)},\n \t\t\t\t\t\t... // 3 identical entries\n \t\t\t\t\t},\n \t\t\t\t\t\"replicas\": int64(2),\n \t\t\t\t\t\"roleName\": string(\"sglangprefill\"),\n \t\t\t\t},\n \t\t\t},\n \t\t\tmap[string]any{\n \t\t\t\t\"annotations\": map[string]any{\"nvidia.com/dynamo-discovery-backend\": string(\"kubernetes\"), \"nvidia.com/dynamo-operator-origin-version\": string(\"1.4.0\")},\n \t\t\t\t\"labels\": map[string]any{\n \t\t\t\t\t... // 5 identical entries\n \t\t\t\t\t\"nvidia.com/dynamo-namespace\": string(\"operatorenv-zkk2r5v9-adjacent-custom-1-5\"),\n \t\t\t\t\t\"nvidia.com/dynamo-sub-component-type\": string(\"decode\"),\n+ \t\t\t\t\t\"nvidia.com/dynamo-worker-hash\": string(\"39ae5e0c\"),\n \t\t\t\t\t\"nvidia.com/metrics-enabled\": string(\"true\"),\n \t\t\t\t\t\"nvidia.com/selector\": string(\"adjacent-custom-1-5-sglangdecode\"),\n \t\t\t\t},\n \t\t\t\t\"name\": string(\"sglangdecode\"),\n \t\t\t\t\"spec\": map[string]any{\n \t\t\t\t\t\"minAvailable\": int64(1),\n \t\t\t\t\t\"podSpec\": map[string]any{\n \t\t\t\t\t\t\"containers\": []any{\n \t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\"args\": []any{string(\"--model-path\"), string(\"Qwen/Qwen3-0.6B\"), string(\"--disaggregation-mode\"), string(\"decode\")},\n \t\t\t\t\t\t\t\t\"command\": []any{string(\"python3\"), string(\"-m\"), string(\"dynamo.sglang\")},\n \t\t\t\t\t\t\t\t\"env\": []any{\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_COMPONENT\"), \"value\": string(\"decode\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_DISCOVERY_BACKEND\"), \"value\": string(\"kubernetes\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_FORWARDPASS_METRIC_PORT\"), \"value\": string(\"20380\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\t\t\"name\": string(\"DYN_HEALTH_CHECK_ENABLED\"),\n- \t\t\t\t\t\t\t\t\t\t\"value\": string(\"false\"),\n+ \t\t\t\t\t\t\t\t\t\t\"value\": string(\"true\"),\n \t\t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9-adjacent-custom-1-5\")},\n+ \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE_WORKER_SUFFIX\"), \"value\": string(\"39ae5e0c\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAME\"), \"value\": string(\"adjacent-custom-1-5\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9\")},\n \t\t\t\t\t\t\t\t\t... // 9 identical elements\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"image\": string(\"registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro\"),\n \t\t\t\t\t\t\t\t\"livenessProbe\": map[string]any{\n- \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(1),\n+ \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(3),\n \t\t\t\t\t\t\t\t\t\"httpGet\": map[string]any{\"path\": string(\"/live\"), \"port\": string(\"system\")},\n \t\t\t\t\t\t\t\t\t\"periodSeconds\": int64(5),\n \t\t\t\t\t\t\t\t\t\"timeoutSeconds\": int64(4),\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"name\": string(\"main\"),\n \t\t\t\t\t\t\t\t\"ports\": []any{map[string]any{\"containerPort\": int64(9090), \"name\": string(\"system\"), \"protocol\": string(\"TCP\")}, map[string]any{\"containerPort\": int64(19090), \"name\": string(\"nixl\"), \"protocol\": string(\"TCP\")}},\n \t\t\t\t\t\t\t\t... // 4 identical entries\n \t\t\t\t\t\t\t},\n \t\t\t\t\t\t},\n \t\t\t\t\t\t\"restartPolicy\": string(\"Always\"),\n \t\t\t\t\t\t\"securityContext\": map[string]any{\"fsGroup\": int64(1000)},\n \t\t\t\t\t\t... // 3 identical entries\n \t\t\t\t\t},\n \t\t\t\t\t\"replicas\": int64(2),\n \t\t\t\t\t\"roleName\": string(\"sglangdecode\"),\n \t\t\t\t},\n \t\t\t},\n \t\t},\n \t\t\"headlessServiceConfig\": map[string]any{\"publishNotReadyAddresses\": bool(true)},\n \t\t\"terminationDelay\": string(\"15m0s\"),\n \t},\n }\n"}
2026-10-17T10:33:28+08:00 INFO Unstructured updated. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Updated Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-custom-1-5-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-custom-1-5-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-sglangprefill","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-sglangdecode","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "65f2dd94-0559-46c1-8f81-fe1aee136cc2"}
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Skipping update Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 ERROR unable to persist workload program status {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "error": "update DynamoGraphDeployment status: Operation cannot be fulfilled on dynamographdeployments.nvidia.com \"adjacent-custom-1-5\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:197
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 ERROR Reconciler error {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "7a64af10-4084-4e2c-8ff7-83e12a794037", "error": "update DynamoGraphDeployment status: Operation cannot be fulfilled on dynamographdeployments.nvidia.com \"adjacent-custom-1-5\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "2a1eec3d-fb6a-4ed6-a1c4-45be1311de1e", "name": "adjacent-official-1-4", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-4-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-4-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-4-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Skipping update Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-4-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-4-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-4-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-4-sglangprefill","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-4-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-4-sglangdecode","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-4","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-4", "reconcileID": "ada19f58-cb9d-47b0-b639-156ac7f6aea4"}
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Skipping update Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "728c17a8-c598-440c-8deb-2173fef180fc", "name": "adjacent-custom-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Worker spec change detected but rolling update not supported for this pathway {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "isGrove": true, "hasMultinode": false}
2026-10-17T10:33:28+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "DynamoGraphDeployment", "apiVersion": "nvidia.com/v1beta1", "type": "Warning", "reason": "RollingUpdateNotSupported", "action": "Update", "note": "Worker spec changed but custom rolling updates are not supported for Grove/multinode deployments"}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-custom-1-5-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-custom-1-5-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-custom-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-sglangprefill","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-custom-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-custom-1-5-sglangdecode","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-custom-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-custom-1-5", "reconcileID": "d3712613-4ca7-42dc-9b04-15138c77dc05"}
2026-10-17T10:33:28+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "1ac066c0-a080-4afb-93fa-8b02faa50204", "name": "adjacent-official-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec changes detected {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5", "resourceType": "Unstructured", "diff": " map[string]any{\n \t\"replicas\": int64(1),\n \t\"template\": map[string]any{\n \t\t\"cliqueStartupType\": string(\"CliqueStartupTypeAnyOrder\"),\n \t\t\"cliques\": []any{\n \t\t\tmap[string]any{\n \t\t\t\t\"annotations\": map[string]any{\"nvidia.com/dynamo-discovery-backend\": string(\"kubernetes\"), \"nvidia.com/dynamo-operator-origin-version\": string(\"1.4.0\")},\n \t\t\t\t\"labels\": map[string]any{\n \t\t\t\t\t... // 5 identical entries\n \t\t\t\t\t\"nvidia.com/dynamo-namespace\": string(\"operatorenv-zkk2r5v9-adjacent-official-1-5\"),\n \t\t\t\t\t\"nvidia.com/dynamo-sub-component-type\": string(\"prefill\"),\n+ \t\t\t\t\t\"nvidia.com/dynamo-worker-hash\": string(\"37070ad2\"),\n \t\t\t\t\t\"nvidia.com/metrics-enabled\": string(\"true\"),\n \t\t\t\t\t\"nvidia.com/selector\": string(\"adjacent-official-1-5-sglangprefill\"),\n \t\t\t\t},\n \t\t\t\t\"name\": string(\"sglangprefill\"),\n \t\t\t\t\"spec\": map[string]any{\n \t\t\t\t\t\"minAvailable\": int64(1),\n \t\t\t\t\t\"podSpec\": map[string]any{\n \t\t\t\t\t\t\"containers\": []any{\n \t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\"args\": []any{string(\"--model-path\"), string(\"Qwen/Qwen3-0.6B\"), string(\"--disaggregation-mode\"), string(\"prefill\")},\n \t\t\t\t\t\t\t\t\"command\": []any{string(\"python3\"), string(\"-m\"), string(\"dynamo.sglang\")},\n \t\t\t\t\t\t\t\t\"env\": []any{\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_COMPONENT\"), \"value\": string(\"prefill\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_DISCOVERY_BACKEND\"), \"value\": string(\"kubernetes\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_FORWARDPASS_METRIC_PORT\"), \"value\": string(\"20380\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\t\t\"name\": string(\"DYN_HEALTH_CHECK_ENABLED\"),\n- \t\t\t\t\t\t\t\t\t\t\"value\": string(\"false\"),\n+ \t\t\t\t\t\t\t\t\t\t\"value\": string(\"true\"),\n \t\t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9-adjacent-official-1-5\")},\n+ \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE_WORKER_SUFFIX\"), \"value\": string(\"37070ad2\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAME\"), \"value\": string(\"adjacent-official-1-5\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9\")},\n \t\t\t\t\t\t\t\t\t... // 9 identical elements\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"image\": string(\"nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0\"),\n \t\t\t\t\t\t\t\t\"livenessProbe\": map[string]any{\n- \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(1),\n+ \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(3),\n \t\t\t\t\t\t\t\t\t\"httpGet\": map[string]any{\"path\": string(\"/live\"), \"port\": string(\"system\")},\n \t\t\t\t\t\t\t\t\t\"periodSeconds\": int64(5),\n \t\t\t\t\t\t\t\t\t\"timeoutSeconds\": int64(4),\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"name\": string(\"main\"),\n \t\t\t\t\t\t\t\t\"ports\": []any{map[string]any{\"containerPort\": int64(9090), \"name\": string(\"system\"), \"protocol\": string(\"TCP\")}, map[string]any{\"containerPort\": int64(19090), \"name\": string(\"nixl\"), \"protocol\": string(\"TCP\")}},\n \t\t\t\t\t\t\t\t... // 4 identical entries\n \t\t\t\t\t\t\t},\n \t\t\t\t\t\t},\n \t\t\t\t\t\t\"restartPolicy\": string(\"Always\"),\n \t\t\t\t\t\t\"securityContext\": map[string]any{\"fsGroup\": int64(1000)},\n \t\t\t\t\t\t... // 3 identical entries\n \t\t\t\t\t},\n \t\t\t\t\t\"replicas\": int64(2),\n \t\t\t\t\t\"roleName\": string(\"sglangprefill\"),\n \t\t\t\t},\n \t\t\t},\n \t\t\tmap[string]any{\n \t\t\t\t\"annotations\": map[string]any{\"nvidia.com/dynamo-discovery-backend\": string(\"kubernetes\"), \"nvidia.com/dynamo-operator-origin-version\": string(\"1.4.0\")},\n \t\t\t\t\"labels\": map[string]any{\n \t\t\t\t\t... // 5 identical entries\n \t\t\t\t\t\"nvidia.com/dynamo-namespace\": string(\"operatorenv-zkk2r5v9-adjacent-official-1-5\"),\n \t\t\t\t\t\"nvidia.com/dynamo-sub-component-type\": string(\"decode\"),\n+ \t\t\t\t\t\"nvidia.com/dynamo-worker-hash\": string(\"37070ad2\"),\n \t\t\t\t\t\"nvidia.com/metrics-enabled\": string(\"true\"),\n \t\t\t\t\t\"nvidia.com/selector\": string(\"adjacent-official-1-5-sglangdecode\"),\n \t\t\t\t},\n \t\t\t\t\"name\": string(\"sglangdecode\"),\n \t\t\t\t\"spec\": map[string]any{\n \t\t\t\t\t\"minAvailable\": int64(1),\n \t\t\t\t\t\"podSpec\": map[string]any{\n \t\t\t\t\t\t\"containers\": []any{\n \t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\"args\": []any{string(\"--model-path\"), string(\"Qwen/Qwen3-0.6B\"), string(\"--disaggregation-mode\"), string(\"decode\")},\n \t\t\t\t\t\t\t\t\"command\": []any{string(\"python3\"), string(\"-m\"), string(\"dynamo.sglang\")},\n \t\t\t\t\t\t\t\t\"env\": []any{\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_COMPONENT\"), \"value\": string(\"decode\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_DISCOVERY_BACKEND\"), \"value\": string(\"kubernetes\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_FORWARDPASS_METRIC_PORT\"), \"value\": string(\"20380\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\n \t\t\t\t\t\t\t\t\t\t\"name\": string(\"DYN_HEALTH_CHECK_ENABLED\"),\n- \t\t\t\t\t\t\t\t\t\t\"value\": string(\"false\"),\n+ \t\t\t\t\t\t\t\t\t\t\"value\": string(\"true\"),\n \t\t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9-adjacent-official-1-5\")},\n+ \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_NAMESPACE_WORKER_SUFFIX\"), \"value\": string(\"37070ad2\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAME\"), \"value\": string(\"adjacent-official-1-5\")},\n \t\t\t\t\t\t\t\t\tmap[string]any{\"name\": string(\"DYN_PARENT_DGD_K8S_NAMESPACE\"), \"value\": string(\"operatorenv-zkk2r5v9\")},\n \t\t\t\t\t\t\t\t\t... // 9 identical elements\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"image\": string(\"nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0\"),\n \t\t\t\t\t\t\t\t\"livenessProbe\": map[string]any{\n- \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(1),\n+ \t\t\t\t\t\t\t\t\t\"failureThreshold\": int64(3),\n \t\t\t\t\t\t\t\t\t\"httpGet\": map[string]any{\"path\": string(\"/live\"), \"port\": string(\"system\")},\n \t\t\t\t\t\t\t\t\t\"periodSeconds\": int64(5),\n \t\t\t\t\t\t\t\t\t\"timeoutSeconds\": int64(4),\n \t\t\t\t\t\t\t\t},\n \t\t\t\t\t\t\t\t\"name\": string(\"main\"),\n \t\t\t\t\t\t\t\t\"ports\": []any{map[string]any{\"containerPort\": int64(9090), \"name\": string(\"system\"), \"protocol\": string(\"TCP\")}, map[string]any{\"containerPort\": int64(19090), \"name\": string(\"nixl\"), \"protocol\": string(\"TCP\")}},\n \t\t\t\t\t\t\t\t... // 4 identical entries\n \t\t\t\t\t\t\t},\n \t\t\t\t\t\t},\n \t\t\t\t\t\t\"restartPolicy\": string(\"Always\"),\n \t\t\t\t\t\t\"securityContext\": map[string]any{\"fsGroup\": int64(1000)},\n \t\t\t\t\t\t... // 3 identical entries\n \t\t\t\t\t},\n \t\t\t\t\t\"replicas\": int64(2),\n \t\t\t\t\t\"roleName\": string(\"sglangdecode\"),\n \t\t\t\t},\n \t\t\t},\n \t\t},\n \t\t\"headlessServiceConfig\": map[string]any{\"publishNotReadyAddresses\": bool(true)},\n \t\t\"terminationDelay\": string(\"15m0s\"),\n \t},\n }\n"}
2026-10-17T10:33:28+08:00 INFO Unstructured updated. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Updated Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-5-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-5-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-sglangprefill","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-sglangdecode","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "ca4ac8d0-ed98-4cff-bceb-954007f031fe"}
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Skipping update Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 ERROR unable to persist workload program status {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "error": "update DynamoGraphDeployment status: Operation cannot be fulfilled on dynamographdeployments.nvidia.com \"adjacent-official-1-5\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:197
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 ERROR Reconciler error {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "522c96df-f5e1-4a4a-9e0f-3293efa93a82", "error": "update DynamoGraphDeployment status: Operation cannot be fulfilled on dynamographdeployments.nvidia.com \"adjacent-official-1-5\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:494
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
zz_adjacent_api_test.go:170: release=1.5.0 case=custom-1.5 persistedHash=39ae5e0c PCS-generation=2
2026-10-17T10:33:28+08:00 INFO Reconciling Grove resources {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "hasMultinode": false}
2026-10-17T10:33:28+08:00 INFO K8s discovery is enabled {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO ServiceAccount spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery", "resourceType": "ServiceAccount"}
2026-10-17T10:33:28+08:00 INFO Role found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery","namespace":"operatorenv-zkk2r5v9"}, "kind": "ServiceAccount", "apiVersion": "v1", "type": "Normal", "reason": "UpdateServiceAccount", "action": "Update", "note": "Skipping update ServiceAccount operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Role spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-role", "resourceType": "Role"}
2026-10-17T10:33:28+08:00 INFO RoleBinding found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery-role","namespace":"operatorenv-zkk2r5v9"}, "kind": "Role", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRole", "action": "Update", "note": "Skipping update Role operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO RoleBinding spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-k8s-service-discovery-binding", "resourceType": "RoleBinding"}
2026-10-17T10:33:28+08:00 DEBUG No EPP service defined, skipping EPP resource reconciliation {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-k8s-service-discovery-binding","namespace":"operatorenv-zkk2r5v9"}, "kind": "RoleBinding", "apiVersion": "rbac.authorization.k8s.io/v1", "type": "Normal", "reason": "UpdateRoleBinding", "action": "Update", "note": "Skipping update RoleBinding operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Unstructured spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5", "resourceType": "Unstructured"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "PodCliqueSet", "apiVersion": "grove.io/v1alpha1", "type": "Normal", "reason": "UpdateUnstructured", "action": "Update", "note": "Skipping update Unstructured operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO admission.dynamographdeployment-validating-webhook validate update {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "resource": {"group":"nvidia.com","version":"v1beta1","resource":"dynamographdeployments"}, "user": "admin", "requestID": "7680e89a-976f-4300-a0ea-fdb518a7e2c5", "name": "adjacent-official-1-5", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Worker spec change detected but rolling update not supported for this pathway {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "isGrove": true, "hasMultinode": false}
2026-10-17T10:33:28+08:00 DEBUG Reconciling Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "kind": "DynamoGraphDeployment", "apiVersion": "nvidia.com/v1beta1", "type": "Warning", "reason": "RollingUpdateNotSupported", "action": "Update", "note": "Worker spec changed but custom rolling updates are not supported for Grove/multinode deployments"}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-5-0-sglangprefill\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangprefill", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Scaling resource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "replicas": 2}
2026-10-17T10:33:28+08:00 ERROR Failed to get current scale - resource may not support scale subresource {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9", "groupResource": "podcliques.grove.io", "error": "podcliques.grove.io \"adjacent-official-1-5-0-sglangdecode\" not found"}
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common.ScaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller_common/scale.go:54
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).scaleResource
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:111
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveScaler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_scaler.go:81
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveWorkloadsReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_workloads_reconciler.go:123
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*groveProgram).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dgd_grove_program.go:129
github.com/ai-dynamo/dynamo/deploy/operator/internal/controller.(*DynamoGraphDeploymentReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/controller/dynamographdeployment_controller.go:192
github.com/ai-dynamo/dynamo/deploy/operator/internal/observability.(*ObservedReconciler).Reconcile
<REPRO_ROOT>/source/1.5.0/deploy/operator/internal/observability/reconciler_wrapper.go:46
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
<GOMODCACHE>/sigs.k8s.io/controller-runtime@v0.24.1/pkg/internal/controller/controller.go:312
2026-10-17T10:33:28+08:00 DEBUG Grove resource not found yet, skipping scaling for now {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "gvr": "grove.io/v1alpha1, Resource=podcliques", "name": "adjacent-official-1-5-0-sglangdecode", "namespace": "operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 DEBUG Successfully reconciled Grove scaling operations {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangprefill", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-sglangprefill","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO Service found. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 INFO Service spec is the same. Skipping update. {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183", "namespace": "operatorenv-zkk2r5v9", "resourceName": "adjacent-official-1-5-sglangdecode", "resourceType": "Service"}
2026-10-17T10:33:28+08:00 DEBUG events Event occurred {"object": {"name":"adjacent-official-1-5-sglangdecode","namespace":"operatorenv-zkk2r5v9"}, "kind": "Service", "apiVersion": "v1", "type": "Normal", "reason": "UpdateService", "action": "Update", "note": "Skipping update Service operatorenv-zkk2r5v9"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
2026-10-17T10:33:28+08:00 INFO nvidia.com/v1alpha1 DynamoGraphDeploymentScalingAdapter is deprecated; use nvidia.com/v1beta1 DynamoGraphDeploymentScalingAdapter {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
2026-10-17T10:33:28+08:00 INFO Reconciliation done {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment", "DynamoGraphDeployment": {"name":"adjacent-official-1-5","namespace":"operatorenv-zkk2r5v9"}, "namespace": "operatorenv-zkk2r5v9", "name": "adjacent-official-1-5", "reconcileID": "281e155e-db63-4a00-84fc-2cec5af29183"}
zz_adjacent_api_test.go:170: release=1.5.0 case=official-1.5 persistedHash=37070ad2 PCS-generation=2
zz_adjacent_api_test.go:170: release=1.5.0 case=official-1.4 persistedHash=3cf136a2 PCS-generation=1
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for warmup runnables
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for non leader election runnables
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for leader election runnables
2026-10-17T10:33:29+08:00 INFO Shutdown signal received, waiting for all workers to finish {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment"}
2026-10-17T10:33:29+08:00 INFO All workers finished {"controller": "DynamoComponentDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoComponentDeployment"}
2026-10-17T10:33:29+08:00 INFO Shutdown signal received, waiting for all workers to finish {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment"}
2026-10-17T10:33:29+08:00 INFO All workers finished {"controller": "DynamoGraphDeployment", "controllerGroup": "nvidia.com", "controllerKind": "DynamoGraphDeployment"}
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for caches
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for webhooks
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for HTTP servers
2026-10-17T10:33:29+08:00 INFO Wait completed, proceeding to shutdown the manager
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for non leader election runnables
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for leader election runnables
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for caches
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for warmup runnables
2026-10-17T10:33:29+08:00 INFO Stopping and waiting for webhooks
2026-10-17T10:33:29+08:00 INFO controller-runtime.webhook Shutting down webhook server with timeout of 1 minute
2026-10-17T10:33:30+08:00 INFO Stopping and waiting for HTTP servers
2026-10-17T10:33:30+08:00 INFO Wait completed, proceeding to shutdown the manager
--- PASS: TestAdjacentAPI (3.80s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/controller 4.553s
Raw
api-after.json
{
"custom-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "39ae5e0c",
"nvidia.com/dynamo-operator-origin-version": "1.4.0",
"nvidia.com/workload-provider": "grove"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {},
"f:nvidia.com/workload-provider": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:28Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:gpusPerEngine": {},
"f:gpusPerReplica": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:gpusPerEngine": {},
"f:gpusPerReplica": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:observedGeneration": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:28Z"
}
],
"name": "adjacent-custom-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "373",
"uid": "92e2d420-94bd-4ab4-b610-c6c39924967f"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangdecode"
],
"gpusPerEngine": 0,
"gpusPerReplica": 0,
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangprefill"
],
"gpusPerEngine": 0,
"gpusPerReplica": 0,
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-custom-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"observedGeneration": 1,
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"pcs": {
"metadata": {
"name": "adjacent-custom-1-5",
"namespace": "operatorenv-zkk2r5v9",
"uid": "40164c23-88ac-41d6-8e74-e9d57798c2bf",
"resourceVersion": "355",
"generation": 2,
"creationTimestamp": "2026-10-17T02:33:21Z",
"labels": {
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5"
},
"annotations": {
"nvidia.com/last-applied-generation": "2",
"nvidia.com/last-applied-hash": "0ba50b9e2239db6b27590f5d15d82213bdc3289b75f249ab73dd4a24e38bfab1"
},
"ownerReferences": [
{
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"name": "adjacent-custom-1-5",
"uid": "92e2d420-94bd-4ab4-b610-c6c39924967f",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "controller.test",
"operation": "Update",
"apiVersion": "grove.io/v1alpha1",
"time": "2026-10-17T02:33:28Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:nvidia.com/last-applied-generation": {},
"f:nvidia.com/last-applied-hash": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-graph-deployment-name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"92e2d420-94bd-4ab4-b610-c6c39924967f\"}": {}
}
},
"f:spec": {
".": {},
"f:replicas": {},
"f:template": {
".": {},
"f:cliqueStartupType": {},
"f:cliques": {
".": {},
"k:{\"name\":\"sglangdecode\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
},
"k:{\"name\":\"sglangprefill\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
}
},
"f:headlessServiceConfig": {
".": {},
"f:publishNotReadyAddresses": {}
},
"f:terminationDelay": {}
}
},
"f:status": {
".": {},
"f:availableReplicas": {},
"f:updatedReplicas": {}
}
}
}
]
},
"spec": {
"replicas": 1,
"template": {
"cliques": [
{
"name": "sglangprefill",
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-component-type": "prefill",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "39ae5e0c",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-custom-1-5-sglangprefill"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangprefill",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "39ae5e0c"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-custom-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
},
{
"name": "sglangdecode",
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-component-type": "decode",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "39ae5e0c",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-custom-1-5-sglangdecode"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangdecode",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "39ae5e0c"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-custom-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
}
],
"cliqueStartupType": "CliqueStartupTypeAnyOrder",
"headlessServiceConfig": {
"publishNotReadyAddresses": true
},
"terminationDelay": "15m0s"
}
},
"status": {
"updatedReplicas": 0,
"availableReplicas": 0
}
}
},
"official-1.4": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "3cf136a2",
"nvidia.com/dynamo-operator-origin-version": "1.4.0",
"nvidia.com/workload-provider": "grove"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {},
"f:nvidia.com/workload-provider": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:28Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:gpusPerEngine": {},
"f:gpusPerReplica": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:gpusPerEngine": {},
"f:gpusPerReplica": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:observedGeneration": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:28Z"
}
],
"name": "adjacent-official-1-4",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "371",
"uid": "86a9f489-d9fc-4612-804d-977261c1df0b"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangdecode"
],
"gpusPerEngine": 0,
"gpusPerReplica": 0,
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangprefill"
],
"gpusPerEngine": 0,
"gpusPerReplica": 0,
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-4: SglangPrefill: resource not found; SglangDecode: resource not found",
"observedGeneration": 1,
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"pcs": {
"metadata": {
"name": "adjacent-official-1-4",
"namespace": "operatorenv-zkk2r5v9",
"uid": "8a4586e1-e198-4c1d-8c3a-6704f19a6828",
"resourceVersion": "320",
"generation": 1,
"creationTimestamp": "2026-10-17T02:33:21Z",
"labels": {
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4"
},
"annotations": {
"nvidia.com/last-applied-generation": "1",
"nvidia.com/last-applied-hash": "f3da4ebc082646d8cb9adf65fc11a7c27c7b19e527b889db8bce4c53113ac4dd"
},
"ownerReferences": [
{
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"name": "adjacent-official-1-4",
"uid": "86a9f489-d9fc-4612-804d-977261c1df0b",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "controller.test",
"operation": "Update",
"apiVersion": "grove.io/v1alpha1",
"time": "2026-10-17T02:33:21Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:nvidia.com/last-applied-generation": {},
"f:nvidia.com/last-applied-hash": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-graph-deployment-name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"86a9f489-d9fc-4612-804d-977261c1df0b\"}": {}
}
},
"f:spec": {
".": {},
"f:replicas": {},
"f:template": {
".": {},
"f:cliqueStartupType": {},
"f:cliques": {
".": {},
"k:{\"name\":\"sglangdecode\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
},
"k:{\"name\":\"sglangprefill\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
}
},
"f:headlessServiceConfig": {
".": {},
"f:publishNotReadyAddresses": {}
},
"f:terminationDelay": {}
}
}
}
}
]
},
"spec": {
"replicas": 1,
"template": {
"cliques": [
{
"name": "sglangprefill",
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-component-type": "prefill",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-4-sglangprefill"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangprefill",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-4-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
},
{
"name": "sglangdecode",
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-component-type": "decode",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-4-sglangdecode"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangdecode",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-4-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
}
],
"cliqueStartupType": "CliqueStartupTypeAnyOrder",
"headlessServiceConfig": {
"publishNotReadyAddresses": true
},
"terminationDelay": "15m0s"
}
},
"status": {
"updatedReplicas": 0,
"availableReplicas": 0
}
}
},
"official-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "37070ad2",
"nvidia.com/dynamo-operator-origin-version": "1.4.0",
"nvidia.com/workload-provider": "grove"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {},
"f:nvidia.com/workload-provider": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:28Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:gpusPerEngine": {},
"f:gpusPerReplica": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:gpusPerEngine": {},
"f:gpusPerReplica": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:observedGeneration": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:28Z"
}
],
"name": "adjacent-official-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "391",
"uid": "817618d5-15c8-4d37-896c-565ee53d661d"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangdecode"
],
"gpusPerEngine": 0,
"gpusPerReplica": 0,
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangprefill"
],
"gpusPerEngine": 0,
"gpusPerReplica": 0,
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"observedGeneration": 1,
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"pcs": {
"metadata": {
"name": "adjacent-official-1-5",
"namespace": "operatorenv-zkk2r5v9",
"uid": "7deff7d4-4dfb-4bf3-b104-3b3be3853584",
"resourceVersion": "381",
"generation": 2,
"creationTimestamp": "2026-10-17T02:33:21Z",
"labels": {
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5"
},
"annotations": {
"nvidia.com/last-applied-generation": "2",
"nvidia.com/last-applied-hash": "2f0e7efcf8bf3196242589bfe9e0e371af36edcff876a30405f03fedf3c90945"
},
"ownerReferences": [
{
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"name": "adjacent-official-1-5",
"uid": "817618d5-15c8-4d37-896c-565ee53d661d",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "controller.test",
"operation": "Update",
"apiVersion": "grove.io/v1alpha1",
"time": "2026-10-17T02:33:28Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:nvidia.com/last-applied-generation": {},
"f:nvidia.com/last-applied-hash": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-graph-deployment-name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"817618d5-15c8-4d37-896c-565ee53d661d\"}": {}
}
},
"f:spec": {
".": {},
"f:replicas": {},
"f:template": {
".": {},
"f:cliqueStartupType": {},
"f:cliques": {
".": {},
"k:{\"name\":\"sglangdecode\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
},
"k:{\"name\":\"sglangprefill\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
}
},
"f:headlessServiceConfig": {
".": {},
"f:publishNotReadyAddresses": {}
},
"f:terminationDelay": {}
}
},
"f:status": {
".": {},
"f:availableReplicas": {},
"f:updatedReplicas": {}
}
}
}
]
},
"spec": {
"replicas": 1,
"template": {
"cliques": [
{
"name": "sglangprefill",
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-component-type": "prefill",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "37070ad2",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-5-sglangprefill"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangprefill",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "37070ad2"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
},
{
"name": "sglangdecode",
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-component-type": "decode",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "37070ad2",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-5-sglangdecode"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangdecode",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "37070ad2"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 3
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
}
],
"cliqueStartupType": "CliqueStartupTypeAnyOrder",
"headlessServiceConfig": {
"publishNotReadyAddresses": true
},
"terminationDelay": "15m0s"
}
},
"status": {
"updatedReplicas": 0,
"availableReplicas": 0
}
}
}
}
Raw
api-before.json
{
"custom-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "e2fff36a",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-custom-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "282",
"uid": "92e2d420-94bd-4ab4-b610-c6c39924967f"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-custom-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"pcs": {
"metadata": {
"name": "adjacent-custom-1-5",
"namespace": "operatorenv-zkk2r5v9",
"uid": "40164c23-88ac-41d6-8e74-e9d57798c2bf",
"resourceVersion": "272",
"generation": 1,
"creationTimestamp": "2026-10-17T02:33:21Z",
"labels": {
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5"
},
"annotations": {
"nvidia.com/last-applied-generation": "1",
"nvidia.com/last-applied-hash": "6cb8618f22c19a3f9093b41e88d846b1815a8a6857391e03be70b7beed8965ae"
},
"ownerReferences": [
{
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"name": "adjacent-custom-1-5",
"uid": "92e2d420-94bd-4ab4-b610-c6c39924967f",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "controller.test",
"operation": "Update",
"apiVersion": "grove.io/v1alpha1",
"time": "2026-10-17T02:33:21Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:nvidia.com/last-applied-generation": {},
"f:nvidia.com/last-applied-hash": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-graph-deployment-name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"92e2d420-94bd-4ab4-b610-c6c39924967f\"}": {}
}
},
"f:spec": {
".": {},
"f:replicas": {},
"f:template": {
".": {},
"f:cliqueStartupType": {},
"f:cliques": {
".": {},
"k:{\"name\":\"sglangdecode\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
},
"k:{\"name\":\"sglangprefill\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
}
},
"f:headlessServiceConfig": {
".": {},
"f:publishNotReadyAddresses": {}
},
"f:terminationDelay": {}
}
}
}
}
]
},
"spec": {
"replicas": 1,
"template": {
"cliques": [
{
"name": "sglangprefill",
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-component-type": "prefill",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-custom-1-5-sglangprefill"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangprefill",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-custom-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
},
{
"name": "sglangdecode",
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-component-type": "decode",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-custom-1-5-sglangdecode"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangdecode",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-custom-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
}
],
"cliqueStartupType": "CliqueStartupTypeAnyOrder",
"headlessServiceConfig": {
"publishNotReadyAddresses": true
},
"terminationDelay": "15m0s"
}
},
"status": {
"updatedReplicas": 0,
"availableReplicas": 0
}
}
},
"official-1.4": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "3cf136a2",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-official-1-4",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "330",
"uid": "86a9f489-d9fc-4612-804d-977261c1df0b"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-4: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"pcs": {
"metadata": {
"name": "adjacent-official-1-4",
"namespace": "operatorenv-zkk2r5v9",
"uid": "8a4586e1-e198-4c1d-8c3a-6704f19a6828",
"resourceVersion": "320",
"generation": 1,
"creationTimestamp": "2026-10-17T02:33:21Z",
"labels": {
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4"
},
"annotations": {
"nvidia.com/last-applied-generation": "1",
"nvidia.com/last-applied-hash": "f3da4ebc082646d8cb9adf65fc11a7c27c7b19e527b889db8bce4c53113ac4dd"
},
"ownerReferences": [
{
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"name": "adjacent-official-1-4",
"uid": "86a9f489-d9fc-4612-804d-977261c1df0b",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "controller.test",
"operation": "Update",
"apiVersion": "grove.io/v1alpha1",
"time": "2026-10-17T02:33:21Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:nvidia.com/last-applied-generation": {},
"f:nvidia.com/last-applied-hash": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-graph-deployment-name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"86a9f489-d9fc-4612-804d-977261c1df0b\"}": {}
}
},
"f:spec": {
".": {},
"f:replicas": {},
"f:template": {
".": {},
"f:cliqueStartupType": {},
"f:cliques": {
".": {},
"k:{\"name\":\"sglangdecode\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
},
"k:{\"name\":\"sglangprefill\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
}
},
"f:headlessServiceConfig": {
".": {},
"f:publishNotReadyAddresses": {}
},
"f:terminationDelay": {}
}
}
}
}
]
},
"spec": {
"replicas": 1,
"template": {
"cliques": [
{
"name": "sglangprefill",
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-component-type": "prefill",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-4-sglangprefill"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangprefill",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-4-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
},
{
"name": "sglangdecode",
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-component-type": "decode",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-4-sglangdecode"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangdecode",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-4"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-4-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
}
],
"cliqueStartupType": "CliqueStartupTypeAnyOrder",
"headlessServiceConfig": {
"publishNotReadyAddresses": true
},
"terminationDelay": "15m0s"
}
},
"status": {
"updatedReplicas": 0,
"availableReplicas": 0
}
}
},
"official-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "90f01928",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-official-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "306",
"uid": "817618d5-15c8-4d37-896c-565ee53d661d"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"pcs": {
"metadata": {
"name": "adjacent-official-1-5",
"namespace": "operatorenv-zkk2r5v9",
"uid": "7deff7d4-4dfb-4bf3-b104-3b3be3853584",
"resourceVersion": "296",
"generation": 1,
"creationTimestamp": "2026-10-17T02:33:21Z",
"labels": {
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5"
},
"annotations": {
"nvidia.com/last-applied-generation": "1",
"nvidia.com/last-applied-hash": "93c6b37e4b7001b21ab115da8c7c104a28eafb76f30f78de33f0c2c0062ca6eb"
},
"ownerReferences": [
{
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"name": "adjacent-official-1-5",
"uid": "817618d5-15c8-4d37-896c-565ee53d661d",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "controller.test",
"operation": "Update",
"apiVersion": "grove.io/v1alpha1",
"time": "2026-10-17T02:33:21Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:nvidia.com/last-applied-generation": {},
"f:nvidia.com/last-applied-hash": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-graph-deployment-name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"817618d5-15c8-4d37-896c-565ee53d661d\"}": {}
}
},
"f:spec": {
".": {},
"f:replicas": {},
"f:template": {
".": {},
"f:cliqueStartupType": {},
"f:cliques": {
".": {},
"k:{\"name\":\"sglangdecode\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
},
"k:{\"name\":\"sglangprefill\"}": {
".": {},
"f:annotations": {
".": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-operator-origin-version": {}
},
"f:labels": {
".": {},
"f:nvidia.com/dynamo-component": {},
"f:nvidia.com/dynamo-component-type": {},
"f:nvidia.com/dynamo-discovery-backend": {},
"f:nvidia.com/dynamo-discovery-enabled": {},
"f:nvidia.com/dynamo-graph-deployment-name": {},
"f:nvidia.com/dynamo-namespace": {},
"f:nvidia.com/dynamo-sub-component-type": {},
"f:nvidia.com/metrics-enabled": {},
"f:nvidia.com/selector": {}
},
"f:name": {},
"f:spec": {
".": {},
"f:minAvailable": {},
"f:podSpec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:env": {
".": {},
"k:{\"name\":\"DYN_COMPONENT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_DISCOVERY_BACKEND\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_FORWARDPASS_METRIC_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_HEALTH_CHECK_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_PARENT_DGD_K8S_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_ENABLED\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_ENABLE\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_EXPORTER\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"NIXL_TELEMETRY_PROMETHEUS_PORT\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"POD_NAME\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_NAMESPACE\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
},
"k:{\"name\":\"POD_UID\"}": {
".": {},
"f:name": {},
"f:valueFrom": {
".": {},
"f:fieldRef": {}
}
}
},
"f:image": {},
"f:livenessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":19090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
},
"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:name": {},
"f:protocol": {}
}
},
"f:readinessProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:resources": {},
"f:startupProbe": {
".": {},
"f:failureThreshold": {},
"f:httpGet": {
".": {},
"f:path": {},
"f:port": {}
},
"f:periodSeconds": {},
"f:timeoutSeconds": {}
},
"f:volumeMounts": {
".": {},
"k:{\"mountPath\":\"/dev/shm\"}": {
".": {},
"f:mountPath": {},
"f:name": {}
}
}
}
},
"f:restartPolicy": {},
"f:securityContext": {
".": {},
"f:fsGroup": {}
},
"f:serviceAccountName": {},
"f:terminationGracePeriodSeconds": {},
"f:volumes": {
".": {},
"k:{\"name\":\"shared-memory\"}": {
".": {},
"f:emptyDir": {
".": {},
"f:medium": {},
"f:sizeLimit": {}
},
"f:name": {}
}
}
},
"f:replicas": {},
"f:roleName": {}
}
}
},
"f:headlessServiceConfig": {
".": {},
"f:publishNotReadyAddresses": {}
},
"f:terminationDelay": {}
}
}
}
}
]
},
"spec": {
"replicas": 1,
"template": {
"cliques": [
{
"name": "sglangprefill",
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-component-type": "prefill",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-5-sglangprefill"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangprefill",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
},
{
"name": "sglangdecode",
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-component-type": "decode",
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-discovery-enabled": "true",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/metrics-enabled": "true",
"nvidia.com/selector": "adjacent-official-1-5-sglangdecode"
},
"annotations": {
"nvidia.com/dynamo-discovery-backend": "kubernetes",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"spec": {
"roleName": "sglangdecode",
"podSpec": {
"volumes": [
{
"name": "shared-memory",
"emptyDir": {
"medium": "Memory",
"sizeLimit": "8Gi"
}
}
],
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"ports": [
{
"name": "system",
"containerPort": 9090,
"protocol": "TCP"
},
{
"name": "nixl",
"containerPort": 19090,
"protocol": "TCP"
}
],
"env": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "shared-memory",
"mountPath": "/dev/shm"
}
],
"livenessProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 5,
"failureThreshold": 1
},
"readinessProbe": {
"httpGet": {
"path": "/health",
"port": "system"
},
"timeoutSeconds": 4,
"periodSeconds": 10,
"failureThreshold": 3
},
"startupProbe": {
"httpGet": {
"path": "/live",
"port": "system"
},
"timeoutSeconds": 5,
"periodSeconds": 10,
"failureThreshold": 720
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 60,
"serviceAccountName": "adjacent-official-1-5-k8s-service-discovery",
"securityContext": {
"fsGroup": 1000
}
},
"replicas": 2,
"minAvailable": 1
}
}
],
"cliqueStartupType": "CliqueStartupTypeAnyOrder",
"headlessServiceConfig": {
"publishNotReadyAddresses": true
},
"terminationDelay": "15m0s"
}
},
"status": {
"updatedReplicas": 0,
"availableReplicas": 0
}
}
}
}
Raw
api-decisions.json
{
"custom-1.5": {
"desiredHash": "39ae5e0c",
"hashChanged": true,
"oldHash": "e2fff36a",
"sameDGDUID": true,
"sameStoredSpecBeforeTakeover": true
},
"official-1.4": {
"desiredHash": "3cf136a2",
"hashChanged": false,
"oldHash": "3cf136a2",
"sameDGDUID": true,
"sameStoredSpecBeforeTakeover": true
},
"official-1.5": {
"desiredHash": "37070ad2",
"hashChanged": true,
"oldHash": "90f01928",
"sameDGDUID": true,
"sameStoredSpecBeforeTakeover": true
}
}
Raw
api-summary.json
{
"custom-1.5": {
"desiredHash": "39ae5e0c",
"hashChanged": true,
"oldHash": "e2fff36a",
"sameDGDUID": true,
"sameStoredSpecBeforeTakeover": true,
"sameDGDSpecAfterTakeover": true,
"samePCSUID": true,
"originVersionBefore": "1.4.0",
"originVersionAfter": "1.4.0",
"pcsGenerationBefore": 1,
"pcsGenerationAfter": 2,
"storedHashInputDiff": [
{
"path": "/SglangDecode/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
},
{
"path": "/SglangPrefill/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
}
],
"workers": [
{
"name": "sglangprefill",
"imageUnchanged": true,
"hashLabelBefore": null,
"hashLabelAfter": "39ae5e0c",
"namespaceSuffixBefore": null,
"namespaceSuffixAfter": "39ae5e0c",
"envDiff": [
{
"path": "/DYN_HEALTH_CHECK_ENABLED",
"before": "false",
"after": "true"
},
{
"path": "/DYN_NAMESPACE_WORKER_SUFFIX",
"before": "<absent>",
"after": "39ae5e0c"
}
],
"podSpecDiff": [
{
"path": "/containers/0/env",
"before": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"after": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "39ae5e0c"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
]
},
{
"path": "/containers/0/livenessProbe/failureThreshold",
"before": 1,
"after": 3
}
]
},
{
"name": "sglangdecode",
"imageUnchanged": true,
"hashLabelBefore": null,
"hashLabelAfter": "39ae5e0c",
"namespaceSuffixBefore": null,
"namespaceSuffixAfter": "39ae5e0c",
"envDiff": [
{
"path": "/DYN_HEALTH_CHECK_ENABLED",
"before": "false",
"after": "true"
},
{
"path": "/DYN_NAMESPACE_WORKER_SUFFIX",
"before": "<absent>",
"after": "39ae5e0c"
}
],
"podSpecDiff": [
{
"path": "/containers/0/env",
"before": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"after": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-custom-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "39ae5e0c"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-custom-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
]
},
{
"path": "/containers/0/livenessProbe/failureThreshold",
"before": 1,
"after": 3
}
]
}
]
},
"official-1.4": {
"desiredHash": "3cf136a2",
"hashChanged": false,
"oldHash": "3cf136a2",
"sameDGDUID": true,
"sameStoredSpecBeforeTakeover": true,
"sameDGDSpecAfterTakeover": true,
"samePCSUID": true,
"originVersionBefore": "1.4.0",
"originVersionAfter": "1.4.0",
"pcsGenerationBefore": 1,
"pcsGenerationAfter": 1,
"storedHashInputDiff": [],
"workers": [
{
"name": "sglangprefill",
"imageUnchanged": true,
"hashLabelBefore": null,
"hashLabelAfter": null,
"namespaceSuffixBefore": null,
"namespaceSuffixAfter": null,
"envDiff": [],
"podSpecDiff": []
},
{
"name": "sglangdecode",
"imageUnchanged": true,
"hashLabelBefore": null,
"hashLabelAfter": null,
"namespaceSuffixBefore": null,
"namespaceSuffixAfter": null,
"envDiff": [],
"podSpecDiff": []
}
]
},
"official-1.5": {
"desiredHash": "37070ad2",
"hashChanged": true,
"oldHash": "90f01928",
"sameDGDUID": true,
"sameStoredSpecBeforeTakeover": true,
"sameDGDSpecAfterTakeover": true,
"samePCSUID": true,
"originVersionBefore": "1.4.0",
"originVersionAfter": "1.4.0",
"pcsGenerationBefore": 1,
"pcsGenerationAfter": 2,
"storedHashInputDiff": [
{
"path": "/SglangDecode/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
},
{
"path": "/SglangPrefill/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
}
],
"workers": [
{
"name": "sglangprefill",
"imageUnchanged": true,
"hashLabelBefore": null,
"hashLabelAfter": "37070ad2",
"namespaceSuffixBefore": null,
"namespaceSuffixAfter": "37070ad2",
"envDiff": [
{
"path": "/DYN_HEALTH_CHECK_ENABLED",
"before": "false",
"after": "true"
},
{
"path": "/DYN_NAMESPACE_WORKER_SUFFIX",
"before": "<absent>",
"after": "37070ad2"
}
],
"podSpecDiff": [
{
"path": "/containers/0/env",
"before": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"after": [
{
"name": "DYN_COMPONENT",
"value": "prefill"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "37070ad2"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
]
},
{
"path": "/containers/0/livenessProbe/failureThreshold",
"before": 1,
"after": 3
}
]
},
{
"name": "sglangdecode",
"imageUnchanged": true,
"hashLabelBefore": null,
"hashLabelAfter": "37070ad2",
"namespaceSuffixBefore": null,
"namespaceSuffixAfter": "37070ad2",
"envDiff": [
{
"path": "/DYN_HEALTH_CHECK_ENABLED",
"before": "false",
"after": "true"
},
{
"path": "/DYN_NAMESPACE_WORKER_SUFFIX",
"before": "<absent>",
"after": "37070ad2"
}
],
"podSpecDiff": [
{
"path": "/containers/0/env",
"before": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "false"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
],
"after": [
{
"name": "DYN_COMPONENT",
"value": "decode"
},
{
"name": "DYN_DISCOVERY_BACKEND",
"value": "kubernetes"
},
{
"name": "DYN_FORWARDPASS_METRIC_PORT",
"value": "20380"
},
{
"name": "DYN_HEALTH_CHECK_ENABLED",
"value": "true"
},
{
"name": "DYN_NAMESPACE",
"value": "operatorenv-zkk2r5v9-adjacent-official-1-5"
},
{
"name": "DYN_NAMESPACE_WORKER_SUFFIX",
"value": "37070ad2"
},
{
"name": "DYN_PARENT_DGD_K8S_NAME",
"value": "adjacent-official-1-5"
},
{
"name": "DYN_PARENT_DGD_K8S_NAMESPACE",
"value": "operatorenv-zkk2r5v9"
},
{
"name": "DYN_SYSTEM_ENABLED",
"value": "true"
},
{
"name": "DYN_SYSTEM_PORT",
"value": "9090"
},
{
"name": "DYN_SYSTEM_USE_ENDPOINT_HEALTH_STATUS",
"value": "[\"generate\"]"
},
{
"name": "NIXL_TELEMETRY_ENABLE",
"value": "n"
},
{
"name": "NIXL_TELEMETRY_EXPORTER",
"value": "prometheus"
},
{
"name": "NIXL_TELEMETRY_PROMETHEUS_PORT",
"value": "19090"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "POD_UID",
"valueFrom": {
"fieldRef": {
"fieldPath": "metadata.uid"
}
}
}
]
},
{
"path": "/containers/0/livenessProbe/failureThreshold",
"before": 1,
"after": 3
}
]
}
]
}
}
Raw
api_test.go.tmpl
package controller
import (
"context"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"testing"
"time"
config "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1"
api "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/consts"
common "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/features"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/podcache"
"github.com/ai-dynamo/dynamo/deploy/operator/internal/testing/operatorenv"
grove "github.com/ai-dynamo/grove/operator/api/core/v1alpha1"
"github.com/stretchr/testify/require"
"k8s.io/client-go/tools/clientcmd"
clientcmdapi "k8s.io/client-go/tools/clientcmd/api"
"k8s.io/utils/ptr"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
"sigs.k8s.io/yaml"
)
type adjacentAPISnapshot struct {
DGD api.DynamoGraphDeployment `json:"dgd"`
PCS grove.PodCliqueSet `json:"pcs"`
}
func adjacentWrite(t *testing.T,path string,value any) {
t.Helper()
data,err:=json.MarshalIndent(value,""," ");require.NoError(t,err)
require.NoError(t,os.WriteFile(path,data,0600))
}
func adjacentStartManager(t *testing.T,env *operatorenv.TestEnv,namespace string) func() {
t.Helper()
cfg:=env.OperatorConfig().DeepCopy();cfg.Namespace.Restricted=namespace
cacheOpts:=cache.Options{DefaultNamespaces:map[string]cache.Config{namespace:{}}}
require.NoError(t,podcache.Configure(&cacheOpts))
mgr,err:=ctrl.NewManager(env.RESTConfig(),ctrl.Options{Scheme:env.Client().Scheme(),Metrics:metricsserver.Options{BindAddress:"0"},Cache:cacheOpts})
require.NoError(t,err)
opts:=SetupOptions{Config:cfg,RuntimeConfig:env.RuntimeConfig()}
dgdOpts:=DynamoGraphDeploymentSetupOptions{SetupOptions:opts}
// SCALE_CLIENT
require.NoError(t,SetupDynamoGraphDeployment(mgr,dgdOpts))
require.NoError(t,SetupDynamoComponentDeployment(mgr,DynamoComponentDeploymentSetupOptions{SetupOptions:opts}))
ctx,cancel:=context.WithCancel(context.Background())
done:=make(chan error,1);go func(){done<-mgr.Start(ctx)}()
var once sync.Once
stop:=func(){once.Do(func(){cancel();require.NoError(t,<-done)})}
t.Cleanup(stop)
timeout,cancelWait:=context.WithTimeout(context.Background(),30*time.Second);defer cancelWait()
require.True(t,mgr.GetCache().WaitForCacheSync(timeout))
return stop
}
func TestAdjacentAPI(t *testing.T) {
ctrl.SetLogger(zap.New(zap.UseDevMode(true)))
root:=os.Getenv("ADJACENT_ROOT")
source:=os.Getenv("ADJACENT_STAGE")=="source"
version:=os.Getenv("ADJACENT_RELEASE")
cfg:=&config.OperatorConfiguration{};config.SetDefaultsOperatorConfiguration(cfg)
cfg.GPU.DiscoveryEnabled=ptr.To(false)
env:=operatorenv.New(operatorenv.Options{
Admission:operatorenv.AdmissionWebhooks{Mutating:true,Validating:true},
SetupWebhooks:setupProductionWebhooks,OperatorVersion:version,Config:cfg,
RuntimeConfig:&common.RuntimeConfig{Gate:features.Gates{Grove:true}},
EventuallyTimeout:30*time.Second,
}).RunT(t)
c:=env.Client();ctx:=context.Background()
before:=map[string]adjacentAPISnapshot{}
namespace:=env.Namespace()
if !source {
data,err:=os.ReadFile(filepath.Join(root,"api-before.json"));require.NoError(t,err)
require.NoError(t,json.Unmarshal(data,&before))
namespace=before["custom-1.5"].DGD.Namespace
t.Log("Candidate uses the SAME API server and existing objects; no historical state is recreated")
decisions:=map[string]any{}
for profile,old:=range before {
var current api.DynamoGraphDeployment
require.NoError(t,c.Get(ctx,client.ObjectKeyFromObject(&old.DGD),&current))
require.Equal(t,old.DGD.UID,current.UID)
require.Equal(t,old.DGD.Spec,current.Spec)
desired,err:=dynamo.ComputeDGDWorkersSpecHash(&current);require.NoError(t,err)
hashChanged:=false
// PLAN_TRANSITION
decisions[profile]=map[string]any{"oldHash":old.DGD.Annotations[consts.AnnotationCurrentWorkerHashV2],"desiredHash":desired,"hashChanged":hashChanged,"sameDGDUID":true,"sameStoredSpecBeforeTakeover":true}
}
adjacentWrite(t,filepath.Join(root,"api-decisions.json"),decisions)
}
stop:=adjacentStartManager(t,env,namespace)
after:=map[string]adjacentAPISnapshot{}
for _,profile:=range []string{"custom-1.5","official-1.5","official-1.4"} {
var dgd api.DynamoGraphDeployment
if source {
data,err:=os.ReadFile(filepath.Join(root,profile+".yaml"));require.NoError(t,err)
require.NoError(t,yaml.UnmarshalStrict(data,&dgd))
dgd.Namespace=namespace
dgd.Name="adjacent-"+strings.ReplaceAll(profile,".","-")
t.Logf("Create %s through production %s admission",profile,version)
require.NoError(t,c.Create(ctx,&dgd))
} else {dgd=before[profile].DGD}
var observed api.DynamoGraphDeployment
var pcs grove.PodCliqueSet
require.Eventually(t,func() bool {
if c.Get(ctx,client.ObjectKeyFromObject(&dgd),&observed)!=nil{return false}
expected,err:=dynamo.ComputeDGDWorkersSpecHash(&observed);if err!=nil{return false}
if observed.Annotations[consts.AnnotationCurrentWorkerHashV2]!=expected{return false}
if observed.Status.ObservedGeneration!=observed.Generation{return false}
key:=client.ObjectKey{Namespace:namespace,Name:dynamo.PCSNameForDGD(observed.Name,observed.Spec.Components)}
if c.Get(ctx,key,&pcs)!=nil{return false}
if len(pcs.Spec.Template.Cliques)!=2{return false}
if !source && profile!="official-1.4" {
for _,clique:=range pcs.Spec.Template.Cliques {
if clique.Labels[consts.KubeLabelDynamoWorkerHash]!=expected{return false}
}
}
return true
},90*time.Second,200*time.Millisecond,"%s must reach the release controller's persisted worker state",profile)
after[profile]=adjacentAPISnapshot{DGD:observed,PCS:pcs}
if !source {
old:=before[profile]
require.Equal(t,old.DGD.Spec,observed.Spec,"DGD intent changed")
require.Equal(t,old.DGD.UID,observed.UID)
require.Equal(t,old.PCS.UID,pcs.UID)
for _,clique:=range pcs.Spec.Template.Cliques {
require.Equal(t,old.DGD.Spec.Components[0].PodTemplate.Spec.Containers[0].Image,clique.Spec.PodSpec.Containers[0].Image)
}
}
t.Logf("release=%s case=%s persistedHash=%s PCS-generation=%d",version,profile,observed.Annotations[consts.AnnotationCurrentWorkerHashV2],pcs.Generation)
}
stop()
if source {
adjacentWrite(t,filepath.Join(root,"api-before.json"),after)
t.Log("Old controller managers stopped; keep etcd/API server alive while candidate takes over")
rest:=env.RESTConfig()
kubeconfig:=clientcmdapi.Config{
Clusters:map[string]*clientcmdapi.Cluster{"adjacent-local":{Server:rest.Host,CertificateAuthorityData:rest.CAData}},
AuthInfos:map[string]*clientcmdapi.AuthInfo{"adjacent-local":{ClientCertificateData:rest.CertData,ClientKeyData:rest.KeyData}},
Contexts:map[string]*clientcmdapi.Context{"adjacent-local":{Cluster:"adjacent-local",AuthInfo:"adjacent-local"}},
CurrentContext:"adjacent-local",
}
path:=filepath.Join(root,"local-api-kubeconfig")
require.NoError(t,clientcmd.WriteToFile(kubeconfig,path));defer os.Remove(path)
cmd:=exec.Command("go","test","-p","2","./internal/controller","-run","^TestAdjacentAPI$","-count=1","-v","-timeout","5m")
cmd.Dir=filepath.Join(root,"source","1.5.0","deploy","operator")
cmd.Env=append(os.Environ(),"ADJACENT_STAGE=target","ADJACENT_RELEASE=1.5.0","USE_EXISTING_CLUSTER=true","KUBECONFIG="+path)
logFile,err:=os.Create(filepath.Join(root,"api-1.5.log"));require.NoError(t,err);defer logFile.Close()
cmd.Stdout=logFile;cmd.Stderr=logFile
require.NoError(t,cmd.Run(),"candidate log: api-1.5.log")
} else {adjacentWrite(t,filepath.Join(root,"api-after.json"),after)}
}
Raw
custom-1.5.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# The generator substitutes only the catalog's image before the old operator runs.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: hash-upgrade-replay
namespace: default
spec:
backendFramework: sglang
components:
- name: SglangPrefill
type: prefill
replicas: 2
podTemplate:
spec:
containers:
- name: main
image: registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro
command: [python3, -m, dynamo.sglang]
args: [--model-path, Qwen/Qwen3-0.6B, --disaggregation-mode, prefill]
- name: SglangDecode
type: decode
replicas: 2
podTemplate:
spec:
containers:
- name: main
image: registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro
command: [python3, -m, dynamo.sglang]
args: [--model-path, Qwen/Qwen3-0.6B, --disaggregation-mode, decode]
Raw
grove-crd-provenance.json
{
"1.4.0": {
"moduleVersion": "v0.1.0-alpha.12-rc1",
"crdSHA256": {
"grove.io_podcliques.yaml": "817388b0b1287b9140de3c44c48d31982db213f98a2475e45ec18638f3c83e04",
"grove.io_podcliquescalinggroups.yaml": "4d8f9bb5768d5ce43d2a81ba3fd046df4d5622b93ba5256676b38b3de8387812",
"grove.io_clustertopologybindings.yaml": "917792ec71a94dc3f87e220961852c7fd192f25bb50cbc8226933cb55f30f16a",
"grove.io_podcliquesets.yaml": "2a7622f2086fcf9fdd76e03de26b318860340bb27b6f4093700783da0220a466"
}
},
"1.5.0": {
"moduleVersion": "v0.1.0-alpha.13",
"crdSHA256": {
"grove.io_podcliques.yaml": "b37985c105ec9a0517604d40a30fac3f01891c54ffd5764440eae0239f3e9866",
"grove.io_podgangmaps.yaml": "d45a8297abacd44999216663351dc5d12c32e19662604ea56acff2e344496634",
"grove.io_podcliquescalinggroups.yaml": "4d8f9bb5768d5ce43d2a81ba3fd046df4d5622b93ba5256676b38b3de8387812",
"grove.io_clustertopologybindings.yaml": "8d30113e0d268eb19080549267b909b0292525c7996df061a352edbc9fe60db3",
"grove.io_podcliquesets.yaml": "2a7622f2086fcf9fdd76e03de26b318860340bb27b6f4093700783da0220a466"
}
}
}
Raw
hash-logs.txt
=== hash-1.2.0.log ===
=== RUN TestAdjacentReleaseHash
=== RUN TestAdjacentReleaseHash/custom-1.5
zz_adjacent_hash_test.go:59: release=1.2.0 profile=custom-1.5 hash=309e9927
=== RUN TestAdjacentReleaseHash/official-1.5
zz_adjacent_hash_test.go:59: release=1.2.0 profile=official-1.5 hash=53549949
=== RUN TestAdjacentReleaseHash/official-1.4
zz_adjacent_hash_test.go:59: release=1.2.0 profile=official-1.4 hash=7be83b1f
--- PASS: TestAdjacentReleaseHash (0.00s)
--- PASS: TestAdjacentReleaseHash/custom-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.4 (0.00s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo 0.769s
=== hash-1.3.0.log ===
=== RUN TestAdjacentReleaseHash
=== RUN TestAdjacentReleaseHash/custom-1.5
zz_adjacent_hash_test.go:59: release=1.3.0 profile=custom-1.5 hash=309e9927
=== RUN TestAdjacentReleaseHash/official-1.5
zz_adjacent_hash_test.go:59: release=1.3.0 profile=official-1.5 hash=53549949
=== RUN TestAdjacentReleaseHash/official-1.4
zz_adjacent_hash_test.go:59: release=1.3.0 profile=official-1.4 hash=7be83b1f
--- PASS: TestAdjacentReleaseHash (0.01s)
--- PASS: TestAdjacentReleaseHash/custom-1.5 (0.01s)
--- PASS: TestAdjacentReleaseHash/official-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.4 (0.00s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo 0.695s
=== hash-1.4.0.log ===
=== RUN TestAdjacentReleaseHash
=== RUN TestAdjacentReleaseHash/custom-1.5
zz_adjacent_hash_test.go:59: release=1.4.0 profile=custom-1.5 hash=309e9927
=== RUN TestAdjacentReleaseHash/official-1.5
zz_adjacent_hash_test.go:59: release=1.4.0 profile=official-1.5 hash=53549949
=== RUN TestAdjacentReleaseHash/official-1.4
zz_adjacent_hash_test.go:59: release=1.4.0 profile=official-1.4 hash=7be83b1f
--- PASS: TestAdjacentReleaseHash (0.01s)
--- PASS: TestAdjacentReleaseHash/custom-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.4 (0.00s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo 0.795s
=== hash-1.5.0.log ===
=== RUN TestAdjacentReleaseHash
=== RUN TestAdjacentReleaseHash/custom-1.5
zz_adjacent_hash_test.go:59: release=1.5.0 profile=custom-1.5 hash=142dbdff
=== RUN TestAdjacentReleaseHash/official-1.5
zz_adjacent_hash_test.go:59: release=1.5.0 profile=official-1.5 hash=e332eabb
=== RUN TestAdjacentReleaseHash/official-1.4
zz_adjacent_hash_test.go:59: release=1.5.0 profile=official-1.4 hash=7be83b1f
--- PASS: TestAdjacentReleaseHash (0.01s)
--- PASS: TestAdjacentReleaseHash/custom-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.4 (0.00s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo 0.703s
=== stored-hash-1.4.0.log ===
=== RUN TestAdjacentReleaseHash
=== RUN TestAdjacentReleaseHash/custom-1.5
zz_adjacent_hash_test.go:59: release=1.4.0 profile=custom-1.5 hash=e2fff36a
=== RUN TestAdjacentReleaseHash/official-1.5
zz_adjacent_hash_test.go:59: release=1.4.0 profile=official-1.5 hash=90f01928
=== RUN TestAdjacentReleaseHash/official-1.4
zz_adjacent_hash_test.go:59: release=1.4.0 profile=official-1.4 hash=3cf136a2
--- PASS: TestAdjacentReleaseHash (0.01s)
--- PASS: TestAdjacentReleaseHash/custom-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.4 (0.00s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo 0.819s
=== stored-hash-1.5.0.log ===
=== RUN TestAdjacentReleaseHash
=== RUN TestAdjacentReleaseHash/custom-1.5
zz_adjacent_hash_test.go:59: release=1.5.0 profile=custom-1.5 hash=39ae5e0c
=== RUN TestAdjacentReleaseHash/official-1.5
zz_adjacent_hash_test.go:59: release=1.5.0 profile=official-1.5 hash=37070ad2
=== RUN TestAdjacentReleaseHash/official-1.4
zz_adjacent_hash_test.go:59: release=1.5.0 profile=official-1.4 hash=3cf136a2
--- PASS: TestAdjacentReleaseHash (0.01s)
--- PASS: TestAdjacentReleaseHash/custom-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.5 (0.00s)
--- PASS: TestAdjacentReleaseHash/official-1.4 (0.00s)
PASS
ok github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo 0.689s
Raw
hash-summary.json
{
"revisions": {
"1.2.0": "30b7c5c38a5fa0fb2acecd66086de7963eee0e35",
"1.3.0": "8ce9e22f11576402102ea9d8b8e46233f5430a0d",
"1.4.0": "03014943323e78feb5bd672ef08b72caea0918ac",
"1.5.0": "b83b1d9304ebfc624709ac46db32b1b6f1ff1615"
},
"cases": {
"custom-1.5": {
"yamlSHA256": "e9a98683e2ac7ef7efd3b347f0ed583db9088d012073ac26642540deecd0ebf3",
"sameDGDDecodedAcrossAllVersions": true,
"hashes": {
"1.2.0": "309e9927",
"1.3.0": "309e9927",
"1.4.0": "309e9927",
"1.5.0": "142dbdff"
},
"transitions": [
{
"from": "1.2.0",
"to": "1.3.0",
"oldHash": "309e9927",
"newHash": "309e9927",
"inputDiff": []
},
{
"from": "1.3.0",
"to": "1.4.0",
"oldHash": "309e9927",
"newHash": "309e9927",
"inputDiff": []
},
{
"from": "1.4.0",
"to": "1.5.0",
"oldHash": "309e9927",
"newHash": "142dbdff",
"inputDiff": [
{
"path": "/SglangDecode/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
},
{
"path": "/SglangPrefill/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
}
]
}
]
},
"official-1.5": {
"yamlSHA256": "0cd3349053ab805aa960d31cb2e271b841caaec0a58bbed36711e722e4b25f9c",
"sameDGDDecodedAcrossAllVersions": true,
"hashes": {
"1.2.0": "53549949",
"1.3.0": "53549949",
"1.4.0": "53549949",
"1.5.0": "e332eabb"
},
"transitions": [
{
"from": "1.2.0",
"to": "1.3.0",
"oldHash": "53549949",
"newHash": "53549949",
"inputDiff": []
},
{
"from": "1.3.0",
"to": "1.4.0",
"oldHash": "53549949",
"newHash": "53549949",
"inputDiff": []
},
{
"from": "1.4.0",
"to": "1.5.0",
"oldHash": "53549949",
"newHash": "e332eabb",
"inputDiff": [
{
"path": "/SglangDecode/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
},
{
"path": "/SglangPrefill/runtimeVersion",
"before": "<absent>",
"after": "1.5.0"
}
]
}
]
},
"official-1.4": {
"yamlSHA256": "b7c89360a9bd24e0c1e2d437368cab70ac89b9f514863799d7b70646bbdfaa40",
"sameDGDDecodedAcrossAllVersions": true,
"hashes": {
"1.2.0": "7be83b1f",
"1.3.0": "7be83b1f",
"1.4.0": "7be83b1f",
"1.5.0": "7be83b1f"
},
"transitions": [
{
"from": "1.2.0",
"to": "1.3.0",
"oldHash": "7be83b1f",
"newHash": "7be83b1f",
"inputDiff": []
},
{
"from": "1.3.0",
"to": "1.4.0",
"oldHash": "7be83b1f",
"newHash": "7be83b1f",
"inputDiff": []
},
{
"from": "1.4.0",
"to": "1.5.0",
"oldHash": "7be83b1f",
"newHash": "7be83b1f",
"inputDiff": []
}
]
}
}
}
Raw
hash_test.go.tmpl
package dynamo
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"testing"
api "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1"
"github.com/stretchr/testify/require"
"sigs.k8s.io/yaml"
)
func TestAdjacentReleaseHash(t *testing.T) {
root := os.Getenv("ADJACENT_ROOT")
release := os.Getenv("ADJACENT_RELEASE")
for _, profile := range []string{"custom-1.5", "official-1.5", "official-1.4"} {
t.Run(profile, func(t *testing.T) {
data, err := os.ReadFile(filepath.Join(root, profile+".yaml"))
require.NoError(t, err)
var dgd api.DynamoGraphDeployment
require.NoError(t, yaml.UnmarshalStrict(data, &dgd))
original := dgd.DeepCopy()
hash, err := ComputeDGDWorkersSpecHash(&dgd)
require.NoError(t, err)
dcds, err := GenerateDynamoComponentsDeployments(&dgd, nil, nil, RollingUpdateContext{NewWorkerHash:dgdWorkerHashPlaceholderValue})
require.NoError(t, err)
type workerTemplate struct {
Labels map[string]string `json:"labels,omitempty"`
Annotations map[string]string `json:"annotations,omitempty"`
RuntimeVersion string `json:"runtimeVersion,omitempty"`
Spec api.DynamoComponentDeploymentSpec `json:"spec"`
}
inputs := map[string]workerTemplate{}
for _, dcd := range dcds {
if dcd == nil || !IsWorkerComponent(string(dcd.Spec.ComponentType)) { continue }
version := ""
// RUNTIME_VERSION
inputs[GetDCDComponentName(dcd)] = workerTemplate{
Labels:GetDCDKubeLabels(dcd), Annotations:GetDCDKubeAnnotations(dcd),
RuntimeVersion:version, Spec:workerHashSpec(dcd),
}
}
require.Len(t, inputs, 2)
canonical, err := json.Marshal(inputs)
require.NoError(t, err)
sum := sha256.Sum256(canonical)
require.Equal(t, hash, hex.EncodeToString(sum[:])[:8])
require.Equal(t, original, &dgd)
output, err := json.MarshalIndent(map[string]any{"release":release, "hash":hash, "dgd":dgd, "hashInput":inputs},""," ")
require.NoError(t, err)
out := filepath.Join(root, "hash-results", release)
require.NoError(t, os.MkdirAll(out,0755))
require.NoError(t, os.WriteFile(filepath.Join(out,profile+".json"),output,0644))
t.Logf("release=%s profile=%s hash=%s",release,profile,hash)
})
}
}
Raw
normalized-hash-inputs.json
{
"identicalYAML": {
"1.2.0": {
"custom-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "309e9927",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.2.0"
},
"official-1.4": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "7be83b1f",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.2.0"
},
"official-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "53549949",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.2.0"
}
},
"1.3.0": {
"custom-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "309e9927",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.3.0"
},
"official-1.4": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "7be83b1f",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.3.0"
},
"official-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "53549949",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.3.0"
}
},
"1.4.0": {
"custom-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "309e9927",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.4.0"
},
"official-1.4": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "7be83b1f",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.4.0"
},
"official-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "53549949",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.4.0"
}
},
"1.5.0": {
"custom-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "142dbdff",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.5.0"
},
"official-1.4": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "7be83b1f",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.5.0"
},
"official-1.5": {
"dgd": {
"apiVersion": "nvidia.com/v1beta1",
"kind": "DynamoGraphDeployment",
"metadata": {
"name": "hash-upgrade-replay",
"namespace": "default"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"state": ""
}
},
"hash": "e332eabb",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "hash-upgrade-replay",
"nvidia.com/dynamo-namespace": "default-hash-upgrade-replay",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.5.0"
}
}
},
"actualStoredDGD": {
"1.4.0": {
"custom-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "e2fff36a",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-custom-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "282",
"uid": "92e2d420-94bd-4ab4-b610-c6c39924967f"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-custom-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"hash": "e2fff36a",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.4.0"
},
"official-1.4": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "3cf136a2",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-official-1-4",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "330",
"uid": "86a9f489-d9fc-4612-804d-977261c1df0b"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-4: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"hash": "3cf136a2",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.4.0"
},
"official-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "90f01928",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-official-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "306",
"uid": "817618d5-15c8-4d37-896c-565ee53d661d"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"hash": "90f01928",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.4.0"
}
},
"1.5.0": {
"custom-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "e2fff36a",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-custom-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "282",
"uid": "92e2d420-94bd-4ab4-b610-c6c39924967f"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-custom-1-5-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-custom-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"hash": "39ae5e0c",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-custom-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-custom-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "registry.example.com/dynamo/sglang-runtime:v1.5.0-9aa3-repro",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.5.0"
},
"official-1.4": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "3cf136a2",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-official-1-4",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "330",
"uid": "86a9f489-d9fc-4612-804d-977261c1df0b"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-4-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-4: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"hash": "3cf136a2",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-4",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-4",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.5.0"
},
"official-1.5": {
"dgd": {
"metadata": {
"annotations": {
"nvidia.com/current-worker-hash-v2": "90f01928",
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
},
"creationTimestamp": "2026-10-17T02:33:21Z",
"finalizers": [
"nvidia.com/finalizer"
],
"generation": 1,
"managedFields": [
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:nvidia.com/current-worker-hash-v2": {}
},
"f:finalizers": {
".": {},
"v:\"nvidia.com/finalizer\"": {}
}
},
"f:spec": {
".": {},
"f:backendFramework": {},
"f:components": {
".": {},
"k:{\"name\":\"SglangDecode\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
},
"k:{\"name\":\"SglangPrefill\"}": {
".": {},
"f:name": {},
"f:podTemplate": {
".": {},
"f:spec": {
".": {},
"f:containers": {
".": {},
"k:{\"name\":\"main\"}": {
".": {},
"f:args": {},
"f:command": {},
"f:image": {},
"f:name": {}
}
}
}
},
"f:replicas": {},
"f:type": {}
}
}
}
},
"manager": "controller.test",
"operation": "Update",
"time": "2026-10-17T02:33:21Z"
},
{
"apiVersion": "nvidia.com/v1beta1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:components": {
".": {},
"f:SglangDecode": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
},
"f:SglangPrefill": {
".": {},
"f:componentKind": {},
"f:componentNames": {},
"f:replicas": {},
"f:runtimeNamespace": {},
"f:updatedReplicas": {}
}
},
"f:conditions": {
".": {},
"k:{\"type\":\"Ready\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
},
"f:observedGeneration": {},
"f:state": {}
}
},
"manager": "controller.test",
"operation": "Update",
"subresource": "status",
"time": "2026-10-17T02:33:21Z"
}
],
"name": "adjacent-official-1-5",
"namespace": "operatorenv-zkk2r5v9",
"resourceVersion": "306",
"uid": "817618d5-15c8-4d37-896c-565ee53d661d"
},
"spec": {
"backendFramework": "sglang",
"components": [
{
"minAvailable": 1,
"name": "SglangPrefill",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "prefill"
},
{
"minAvailable": 1,
"name": "SglangDecode",
"podTemplate": {
"spec": {
"containers": [
{
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"name": "main"
}
]
}
},
"replicas": 2,
"type": "decode"
}
]
},
"status": {
"components": {
"SglangDecode": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangdecode"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
},
"SglangPrefill": {
"componentKind": "PodClique",
"componentNames": [
"adjacent-official-1-5-0-sglangprefill"
],
"replicas": 0,
"runtimeNamespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"updatedReplicas": 0
}
},
"conditions": [
{
"lastTransitionTime": "2026-10-17T02:33:21Z",
"message": "Resources not ready: adjacent-official-1-5: SglangPrefill: resource not found; SglangDecode: resource not found",
"reason": "some_resources_are_not_ready",
"status": "False",
"type": "Ready"
}
],
"observedGeneration": 1,
"state": "pending"
}
},
"hash": "37070ad2",
"hashInput": {
"SglangDecode": {
"labels": {
"nvidia.com/dynamo-component": "SglangDecode",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "decode",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangDecode",
"type": "decode",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"decode"
],
"resources": {}
}
]
}
}
}
},
"SglangPrefill": {
"labels": {
"nvidia.com/dynamo-component": "SglangPrefill",
"nvidia.com/dynamo-graph-deployment-name": "adjacent-official-1-5",
"nvidia.com/dynamo-namespace": "operatorenv-zkk2r5v9-adjacent-official-1-5",
"nvidia.com/dynamo-sub-component-type": "prefill",
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"runtimeVersion": "1.5.0",
"spec": {
"backendFramework": "sglang",
"name": "SglangPrefill",
"type": "prefill",
"podTemplate": {
"metadata": {
"labels": {
"nvidia.com/dynamo-worker-hash": "worker-hash-placeholder"
},
"annotations": {
"nvidia.com/dynamo-operator-origin-version": "1.4.0"
}
},
"spec": {
"containers": [
{
"name": "main",
"image": "nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0",
"command": [
"python3",
"-m",
"dynamo.sglang"
],
"args": [
"--model-path",
"Qwen/Qwen3-0.6B",
"--disaggregation-mode",
"prefill"
],
"resources": {}
}
]
}
}
}
}
},
"release": "1.5.0"
}
}
}
}
Raw
official-1.4.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# The generator substitutes only the catalog's image before the old operator runs.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: hash-upgrade-replay
namespace: default
spec:
backendFramework: sglang
components:
- name: SglangPrefill
type: prefill
replicas: 2
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0
command: [python3, -m, dynamo.sglang]
args: [--model-path, Qwen/Qwen3-0.6B, --disaggregation-mode, prefill]
- name: SglangDecode
type: decode
replicas: 2
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.4.0
command: [python3, -m, dynamo.sglang]
args: [--model-path, Qwen/Qwen3-0.6B, --disaggregation-mode, decode]
Raw
official-1.5.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# The generator substitutes only the catalog's image before the old operator runs.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: hash-upgrade-replay
namespace: default
spec:
backendFramework: sglang
components:
- name: SglangPrefill
type: prefill
replicas: 2
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
command: [python3, -m, dynamo.sglang]
args: [--model-path, Qwen/Qwen3-0.6B, --disaggregation-mode, prefill]
- name: SglangDecode
type: decode
replicas: 2
podTemplate:
spec:
containers:
- name: main
image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
command: [python3, -m, dynamo.sglang]
args: [--model-path, Qwen/Qwen3-0.6B, --disaggregation-mode, decode]
Raw
prepare_sources.py
#!/usr/bin/env python3
"""Export the pinned releases and install their own Grove CRDs for the audit."""
import argparse
import io
import json
import os
from pathlib import Path
import shutil
import subprocess
import tarfile
ROOT = Path(__file__).resolve().parent
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repo", type=Path, required=True, help="Existing ai-dynamo/dynamo Git clone")
args = parser.parse_args()
env = dict(os.environ, GOTOOLCHAIN="go1.26.8")
for version, sha in json.loads((ROOT / "revisions.json").read_text()).items():
present = subprocess.run(["git", "cat-file", "-e", sha + "^{commit}"], cwd=args.repo,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if present.returncode:
subprocess.run(["git", "fetch", "--no-tags", "https://github.com/ai-dynamo/dynamo.git", sha],
cwd=args.repo, check=True)
destination = ROOT / "source" / version
destination.mkdir(parents=True, exist_ok=True)
paths = ["deploy/operator", "deploy/helm"]
if version != "1.5.0":
paths.append("deploy/snapshot")
data = subprocess.check_output(["git", "archive", sha, *paths], cwd=args.repo)
with tarfile.open(fileobj=io.BytesIO(data)) as archive:
archive.extractall(destination, filter="data")
if version in ("1.4.0", "1.5.0"):
operator = destination / "deploy/operator"
subprocess.run(["go", "mod", "download", "github.com/ai-dynamo/grove/operator/api"],
cwd=operator, env=env, check=True)
module = json.loads(subprocess.check_output(
["go", "list", "-m", "-json", "github.com/ai-dynamo/grove/operator/api"],
cwd=operator, env=env))
crds = operator / "internal/controller/testing/grove.io"
crds.mkdir(parents=True, exist_ok=True)
for path in (Path(module["Dir"]) / "core/v1alpha1/crds").glob("*.yaml"):
shutil.copyfile(path, crds / path.name)
print(version, sha)
Raw
revisions.json
{
"1.2.0": "30b7c5c38a5fa0fb2acecd66086de7963eee0e35",
"1.3.0": "8ce9e22f11576402102ea9d8b8e46233f5430a0d",
"1.4.0": "03014943323e78feb5bd672ef08b72caea0918ac",
"1.5.0": "b83b1d9304ebfc624709ac46db32b1b6f1ff1615"
}
Raw
run_api.py
#!/usr/bin/env python3
"""Run production admission/controllers across one persistent local envtest API server."""
import os
from pathlib import Path
import subprocess
ROOT = Path(__file__).resolve().parent
for version in ["1.4.0", "1.5.0"]:
source = (ROOT / "api_test.go.tmpl").read_text()
plan = ""
scale = ""
if version == "1.5.0":
plan = "rollout:=newDGDWorkerRolloutReconciler(c, nil); transition,err:=rollout.planUnsupportedWorkerHashTransition(&current); require.NoError(t,err); hashChanged=transition.hashChanged"
else:
scale = "scaleClient,err:=env.ScaleClient();require.NoError(t,err);dgdOpts.ScaleClient=scaleClient"
path = ROOT / "source" / version / "deploy/operator/internal/controller/zz_adjacent_api_test.go"
path.write_text(source.replace("// PLAN_TRANSITION", plan).replace("// SCALE_CLIENT", scale))
subprocess.run(["gofmt", "-w", str(path)], check=True)
env = dict(os.environ, GOTOOLCHAIN="go1.26.8", ADJACENT_ROOT=str(ROOT),
ADJACENT_RELEASE="1.4.0", ADJACENT_STAGE="source",
KUBEBUILDER_ASSETS=os.environ["KUBEBUILDER_ASSETS"],
USE_EXISTING_CLUSTER="false")
with (ROOT / "api-1.4.log").open("w") as log:
result = subprocess.run(["go", "test", "-p", "2", "./internal/controller",
"-run", "^TestAdjacentAPI$", "-count=1", "-v", "-timeout", "8m"],
cwd=ROOT / "source/1.4.0/deploy/operator", env=env,
stdout=log, stderr=subprocess.STDOUT)
print("Full adjacent API-server handoff exit:", result.returncode)
for name in ["api-1.4.log", "api-1.5.log"]:
path = ROOT / name
if path.exists():
print(path.read_text()[-9000:])
raise SystemExit(result.returncode)
Raw
run_hashes.py
#!/usr/bin/env python3
"""Compare unchanged DGD hash inputs with each pinned release implementation."""
import json
import os
from pathlib import Path
import subprocess
ROOT = Path(__file__).resolve().parent
for version in json.loads((ROOT / "revisions.json").read_text()):
operator = ROOT / "source" / version / "deploy/operator"
template = (ROOT / "hash_test.go.tmpl").read_text()
expression = "version = resolvedRuntimeVersionForHash(&dcd.Spec.DynamoComponentDeploymentSharedSpec)" if version == "1.5.0" else ""
test = operator / "internal/dynamo/zz_adjacent_hash_test.go"
test.write_text(template.replace("// RUNTIME_VERSION", expression))
subprocess.run(["gofmt", "-w", str(test)], check=True)
env = dict(os.environ, GOTOOLCHAIN="go1.26.8", ADJACENT_ROOT=str(ROOT), ADJACENT_RELEASE=version)
with (ROOT / f"hash-{version}.log").open("w") as output:
result = subprocess.run(["go", "test", "-p", "2", "./internal/dynamo",
"-run", "^TestAdjacentReleaseHash$", "-count=1", "-v"],
cwd=operator, env=env, stdout=output, stderr=subprocess.STDOUT)
print((ROOT / f"hash-{version}.log").read_text(), flush=True)
if result.returncode:
raise SystemExit(result.returncode)
Raw
SHA256SUMS
37e1200d2a71d5f10cc3ac8b8888b2ab3aba49ce72d8cc16eb0e41a29a800f02 README.md
18ee7f0f7c58279174d4d96eac13e8cb6a53b097ae36e489a4abe01c0f78ff8d api-1.4.log
215d44313f8cab3122c2aa54212dc7d5f45250665bce7afb9ad08e4b439acf72 api-1.5.log
c190771b31f48eb18342920f9a4446432bbad512dc0d4915c0c234258865338c api-after.json
345c1b4e37a95c257fa054748bbf0923e28a9dde31790c6a0c7b3d7fac6acd89 api-before.json
ea6f26ee96ef317d9f01e10432217778f2e254d5969c621c39841408ac6f2559 api-decisions.json
f4f6879a95b6ceea78db9050386964700daf5d2092ff1a998bf044339eaba665 api-summary.json
a845ab848555693b494363f228693f8a0b76bfc88931f604b87704750d39d70d api_test.go.tmpl
e9a98683e2ac7ef7efd3b347f0ed583db9088d012073ac26642540deecd0ebf3 custom-1.5.yaml
479cabb87af2ccff4274bfc500cd5472b3b198e819201f420d5b7a239d2eed4d grove-crd-provenance.json
0b7af2512c4d30265da809ecb24417dc4b5ee73ffa73d5efe989fcd3ccf4c765 hash-logs.txt
5561f3a76e768e6239006fa9039fff5f49880b2259abebc0f80ce00d30b45b66 hash-summary.json
d7406366215c4bc560b7c090e24bc531fdcaa683bfcc69ae7dcacc72e6485ba2 hash_test.go.tmpl
7d4c18833c108308477c275a84f58d4da0c6e22db20d37600c815c4191415a46 normalized-hash-inputs.json
b7c89360a9bd24e0c1e2d437368cab70ac89b9f514863799d7b70646bbdfaa40 official-1.4.yaml
0cd3349053ab805aa960d31cb2e271b841caaec0a58bbed36711e722e4b25f9c official-1.5.yaml
75c14e0da096b0c82e95c6d46f80af6eb42c5a4d6fb14d2f005fd8acb78d5311 prepare_sources.py
d4e759ff0fae9c5f1f15284b9790994a0dc032b6824f3032b910d2807e615c58 revisions.json
c1c67dee007edc7651cd9f857e7a2fc61da06f6b3b4003de7454b111342da1f0 run_api.py
8391ca4f26b8b7854b0171f199c16809082ff09419fbb121ac5d8863f86957ef run_hashes.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment