Run Hermes/OpenClaw/any CLI agent on GKE with Agent Substrate + AX — scripted, digest-pinned, verification-gated playbook

README.md

Agents on GKE with Agent Substrate + AX

Layout: this gist is flat. Recreate the tree by replacing __ with / in filenames (e.g. scripts__install.sh -> k8s/scripts/install.sh):

mkdir -p k8s && for f in *; do [ "$f" = README.md ] && d=k8s/README.md || d="k8s/${f//__//}"; mkdir -p "$(dirname "$d")"; cp "$f" "$d"; done; chmod +x k8s/scripts/*.sh

Run your own agents (Hermes, OpenClaw, or any CLI agent) as AX Tasks on Agent Substrate on GKE: gVisor-sandboxed, suspend-to-GCS, resume on any node. Always-on services (LLM gateway, databases) stay plain Kubernetes workloads next to them.

Everything here is scripted, idempotent, and pinned by digest. install.sh ends with verification gates. If any gate fails, the install fails.

What you get

Layer	What runs	Durable state
GKE 1.36, gVisor node pool (nested virt)	created by Substrate's tools/setup-gcp	—
Agent Substrate v0.5.0	ate-system: api-server, controller, atelet, atenet router/egress, credential provider, postgres	Postgres PVC; actor snapshots in GCS
AX 0f085aa (the pair AX's go.mod pins)	ax-system: ax-server, redis, WorkerPool	Redis StatefulSet + PVC (AOF)
Your agents	AX Tasks from your own image	/workspace volume, snapshotted to GCS on suspend

Prerequisites

• A GCP project where you can create GKE clusters, GCS buckets, Artifact Registry repos, and IAM bindings (roles/owner on a sandbox project is simplest).
• A Linux build host (not your laptop's distro Docker) with:
  • docker-ce + docker-buildx-plugin from download.docker.com. Distro docker.io lacks buildx, and the Substrate image build fails with exit 125.
  • Go ≥ 1.27.1, git, make, gettext-base (for envsubst), gcloud, kubectl, gke-gcloud-auth-plugin.
  • On Ubuntu 24.04 that is one apt line plus a Go tarball (see "Build host" below).

Run it

cp k8s/env.example k8s/env.local      # set PROJECT_ID, CLUSTER_NAME, region
./k8s/scripts/install.sh               # ~25 min first run

Or run the steps one at a time. Each one is safe to re-run:

Step	Does
00-prereqs.sh	checks tools; clones Substrate + AX at the pinned refs
10-cluster.sh	APIs, AR repo, cluster + bucket + Workload Identity (setup-gcp bootstrap); checks node kernel ≥ 6.8 and that the beta cert APIs are served
20-substrate.sh	builds Substrate images into your AR and installs ate-system
30-images.sh	builds AX + agent images, mirrors third-party images into AR, writes k8s/images.lock (digests)
40-ax.sh	deploys AX with durable Redis and narrowed RBAC
50-verify.sh	gates (below)
99-teardown.sh	deletes cluster, bucket, repo (asks first)

Verification gates (50-verify.sh)

1. AX upstream e2e suite (8 tests) against your cluster: task lifecycle, guest exec, workspace git/files, suspend/resume persistence, atespace isolation.
2. Durability: a task writes 20 MB to /workspace and is suspended. Then every worker pod is deleted. On resume the task lands on a new worker, restored from GCS, and the sha256 must match.
3. Control-plane durability: the Redis pod is deleted and ax-server restarted. Task records must survive.
4. Agent images: Hermes and OpenClaw tasks start in the sandbox and run their CLI.

Your own agent image

AX always starts /usr/local/bin/ax-task-runner as PID 1. The runner prepares /workspace, runs spec.command, and serves health and guest-exec. So an agent image is your base + the runner binary:

ARG RUNNER_IMAGE                       # AX_DEFAULT_TASK_IMAGE from images.lock
FROM ${RUNNER_IMAGE} AS runner
FROM nousresearch/hermes-agent:v0.22.5@sha256:...   # or ghcr.io/openclaw/openclaw:<ver>, or your own
COPY --from=runner /usr/local/bin/ax-task-runner /usr/local/bin/ax-task-runner
ENV HERMES_HOME=/workspace/.hermes     # keep agent state on the durable volume
ENV HERMES_DOCKER_EXEC_AS_ROOT=1       # hermes only: skip the s6 privilege drop (fails in gVisor)
WORKDIR /workspace
ENTRYPOINT ["/usr/local/bin/ax-task-runner"]

The image needs git and a POSIX shell. Ready-made ones are in k8s/agents/{hermes,openclaw,custom}. Add yours under k8s/agents/<name>/ and build it with AGENTS="hermes openclaw <name>" ./k8s/scripts/30-images.sh.

Run it:

# task.yaml
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata: {name: my-ws, atespace: default}
spec:
  git: [{repo: https://github.com/you/your-repo.git}]
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata: {name: my-agent, atespace: default}
spec:
  image: "<AGENT_..._IMAGE from images.lock>"
  command: ["hermes", "-z", "summarize README.md"]
  workspace: {name: my-ws, path: /workspace}
  debug: true             # enables `ax ssh`; turn off for unattended tasks

ax apply -f task.yaml
ax resume task my-agent
ax ssh my-agent -- ls /workspace
ax suspend task my-agent   # snapshot /workspace to GCS, free the worker
ax resume task my-agent    # restore on any worker

Secrets

• Never put secrets in spec.env. They are stored in AX's Redis and baked into the per-task ActorTemplate.
• Preferred: Substrate egress credential injection. The agent sends a placeholder Authorization header. The egress gateway swaps in the real value from a Kubernetes Secret (or Secret Manager). The secret never enters the sandbox or a snapshot. See Substrate docs/egress-credential-injection.md. Grant each atespace namespace access in the k8s-credential-provider-namespace-policy ConfigMap (default-deny).
• Simplest: an in-cluster LLM gateway (e.g. LiteLLM as a Deployment) holding the provider keys. Agents get only a scoped gateway key.

Known limits (as of AX 0f085aa / Substrate v0.5.0)

Limit	Impact	What we do
AX tasks snapshot with VOLUMES fidelity	/workspace survives suspend; processes and /tmp do not	keep agent state under /workspace; agents must cold-start cleanly
AX hardcodes the gVisor sandbox class	no microVM via AX	fine for CLI agents
AX's default snapshot bucket points at someone else's bucket	snapshots fail or leak if unset	AX_SNAPSHOTS_BUCKET is always set by 40-ax.sh
AX's e2e Redis is an ephemeral Deployment	lose the pod, lose all task records	replaced with StatefulSet + PVC + AOF
AX's ClusterRole reads all Secrets	over-broad	narrowed to get on gemini-api-secret
A freshly started worker needs ~30 s before it accepts actors	resume right after a worker restart returns ResourceExhausted: no worker has room	keep ≥ 1 spare worker; retry resume (scripts do)
Upstream Hermes image's hermes shim drops root via s6-setuidgid	gVisor forbids setgroups: s6-applyuidgid: unable to set supplementary group list	k8s/agents/hermes sets HERMES_DOCKER_EXEC_AS_ROOT=1 (no s6 tree under AX; the sandbox is the boundary)
Node kernel must be ≥ 6.8 (undocumented)	on 6.1, every resume fails: creating overlay superblock: invalid argument	10-cluster.sh checks it
GKE 1.36 beta cert APIs are create-time only	install hangs at "Waiting for ClusterTrustBundles"	let setup-gcp create the cluster
Docker's containerd image store reports the upstream index digest in RepoDigests for a re-tagged multi-arch image, but docker push uploads only the local platform	pinning that digest -> ImagePullBackOff: not found	digests are read back from the registry (imagetools inspect); third-party images are mirrored registry-to-registry (imagetools create, digest identical to upstream)
Neither project publishes images	—	build into your AR, pin digests (images.lock)

Build host (Ubuntu 24.04)

sudo install -m0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc >/dev/null
echo "deb [signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" | sudo tee /etc/apt/sources.list.d/docker.list
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin \
  git make gettext-base jq build-essential google-cloud-cli google-cloud-cli-gke-gcloud-auth-plugin kubectl
curl -fsSL https://go.dev/dl/go1.27.1.linux-amd64.tar.gz | sudo tar -C /usr/local -xz && export PATH=/usr/local/go/bin:$PATH

agents__custom__Dockerfile

# Template: any agent CLI as an AX task. Needs git + a shell in the image.
ARG RUNNER_IMAGE
FROM ${RUNNER_IMAGE} AS runner
FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends git curl ca-certificates bash \
    && rm -rf /var/lib/apt/lists/*
# RUN pip install --no-cache-dir my-agent   # <- your agent here
COPY --from=runner /usr/local/bin/ax-task-runner /usr/local/bin/ax-task-runner
WORKDIR /workspace
ENTRYPOINT ["/usr/local/bin/ax-task-runner"]

agents__hermes__Dockerfile

# Hermes Agent as an AX task / Substrate actor.
# Upstream image + AX's runner as PID 1 (AX always starts /usr/local/bin/ax-task-runner).
# Build: k8s/scripts/30-images.sh (pass RUNNER_IMAGE + BASE_IMAGE as build args).
ARG RUNNER_IMAGE
ARG BASE_IMAGE=nousresearch/hermes-agent:v0.22.5@sha256:9c5561579f3ffdf333401797094ac3a743738d6c25d40f4331c7bf77b694238d
FROM ${RUNNER_IMAGE} AS runner
FROM ${BASE_IMAGE}
COPY --from=runner /usr/local/bin/ax-task-runner /usr/local/bin/ax-task-runner
# Durable state lives on the AX /workspace volume (snapshotted on suspend).
# HERMES_DOCKER_EXEC_AS_ROOT: upstream's /opt/hermes/bin/hermes shim drops root to
# the `hermes` user via s6-setuidgid so files stay uid-aligned with its s6-supervised
# gateway. Here ax-task-runner is PID 1 (no s6 tree, no gateway), everything runs as
# root inside the gVisor sandbox, and the sandbox forbids setgroups(2) — so the drop
# fails with "s6-applyuidgid: unable to set supplementary group list". Opt out.
ENV HERMES_HOME=/workspace/.hermes \
    HERMES_WRITE_SAFE_ROOT=/workspace \
    HERMES_DOCKER_EXEC_AS_ROOT=1 \
    HOME=/workspace
WORKDIR /workspace
ENTRYPOINT ["/usr/local/bin/ax-task-runner"]

agents__openclaw__Dockerfile

# OpenClaw as an AX task / Substrate actor.
# Upstream image + AX's runner as PID 1 (AX always starts /usr/local/bin/ax-task-runner).
ARG RUNNER_IMAGE
ARG BASE_IMAGE=ghcr.io/openclaw/openclaw:2026.10.7@sha256:fb26f0841659dc757f23feec9adeab51f11d76779a409db5d22e191b43841afa
FROM ${RUNNER_IMAGE} AS runner
FROM ${BASE_IMAGE}
USER root
COPY --from=runner /usr/local/bin/ax-task-runner /usr/local/bin/ax-task-runner
# Durable state lives on the AX /workspace volume (snapshotted on suspend).
ENV OPENCLAW_STATE_DIR=/workspace/.openclaw \
    HOME=/workspace
WORKDIR /workspace
ENTRYPOINT ["/usr/local/bin/ax-task-runner"]

env.example

# k8s/env.example — copy to k8s/env.local (gitignored) and fill in.
# Every script sources this file. No secrets belong here.

# --- GCP ---
export PROJECT_ID=my-gcp-project
export PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)' 2>/dev/null)
export GCE_REGION=us-central1
export CLUSTER_LOCATION=us-central1-a
export NETWORK=default
export SUBNETWORK=default

# --- Cluster (Substrate needs GKE >= 1.36 and node kernel >= 6.8) ---
export CLUSTER_NAME=ax-cluster
# Pick a 1.36.x from: gcloud container get-server-config --zone $CLUSTER_LOCATION
export CLUSTER_VERSION=1.36.4-gke.2046003
export NODE_POOL_NAME=gvisor-pool
export NODE_POOL_VERSION=$CLUSTER_VERSION
export NODE_MACHINE_TYPE=c3-standard-8
export ENABLE_NESTED_VIRTUALIZATION=true
export KUBE_CONTEXT=gke_${PROJECT_ID}_${CLUSTER_LOCATION}_${CLUSTER_NAME}

# --- Storage + images (both created by 10-cluster.sh) ---
export BUCKET_NAME=${CLUSTER_NAME}-snapshots-${PROJECT_ID}
export AR_REPO=${CLUSTER_NAME}
export REGISTRY=${GCE_REGION}-docker.pkg.dev/${PROJECT_ID}/${AR_REPO}
export KO_DOCKER_REPO=$REGISTRY
export KO_DEFAULTPLATFORMS=linux/amd64

# --- Pinned, tested pair (AX go.mod pins this Substrate tag) ---
export SUBSTRATE_REF=v0.5.0           # e8d40c33f96f4b7a4984bfea7334a36ab5b45b7b
export AX_REF=0f085aa063db6de888e37def7a8943f7901e6c1e
export SRC_DIR=${SRC_DIR:-$HOME/src}  # where substrate/ and ax/ get cloned

# --- Substrate egress credential injection ---
export ATE_CREDENTIAL_PROVIDER='{"name":"k8s.io"}'

# --- AX ---
export AX_SYSTEM_NAMESPACE=ax-system
export AX_WORKER_REPLICAS=3           # keep >= 1 spare: a fresh worker needs ~30s warmup
export AX_SNAPSHOTS_BUCKET=gs://${BUCKET_NAME}/ax/
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

manifests__ax__ax-server.yaml

# ax-server. Derived from google/ax test/e2e/manifests/ax-server.yaml (Apache-2.0).
# Changes: secrets RBAC narrowed; images pinned by digest via envsubst.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ax-server
  namespace: ${AX_SYSTEM_NAMESPACE}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ax-server
rules:
  # Narrowed from AX's default (get/list/watch on ALL secrets): AX only GETs
  # the per-atespace Gemini key secret by name.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["gemini-api-secret"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ax-server
subjects:
  - kind: ServiceAccount
    name: ax-server
    namespace: ${AX_SYSTEM_NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ax-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ax-server
  namespace: ${AX_SYSTEM_NAMESPACE}
  labels:
    app.kubernetes.io/name: ax-server
    app.kubernetes.io/part-of: ax
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ax-server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ax-server
    spec:
      serviceAccountName: ax-server
      containers:
        - name: ax-server
          image: ${AX_SERVER_IMAGE}
          imagePullPolicy: IfNotPresent
          args:
            - "--addr=:8080"
            - "--redis-addr=ax-redis.${AX_SYSTEM_NAMESPACE}.svc.cluster.local:6379"
            - "--substrate-endpoint=api.ate-system.svc.cluster.local:443"
            - "--substrate-authority=api.ate-system.svc"
            - "--substrate-token-file=/var/run/secrets/ateapi/token"
            - "--substrate-ca-file=/run/servicedns-ca/trust-bundle.pem"
            - "--template=default-template"
            - "--template-atespace=${AX_SYSTEM_NAMESPACE}"
          env:
            - name: ATENET_ROUTER_ADDR
              value: "atenet-router.ate-system.svc.cluster.local:80"
            - name: AX_SNAPSHOTS_BUCKET
              value: "${AX_SNAPSHOTS_BUCKET}"
            - name: AX_DEFAULT_TASK_IMAGE
              value: "${AX_DEFAULT_TASK_IMAGE}"
          ports:
            - containerPort: 8080
              name: http
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 2
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          volumeMounts:
            - mountPath: /var/run/secrets/ateapi
              name: ate-token
              readOnly: true
            - mountPath: /run/servicedns-ca
              name: servicedns-ca
              readOnly: true
      volumes:
        - name: ate-token
          projected:
            defaultMode: 420
            sources:
              - serviceAccountToken:
                  audience: api.ate-system.svc
                  expirationSeconds: 7200
                  path: token
        - name: servicedns-ca
          projected:
            defaultMode: 420
            sources:
              - clusterTrustBundle:
                  labelSelector:
                    matchLabels:
                      podcert.ate.dev/canarying: live
                  path: trust-bundle.pem
                  signerName: servicedns.podcert.ate.dev/identity
---
apiVersion: v1
kind: Service
metadata:
  name: ax-server
  namespace: ${AX_SYSTEM_NAMESPACE}
  labels:
    app.kubernetes.io/name: ax-server
    app.kubernetes.io/part-of: ax
spec:
  ports:
    - port: 8080
      targetPort: 8080
      name: http
  selector:
    app.kubernetes.io/name: ax-server

manifests__ax__namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ${AX_SYSTEM_NAMESPACE}
  labels:
    ate.dev/api-access: "true"

manifests__ax__redis.yaml

# AX state store. AX keeps Tasks/Workspaces/Models in Redis, so it must be
# durable: StatefulSet + PVC + AOF. (AX's e2e manifest uses an ephemeral
# Deployment; losing that pod loses every Task record.)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ax-redis
  namespace: ${AX_SYSTEM_NAMESPACE}
  labels: {app.kubernetes.io/name: ax-redis, app.kubernetes.io/part-of: ax}
spec:
  serviceName: ax-redis
  replicas: 1
  selector:
    matchLabels: {app.kubernetes.io/name: ax-redis}
  template:
    metadata:
      labels: {app.kubernetes.io/name: ax-redis, app.kubernetes.io/part-of: ax}
    spec:
      securityContext: {fsGroup: 999, runAsUser: 999, runAsNonRoot: true}
      containers:
      - name: redis
        image: ${REDIS_IMAGE}
        args: ["--appendonly", "yes", "--appendfsync", "everysec", "--dir", "/data"]
        ports: [{containerPort: 6379, name: redis}]
        readinessProbe: {exec: {command: ["redis-cli", "ping"]}, periodSeconds: 5}
        livenessProbe: {exec: {command: ["redis-cli", "ping"]}, periodSeconds: 10, initialDelaySeconds: 10}
        resources:
          requests: {cpu: 100m, memory: 128Mi}
          limits: {cpu: "1", memory: 1Gi}
        volumeMounts: [{name: data, mountPath: /data}]
  volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: standard-rwo
      resources: {requests: {storage: 10Gi}}
---
apiVersion: v1
kind: Service
metadata:
  name: ax-redis
  namespace: ${AX_SYSTEM_NAMESPACE}
  labels: {app.kubernetes.io/name: ax-redis, app.kubernetes.io/part-of: ax}
spec:
  ports: [{port: 6379, targetPort: 6379, name: redis}]
  selector: {app.kubernetes.io/name: ax-redis}

manifests__ax__workerpool.yaml

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: ax-workers
  namespace: ${AX_SYSTEM_NAMESPACE}
spec:
  replicas: ${AX_WORKER_REPLICAS}
  workerImage: ${ATEOM_GVISOR_IMAGE}
  sandboxClasses:
  - name: gvisor
    configRef: {name: gvisor-default}

scripts__00-prereqs.sh

#!/usr/bin/env bash
# Checks tools, clones the pinned Substrate + AX sources.
source "$(dirname "$0")/lib.sh"
need gcloud kubectl docker git go envsubst gke-gcloud-auth-plugin
docker buildx version >/dev/null 2>&1 || die "docker buildx missing (install docker-ce + docker-buildx-plugin; distro docker.io lacks it)"
gv=$(go env GOVERSION); log "go $gv (AX needs >= 1.27.1)"
mkdir -p "$SRC_DIR"
clone() { # clone <url> <dir> <ref>
  [[ -d "$2/.git" ]] || git clone -q "$1" "$2"
  git -C "$2" fetch -q --tags origin
  git -C "$2" -c advice.detachedHead=false checkout -q "$3"
  echo "  $(basename "$2") @ $(git -C "$2" rev-parse --short HEAD)"
}
log "sources"
clone https://github.com/agent-substrate/substrate.git "$SRC_DIR/substrate" "$SUBSTRATE_REF"
clone https://github.com/google/ax.git "$SRC_DIR/ax" "$AX_REF"
grep -q "agent-substrate/substrate $SUBSTRATE_REF" "$SRC_DIR/ax/go.mod" \
  || echo "WARN: AX go.mod does not pin $SUBSTRATE_REF — not a tested pair"
log "prereqs OK"

scripts__10-cluster.sh

#!/usr/bin/env bash
# Creates (idempotently) the GKE cluster, snapshot bucket, Workload Identity IAM,
# and Artifact Registry repo, using Substrate's own provisioning tool.
source "$(dirname "$0")/lib.sh"
log "APIs"
gcloud services enable container.googleapis.com artifactregistry.googleapis.com \
  compute.googleapis.com storage.googleapis.com iamcredentials.googleapis.com --project "$PROJECT_ID"
log "Artifact Registry $AR_REPO"
gcloud artifacts repositories describe "$AR_REPO" --location "$GCE_REGION" --project "$PROJECT_ID" >/dev/null 2>&1 \
  || gcloud artifacts repositories create "$AR_REPO" --repository-format=docker --location "$GCE_REGION" --project "$PROJECT_ID"
gcloud auth configure-docker "${GCE_REGION}-docker.pkg.dev" --quiet >/dev/null 2>&1
log "cluster + bucket + IAM via substrate tools/setup-gcp bootstrap (idempotent; ~10 min first time)"
# GKE 1.36 needs the PodCertificateRequest/ClusterTrustBundle beta APIs at CREATE time;
# setup-gcp sets them. Don't hand-create the cluster unless you pass them yourself.
(cd "$SRC_DIR/substrate" && go run ./tools/setup-gcp bootstrap)
gcloud container clusters get-credentials "$CLUSTER_NAME" --zone "$CLUSTER_LOCATION" --project "$PROJECT_ID"
log "verify"
k get nodes -o wide
res=$(k api-resources -o name)   # capture first: grep -q + pipefail = SIGPIPE false negative
grep -qx "clustertrustbundles.certificates.k8s.io" <<<"$res" || die "ClusterTrustBundle API not served — recreate cluster (beta APIs are create-time only)"
kern=$(k get nodes -o jsonpath='{.items[0].status.nodeInfo.kernelVersion}')
maj=${kern%%.*}; rest=${kern#*.}; min=${rest%%.*}
(( maj > 6 || (maj == 6 && min >= 8) )) || die "node kernel $kern < 6.8: Substrate overlay mounts (lowerdir+) will fail"
echo "  node kernel $kern OK"
log "cluster OK"

scripts__20-substrate.sh

#!/usr/bin/env bash
# Builds Substrate images into $REGISTRY and installs ate-system.
source "$(dirname "$0")/lib.sh"
log "install Substrate $SUBSTRATE_REF"
current=$(kubectl config current-context)
[[ "$current" == "$KUBE_CONTEXT" ]] || kubectl config use-context "$KUBE_CONTEXT"
(cd "$SRC_DIR/substrate" && KUBECTL_CONTEXT="$KUBE_CONTEXT" \
  ./hack/install-ate.sh --deploy-ate-system --credential-provider="$ATE_CREDENTIAL_PROVIDER")
for d in ate-api-server ate-controller atenet-router atenet-egress k8s-credential-provider; do
  k -n ate-system rollout status deploy/$d --timeout=300s
done
k -n ate-system rollout status ds/atelet --timeout=300s
log "Substrate OK"

scripts__30-images.sh

#!/usr/bin/env bash
# Builds AX + agent images into $REGISTRY, pins every one by digest in k8s/images.lock.
# Re-runnable: same sources -> same layers; images.lock is the single source of truth.
source "$(dirname "$0")/lib.sh"
AX="$SRC_DIR/ax"; SUB="$SRC_DIR/substrate"
gcloud auth configure-docker "${GCE_REGION}-docker.pkg.dev" --quiet >/dev/null 2>&1
TAG=ax-$(git -C "$AX" rev-parse --short HEAD)

log "AX binaries ($TAG)"
(cd "$AX" && mkdir -p bin/linux_amd64 .tmp/bin \
  && CGO_ENABLED=0 go build -o .tmp/bin/ax ./cmd/ax \
  && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/linux_amd64/ax-server ./cmd/ax-server \
  && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/linux_amd64/ax-task-runner ./cmd/ax-task-runner)
install -m 0755 "$AX/.tmp/bin/ax" "$K8S_DIR/.bin-ax" 2>/dev/null || true

log "AX images"
docker build -q -t "$REGISTRY/ax-server:$TAG" --build-arg TARGETARCH=amd64 -f "$AX/test/e2e/Dockerfile.server" "$AX" >/dev/null
save_image AX_SERVER_IMAGE "$(push_digest "$REGISTRY/ax-server:$TAG")"
docker build -q -t "$REGISTRY/ax-task-runner:$TAG" --build-arg TARGETARCH=amd64 -f "$AX/Dockerfile.task-runner" "$AX" >/dev/null
save_image AX_DEFAULT_TASK_IMAGE "$(push_digest "$REGISTRY/ax-task-runner:$TAG")"
load_images

log "Substrate worker image (ateom-gvisor) via ko"
ATEOM=$(cd "$SUB" && printf 'image: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor\n' \
  | KO_DOCKER_REPO="$REGISTRY" ./hack/run-tool.sh ko resolve --platform=linux/amd64 -f - | awk '/image:/{print $2}')
[[ "$ATEOM" == *@sha256:* ]] || die "ko did not return a digest: $ATEOM"
save_image ATEOM_GVISOR_IMAGE "$ATEOM"

log "third-party images (mirrored into $REGISTRY so the cluster never pulls from Docker Hub)"
mirror() { # mirror <VAR> <upstream@sha256:...> <name>
  # Registry-to-registry copy of the full manifest list: digest stays identical to upstream.
  docker buildx imagetools create -t "$REGISTRY/$3" "$2" >/dev/null || die "mirror $2 failed"
  local d; d=$(registry_digest "$REGISTRY/$3")
  [[ "$d" == "${2##*@}" ]] || die "mirror digest mismatch for $2: got $d"
  save_image "$1" "${REGISTRY}/${3%:*}@$d"
}
mirror REDIS_IMAGE "redis:7.4-alpine@sha256:57e4255cc971b1c8d6f8a6ca283ac01bc3ecdd914273ef6bfaede87aaecd6541" redis:7.4-alpine

log "agent images (upstream base + AX runner)"
for a in ${AGENTS:-hermes openclaw}; do
  docker build -q -t "$REGISTRY/agent-$a:$TAG" --build-arg RUNNER_IMAGE="$AX_DEFAULT_TASK_IMAGE" \
    -f "$K8S_DIR/agents/$a/Dockerfile" "$K8S_DIR/agents/$a" >/dev/null
  save_image "AGENT_${a^^}_IMAGE" "$(push_digest "$REGISTRY/agent-$a:$TAG")"
done
log "images.lock"; cat "$IMAGES_FILE"

scripts__40-ax.sh

#!/usr/bin/env bash
# Deploys AX (durable Redis, WorkerPool, ax-server) from k8s/manifests/ax with pinned digests.
source "$(dirname "$0")/lib.sh"
load_images
for v in AX_SERVER_IMAGE AX_DEFAULT_TASK_IMAGE ATEOM_GVISOR_IMAGE REDIS_IMAGE; do
  [[ "${!v:-}" == *@sha256:* ]] || die "$v not pinned — run 30-images.sh"
done
export AX_SERVER_IMAGE AX_DEFAULT_TASK_IMAGE ATEOM_GVISOR_IMAGE REDIS_IMAGE
VARS='${AX_SYSTEM_NAMESPACE} ${AX_WORKER_REPLICAS} ${AX_SERVER_IMAGE} ${AX_DEFAULT_TASK_IMAGE} ${AX_SNAPSHOTS_BUCKET} ${ATEOM_GVISOR_IMAGE} ${REDIS_IMAGE}'
render() { envsubst "$VARS" < "$K8S_DIR/manifests/ax/$1"; }
log "namespace + WorkerPool ($AX_WORKER_REPLICAS workers)"
render namespace.yaml | k apply -f -
render workerpool.yaml | k apply -f -
retry 60 2 k -n "$AX_SYSTEM_NAMESPACE" get deploy ax-workers >/dev/null
k -n "$AX_SYSTEM_NAMESPACE" rollout status deploy/ax-workers --timeout=300s
log "durable redis"
# Migrate from AX's ephemeral Deployment if present (one-time; data there is not durable anyway).
k -n "$AX_SYSTEM_NAMESPACE" delete deploy ax-redis --ignore-not-found
render redis.yaml | k apply -f -
k -n "$AX_SYSTEM_NAMESPACE" rollout status sts/ax-redis --timeout=300s
log "ax-server"
render ax-server.yaml | k apply -f -
k -n "$AX_SYSTEM_NAMESPACE" rollout status deploy/ax-server --timeout=300s
k -n "$AX_SYSTEM_NAMESPACE" get pods -o wide
log "AX OK"

scripts__50-verify.sh

#!/usr/bin/env bash
# Verification gates. All must pass before trusting the install.
#   1. AX upstream e2e suite (8 tests) against this cluster
#   2. Durability: write -> suspend -> delete ALL worker pods -> resume -> sha256 match (GCS restore)
#   3. Control-plane durability: restart ax-redis + ax-server -> task records survive
#   4. Agent images: hermes/openclaw tasks run their CLI inside the sandbox
source "$(dirname "$0")/lib.sh"
load_images
AXBIN="$SRC_DIR/ax/.tmp/bin/ax"; [[ -x "$AXBIN" ]] || die "build ax first (30-images.sh)"
export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config}" AX_HOME="${AX_HOME:-$(mktemp -d)}"
kubectl config use-context "$KUBE_CONTEXT" >/dev/null
ax() { "$AXBIN" "$@"; }
PASS=(); FAIL=()
gate() { local name=$1; shift; log "GATE: $name"; if "$@"; then PASS+=("$name"); else FAIL+=("$name"); fi; }
NS=$AX_SYSTEM_NAMESPACE

resume_retry() { retry 10 6 ax resume task "$1" >/dev/null; }
wait_running() { retry 30 2 sh -c "'$AXBIN' get task '$1' -o yaml | grep -q 'phase: Running'"; }

g_e2e() {
  local AX="$SRC_DIR/ax" base e2e rc
  base=$AX_DEFAULT_TASK_IMAGE
  docker build -q -t "$REGISTRY/ax-task-runner:e2e" --build-arg BASE_IMAGE="$base" \
    -f "$AX/test/e2e/Dockerfile.task-runner.e2e" "$AX" >/dev/null
  e2e=$(push_digest "$REGISTRY/ax-task-runner:e2e")
  k -n $NS set env deploy/ax-server AX_DEFAULT_TASK_IMAGE="$e2e" >/dev/null
  k -n $NS rollout status deploy/ax-server --timeout=180s >/dev/null
  rm -f "$AX/.tmp/e2e.env"
  (cd "$AX" && KUBE_CONTEXT=$KUBE_CONTEXT AX_SYSTEM_NAMESPACE=$NS AX_BIN=$AXBIN ./hack/run-e2e.sh) > /tmp/ax-e2e.log 2>&1; rc=$?
  k -n $NS set env deploy/ax-server AX_DEFAULT_TASK_IMAGE="$base" >/dev/null
  k -n $NS rollout status deploy/ax-server --timeout=180s >/dev/null
  grep -E '^--- (PASS|FAIL)' /tmp/ax-e2e.log
  return $rc
}

g_durability() {
  local n=gate-dur-$RANDOM before after w1 w2
  printf 'apiVersion: ax.io/v1alpha1\nkind: Workspace\nmetadata: {name: %s-ws, atespace: default}\nspec: {files: [{path: seed.txt, content: seed}]}\n---\napiVersion: ax.io/v1alpha1\nkind: Task\nmetadata: {name: %s, atespace: default}\nspec: {debug: true, workspace: {name: %s-ws, path: /workspace}}\n' $n $n $n | ax apply -f - >/dev/null
  resume_retry $n && wait_running $n || return 1
  w1=$(ax get task $n -o yaml | awk '/workerIP/{print $2}')
  before=$(ax ssh $n -- sh -c 'dd if=/dev/urandom of=/workspace/blob bs=1M count=20 2>/dev/null; sha256sum /workspace/blob' | awk '{print $1}')
  ax suspend task $n >/dev/null
  echo "  GCS objects: $(gcloud storage ls -r "gs://$BUCKET_NAME/ax/**" 2>/dev/null | wc -l)"
  k -n $NS delete pod -l ate.dev/worker-pool=ax-workers --wait=true >/dev/null
  k -n $NS rollout status deploy/ax-workers --timeout=300s >/dev/null
  resume_retry $n && wait_running $n || { ax describe task $n | tail -5; return 1; }
  w2=$(ax get task $n -o yaml | awk '/workerIP/{print $2}')
  after=$(ax ssh $n -- sha256sum /workspace/blob | awk '{print $1}')
  echo "  worker $w1 -> $w2 (all worker pods replaced); sha256 $before -> $after"
  ax delete task $n >/dev/null; ax delete workspace $n-ws >/dev/null
  [[ -n "$before" && "$before" == "$after" ]]
}

g_control_plane() {
  local n=gate-cp-$RANDOM
  printf 'apiVersion: ax.io/v1alpha1\nkind: Task\nmetadata: {name: %s, atespace: default}\nspec: {debug: true}\n' $n | ax apply -f - >/dev/null
  k -n $NS delete pod -l app.kubernetes.io/name=ax-redis --wait=true >/dev/null
  k -n $NS rollout status sts/ax-redis --timeout=300s >/dev/null
  k -n $NS rollout restart deploy/ax-server >/dev/null
  k -n $NS rollout status deploy/ax-server --timeout=180s >/dev/null
  retry 10 3 sh -c "'$AXBIN' get task $n >/dev/null" || return 1
  echo "  task $n survived redis pod deletion + ax-server restart"
  ax delete task $n >/dev/null
}

g_agent() { # g_agent <name> <image-var> <cli check>
  local a=$1 img=${!2:-} chk=$3 n=gate-$1-$RANDOM out
  [[ -n "$img" ]] || { echo "  $2 not built"; return 1; }
  printf 'apiVersion: ax.io/v1alpha1\nkind: Task\nmetadata: {name: %s, atespace: default}\nspec:\n  debug: true\n  image: "%s"\n' $n "$img" | ax apply -f - >/dev/null
  resume_retry $n && wait_running $n || { ax describe task $n | tail -5; return 1; }
  out=$(ax ssh $n -- sh -c "$chk" 2>&1); echo "  $out" | head -3
  ax delete task $n >/dev/null
  [[ "$out" == *"$4"* ]]
}

gate "ax-e2e (upstream suite)" g_e2e
gate "durability (GCS restore on new workers)" g_durability
gate "control-plane durability (redis PVC)" g_control_plane
gate "hermes image runs in sandbox" g_agent hermes AGENT_HERMES_IMAGE 'hermes --version' 'Hermes Agent'
gate "openclaw image runs in sandbox" g_agent openclaw AGENT_OPENCLAW_IMAGE 'openclaw --version' 'OpenClaw'

log "RESULT: ${#PASS[@]} passed, ${#FAIL[@]} failed"
for p in "${PASS[@]}"; do echo "  PASS  $p"; done
for f in "${FAIL[@]:-}"; do [[ -n "$f" ]] && echo "  FAIL  $f"; done
(( ${#FAIL[@]} == 0 ))

scripts__99-teardown.sh

#!/usr/bin/env bash
# Deletes the cluster, bucket and AR repo. Asks first.
source "$(dirname "$0")/lib.sh"
read -r -p "Delete cluster $CLUSTER_NAME, bucket $BUCKET_NAME, repo $AR_REPO in $PROJECT_ID? type the cluster name: " ok
[[ "$ok" == "$CLUSTER_NAME" ]] || die "aborted"
gcloud container clusters delete "$CLUSTER_NAME" --zone "$CLUSTER_LOCATION" --project "$PROJECT_ID" --quiet || true
gcloud storage rm -r "gs://$BUCKET_NAME" --quiet || true
gcloud artifacts repositories delete "$AR_REPO" --location "$GCE_REGION" --project "$PROJECT_ID" --quiet || true

scripts__install.sh

#!/usr/bin/env bash
# One-shot: prereqs -> cluster -> substrate -> images -> ax -> verify. Every step is idempotent.
set -euo pipefail
d="$(dirname "$0")"
for s in 00-prereqs 10-cluster 20-substrate 30-images 40-ax 50-verify; do
  echo; echo "################ $s ################"
  "$d/$s.sh"
done

scripts__lib.sh

# Shared helpers. Sourced by every step script.
set -euo pipefail
K8S_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ENV_FILE="${ENV_FILE:-$K8S_DIR/env.local}"
[[ -f "$ENV_FILE" ]] || { echo "missing $ENV_FILE (copy k8s/env.example)" >&2; exit 1; }
# shellcheck disable=SC1090
source "$ENV_FILE"
log()  { printf '\n==> %s\n' "$*"; }
die()  { echo "ERROR: $*" >&2; exit 1; }
k()    { kubectl --context "$KUBE_CONTEXT" "$@"; }
need() { for c in "$@"; do command -v "$c" >/dev/null || die "need '$c' on PATH"; done; }
# Retry: retry <tries> <sleep> cmd...
retry() { local n=$1 s=$2; shift 2; local i; for ((i=1;i<=n;i++)); do "$@" && return 0; echo "  retry $i/$n: $*" >&2; sleep "$s"; done; return 1; }
# Push a local tag and print its pinned digest ref, as the REGISTRY reports it.
# Don't trust `docker inspect .RepoDigests`: with the containerd image store it lists the
# upstream index digest under the pushed repo too, which may not exist there (multi-arch pull
# by digest keeps one platform; push uploads only that) -> ImagePullBackOff "not found".
registry_digest() { docker buildx imagetools inspect "$1" --format '{{.Manifest.Digest}}'; }
push_digest() { docker push -q "$1" >/dev/null || die "push $1 failed"; local d; d=$(registry_digest "$1") \
  && [[ "$d" == sha256:* ]] || die "no registry digest for $1"; echo "${1%:*}@$d"; }
# Persist resolved image digests so later steps and re-runs use the same bits.
IMAGES_FILE="$K8S_DIR/images.lock"
save_image() { touch "$IMAGES_FILE"; grep -v "^$1=" "$IMAGES_FILE" > "$IMAGES_FILE.tmp" || true; echo "$1=$2" >> "$IMAGES_FILE.tmp"; sort -o "$IMAGES_FILE" "$IMAGES_FILE.tmp"; rm -f "$IMAGES_FILE.tmp"; }
load_images() { [[ -f "$IMAGES_FILE" ]] && set -a && source "$IMAGES_FILE" && set +a || true; }