Cluster API TIC Live 2 Demo

clusterapi-demo.md

Local Cluster API (CAPI) Hands-On Lab with Colima on Apple Silicon

A step-by-step guide to setting up a fully functional Cluster API lab on macOS (M1/M2/M3) using Colima, Kind, and the Docker Infrastructure Provider (CAPD).

This lab covers provisioning, networking, day-2 operations (scaling/upgrades), automated remediation (MHC), management pivoting, and graceful cluster teardown.

---

Prerequisites

Install required tools via Homebrew:

brew install colima docker kubectl kind clusterctl

Start Colima with sufficient resources for nested container nodes:

colima start --cpu 4 --memory 8 --disk 50

Set up your shell environment variables:

# Enable topology feature gate and avoid GitHub rate limits
export CLUSTER_TOPOLOGY=true
export GITHUB_TOKEN="ghp_your_personal_access_token"

---

1. Create the Bootstrap Management Cluster

Because CAPD provisions nodes as Docker containers, the management cluster needs access to your Mac's Docker socket inside Colima.

# Create a Kind config referencing your user's Colima socket path
cat <<EOF> kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /Users/$(whoami)/.colima/default/docker.sock
      containerPath: /var/run/docker.sock
EOF

# Spin up the management cluster
kind create cluster --name capi-mgmt --config kind-config.yaml

# Initialize Cluster API with the Docker Infrastructure Provider
clusterctl init --infrastructure docker

---

2. Provision the Workload Cluster

Generate and apply the workload cluster manifest:

# Generate declarative template
clusterctl generate cluster workload-demo \
  --kubernetes-version v1.30.0 \
  --control-plane-machine-count 1 \
  --worker-machine-count 2 \
  --flavor development > workload-demo.yaml

# Apply manifest
kubectl apply -f workload-demo.yaml

# Monitor cluster creation
clusterctl describe cluster workload-demo

---

3. Configure Workload Cluster Access & CNI

Cluster API provisions bare nodes. Extract the kubeconfig, adjust network routing for macOS, and install Calico CNI.

# 1. Fetch kubeconfig
clusterctl get kubeconfig workload-demo > workload-demo.kubeconfig

# 2. Map internal Docker load balancer port to 127.0.0.1 for macOS host routing
LB_PORT=$(docker port workload-demo-lb 6443/tcp | awk -F: '{print $NF}')
sed -i '' "s|https://.*:6443|[https://127.0.0.1](https://127.0.0.1):${LB_PORT}|g" workload-demo.kubeconfig

# 3. Install Calico CNI
kubectl --kubeconfig=workload-demo.kubeconfig apply -f [https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml](https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml)

# 4. Confirm nodes transition to Ready
kubectl --kubeconfig=workload-demo.kubeconfig get nodes -w

---

4. Day-2 Operations: Scaling & Rolling Upgrades

When using Cluster Topology (ClusterClass), always patch the top-level Cluster resource as the single source of truth.

Scale Worker Pool (2 to 3 Nodes)

kubectl patch cluster workload-demo --type merge -p '{
  "spec": {
    "topology": {
      "workers": {
        "machineDeployments": [
          {
            "name": "md-0",
            "replicas": 3
          }
        ]
      }
    }
  }
}'

Perform Rolling Kubernetes Upgrade (v1.30.0 -> v1.31.0)

kubectl patch cluster workload-demo --type merge -p '{
  "spec": {
    "topology": {
      "version": "v1.31.0"
    }
  }
}'

# Watch rolling replacement of nodes in real time
kubectl get machines -w

---

5. Automated Remediation via Cluster Topology (MHC)

Note: Direct modifications to MachineHealthCheck objects generated by ClusterClass will be reconciled back to the topology spec. Always configure MHC inside spec.topology.

Configure MHC via Cluster Topology

Patch the Cluster spec to lower the unhealthy node detection timeout to 30s:

kubectl patch cluster workload-demo --type merge -p '{
  "spec": {
    "topology": {
      "workers": {
        "machineDeployments": [
          {
            "name": "md-0",
            "machineHealthCheck": {
              "nodeStartupTimeout": "10m",
              "maxUnhealthy": "100%",
              "unhealthyConditions": [
                {
                  "type": "Ready",
                  "status": "False",
                  "timeout": "30s"
                },
                {
                  "type": "Ready",
                  "status": "Unknown",
                  "timeout": "30s"
                }
              ]
            }
          }
        ]
      }
    }
  }
}'

Test Automated Self-Healing

# 1. Forcefully stop a worker node container
WORKER_CONTAINER=$(docker ps --format '{{.Names}}' | grep workload-demo-md | head -n 1)
docker stop $WORKER_CONTAINER

# 2. Watch CAPI detect failure and recreate the machine
kubectl get machines -w

---

6. Management Pivoting (clusterctl move)

Transfer control plane management from the initial bootstrap cluster to a new management cluster without workload downtime.

# 1. Create target management cluster
kind create cluster --name capi-mgmt-2 --config kind-config.yaml

# 2. Initialize CAPI on the target cluster
clusterctl init --kubeconfig-context kind-capi-mgmt-2 --infrastructure docker

# 3. Ensure context is set to source cluster
kubectl config use-context kind-capi-mgmt

# 4. Pivot resources
clusterctl move --to-kubeconfig ~/.kube/config --to-kubeconfig-context kind-capi-mgmt-2

# 5. Verify source is empty and target holds state
kubectl --context kind-capi-mgmt get clusters
kubectl --context kind-capi-mgmt-2 get clusters,machines

# 6. Delete old bootstrap cluster safely
kind delete cluster --name capi-mgmt

---

7. Graceful Teardown

Cluster API performs dependency-aware garbage collection via Kubernetes finalizers.

# Delete workload cluster
kubectl --context kind-capi-mgmt-2 delete cluster workload-demo

# Watch ordered teardown
kubectl --context kind-capi-mgmt-2 get machines -w

# Cleanup remaining management resources
kind delete cluster --name capi-mgmt-2
colima stop