Cloud Api Adaptor image decryption with cc_kbc

busybox-encrypted.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: busybox-encrypted
  name: busybox-encrypted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-encrypted
  template:
    metadata:
      labels:
        app: busybox-encrypted
    spec:
      runtimeClassName: kata-remote
      containers:
      - image: ghcr.io/mkulke/busybox-encrypted:demo
        name: busybox
        command: ["sleep", "infinity"]

Dockerfile.kbs

FROM rust:1.71-slim-bullseye as builder
ARG KBS_COMMIT=81f3de7

RUN apt-get update && apt-get install -y curl gnupg2
RUN curl -L https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | apt-key add -
RUN echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu focal main' > /etc/apt/sources.list.d/intel-sgx.list
RUN apt-get update && apt-get install -y \
	build-essential \
	clang \
	git \
	libsgx-dcap-quote-verify-dev \
	libtdx-attest-dev \
	libtss2-dev \
	openssl \
	pkg-config \
	protobuf-compiler
RUN curl -LO https://go.dev/dl/go1.20.7.linux-amd64.tar.gz
RUN tar -C /usr/local -xzf go1.20.7.linux-amd64.tar.gz
ENV PATH=$PATH:/usr/local/go/bin
RUN git clone https://github.com/confidential-containers/kbs.git -b main /build
WORKDIR /build
RUN git checkout $KBS_COMMIT
RUN cargo b --release -p kbs

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl gnupg2
RUN curl -L https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | apt-key add -
RUN echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu focal main' > /etc/apt/sources.list.d/intel-sgx.list
RUN apt-get update && apt-get install -y \
	libsgx-dcap-quote-verify \
	libtss2-esys-3.0.2-0 \
	libtss2-tctildr0 \
	openssl
ENV KBS_REPO_PATH=/opt/confidential-containers/kbs/repository
RUN mkdir -p "${KBS_REPO_PATH}/one/two"
RUN echo "ohai" > "${KBS_REPO_PATH}/one/two/key"
RUN openssl genpkey -algorithm ed25519 > /kbs.key
RUN openssl pkey -in /kbs.key -pubout -out /kbs.pem
COPY --from=builder /build/target/release/kbs /kbs
RUN mkdir -p "${KBS_REPO_PATH}/default/key"
RUN mkdir -p "${KBS_REPO_PATH}/default/credential"
# we need a default credential object for cc_kbc to work
RUN echo '{"auths":{}}' > "${KBS_REPO_PATH}/default/credential/test"
CMD ["/kbs", "--socket", "0.0.0.0:8080", "--insecure-http", "--auth-public-key", "/kbs.pem" ]

instructions.md

Instructions

The following instructions depict a setup for securely decrypting image layers via the cc_kbc module using Cloud API Adaptor on Azure SNP CVMs. The instructions assume a functional k8s cluster configured with Cloud Api Adaptor v0.7.0.

Deploy KBS

Build kbs image:

KBS_IMAGE=ghcr.io/mkulke/kbs:demo
docker build -t "$KBS_IMAGE" -f Dockerfile.kbs .
docker push "$KBS_IMAGE"

Deploy kbs and expose it in the cluster:

kubectl create deploy kbs --image "$KBS_IMAGE" --port 8080
kubectl expose deploy kbs

KBS needs to be exposed on a Node Port, since CAA's Attestation-Agent doesn't run in the pod network namespace. We'll pick port 30080.

kubectl apply -f kbs-nodeport-svc.yaml

Take note of a node's address:

kubectl get node -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'
10.200.28.5

Thus aa_kbc_params for kata-agent would be cc_kbc::http://10.200.28.5:30080.

Build CAA PodVM

Please refer to this documentation for generic PodVM image build instructions.

We need to bake aa_kbc_params into the PodVM image currently, we will specify the endpoint from the nodeport service in the prior step:

git clone https://github.com/confidential-containers/guest-components.git -b main
pushd guest-components
git checkout v0.7.0
popd
git clone https://github.com/kata-containers/kata-containers.git -b CCv0
pushd kata-containers
git checkout CC-0.7.0
rustup override set 1.69.0
popd
git clone https://github.com/confidential-containers/cloud-api-adaptor.git
cd cloud-api-adaptor/azure/image
git checkout 1297e19
AA_KBC=cc_kbc KBC_URI=http://10.200.28.5:30080 LIBC=gnu PODVM_DISTRO=ubuntu make image

Configure CAA to use the cc_kbc PodVM image

Add the resulting IMAGE_ID from the previous step into AZURE_IMAGE_ID value in caa's configmap and restart caa. Confirm that the correct values have been set using the daemonset logs:

kubectl edit cm peer-pods-cm -n confidential-containers-system
kubectl delete po -l app=cloud-api-adaptor -n confidential-containers-system
kubectl logs ds/cloud-api-adaptor-daemonset -n confidential-containers-system

Encrypt Image

Build coco_keyprovider:

cd guest-components
cargo b --release -p coco_keyprovider

Start coco_provider daemon in bg:

KEYPROVIDER_SOCKET=127.0.0.1:50000
./target/release/coco_keyprovider --socket "$KEYPROVIDER_SOCKET" &
KEYPROVIDER_PID=$!

Encrypt an image with a random secret using skopeo and ocicrypt:

note: the key file needs to have a length of exactly 32 bytes.

KEY_FILE="${PWD}/image_key"
head -c 32 /dev/urandom | openssl enc > "$KEY_FILE"
echo '{"key-providers": {"attestation-agent": {"grpc": "'"$KEYPROVIDER_SOCKET"'"}}}' > ocicrypt.conf
IMAGE_DESTINATION="ghcr.io/mkulke/busybox-encrypted:demo"
KEY_ID="default/key/busybox-encrypted"
KEYPROVIDER_PARAMS="provider:attestation-agent:keypath=${KEY_FILE}::keyid=kbs:///${KEY_ID}::algorithm=A256GCM"
OCICRYPT_KEYPROVIDER_CONFIG="${PWD}/ocicrypt.conf" skopeo copy --insecure-policy --encryption-key "$KEYPROVIDER_PARAMS" docker://busybox:stable "docker://${IMAGE_DESTINATION}"
kill "$KEYPROVIDER_PID"

Provision Key to KBS

cat "$KEY_FILE" | kubectl exec -i deploy/kbs -- tee "/opt/confidential-containers/kbs/repository/${KEY_ID}" > /dev/null

Deploy encrypted image

kubectl apply -f busybox-encrypted.yaml

Look at daemonset logs to confirm or diagnose problems:

kubectl logs --tail 3 ds/cloud-api-adaptor-daemonset -n confidential-containers-system
2023/08/08 10:58:11 [adaptor/proxy] CreateContainer: calling PullImage for "ghcr.io/mkulke/busybox-encrypted:demo" before CreateContainer (cid: "7a7210f09d0c43f2d0f800d92a22575e9a0fca033fa425612a7cb2dfb4ec897a")
2023/08/08 10:58:11 [adaptor/proxy] CreateContainer: successfully pulled image "ghcr.io/mkulke/busybox-encrypted:demo"
2023/08/08 10:58:11 [adaptor/proxy] StartContainer: containerID:7a7210f09d0c43f2d0f800d92a22575e9a0fca033fa425612a7cb2dfb4ec897a

Confirm with kbs logs the we requested an image key:

kubectl logs --tail=3 deploy/kbs
[2023-08-08T10:58:11Z INFO  api_server::http::resource] Get pkey from session.
[2023-08-08T10:58:11Z INFO  api_server::http::resource] Resource description: ResourceDesc { repository_name: "default", resource_type: "key", resource_tag: "busybox-encrypted" }
[2023-08-08T10:58:11Z INFO  actix_web::middleware::logger] 10.244.0.1 "GET /kbs/v0/resource/default/key/busybox-encrypted HTTP/1.1" 200 530 "-" "attestation-agent-kbs-client/0.1.0" 0.000779

kbs-nodeport-svc.yaml

kubectl apply -f 
apiVersion: v1
kind: Service
metadata:
  name: kbs-nodeport
spec:
  type: NodePort
  selector:
    app: kbs
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30080

PodVM attesation-agent service needs to be updated

diff --git a/podvm/files/etc/systemd/system/attestation-agent.service b/podvm/files/etc/systemd/system/attestation-agent.service
index 23e7246..f0f988d 100644
--- a/podvm/files/etc/systemd/system/attestation-agent.service
+++ b/podvm/files/etc/systemd/system/attestation-agent.service
@@ -6,7 +6,7 @@ After=network.target kata-agent.service [email protected]
 [Service]
 Type=simple
 NetworkNamespacePath=/run/netns/podns
-ExecStart=/usr/local/bin/attestation-agent --getresource_sock 127.0.0.1:50001
+ExecStart=/usr/local/bin/attestation-agent --getresource_sock unix:///run/confidential-containers/attestation-agent/getresource.sock
 Restart=always

 [Install]