Note: this description is deprecated if you use the latest DANM binary with this PR merged: nokia/danm#34
Kubernetes setup with Danm
This is a rough step list how I managed to upgrade a simple Flannel based Kubernetes cluster to use Danm, while all common (platform?) services are still working (such as Helm, CoreDNS, Prometheus). The goal is to still use Flannel for the platform services. Flannel allows them to access K8S API or native services running on the host. An alternative could be to use host network namespace for the platform services, but it is not that fun 😊. Since Danmnet CRD objects are namespaced, we have to delegate the CNI setup to Flannel in all K8S namespaces where needed, like kube-system.
- Started with kubeadm installed 3 nodes cluster
- Centos 7.4 with 4.19 kernel
- Single master, schedulable
- Kubernetes version 11.1.2
- Flannel installed
- CoreDNS installed
- Danm compiled from source (according to the readme), Danm Daemonsets created with example manifests
- For all nodes:
- Load docker images, copy danm and other binaries according to the Danm readme
- Flannel conflist deleted from /etc/cni/net.d/
- New flannel CNI configuration created for delegation under
/etc/cni/net.d/flannel.conf
{
"name": "cbr0",
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
}
- Danm CNI config created under
/etc/cni/net.d/00-danm.conf. Change the last line to your own kubernetes admin config path.
{
"name": "meta_cni",
"type": "danm",
"kubeconfig": "/etc/kubernetes/admin.conf"
}
- Create the delegated Flannel Danmnet in kube-system with
kubectl apply -n kube-system -f flannel.yaml. The content should be:
apiVersion: danm.k8s.io/v1
kind: DanmNet
metadata:
name: flannel
spec:
NetworkID: flannel
NetworkType: flannel
- Now the challenge is to deploy all platform services with Pod annotations pointing to this Flannel network.
- CoreDNS: Since it is already running, we can edit the object in place:
kubectl edit deployment -n kube-system coredns- Add the needed annotation under spec/template/metadata/annotations:
danm.k8s.io/interfaces: '[{"network":"flannel"}]' - You should see coredns Pod(s) are re-created and not crashing 😊
- Add the needed annotation under spec/template/metadata/annotations:
- Helm:
- Install the Helm CLI binary from the latest release.
- Install the RBAC related things.
- Unfortunately Helm's tiller installer command does not offer to change the annotations, first we should ask Helm to dump the manifest instead of install it:
helm init --service-account helm --upgrade --dry-run --debug > tiller-deploy.yml - Add the same needed annotation in the Deployment under spec/template/metadata/annotations:
danm.k8s.io/interfaces: '[{"network":"flannel"}]' - Delete the garbage
...from the last line of the manifest if there it is. - Deploy tiller manually with
kubectl apply -f tiller-deploy.yaml.
- Prometheus:
- I use Prometheus operator and the related kube-prometheus manifests to bootstrap it.
- After getting the repo, in the following files you should add the same annotation in the Deployments under spec/template/metadata/annotations:
danm.k8s.io/interfaces: '[{"network":"flannel"}]'
manifests/0prometheus-operator-deployment.yaml
manifests/grafana-deployment.yaml
manifests/kube-state-metrics-deployment.yaml
manifests/prometheus-adapter-deployment.yaml
- There are two remaining CRD instances, which are read by the Prometheus operator and based on them it deploys the Prometheus server and alarmmanager StatefulSets.
- So we should instruct the operator to create the annotations. After digging myself into the specs of the CRD schema, the following changes needed:
podMetadata:
annotations:
'danm.k8s.io/interfaces': '[{"network":"flannel"}]'
- Add this snippet to the following two CRDs:
manifests/prometheus-prometheus.yaml
manifests/alertmanager-alertmanager.yaml
- Based on the Quickstart section, we are now able to deploy the Prometheus operator and all the components with it.
After all this fun, we have the following setup:
# kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-9b7b4fb88-5jtbc 1/1 Running 0 59m
kube-system coredns-9b7b4fb88-wpvsr 1/1 Running 0 59m
kube-system etcd-master 1/1 Running 10 91d
kube-system kube-apiserver-master 1/1 Running 7 80d
kube-system kube-controller-manager-master 1/1 Running 8 91d
kube-system kube-flannel-ds-c7qtm 1/1 Running 3 91d
kube-system kube-flannel-ds-t7mnf 1/1 Running 7 91d
kube-system kube-flannel-ds-vng5p 1/1 Running 5 91d
kube-system kube-proxy-n9nb2 1/1 Running 3 80d
kube-system kube-proxy-nvglq 1/1 Running 2 80d
kube-system kube-proxy-x278s 1/1 Running 4 80d
kube-system kube-scheduler-master 1/1 Running 9 91d
kube-system netwatcher-2xjdt 1/1 Running 4 16d
kube-system netwatcher-6lfmj 1/1 Running 3 16d
kube-system netwatcher-8mwxb 1/1 Running 2 16d
kube-system svcwatcher-vm2c2 1/1 Running 1 16d
kube-system tiller-deploy-647847c877-fqhzw 1/1 Running 0 31m
monitoring alertmanager-main-0 2/2 Running 0 114m
monitoring grafana-85557fccd4-t6g87 1/1 Running 0 4h16m
monitoring kube-state-metrics-8585f7f74b-gfzzn 4/4 Running 0 4h15m
monitoring node-exporter-6jbvp 2/2 Running 0 4h32m
monitoring node-exporter-rpf9r 2/2 Running 0 4h32m
monitoring node-exporter-zqr6d 2/2 Running 0 4h32m
monitoring prometheus-adapter-b8ff66c-262qb 1/1 Running 0 4h27m
monitoring prometheus-k8s-0 3/3 Running 1 113m
monitoring prometheus-operator-96dffc7cd-rqp5w 1/1 Running 0 4h22m
