Created
May 12, 2017 21:09
-
-
Save mml/1a24775b6fceea4685c84aee2f548f6b to your computer and use it in GitHub Desktop.
kubectl drain-based node upgrades!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| == Pre-Upgrade Node OS and Kubelet Versions == | |
| name: "e2e-master", osImage: "Container-Optimized OS from Google", kubeletVersion: "v1.7.0-alpha.3.278+a365e02e018731-dirty" | |
| name: "e2e-minion-group-5f66", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.7.0-alpha.3.278+a365e02e018731-dirty" | |
| name: "e2e-minion-group-ls90", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.7.0-alpha.3.278+a365e02e018731-dirty" | |
| Project: mml-e2e | |
| Zone: us-central1-f | |
| INSTANCE_GROUPS=e2e-minion-group | |
| NODE_NAMES=e2e-minion-group-5f66 e2e-minion-group-ls90 | |
| == Preparing node upgrade (to v1.7.0-alpha.3.272+69ad6addcc5c25). == | |
| == Ensuring that new Node base OS image matched the existing Node base OS image | |
| == Upgrading node environment variables. == | |
| Instance template e2e-minion-template-v1-7-0-alpha-3-272-69ad6addcc5c25 already exists; deleting. | |
| Attempt 1 to create e2e-minion-template-v1-7-0-alpha-3-272-69ad6addcc5c25 | |
| WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#pdperformance. | |
| Created [https://www.googleapis.com/compute/v1/projects/mml-e2e/global/instanceTemplates/e2e-minion-template-v1-7-0-alpha-3-272-69ad6addcc5c25]. | |
| NAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP | |
| e2e-minion-template-v1-7-0-alpha-3-272-69ad6addcc5c25 n1-standard-2 2017-05-12T12:03:25.300-07:00 | |
| Instance template name: e2e-minion-template-v1-7-0-alpha-3-272-69ad6addcc5c25 | |
| == Finished preparing node upgrade (to v1.7.0-alpha.3.272+69ad6addcc5c25). == | |
| == Upgrading nodes to v1.7.0-alpha.3.272+69ad6addcc5c25. == | |
| Updated [https://www.googleapis.com/compute/v1/projects/mml-e2e/zones/us-central1-f/instanceGroupManagers/e2e-minion-group]. | |
| --- | |
| baseInstanceName: e2e-minion-group | |
| creationTimestamp: '2017-05-11T15:46:52.276-07:00' | |
| currentActions: | |
| abandoning: 0 | |
| creating: 0 | |
| creatingWithoutRetries: 0 | |
| deleting: 0 | |
| none: 2 | |
| recreating: 0 | |
| refreshing: 0 | |
| restarting: 0 | |
| fingerprint: NdOyzo5JUqw= | |
| id: '3673119221375129395' | |
| instanceGroup: https://www.googleapis.com/compute/v1/projects/mml-e2e/zones/us-central1-f/instanceGroups/e2e-minion-group | |
| instanceTemplate: https://www.googleapis.com/compute/v1/projects/mml-e2e/global/instanceTemplates/e2e-minion-template-v1-7-0-alpha-3-272-69ad6addcc5c25 | |
| kind: compute#instanceGroupManager | |
| name: e2e-minion-group | |
| selfLink: https://www.googleapis.com/compute/v1/projects/mml-e2e/zones/us-central1-f/instanceGroupManagers/e2e-minion-group | |
| targetSize: 2 | |
| zone: https://www.googleapis.com/compute/v1/projects/mml-e2e/zones/us-central1-f | |
| == Draining e2e-minion-group-5f66. == | |
| node "e2e-minion-group-5f66" cordoned | |
| WARNING: Ignoring DaemonSet-managed pods: fluentd-gcp-v2.0-1ztqr, npd-v0.3.0-stsrj; Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: kube-proxy-e2e-minion-group-5f66; Deleting pods with local storage: monitoring-influxdb-grafana-v4-qwblm | |
| pod "kube-dns-autoscaler-2042644734-dcrft" evicted | |
| pod "kubernetes-dashboard-638781211-fzgwj" evicted | |
| pod "heapster-v1.3.0-514138450-hsrj1" evicted | |
| pod "monitoring-influxdb-grafana-v4-qwblm" evicted | |
| pod "l7-default-backend-1599978672-lfwn4" evicted | |
| pod "kube-dns-2473071502-bgvj1" evicted | |
| pod "kube-dns-2473071502-6slgd" evicted | |
| node "e2e-minion-group-5f66" drained | |
| Updated [https://www.googleapis.com/compute/v1/projects/mml-e2e/zones/us-central1-f/instanceGroupManagers/e2e-minion-group]. | |
| PROJECT ZONE INSTANCE STATUS | |
| mml-e2e us-central1-f e2e-minion-group-5f66 SUCCESS | |
| == Waiting for e2e-minion-group-5f66 to become young. == | |
| Node e2e-minion-group-5f66 not new: 1068 seconds. | |
| Node e2e-minion-group-5f66 not new: 1078 seconds. | |
| Node e2e-minion-group-5f66 not new: 1088 seconds. | |
| Node e2e-minion-group-5f66 not new: 1099 seconds. | |
| Node e2e-minion-group-5f66 not new: 1109 seconds. | |
| Node e2e-minion-group-5f66 not new: 1119 seconds. | |
| Node e2e-minion-group-5f66 not new: 1129 seconds. | |
| Node e2e-minion-group-5f66 not new: 1140 seconds. | |
| Node e2e-minion-group-5f66 not new: 1150 seconds. | |
| Node e2e-minion-group-5f66 not new: 1160 seconds. | |
| Node e2e-minion-group-5f66 not new: 1171 seconds. | |
| Node e2e-minion-group-5f66 not new: 1181 seconds. | |
| Node e2e-minion-group-5f66 not new: 1191 seconds. | |
| Node e2e-minion-group-5f66 not new: 1202 seconds. | |
| Node e2e-minion-group-5f66 not new: 1212 seconds. | |
| Node e2e-minion-group-5f66 not new: 1222 seconds. | |
| Node e2e-minion-group-5f66 recreated. | |
| == Waiting for e2e-minion-group-5f66 to become ready. == | |
| Node e2e-minion-group-5f66 is still not ready: Ready=False | |
| Node e2e-minion-group-5f66 Ready=True | |
| == Draining e2e-minion-group-ls90. == | |
| node "e2e-minion-group-ls90" cordoned | |
| WARNING: Ignoring DaemonSet-managed pods: fluentd-gcp-v2.0-0v7nb, npd-v0.3.0-bvxmt; Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: kube-proxy-e2e-minion-group-ls90; Deleting pods with local storage: monitoring-influxdb-grafana-v4-p6lsq | |
| pod "l7-default-backend-1599978672-2lb64" evicted | |
| pod "kube-dns-autoscaler-2042644734-w9z9k" evicted | |
| pod "heapster-v1.3.0-514138450-phx10" evicted | |
| pod "kubernetes-dashboard-638781211-d6s22" evicted | |
| pod "monitoring-influxdb-grafana-v4-p6lsq" evicted | |
| pod "kube-dns-2473071502-rlw6j" evicted | |
| pod "kube-dns-2473071502-95nh9" evicted | |
| node "e2e-minion-group-ls90" drained | |
| Updated [https://www.googleapis.com/compute/v1/projects/mml-e2e/zones/us-central1-f/instanceGroupManagers/e2e-minion-group]. | |
| PROJECT ZONE INSTANCE STATUS | |
| mml-e2e us-central1-f e2e-minion-group-ls90 SUCCESS | |
| == Waiting for e2e-minion-group-ls90 to become young. == | |
| Node e2e-minion-group-ls90 not new: 999 seconds. | |
| Node e2e-minion-group-ls90 not new: 1009 seconds. | |
| Node e2e-minion-group-ls90 not new: 1019 seconds. | |
| Node e2e-minion-group-ls90 not new: 1029 seconds. | |
| Node e2e-minion-group-ls90 not new: 1040 seconds. | |
| Node e2e-minion-group-ls90 not new: 1050 seconds. | |
| Node e2e-minion-group-ls90 not new: 1060 seconds. | |
| Node e2e-minion-group-ls90 not new: 1071 seconds. | |
| Node e2e-minion-group-ls90 not new: 1081 seconds. | |
| Node e2e-minion-group-ls90 not new: 1091 seconds. | |
| Node e2e-minion-group-ls90 not new: 1102 seconds. | |
| Node e2e-minion-group-ls90 not new: 1112 seconds. | |
| Node e2e-minion-group-ls90 not new: 1122 seconds. | |
| Node e2e-minion-group-ls90 not new: 1133 seconds. | |
| Node e2e-minion-group-ls90 not new: 1143 seconds. | |
| Node e2e-minion-group-ls90 not new: 1153 seconds. | |
| Node e2e-minion-group-ls90 not new: 1163 seconds. | |
| Node e2e-minion-group-ls90 not new: 1174 seconds. | |
| Node e2e-minion-group-ls90 recreated. | |
| == Waiting for e2e-minion-group-ls90 to become ready. == | |
| Node e2e-minion-group-ls90 is still not ready: Ready=False | |
| Node e2e-minion-group-ls90 is still not ready: Ready=False | |
| Node e2e-minion-group-ls90 Ready=True | |
| == Deleting old templates in mml-e2e. == | |
| Deleted [https://www.googleapis.com/compute/v1/projects/mml-e2e/global/instanceTemplates/e2e-minion-template]. | |
| == Finished upgrading nodes to v1.7.0-alpha.3.272+69ad6addcc5c25. == | |
| == Validating cluster post-upgrade == | |
| MYCONFIG=foo | |
| DONE | |
| Validating gce cluster, MULTIZONE= | |
| Project: mml-e2e | |
| Zone: us-central1-f | |
| Found 3 node(s). | |
| NAME STATUS AGE VERSION | |
| e2e-master Ready,SchedulingDisabled 20h v1.7.0-alpha.3.278+a365e02e018731-dirty | |
| e2e-minion-group-5f66 Ready 4m v1.7.0-alpha.3.272+69ad6addcc5c25 | |
| e2e-minion-group-ls90 Ready 30s v1.7.0-alpha.3.272+69ad6addcc5c25 | |
| Validate output: | |
| NAME STATUS MESSAGE ERROR | |
| scheduler Healthy ok | |
| controller-manager Healthy ok | |
| etcd-0 Healthy {"health": "true"} | |
| etcd-1 Healthy {"health": "true"} | |
| Cluster validation succeeded | |
| == Post-Upgrade Node OS and Kubelet Versions == | |
| name: "e2e-master", osImage: "Container-Optimized OS from Google", kubeletVersion: "v1.7.0-alpha.3.278+a365e02e018731-dirty" | |
| name: "e2e-minion-group-5f66", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.7.0-alpha.3.272+69ad6addcc5c25" | |
| name: "e2e-minion-group-ls90", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.7.0-alpha.3.272+69ad6addcc5c25" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment