Drains a Kubernetes node using "rollout restart" instead "kubectl drain". See comments for motiviation and usage.

k8s-drain.sh

#!/bin/bash

NODE_NAME=$1
ROLLOUT_CMD=$2

if [[ "$NODE_NAME" == "" ]]; then
  echo "
  USAGE: ./drain.sh <NODE_NAME>

  Drains a node from its Deployments/Stateful set pods.

  Examples:

    # List Deploy/Sts objects that have pods running on the given node
    ./drain.sh NODE_NAME

    # Perform the actual rollout restart
    ./drain.sh NODE_NAME restart

  Pre-requisite: Kubernetes 1.15.x or newer.
"
  exit 1
fi

function rollout_restart() {

  ## Object types to be processed
  ## This could be deployment ou satefulset.
  OBJTYPE=$1

  ## Loop through all objects of type $OBJTYPE.
  ## Splits their names and namespaces.
  for dp in $(kubectl get $OBJTYPE -A --no-headers | awk '{print $1 "|" $2}'); do
    NAMESPACE=$(echo $dp | sed 's/|.*//')
    DEPLOY=$(echo $dp | sed 's/.*|//')
    
    ## For each Deploy/Sts, acquire the SELECTOR used to select its pods.
    SELECTOR=$(kubectl get $OBJTYPE --no-headers -owide -n $NAMESPACE $DEPLOY -owide | awk '{print $8}')

    ## Using SELECTOR, list all pods from the Deploy/Sts running in the target node
    PODLIST=$(kubectl get pod -owide --no-headers -n $NAMESPACE -l "$SELECTOR" | grep $NODE_NAME | awk '{print $1}')

    ## If we have pods running in NODE_NAME, act accordingly
    if [[ "$PODLIST" != "" ]]; then
      echo "=== $OBJTYPE $NAMESPACE/$DEPLOY ==="
      echo $PODLIST | sed 's/ /\n/g'
      if [[ "$ROLLOUT_CMD" != "" ]]; then
        echo ">> Rollout restart..."
        set -x
        kubectl rollout $ROLLOUT_CMD -n $NAMESPACE $OBJTYPE/$DEPLOY
        set +x
      fi

      echo
    fi
  done
}

if [[ "$ROLLOUT_CMD" == "restart" ]]; then
  set -x
  kubectl cordon $NODE_NAME
  set +x
fi

rollout_restart deploy
rollout_restart sts

The following rollout strategy is very common:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

It allows a new version of the application to be rolled out without any downtime, waiting for a the new pods to become ready before terminating the old ones.

As it stands now, k8s 1.15.x and older, the rolling update strategy is ignored by kubectl drain. Because of that, draining a node leads to unwanted downtime. Even though PodDisruptionBudgets are considered, honoring them means some pods may not be allowed to be evicted. That can lead to a deadlock where kubectl is unable to drain a node because PodDisruptionBudgets prevent it.

Some related issues:

• kubernetes/kubernetes#48307
• kubernetes/kubernetes#83307
• kubernetes/kubernetes#31487

This bash script provides a quick and dirty alternative to drain a node. Aside from cordoning the node to prevent new pods from coming in, it will find a list of all Deployments/Statefulsets in the cluster. For each one, if any of its pods are running in the specified node, the script will issue a kubectl rollout restart. This triggers a rolling update, honoring the Rollout Strategy of the Deployment/Statefulset.

Drawbacks: The rollout restart causes all related pods from a given Deployment/Statefulset to restart, not only the ones running in the target node. Please consider this at your scale, before running wild.

👏 👏

This is brilliant and perfect for our setup. Thank you @juliohm1978 !

Is there a typo on line 61 though? kubetl should be kubectl (I changed to kubectl locally and working fine).

Yes, thanks!
Fixed line 61 👍

Brilliant.
You should make a PR to make this available in k8s API server!

Great, was wondering the same thing and this confirmed it, how come doesn't exist as functionality.