Install Script

README.md

Installation Script Documentation

Overview

The install.sh script is designed to bootstrap a new virtual machine (VM) by performing various system configurations and installations. It can execute locally or bootstrap remote servers by connecting via SSH and executing the script remotely. This script automates the setup process, ensuring consistency and reducing manual intervention.

---

Author: Troy Klly
Contact: [email protected]
Date: Saturday 2 November 2024

---

Features

• Local Execution: Configures the local machine with necessary packages, users, and settings.
• Remote Bootstrapping: Connects to remote servers over SSH to perform the same configuration.
• Idempotent Design: Safe to run multiple times without causing adverse effects.
• Error Logging: Collects and displays errors encountered during execution.
• Customizable User: Uses an operations user by default, which can be customized.

Table of Contents

• Prerequisites
• Usage
  • Local Execution
  • Remote Bootstrapping
• Script Functions
• Environment Variables
• Notes
• License

Prerequisites

• Bash Shell: The script is written for the Bash shell.
• Root Privileges: Required when executing locally.
• Expect Utility: Required on the local machine for remote bootstrapping.
• SSH Access: The local machine must have network access to the remote servers.
• Supported Systems: Designed for Debian-based systems (e.g., Ubuntu).

Usage

Local Execution

To execute the script on the local machine:

1. Ensure Root Privileges: The script must be run as root.

   sudo ./install.sh

2. Execution: Run the script.

   ./install.sh

Remote Bootstrapping

To bootstrap remote servers:

1. Install expect: Ensure that the expect utility is installed on the local machine.

   # For Debian-based systems
   sudo apt-get install expect

2. Set Environment Variables: Before running the script, set the following environment variables:

   • BOOTSTRAP_SERVERS: Comma-separated list of Fully Qualified Domain Names (FQDNs) of the remote servers.
   • BOOTSTRAP_OPERATIONS_PASSWORD: Password for the operations user on the remote servers.
   • BOOTSTRAP_ROOT_PASSWORD: Password for the root user on the remote servers.

   export BOOTSTRAP_SERVERS="server1.example.com, server2.example.com"
   export BOOTSTRAP_OPERATIONS_PASSWORD="operations_password"
   export BOOTSTRAP_ROOT_PASSWORD="root_password"

3. Execution: Run the script without root privileges.

   bash <(curl -fsSL https://gist.githubusercontent.com/troykelly/f76efa97a3bf77ae0d4b0e3c82e61455/raw/install.sh)

   The script will:

   • Connect to each server as the operations user.
   • Elevate to root user.
   • Install curl if it's not already installed.
   • Execute the install script on the remote server.

Script Functions

The script consists of several functions that perform specific tasks:

• bootstrap_remote_servers: Bootstraps remote servers by connecting over SSH and executing the installation steps.
• update_apt: Updates package lists using apt-get update.
• full_upgrade: Performs a full system upgrade.
• install_packages: Installs required packages.
• configure_sudoers: Configures sudo privileges for the operations user.
• configure_rsyslog: Sets up remote syslog forwarding.
• setup_ssh: Configures SSH access for the operations user.
• setup_zshrc: Configures the Zsh shell environment for root and operations users.
• set_zsh_as_default_shell: Sets Zsh as the default shell.
• disable_swap: Disables swap, which is recommended for Kubernetes.
• configure_modules: Loads necessary kernel modules.
• configure_sysctl: Configures sysctl parameters for networking.
• configure_containerd: Configures the containerd runtime.
• restart_containerd: Restarts and enables the containerd service.
• add_kubernetes_repo: Adds the Kubernetes apt repository.

Environment Variables

The script behavior can be controlled using the following environment variables:

• BOOTSTRAP_SERVERS: A comma-separated list of FQDNs of the remote servers to bootstrap.
• BOOTSTRAP_OPERATIONS_PASSWORD: The password for the operations user on remote servers.
• BOOTSTRAP_ROOT_PASSWORD: The root password on the remote servers.

Notes

• Non-Interactive Modes: The script is designed to run non-interactively and should not prompt for input during execution.
• Error Handling: All errors are logged, and a summary is displayed at the end of execution.
• Profiles and Shell Configurations: The script avoids triggering user profiles that may initiate updates or other interactive behaviors.

Troubleshooting

• Permission Denied Errors: Ensure you have the necessary permissions and that password authentication is enabled on remote servers.
• SSH Host Key Verification: The script ignores SSH host key verification for automation purposes.
• Missing Dependencies: If the script reports missing commands (e.g., expect), install them using your system's package manager.

---

Contact Information

For any questions or issues, please contact:

• Name: Troy Klly
• Email: [email protected]

install.sh

#!/usr/bin/env bash
#
# install.sh
#
# This script bootstraps new VMs, performing various system configurations and installations.
# It can also bootstrap remote servers by connecting via SSH and executing this script remotely.
#
# Author: Troy Kelly
# Contact: [email protected]
# Date: Saturday 2 November 2024
#
# Code History:
# - Created on Saturday 2 November 2024 by Troy Kelly
# - Updated on Saturday 2 November 2024 to make the script idempotent
# - Updated on Saturday 2 November 2024 to add bootstrap functionality for remote servers
# - Updated on Saturday 2 November 2024 to fix server parsing in bootstrap function
# - Updated on Saturday 2 November 2024 to fix syntax errors in function calls
# - Updated on Saturday 2 November 2024 to prevent shell profiles from triggering during SSH and su commands
# - Updated on Saturday 2 November 2024 to execute remote commands directly to avoid interactive prompts
# - Updated on Saturday 2 November 2024 to fix quoting issues in the expect script
# - Updated on Saturday 2 November 2024 to fix syntax error due to mismatched 'fi' in configure_sudoers function
# - Updated on Saturday 2 November 2024 to correct variable expansion in the expect script
# - Updated on Saturday 2 November 2024 to replace process substitution with pipe and simplify remote command
# - Updated on Saturday 2 November 2024 to adjust sysctl settings to support Kubernetes dual-stack networking without breaking IPv6 SLAAC
#
# This script adheres to the Google Bash Style Guide and best practices.

# Exit immediately if a command exits with a non-zero status,
# and treat unset variables as an error.
set -euo pipefail

# Global variables
declare -a ERRORS=()
OPERATIONS_USER="operations"

# Function to log errors
log_error() {
  local message="$1"
  echo "ERROR: ${message}" >&2
  ERRORS+=("${message}")
}

# Function to display the markdown report
display_report() {
  echo -e "\n## Installation Report\n"
  if [[ "${#ERRORS[@]}" -eq 0 ]]; then
    echo -e "All tasks completed successfully."
    exit 0
  else
    echo -e "The following errors occurred during installation:\n"
    for error in "${ERRORS[@]}"; do
      echo "- ${error}"
    done
    exit 1
  fi
}

# Function to bootstrap remote servers
bootstrap_remote_servers() {
  echo "Bootstrapping remote servers..."

  # Parse BOOTSTRAP_SERVERS into an array
  # Handle commas, spaces, or comma-space combinations
  IFS=',' read -ra servers <<<"${BOOTSTRAP_SERVERS//[[:space:]]/,}"
  # Remove any empty entries in the array
  servers=("${servers[@]}")

  # Check if 'expect' is installed on local machine
  if ! command -v expect &>/dev/null; then
    echo "Error: 'expect' is required but not installed. Please install it on your local machine."
    exit 1
  fi

  for server in "${servers[@]}"; do
    # Trim whitespace and any leading/trailing commas
    server=$(echo "$server" | tr -d '[:space:]' | sed 's/^,*//;s/,*$//')
    echo "Processing server: ${server}"

    # Define the remote command
    # Replaced process substitution with pipe to avoid shell limitations
    remote_command="bash --noprofile --norc -c 'su -m root -c \"bash --noprofile --norc -c \\\"if ! command -v curl >/dev/null 2>&1; then apt-get update && apt-get install -y curl; fi; curl -fsSL https://gist.githubusercontent.com/troykelly/f76efa97a3bf77ae0d4b0e3c82e61455/raw/install.sh | bash\\\"\"'"

    # Create temporary expect script
    expect_script=$(mktemp)

    cat >"${expect_script}" <<EOF
#!/usr/bin/env expect
# Expect script to automate SSH and 'su' interactions
# Author: Troy Kelly
# Date: Saturday 2 November 2024

set timeout -1
log_user 1

# Variables passed from the shell script
set server "${server}"
set operations_user "${OPERATIONS_USER}"
set operations_password "${BOOTSTRAP_OPERATIONS_PASSWORD}"
set root_password "${BOOTSTRAP_ROOT_PASSWORD}"
set remote_command {${remote_command}}

# Spawn SSH session executing remote command
spawn ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \$operations_user@\${server} \$remote_command

# Handle SSH login
expect {
  "(yes/no)?*" {
    send "yes\r"
    exp_continue
  }
  "*assword:" {
    send "\$operations_password\r"
    exp_continue
  }
}

# Handle 'su' password prompt
expect {
  "*assword:" {
    send "\$root_password\r"
    exp_continue
  }
  eof
}
EOF

    chmod +x "${expect_script}"

    # Execute the expect script
    if "${expect_script}"; then
      echo "Bootstrap completed successfully on ${server}"
    else
      echo "Bootstrap failed on ${server}"
      ERRORS+=("Bootstrap failed on ${server}")
    fi

    # Remove temporary expect script
    rm -f "${expect_script}"
    echo "Finished processing server: ${server}"
  done

  display_report
}

# 1. Update package lists
update_apt() {
  echo "Updating package lists..."
  if ! apt-get update; then
    log_error "Failed to update package lists."
  fi
}

# 2. Perform full upgrade
full_upgrade() {
  echo "Performing full upgrade..."
  if ! DEBIAN_FRONTEND=noninteractive apt-get -y full-upgrade; then
    log_error "Failed to perform full upgrade."
  fi
}

# 3. Install required packages
install_packages() {
  echo "Installing required packages..."
  local packages=(
    apt-transport-https
    build-essential
    ca-certificates
    containerd
    curl
    git
    gpg
    "linux-headers-$(uname -r)"
    rsyslog
    sudo
    unzip
    vim
    wget
    zplug
    zsh
  )
  if ! DEBIAN_FRONTEND=noninteractive apt-get -y install "${packages[@]}"; then
    log_error "Failed to install required packages."
  fi
}

# 4. Configure sudoers for operations user
configure_sudoers() {
  echo "Configuring sudoers for operations user..."
  local sudoers_file="/etc/sudoers.d/${OPERATIONS_USER}"
  if [[ ! -f "${sudoers_file}" ]]; then
    if ! echo "${OPERATIONS_USER} ALL=(ALL) NOPASSWD: ALL" >"${sudoers_file}"; then
      log_error "Failed to configure sudoers file."
    fi
    if ! chmod 0440 "${sudoers_file}"; then
      log_error "Failed to set permissions on sudoers file."
    fi
  else
    echo "Sudoers file for ${OPERATIONS_USER} already exists. Skipping."
  fi
}

# 5. Configure rsyslog
configure_rsyslog() {
  echo "Configuring rsyslog..."
  local rsyslog_conf="/etc/rsyslog.d/log001_sy3.conf"
  local content="*.* @log001.public-servers.sy3.aperim.net:1514;RSYSLOG_SyslogProtocol23Format"
  if [[ ! -f "${rsyslog_conf}" ]]; then
    if ! echo "${content}" >"${rsyslog_conf}"; then
      log_error "Failed to configure rsyslog."
    fi
  else
    echo "Rsyslog configuration already exists. Overwriting."
    if ! echo "${content}" >"${rsyslog_conf}"; then
      log_error "Failed to overwrite rsyslog configuration."
    fi
  fi
  echo "Restarting and enabling rsyslog service..."
  if ! systemctl restart rsyslog; then
    log_error "Failed to restart rsyslog."
  fi
  if ! systemctl enable rsyslog; then
    log_error "Failed to enable rsyslog to start on boot."
  fi
}

# 6-9. Setup SSH authorized_keys for operations user
setup_ssh() {
  echo "Setting up SSH authorized_keys for ${OPERATIONS_USER} user..."
  local ssh_dir="/home/${OPERATIONS_USER}/.ssh"
  local auth_keys="${ssh_dir}/authorized_keys"
  local key="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMamEPOMJGAAw9/zIIrQ2GVzTtI6wkqNQ2GL2Y184Fot Termius k8s-1"

  if ! id "${OPERATIONS_USER}" &>/dev/null; then
    echo "User ${OPERATIONS_USER} does not exist. Creating user."
    if ! useradd -m -s /bin/bash "${OPERATIONS_USER}"; then
      log_error "Failed to create user ${OPERATIONS_USER}."
      return
    fi
  fi

  if ! mkdir -p "${ssh_dir}"; then
    log_error "Failed to create .ssh directory for ${OPERATIONS_USER} user."
  fi
  if ! chown -R "${OPERATIONS_USER}:${OPERATIONS_USER}" "${ssh_dir}"; then
    log_error "Failed to set ownership of .ssh directory."
  fi
  if ! chmod 700 "${ssh_dir}"; then
    log_error "Failed to set permissions on .ssh directory."
  fi
  if [[ -f "${auth_keys}" ]]; then
    if grep -q "${key}" "${auth_keys}"; then
      echo "SSH key already exists in authorized_keys. Skipping."
    else
      echo "Adding SSH key to authorized_keys."
      if ! echo "${key}" >>"${auth_keys}"; then
        log_error "Failed to append SSH key to authorized_keys."
      fi
    fi
  else
    echo "Creating authorized_keys file with the SSH key."
    if ! echo "${key}" >"${auth_keys}"; then
      log_error "Failed to write authorized_keys."
    fi
  fi
  if ! chown "${OPERATIONS_USER}:${OPERATIONS_USER}" "${auth_keys}"; then
    log_error "Failed to set ownership of authorized_keys."
  fi
  if ! chmod 600 "${auth_keys}"; then
    log_error "Failed to set permissions on authorized_keys."
  fi
}

# 10-12. Setup .zshrc for root and operations user
setup_zshrc() {
  echo "Setting up .zshrc for root and ${OPERATIONS_USER} user..."
  local zshrc_url="https://gist.githubusercontent.com/troykelly/5636d6a5d2190ed6152733839387038c/raw/example.zshrc"

  # For root
  if ! curl -L "${zshrc_url}" -o /root/.zshrc.new; then
    log_error "Failed to download .zshrc for root."
  else
    mv /root/.zshrc.new /root/.zshrc
  fi

  # For operations user
  if ! cp /root/.zshrc "/home/${OPERATIONS_USER}/.zshrc.new"; then
    log_error "Failed to copy .zshrc to ${OPERATIONS_USER} user."
  else
    mv "/home/${OPERATIONS_USER}/.zshrc.new" "/home/${OPERATIONS_USER}/.zshrc"
    if ! chown "${OPERATIONS_USER}:${OPERATIONS_USER}" "/home/${OPERATIONS_USER}/.zshrc"; then
      log_error "Failed to set ownership of .zshrc for ${OPERATIONS_USER}."
    fi
  fi
}

# 13. Set zsh as the default shell for root and operations user
set_zsh_as_default_shell() {
  echo "Setting zsh as the default shell for root and ${OPERATIONS_USER} user..."
  local zsh_path
  zsh_path="$(which zsh)"
  if ! chsh -s "${zsh_path}" root; then
    log_error "Failed to change default shell to zsh for root."
  fi
  if ! chsh -s "${zsh_path}" "${OPERATIONS_USER}"; then
    log_error "Failed to change default shell to zsh for ${OPERATIONS_USER}."
  fi
}

# 14. Disable swap
disable_swap() {
  echo "Disabling swap..."

  # Disable all swap devices (immediately)
  if /usr/sbin/swapon --show | grep -q '^'; then
    if ! /usr/sbin/swapoff -a; then
      log_error "Failed to disable swap."
    else
      echo "Swap disabled."
    fi
  else
    echo "Swap is already disabled."
  fi

  # Comment out any swap entries in /etc/fstab
  if grep -Eqs '^\s*[^#]*\s+swap\s' /etc/fstab; then
    if ! sed -i.bak '/\s\+swap\s\+/s/^/#/' /etc/fstab; then
      log_error "Failed to comment out swap entries in /etc/fstab."
    else
      echo "Commented out swap entries in /etc/fstab."
    fi
  else
    echo "No swap entries found in /etc/fstab."
  fi

  # Find and mask any swap units managed by systemd
  echo "Processing systemd swap units..."

  # Get all swap units (both enabled and disabled)
  swap_units=$(systemctl list-unit-files --type=swap --all | awk '/\.swap/ {print $1}')

  for unit in $swap_units; do
    echo "Processing swap unit: $unit"

    # Stop the unit if it's active
    if systemctl is-active --quiet "$unit"; then
      if ! systemctl stop "$unit"; then
        log_error "Failed to stop swap unit $unit"
      else
        echo "Stopped swap unit $unit"
      fi
    fi

    # Disable the unit if it's enabled
    if systemctl is-enabled --quiet "$unit"; then
      if ! systemctl disable "$unit"; then
        log_error "Failed to disable swap unit $unit"
      else
        echo "Disabled swap unit $unit"
      fi
    fi

    # Mask the unit to prevent it from being started
    if ! systemctl mask "$unit"; then
      log_error "Failed to mask swap unit $unit"
    else
      echo "Masked swap unit $unit"
    fi
  done

  # Reload systemd daemon
  if ! systemctl daemon-reload; then
    log_error "Failed to reload systemd daemon."
  fi

  echo "Swap has been disabled and swap units have been masked."
}

# 15-18. Load necessary kernel modules
configure_modules() {
  echo "Loading necessary kernel modules..."
  local modules_conf="/etc/modules-load.d/containerd.conf"
  {
    echo "overlay"
    echo "br_netfilter"
  } >"${modules_conf}"

  if ! /usr/sbin/modprobe overlay; then
    log_error "Failed to load overlay module."
  fi
  if ! /usr/sbin/modprobe br_netfilter; then
    log_error "Failed to load br_netfilter module."
  fi
}

# 19-22. Configure sysctl parameters
configure_sysctl() {
  echo "Configuring sysctl parameters for Kubernetes networking..."
  local sysctl_conf="/etc/sysctl.d/99-kubernetes-k8s.conf"
  {
    echo "net.bridge.bridge-nf-call-ip6tables = 1"
    echo "net.bridge.bridge-nf-call-iptables  = 1"
    echo "net.ipv4.ip_forward                 = 1"
    # Enable IPv6 forwarding on default interfaces to prevent breaking SLAAC
    echo "net.ipv6.conf.default.forwarding    = 1"
    # Ensure accept_ra is set to 2 to accept RAs even if forwarding is enabled
    echo "net.ipv6.conf.all.accept_ra         = 2"
    echo "net.ipv6.conf.default.accept_ra     = 2"
  } >"${sysctl_conf}"

  # Apply sysctl settings
  if ! /usr/sbin/sysctl --system; then
    log_error "Failed to apply sysctl settings."
  fi
}

# 23-24. Configure containerd
configure_containerd() {
  echo "Configuring containerd..."
  local config_file="/etc/containerd/config.toml"
  if [[ ! -f "${config_file}" ]]; then
    if ! containerd config default >"${config_file}"; then
      log_error "Failed to generate default containerd configuration."
    fi
  fi
  if grep -q 'SystemdCgroup = false' "${config_file}"; then
    if ! sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' "${config_file}"; then
      log_error "Failed to update containerd configuration."
    fi
  else
    echo "Containerd is already configured with SystemdCgroup = true."
  fi
}

# 25-26. Restart and enable containerd
restart_containerd() {
  echo "Restarting and enabling containerd service..."
  if ! systemctl restart containerd; then
    log_error "Failed to restart containerd."
  fi
  if ! systemctl enable containerd; then
    log_error "Failed to enable containerd to start on boot."
  fi
}

# 27-28. Add Kubernetes apt repository
add_kubernetes_repo() {
  echo "Adding Kubernetes apt repository..."
  local keyring_dir="/etc/apt/keyrings"
  local keyring_file="${keyring_dir}/kubernetes-apt-keyring.gpg"
  local sources_list="/etc/apt/sources.list.d/kubernetes.list"

  mkdir -p "${keyring_dir}"
  if [[ ! -f "${keyring_file}" ]]; then
    if ! curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o "${keyring_file}"; then
      log_error "Failed to add Kubernetes apt keyring."
    fi
  else
    echo "Kubernetes apt keyring already exists."
  fi

  if [[ ! -f "${sources_list}" ]]; then
    if ! echo "deb [signed-by=${keyring_file}] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" >"${sources_list}"; then
      log_error "Failed to add Kubernetes apt repository."
    fi
  else
    echo "Kubernetes apt repository already exists in sources list."
  fi
}

# Main function to orchestrate the calls
main() {
  # Check for special environment variables
  if [[ -n "${BOOTSTRAP_SERVERS:-}" && -n "${BOOTSTRAP_OPERATIONS_PASSWORD:-}" && -n "${BOOTSTRAP_ROOT_PASSWORD:-}" ]]; then
    # All variables are set, proceed with bootstrap
    bootstrap_remote_servers
  else
    # Ensure script is run as root when executing locally
    if [[ "$EUID" -ne 0 ]]; then
      echo "Please run as root."
      exit 1
    fi
    # Proceed with normal execution
    update_apt
    full_upgrade
    install_packages
    configure_sudoers
    configure_rsyslog
    setup_ssh
    setup_zshrc
    set_zsh_as_default_shell
    disable_swap
    configure_modules
    configure_sysctl
    configure_containerd
    restart_containerd
    add_kubernetes_repo
    display_report
  fi
}

# Invoke main function
main