Notes on setting up a persistent worker on Hetzner

persistent_worker.md

Beefy workers

• Can't use "projects" on Hetzner as they are for cloud only. Therefore just set up servers manually and connect to each, running cirrus-cli in screen

• Testing TSAN job using 1 AX52 Server

• Version of containernetworking-plugins on Jammy 22.04 Ubuntu is broken for podman. Workaround:

  curl -O http://archive.ubuntu.com/ubuntu/pool/universe/g/golang-github-containernetworking-plugins/containernetworking-plugins_1.1.1+ds1-3build1_amd64.deb
  dpkg -i containernetworking-plugins_1.1.1+ds1-3build1_amd64.deb

  Ref: https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2024394

• Cannot extract packages as permissions are incorrect:

  Fetching boost_1_81_0.tar.gz from https://archives.boost.io/release/1.81.0/source/
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
   63  133M   63 85.1M    0     0   103M      0  0:00:01 --:--:--  0:00:01  103M
  100  133M  100  133M    0     0   105M      0  0:00:01  0:00:01 --:--:--  105M
  /ci_container_base/depends/work/download/boost-1.81.0/boost_1_81_0.tar.gz.temp: OK
  Extracting boost...
  /ci_container_base/depends/sources/boost_1_81_0.tar.gz: OK
  tar: boost/multiprecision/concepts: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/multiprecision/cpp_bin_float: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/multiprecision/cpp_int: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/multiprecision/detail/functions: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/multiprecision/detail: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/multiprecision/traits: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/multiprecision: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/core/matcher: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/core: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/dynamic: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/static/transforms: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/static: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/utility/chset: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail/utility: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/detail: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/traits/detail: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive/traits: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/xpressive: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/convert/detail: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/convert: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/align/detail: Cannot change mode to rwxr-xr-x: Operation not permitted
  tar: boost/align: Cannot change mode to rwxr-xr-x: Operation not permitted
  ...
  

• Pods cannot stop themselves properly. Something to do with AppArmor: https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 Fix is not yet out, but could disable AppArmor in the interim (nasty):

  # Check AppArmor status
  sudo aa-status
  # Shutdown apparmor and prevent it from restarting
  sudo systemctl disable apparmor.service --now
  # Unload AppArmor profiles
  sudo service apparmor teardown
  # Re-check AppArmor status
  sudo aa-status

  Fix I used:

  mkdir -p /etc/containers/containers.conf.d
  echo '[CONTAINERS]
  apparmor_profile=""' > /etc/containers/containers.conf.d/disable-apparmor.conf
  systemctl restart apparmor

• Need to install clang-15 for Tidy and centos job. Edit: I had this set using an env var, perhaps this isn't needed.

• Some tests are out of order: p2p_node_network_limited.py --v[1|2]transport seems to often take 5 mins and is started last. Amend this, and print a warning for tests which take unexpectedly long.