ssh mystery nixosify + mindcraft

intro.md

hello folks; I'm working on some github CI/CD pipelines, and getting stomped with some ssh issues. I made a github action called nixosify which is an script which sshes into a VM and install a distribution on it. The nixosify repo has a test (shared in the gist) which works fine, but when I call this in a pipeline for deploying a server in another repo I am working on called mindcraft, I get ssh errors, and I cannot for the life of me figure out why the same code gives an error in one context but not the other.

I already recreated the ssh keys to ensure I didn't make a mistake there but I am having no luck here. In both cases I create the VM using the VPS provider's CLI and reference a public ssh key that I added to my account.

Links:

• Full mindcraft repo: https://github.com/clusterplay/mindcraft
• Full mindcraft pipeline run log: https://github.com/clusterplay/mindcraft/runs/599258535
• Full nixosify repo: https://github.com/sparkletco/nixosify
• Full nixosify pipeline run log: https://github.com/sparkletco/nixosify/runs/598589621

mindcraft-pipeline.yml

on: [push]

env:
  name: 'mindcraft'
  size: 'cx11'
  float_ip: 95.217.25.209
  float_ip_id: 236720
  host_status: 'unknown'
  HCLOUD_TOKEN: ${{ secrets.hcloud_token }}

jobs:
  create-host:
    runs-on: ubuntu-latest
    name: Create minecraft server
    steps:

      # prep environment
      - name: Checkout
        uses: actions/checkout@v2
      - name: Install depedencies
        env:
          hcloud_binary: https://github.com/hetznercloud/cli/releases/download/v1.16.2/hcloud-linux-amd64.tar.gz
        run: |
          curl -L $hcloud_binary --output ./hcloud.tar.gz
          tar -xf ./hcloud.tar.gz
          sudo cp ./hcloud /usr/local/bin/hcloud
          sudo apt-get install jq
      - name: Setup keys
        env:
          authkey: ${{ secrets.mindcraft_key }}
          authkey_pub: ${{ secrets.mindcraft_key_pub }}
        run: |
          echo "${authkey}" > ~/authkey
          chmod 600 ~/authkey
          echo "${authkey_pub}" > ~/authkey.pub
          chmod 644 ~/authkey.pub
      # check current deployment
      - name: Check if server exist
        id: check-existence
        run: |
          if hcloud server describe $name &> /dev/null; 
          then 
            host_status='found'
          else 
            host_status='missing'
          fi
          echo "::set-env name=host_status::$host_status"
      # backup current deployment (if found)
      - name: Stop minecraft from saving
        id: stop-saving
        if: env.host_status == 'found'
        run: |
          echo "stop minecraft from saving"
      - name: Create hcloud snapshot
        if: steps.stop-saving.outcome == 'success'
        run: |
          echo "create minecraft snapshot "
      # create fresh server (if missing)
      - name: Create server
        if: env.host_status == 'missing'
        id: create-server
        run: |
          hcloud server create \
            --name $name \
            --type $size \
            --image ubuntu-18.04 \
            --ssh-key mindcraft
          host_ip=$(hcloud server describe $name -o json | jq -r .public_net.ipv4.ip)
          echo "::set-env name=host_ip::${host_ip}"
          echo "::set-env name=host_status::created"
      - name: Nixosify host
        if: env.host_status == 'created'
        uses: sparkletco/nixosify@a5b9049
        id: nixosify
        with:
          target: ${{ env.host_ip }}
          tempkey: ${{ secrets.mindcraft_key }}
          tempkey_pub: ${{ secrets.mindcraft_key_pub }}
          authkey_pub: ${{ secrets.mindcraft_key_pub }}
      - name: Assign floating ip
        if: steps.nixosify.outcome == 'success'
        id: assign-ip
        run: |
          echo "${float_ip_id}"
          echo "${name}"
          hcloud floating-ip assign ${float_ip_id} ${name}
          ssh -i ~/authkey "root@${host_ip}" "ip addr add ${float_ip} dev eth0"
          echo "::set-env name=host_status::ready"
      # restore latest snapshot

      # rebuild with new configuration
      - name: Upload configuration and rebuild
        if: env.host_status == 'ready'
        id: rebuild-server
        run: |
          scp -i ~/authkey ./configuration.nix "root@${host_ip}:/etc/nixos/configuration.nix"
          ssh -i ~/authkey "root@${host_ip}" 'nixos-rebuild switch'

mindcraft-run.log

<<Pass keys>>


<<Move key to target>>

ssh: connect to host 95.217.6.219 port 22: Connection refused
lost connection

<<Copy nixos kexec tarball>>

ssh: connect to host 95.217.6.219 port 22: Connection refused
lost connection

<<Booting into NixOS with kexec>>

ssh: connect to host 95.217.6.219 port 22: Connection refused

<<Waiting for kexec to boot up NixOS>>

Warning: Permanently added '95.217.6.219' (ECDSA) to the list of known hosts.
Warning: Permanently added '95.217.6.219' (ECDSA) to the list of known hosts.
Warning: Permanently added '95.217.6.219' (ECDSA) to the list of known hosts

nixosify-pipeline.yml

on: [push]

env:
  prefix: 'test-nixosify'
  HCLOUD_TOKEN: ${{ secrets.hcloud_token }}

jobs:
  test-nixosify:
    runs-on: ubuntu-latest
    name: Test nixosify action
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Install depedencies
        env:
          hcloud_binary: https://github.com/hetznercloud/cli/releases/download/v1.16.2/hcloud-linux-amd64.tar.gz
        run: |
          curl -L $hcloud_binary --output ./hcloud.tar.gz
          tar -xf ./hcloud.tar.gz
          sudo cp ./hcloud /usr/local/bin/hcloud
          sudo apt-get install jq
      - name: Create test servers
        id: create-test-servers
        env:
          number: 1
          size: 'cx11'
          image: 'ubuntu-18.04'
        run: |
          host_info=""
          for n in `seq ${number}`
          do
            name="${prefix}-${n}"
            hcloud server create \
              --name ${name} \
              --type ${size} \
              --image ${image} \
              --ssh-key nixosify
            ip=$(hcloud server describe ${name} -o json | jq -r .public_net.ipv4.ip)
            host_info="${host_info}${name}\t${ip}\n"
          done
          echo "::set-output name=host_info::$host_info"
      - name: Get host IP
        id: get-host-ip
        env:
          host_info: ${{ steps.create-test-servers.outputs.host_info }}
        run: |
          host_ip=$(printf $host_info | awk '{print $2}')
          echo "$host_ip"
          echo "::set-output name=host_ip::$host_ip"
      - name: Test nixosify
        uses: ./
        id: nixosify
        with:
          target: ${{ steps.get-host-ip.outputs.host_ip }}
          tempkey: ${{ secrets.nixosify_key }}
          tempkey_pub: ${{ secrets.nixosify_key_pub }}
          authkey: ${{ secrets.nixosify_key_pub }}
      - name: Delete test servers
        if: always()
        run: |
          hcloud server list -o noheader -o columns=name \
            | awk -v pattern="$prefix" '$0 ~ pattern' \
            | while read -r host; 
          do 
            hcloud server delete $host 
          done

nixosify-run.log

<<Pass keys>>


<<Move key to target>>

Warning: Permanently added '95.217.6.219' (ECDSA) to the list of known hosts.

<<Copy nixos kexec tarball>>

Warning: Permanently added '95.217.6.219' (ECDSA) to the list of known hosts.

<<Booting into NixOS with kexec>>

Warning: Permanently added '95.217.6.219' (ECDSA) to the list of known hosts.
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://mirror.hetzner.de/ubuntu/packages bionic InRelease [242 kB

.....