Skip to content

Instantly share code, notes, and snippets.

@johndhancock
Last active December 19, 2019 05:10
Show Gist options
  • Save johndhancock/637b8ec5e544c23f7677fa9a96b17834 to your computer and use it in GitHub Desktop.
Save johndhancock/637b8ec5e544c23f7677fa9a96b17834 to your computer and use it in GitHub Desktop.
How to debug Circle-Ci problems, via a real-world example

Early Oct. 2019, I ran into a problem on our Voter Guide where Circle Ci would fail before tests would even run. I could get a basic sense of what was happening via the Circle-Ci interface, but the problem wasn't immediately evident.

After consulting with Andrew Chavez, he pointed me to a page that explains how to SSH into a build running on Circle-Ci to run the commands step-by-step in an attempt to debug the problem.

After SSH-ing in, and cd-ing into the directory on the running build (in this case, cd voter-guide), you're able to run the commands listed in the circle-ci/config.yaml file in the project and seeing the read-out within the terminal as you go along.

In this instance, part of the setup process of the circle ci box involves running this command, which pulls a setup file for Nodejs v 8.x and installs it:

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -

sudo apt-get install -y nodejs

The next command was to update npm to the latest version:

sudo npm i npm@latest -g

At this point, circle-ci was failing, as it didn't recognize the command npm. This was because the script that gets Nodejs 8.x had been updated to then upate to the latest version of Node after installing it, which was setting the version to 10.x. Version 10.x does not include npm, so, there was no npm command to run.

After doing some digging, I found some other folks had run into this problem. The solution was to inject a command that rolls back Node to a 8.x version after Node is installed and before npm is updated.

apt-get install -y --allow-downgrades nodejs=8.16.1-1nodesource1

This resulted in npm being present, the box getting setup right, and all tests running and passing.

*** NOTE DEC. 2019 ***

The command to roll back to node version 8 keeps changing. To find out what the current node version is, while ssh-ed into the circleci box, you can run this command: apt-cache policy nodejs to find out what the current 8 version is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment