Setup Cloud9 on EC2 and remote Jupyter

remote_jupyter_setup.md

Setup Cloud9 on EC2 and remote Jupyter

Create an EC2 instance on AWS

• Launch: t3.2xlarge ($0.33/h) / m8g.4xlarge ($0.718) / g6.4xlarge ($1.32/h) / p3.2xlarge ($3.02/h)
• Image: Deep Learning AMI (Ubuntu 22.04)
• Configure Security Group:
  • open custom TCP and port 9999
  • open HTTPS, HTTP to anywhere
• Attach an Elastic IP to the instance

ssh into EC2 from MobaXterm and run

sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y && sudo apt install libevent-dev -y;
sudo snap install node --channel=13/stable --classic;

Create SSH environment from Cloud9

1. Add Cloud9's public key to EC2's ~/.ssh/authorized_keys file
2. Specify which node (usually /snap/bin/node) as Node.js binary path under advanced seetings under Cloud9
3. Create ssh environment from Cloud9

Config Jupyter

Concurrently, you can install conda env and jupyter to EC2

mamba update conda -n base -y;
mamba env create a3642578/MLlab -n ml;
source activate ml;
pip install --upgrade jupyterlab-git && jupyter lab build;
echo "conda env installed";
#
echo "generate jupyter password to ~/.jupyter/jupyter_server_config.json"
jupyter server --generate-config;
jupyter server password;
#
echo "generate ssl credentials to ~/.ssh"
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout sslkey.key -out sslcert.pem;
mv sslkey.key ~/.ssh/sslkey.key && mv sslcert.pem ~/.ssh/sslcert.pem;

Add the following to ~/.jupyter/jupyter_server_config.py

# use https
c.ServerApp.certfile = u'/home/ubuntu/.ssh/sslcert.pem'
c.ServerApp.keyfile = u'/home/ubuntu/.ssh/sslkey.key'
# Set ip to '*' to bind on all interfaces (ips) for the public server
c.ServerApp.ip = '*'
c.PasswordIdentityProvider.hashed_password = u'argon2:...<your hashed password here>' #find it in ~/.jupyter/jupyter_server_config.json
c.ServerApp.open_browser = False
c.ServerApp.port = 9999
# At your discretion
c.ServerApp.allow_root = True
c.ServerApp.autoreload = True
c.ContentsManager.allow_hidden = True

Launch Jupyter from Cloud9

Set Cloud9 Project Preferrence Stop my environemnt to never Now launch jupyter from Cloud 9's terminal

Setup tunneling from local

Ubuntu / MacOS:

ssh -i private_key_to_ec2.pem -NfL localhost:9999:localhost:9999 ubuntu@ec2_host

Windows: Using MobaXterm

Enjoy!

https://localhost:9999

Appendix

CodeCommit Tip

git config --global credential.useHttpPath true;
git config --local credential.helper 'store';
git pull;

Mount ephemeral disk

Check available disks.

sudo lsblk -o +UUID

Find disks without a mount point.

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT                  UUID
nvme1n1     259:0    0 209.6G  0 disk                             eecb086f-8fc0-48ba-a7a7-0786fe3d5a56
nvme0n1     259:1    0   128G  0 disk
└─nvme0n1p1 259:2    0   128G  0 part /                           b8faee3a-b82d-4a61-b1f3-e473b6c363cc

Check file system type

sudo lsblk -f

Format and mount an instance store volume

sudo mkfs -t ext4 /dev/nvme1n1
mkdir ~/ephemeral 
sudo mount -t ext4 /dev/nvme1n1 ~/ephemeral 

Automatically mount at reboot

sudo cp /etc/fstab /etc/fstab.bak
## add to 
sudo nano /etc/fstab
UUID=eecb086f-8fc0-48ba-a7a7-0786fe3d5a56  /home/ubuntu/ephemeral  ext4  defaults,nofail  0  2

Test fstab

sudo umount ~/ephemeral
sudo mount -a

Move large files from home to another disk

mv /home/ubuntu/anaconda3/envs/aws_neuron_pytorch_p36 /home/ubuntu/ephemeral/
ln -s /home/ubuntu/ephemeral/aws_neuron_pytorch_p36 /home/ubuntu/anaconda3/envs/aws_neuron_pytorch_p36