SSH & DevOps Crash Course Snippets

express-nginx.md

How to configure HTTPS with Lets Encrypt, Nginx reverse proxy, Express and Node

1. Have a Node app ready for production.
2. Create an app.js file in your project directory:

const express = require('express');
const path = require('path');
const app = express();

// Allow dotfiles - this is required for verification by Lets Encrypt's certbot
app.use(express.static(path.join(__dirname, 'build'), {dotfiles: 'allow'}));

app.get('*', function (req, res) {
  res.sendFile(path.join(__dirname, 'build', 'index.html'));
});

app.listen(3000);

2. Follow this guide to get your SSL certificates
3. Configure Nginx at /etc/nginx/sites-available/default

# Default server configuration
server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name example.com www.example.com;
	return 301 https://$server_name$request_uri;
}

# Virtual Host/SSL/Reverse proxy configuration for example.com

server {
    # Listen on both HTTP and HTTPS - between Nginx and Express the traffic is HTTP but this is not a major
    # security concern as both services are on the same box
    listen 80;
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    include snippets/ssl-example.com.conf;
    include snippets/ssl-params.conf;

    server_name example.com www.example.com;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    
    # Allow location for Acme challenge - you also might need to allow 'dotfiles' in Express (see next section)
    location ~ /.well-known {
        allow all;
	proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
    }	
}

3. Restart Nginx and start your Express server (I recommend PM2 to manage the process):

sudo systemctl restart nginx

In your project directory:

pm2 start app.js

nginx-cloud.md

Setting up your cloud server:

1. Start an aws account, and start up an EC2 instance. Guide here
2. Add HTTP to your inbound rules in your ec2 instance's security group. Guide here
3. Connect to your ec2 instance with ssh with a command like this:

ssh -i <path to ssh key given to you from amazon> ubuntu@<public ip address of your server>

Set up nginx

(based on the guide here):

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install nginx -y
sudo systemctl status nginx    # To check the status of nginx
sudo systemctl start nginx     # To start nginx
sudo systemctl enable nginx
sudo rm /etc/nginx/sites-available/default
sudo vim /etc/nginx/sites-available/default

Then, copy and paste this into the file you opened in vim, and replace "<your IP address>" with either a domain you own if you have one, or just your public IP address (we'll talk about getting a domain below)

server {
    listen 80;
    server_name <your IP address> <your domain>;
    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
     }
}

Check it works, and load the new config:

sudo nginx -t
sudo systemctl reload nginx

Install node version manager, node.js and node package manager:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash

Then, copy and paste this into your terminal as a command:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

Then, run this to install node and npm:

nvm install node

Then create a project with:

mkdir myapp
cd myapp
npm install express
vim app.js

and copy and paste this into the file you opened with vim:

var express = require('express');
var app = express();
app.get('/', function(req, res){
   res.send("Hello World!");
});
app.listen(8080, 'localhost');

Now run:

node app.js

and your app is running! You won't see any output, but you'll notice you won't exit from the program until you do ctrl+C and stop the server. You should be able to see the words "Hello World" when you visit your server's public IP.

Now, to link your public IP to an actual domain, you will need to go to a domain provider, like namecheap.com or domain.com and buy one. They can cost under $10 a year, and you only ever need to buy one to actually get infinite subdomains (like, if you buy alex.com, you can make websites like meet.alex.com or whos.alex.com).

Assuming you used namecheap, you should now follow this guide to add an A name record to your domain. A name records basically direct a request to your domain to go to an IP address you specific. Set an A name record from "@" (which represents when people visit your website with no subdomain, so like, "alex.com"), that goes to your public IP address. Set the TTL to be as short as possible, so it updates more frequently. Make sure you use your PUBLIC ip address for your AWS instance, not your private one, as web domains can only access your public one.

Now, go back to your nginx config with: sudo vim /etc/nginx/sites-available/default

and change it so you include your URL in the "server_names" argument list if you didn't already:

server {
    listen 80;
    server_name <your IP address> <your domain>;  # REMEMBER TO REPLACE WITH YOUR DOMAIN. DO NOT INCLUDE http:// or https://
    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
     }
}

Now run the steps you should run every time you want to deploy new nginx config changes:

sudo nginx -t
sudo systemctl reload nginx

and if you visit your website at your domain, you should be able to see "Hello World" instead of a 404 error. If it doesn't work initially, try using "ctrl+shift+R" (basically a hard refresh to avoid reviewing a cached version of your website) to reload the page.

Now that you have a basic HTTP website, let's add HTTPS.

We will be following this guide:

Install depedencies

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python3-certbot-nginx

Run the certifying process! (This will allow Let's Encrypt to know that you control this server). Replace "my domain url" with your domain url.

sudo certbot --nginx -d <my domain url>

If the process asks if you'd like to redict HTTP traffic to HTTPS, I would generally say yes, you should, just so your users don't accidentally use HTTP.

If that succeeded, now when people visit your website, they will have encryption! Awesome!! You're a top web developer now. Go make something cool!

nginx-nestjs.md

upstream backend { server localhost:8080; }

upstream websocket { server localhost:8081; }

server { listen 80;

location /uploads { root /server; add_header 'Cache-Control' 'max-age=31536000, public'; add_header 'Access-Control-Allow-Origin' '*'; }

location /socket.io { # add_header 'Access-Control-Allow-Origin' '*'; proxy_pass http://websocket; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $remote_addr; client_max_body_size 10g; }

location / { # add_header 'Access-Control-Allow-Origin' '*'; proxy_pass http://backend; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $remote_addr; client_max_body_size 10g; } }

nginx-setup.md

Nginx reverse proxy setup with express server

Install nginx

sudo apt update
sudo apt install nginx
sudo ufw app list
sudo ufw allow 'Nginx HTTP'
sudo ufw status
systemctl status nginx
curl -4 icanhazip.com
http://localhost # server_ip

Update nginx confifuration

sudo nano /etc/nginx/sites-available/express-auth.test

 server {
        listen 80;
        server_name express-test.test;
        location / {
            # express port number
            proxy_pass http://localhost:4000;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
        }
}

sudo ln -s /etc/nginx/sites-available/express-auth.test /etc/nginx/sites-enabled/
sudo unlink /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

If using local computer

Set the server_name inside hosts local file.

nano /etc/hosts
127.0.0.1       express-auth.test

ssh-new-user.md

Creating a new user on Linux

I prefer to use cmder on Windows. It provides most comment linux commends out of the box.

Create a node.js project

# install express-generator
npm install -g express-generator

# create new express project
# use hogan.js and sass
express --view=hjs --css=sass node-app

# install package
cd node-app
yarn

# start server
yarn start

Testing the server by browsing http://localhost:3000 using Chrome Or curl http://localhost:3000 from terminal

Create Vagrant virtual box

# create a folder to save vagrant vm profile
mkdir vagrant

# install Vagrant box
# https://app.vagrantup.com/boxes/
vagrant box add ubuntu/xenial64
vagrant init ubuntu/xenial64

# start vm
vagrant up
vagrant status

# login vm
vagrant ssh

Edit the generated Vagrantfile, change language mode to Ruby

In Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

$script = <<-SCRIPT
#!/bin/bash

echo "Starting Provision..."
sudo apt-get update

echo "Installing nginx..."
sudo apt-get install nginx -y

echo "Installing nodejs..."
curl -sL https://deb.nodesource.com/setup_10.x -o nodesource_setup.sh
sudo bash nodesource_setup.sh
sudo apt-get install nodejs -y

echo "Installing yarn..."
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update
sudo apt-get install yarn -y

echo "Installing pm2..."
sudo npm -g install pm2

echo "Provision completed"
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"

  config.vm.network "private_network", ip: "10.0.0.10"
  config.vm.provision "shell" do |s|
    s.inline = $script
  end
end

Create user

From remote

# add a user called "luke"
sudo adduser luke

# switch to user
sudo su - luke

whoami

# create SSH connection
cd ~
mkdir .ssh
# 700 owner can read, write and execute
chmod 700 .ssh

# do NOT change the file name, `authorized_keys`
touch .ssh/authorized_keys

# owner can read and write
chmod 600 .ssh/authorized_keys

Copy the public key to .ssh/authorized_keys file

From local machine, on Windows

cat C:\Users\<username>\.ssh\id_rsa.pub

On Mac

cat ~/.ssh/id_rsa.pub

From vm

# paste your public key using vim
vim ~/.ssh/authorized_keys

Login as new user

ssh luke@<ip_address>

Create a compressed file to upload our project

# tar compress, zip, file <output_filename>.tar.gz files...
tar czf test-app.tar.gz app.js bin package.json public routes views

Copy file from local to remote

scp test##app.tar.gz [email protected]:~

Extract file from remote

# create a folder
mkdir test-app

# tar eXtract, file <compressed_file> -C (speCified directory) <folder>
tar xf test-app.tar.gz -C test-app

Start node server

DEBUG=test-node-app:server NODE_ENV=development PORT=8081 yarn start

Configurate PM2

# require log in as root
sudo npm install -g pm2

Create config file

touch pm2.config.js
vim pm2.config.js

In pm2.config.js

module.exports = {
  apps: [
    {
      name: 'node-demo',
      script: './bin/www',
      watch: false, // pm2 kills and restarts your application without indication!
      instances: 'max',
      exec_mode: 'cluster',
      env: {
        PORT: 3001,
        NODE_ENV: 'production'
      },
      env_dev: {
        DEBUG: 'node-demo:server',
        PORT: 3001,
        NODE_ENV: 'development'
      }
    }
  ]
}

Start pm2

pm2 start pm2.config.js

# Check current running processes
pm2 list

Run pm2 as a service

# generate startup script
# provide user and home path
pm2 startup systemd -u luke --hp /home/luke

Returns

sudo env PATH=$PATH:/usr/bin pm2 startup systemd -u luke --hp /home/luke

Save your pm2 configuration!

pm2 save

Test pm2 by rebooting the server

sudo reboot

Automated deployment

# create deploy script
touch deploy.sh

# give it execute right
chmod +x deploy.sh

# run it
./deploy.sh

In deploy.sh file

#!/bin/bash

echo "Creating zip file and copying to remove..."
tar czf app.tar.gz bin public routes views app.js package.json pm2.config.js yarn.lock
scp app.tar.gz [email protected]:~
rm app.tar.gz

echo "Login to remove..."
# the command afterward will be executed on remote
ssh [email protected] << 'ENDSSH'
pm2 stop node-demo
pm2 delete node-demo
echo "pm2 is stopped"
rm -rf app
mkdir app
tar xf app.tar.gz -C app
rm app.tar.gz
cd app
yarn
echo "Starting pm2..."
pm2 start pm2.config.js
pm2 save
ENDSSH

Setup nginx as a Reverse Proxy (listening on Port 80)

• Only root can access Port 80, so the deploy user doesn't have access.

What can nginx do?

• HTTPS
• Listening on Port 80, reverse proxy to node.js server
• Serve static files
• Load balancing

nginx setup

Login remote as root

sudo apt-get update && sudo apt-get install nginx -y

cd /etc/nginx/sites-available
sudo vim default

In /etc/nginx/nginx.conf

The # of cores your CPU is running

worker_processes: auto;

Check the # of cores

cat /proc/cpuinfo

Add proxy configuration into server block in /etc/nginx/sites-available/default file

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                proxy_pass http://localhost:3001;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
        }
}

To map another url, e.g. http://dev.mynodeapp.com/app2

server {
  ...
  location / {
    ...
  }

  location /app2 {
    proxy_pass http://localhost:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }
}

Testing current configuration

sudo nginx -t

Restart nginx

sudo systemctl restart nginx

Testing with Apache Bench

ab -c 40 -n 1000 http://10.0.0.10/

Shutdown Vagrant VM

From vagrant folder

vagrant halt default

ssh.md

SSH Cheat Sheet

This sheet goes along with this SSH YouTube tutorial

Login via SSH with password (LOCAL SERVER)

$ ssh [email protected]

Create folder, file, install Apache (Just messing around)

$ mkdir test

$ cd test

$ touch hello.txt

$ sudo apt-get install apache2

Generate Keys (Local Machine)

$ ssh-keygen

Add Key to server in one command

> cat ~/.ssh/id_rsa.pub | ssh [email protected] "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys

Create & copy a file to the server using SCP

$ touch test.txt $ scp ~/test.txt [email protected]:~

DIGITAL OCEAN

Create account->create droplet

Create Keys For Droplet (id_rsa_do)

$ ssh-keygen -t rsa

Add Key When Creating Droplet

Try logging in

$ ssh root@doserver

If it doesn't work

$ ssh-add ~/.ssh/id_rsa_do (or whatever name you used)

Login should now work

$ ssh root@doserver

Update packages

$ sudo apt update

$ sudo apt upgrade

Create new user with sudo

$ adduser brad

$ id brad

$ usermod -aG sudo brad

$ id brad

Login as brad

> ssh brad@doserver

We need to add the key to brads .ssh on the server, log back in as root

$ ssh root@doserver

$ cd /home/brad

$ mkdir .ssh

$ cd .ssh

$ touch authorized_keys

> sudo nano authorized_keys (paste in the id_rsa_do.pub key, exit and log in as brad)

Disable root password login

$ sudo nano /etc/ssh/sshd_config

Set the following

PermitRootLogin no

PasswordAuthentication no

Reload sshd service

$ sudo systemctl reload sshd

Change owner of /home/brad/* to brad

$ sudo chown -R brad:brad /home/brad

May need to set permission

$ chmod 700 /home/brad/.ssh

Install Apache and visit ip

$ sudo apt install apache2 -y

Github

Generate Github Key(On Server)

$ ssh-keygen -t rsa (id_rsa_github or whatever you want)

Add new key

$ ssh-add /home/brad/.ssh/id_rsa_github

If you get a message about auth agent, run this and try again

```$ eval `ssh-agent -s````

Clone repo

$ git clone [email protected]:bradtraversy/react_otka_auth.git

Install Node

$ curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -

$ sudo apt-get install -y nodejs

Install Dependencies

$ npm install

Start Dev Server and visit ip:3000

$ npm start

Build Out React App

$ npm run build

Move static build to web server root

$ sudo mv -v /home/brad/react_otka_auth/build/* /var/www/html