Deploy a web app on a dedicated server with Kamal

DEPLOY_WITH_KAMAL_ON_DEDICATED_SERVER.md

Notes

This guide uses Kamal 2.5.3

Motivation

Kamal was designed with 1 service = 1 droplet/VPS in mind.
But I'm cheap and I want to be able to deploy multiple demo/poc apps apps on my $20/month dedicated server.
What the hell, I'll even host my private container registry on it.

Setup your dedicated server

• https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-22-04
• https://www.digitalocean.com/community/tutorials/how-to-set-up-minio-object-storage-server-in-standalone-mode-on-ubuntu-20-04
• https://min.io/docs/minio/linux/integrations/generate-lets-encrypt-certificate-using-certbot-for-minio.html

Setup your domain names

• Add A/AAAA record in your DNS records for server.mydomain.com, registry.mydomain.com, myapp1.mydomain.com and myapp2.mydomain.com to the IP of your dedicated server

Setup a private container registry

• https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-22-04

Note 1: This tutorial says you need a host server and a client server. In our case, we will use only one server to be both client and server.

Note 2: The port 5000 is already used by datadog-agent by default, so I prefer using 7000:5000 instead.

Install SSL certificates for the registry

Docker will have full control of port 80, so we need to choose another port. I'll pick 8443.

• Run sudo certbot certonly --nginx -d registry.mydomain.com
• Add to /etc/nginx/sites-enabled/registry:

  server {
    ...
  
    listen 8443 ssl;
    listen [::]:8443 ssl;
  
    ssl_certificate /etc/letsencrypt/live/registry.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/registry.domain.com/privkey.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;
    
    location / {
      ...
      
      proxy_pass                          http://localhost:7000;
      ...
    }
  }

Install Kamal in your web apps

• Run Docker locally
• Follow: [https://kamal-deploy.org/docs/installation]
• Add registry/database credentials with EDITOR=vi bin/rails credentials:edit

kamal_registry:
  username: ubuntu
  password: #<REGISTRY_PASSWORD>
production:
  db_username: #<POSTGRES_USER>
  db_password: #<POSTGRES_PASSWORD>

• Update your database.yml file

production:
  primary: &primary_production
    <<: *default
    host: <%= ENV["DB_HOST"] %>
    database: <%= ENV.fetch("POSTGRES_DB", "myapp1_production") %>
    username: <%= ENV["POSTGRES_USER"] %>
    password: <%= ENV["POSTGRES_PASSWORD"] %>
  cache:
    <<: *primary_production
    database: <%= ENV.fetch("POSTGRES_DB", "myapp1_production") %>_cache
    migrations_paths: db/cache_migrate
  queue:
    <<: *primary_production
    database: <%= ENV.fetch("POSTGRES_DB", "myapp1_production") %>_queue
    migrations_paths: db/queue_migrate
  cable:
    <<: *primary_production
    database: <%= ENV.fetch("POSTGRES_DB", "myapp1_production") %>_cable
    migrations_paths: db/cable_migrate

• Update you .kamal/secrets

RAILS_MASTER_KEY=$(cat config/master.key)
KAMAL_REGISTRY_USERNAME=$(bin/rails runner "print Rails.application.credentials.kamal_registry.username")
KAMAL_REGISTRY_PASSWORD=$(bin/rails runner "print Rails.application.credentials.kamal_registry.password")
POSTGRES_USER=$(bin/rails runner "print Rails.application.credentials.production.db_username")
POSTGRES_PASSWORD=$(bin/rails runner "print Rails.application.credentials.production.db_password")

• Create a file config/init.sql

CREATE DATABASE myapp1_production;
CREATE DATABASE myapp1_production_cache;
CREATE DATABASE myapp1_production_queue;
CREATE DATABASE myapp1_production_cable;

• Edit production.rb in your app to enable assume_ssl and disable force_ssl

config.assume_ssl = true
config.force_ssl = false

• Update deploy.yml

The deploy.yml file should look like this:

# Name of your application. Used to uniquely configure containers.
service: myapp1

# Name of the container image.
image: mydomain/myapp1

# Deploy to these servers.
servers:
  web:
    - server.mydomain.com

# On a small dedicated server, it can take up to 40s to boot.
deploy_timeout: 40

# Note: If using Cloudflare, set encryption mode in SSL/TLS setting to "Full" to enable CF-to-app encryption.
proxy:
  ssl: true
  host: myapp1.mydomain.com

# Credentials for your image host.
registry:
  server: registry.mydomain.com:8443

  username:
    - KAMAL_REGISTRY_USERNAME
  password:
    - KAMAL_REGISTRY_PASSWORD

env:
  secret:
    - RAILS_MASTER_KEY
    - POSTGRES_USER
    - POSTGRES_PASSWORD
  clear:
    DB_HOST: myapp1-db
    POSTGRES_DB: myapp1_production
    # Run the Solid Queue Supervisor inside the web server's Puma process to do jobs.
    # When you start using multiple servers, you should split out job processing to a dedicated machine.
    SOLID_QUEUE_IN_PUMA: true

    # Serve static files from Rails
    RAILS_SERVE_STATIC_FILES: 1

# Aliases are triggered with "bin/kamal <alias>". You can overwrite arguments on invocation:
# "bin/kamal logs -r job" will tail logs from the first server in the job section.
aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"

# Use a persistent storage volume for sqlite database files and local Active Storage files.
# Recommended to change this to a mounted volume path that is backed up off server.
volumes:
  - "myapp1_storage:/rails/storage"

# Bridge fingerprinted assets, like JS and CSS, between versions to avoid
# hitting 404 on in-flight requests. Combines all files from new and old
# version inside the asset_path.
asset_path: /rails/public/assets

# Configure the image builder.
builder:
  arch: amd64

# Use a different ssh user than root
ssh:
  user: ubuntu

# Use accessory services (secrets come from .kamal/secrets).
accessories:
  db:
    image: postgres:17
    host: server.mydomain.com
    port: 127.0.0.1:8001:5432 # This port must be unique for each app. Here I chose 8001
    env:
      clear:
        POSTGRES_DB: myapp1_production
        POSTGRES_HOST_AUTH_METHOD: trust
      secret:
        - POSTGRES_USER
        - POSTGRES_PASSWORD
    files:
      - config/init.sql:/docker-entrypoint-initdb.d/setup.sql
    directories:
      - data:/var/lib/postgresql/data

• Run bin/kamal setup
• Repeat the same for your other app with a different port for postgresql (8002)

Visit website

• Open https://app1.mydomain.com in your browser

Run console (for Rails)

• bin/kamal console

Notes

TailwindCSS

There is an issue with Tailwindcss 4. Revert back to Tailwindcss 3 until the issue is fixed.

Min.io

In order to run min.io at startup, run: sudo systemctl enable minio.service
If you want to use other ports than the default ones (9000,9001), edit the file: /etc/default/minio and write: MINIO_OPTS="--certs-dir /home/ubuntu/.minio/certs --address :<NEW API PORT> --console-address :<NEW CONSOLE PORT>"

In order to access min.io from the port 443, use Cloudflare and add the rule: (http.host eq "minio.mydomain.com") => Destination Port/Rewrite to/<MINIO PORT>

It might be necessary to add chmod 711 /home/ubuntu to allow minio-user access to the certs.

Renewing the registry certificate (involves downtime)

docker ps
# find the container ID of kamal-proxy (eg. ae6df7151e5e)
docker stop ae6df7151e5e
sudo certbot certonly --standalone -d registry.mydomain.com
docker start ae6df7151e5e
sudo systemctl restart nginx

Out of curiosity, if each app gets its own traefik instance which use a different host port, why the need for the routing (traefik.http.routers.my-app-1-web.entrypoints: my-app-1-web)? I would imagine that when Kamal deploys, it tells the traefik instance for app1 to bind to the exposed ports on app1, and same for app2.

In other words, I've seen the usage of traefik router when you have 1 traefik instance, and it needs to conditionally route to app1 or app2, but if you have 2 traefik instances, wouldn't they just know from how Kamal deploys them to route everything to/from the container they deploy with?

@gregschmit You're right! I updated the gist!

@superp I managed to make it work with the traefik: false configuration and updated the guide!

Note: Adding a worker service for solid queue will cause a gateway timeout error (no idea why). The only workaround I've found was to run solid queue with puma via plugin :solid_queue

Note for upgrading to Kamal 2:
Use port 8443 for registry because Kamal 2 needs the ports 80 and 443 to be completely available.
We can use Cloudflare to make the registry available to port 443.