Running SI local stack using lima-vm

Running_SI_local_stack_using_lima-vm.md

Running SI local stack using lima-vm

Prerequisites

1. Install lima-vm https://github.com/lima-vm/lima
2. Editor/IDE with availability to open directories by SSH (for example VS Code + Remote SSH)

Launch lima instance

limactl start --name=si-dev --tty=false https://gist.githubusercontent.com/viatcheslavmogilevsky/79f58150dd4d89a5b068e015770e5a8e/raw/e2a0553cb413b39a488bb61331576b0fb0c5c76d/si-dev-profile.yaml

Connecting editor to lima instance

1. Generate SSH client config: limactl show-ssh --format=config si-dev > ./si_dev_ssh_conf
2. Use generated SSH client config file in Remote SSH and open connect to that host
3. Launch terminal in VS Code to clone https://github.com/systeminit/si somewhere at home direcory:

cd
git clone https://github.com/systeminit/si.git

4. Open cloned repo VS Code's explorer and launch terminal there:

direnv allow

5. Run stack as usual:

buck2 run dev:healthcheck
buck2 run dev:up

Stopping stuff

Inside lima instance (i.e using VS Code terminal) - stop/down the setup as usual:

buck2 run dev:stop
# buck2 run dev:down

To stop lima instance and keep cloned repo with local data, run the follwing in regular (host) terminal:

limactl stop si-dev

to start again: limactl start si-dev

NOTE rewrite ssh client config after stop/start lima instance by: limactl show-ssh --format=config si-dev > ./si_dev_ssh_conf and re-open remote window in VS Code

Full cleanup

In regular (host) terminal run: limactl delete si-dev - this will be delete whole lima instance, including docker runtime storage, cloned repo with local build files, etc

si-dev-profile.yaml

# A template to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine

# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...

# This template requires Lima v0.8.0 or later
images:
  # Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release-20230729/ubuntu-22.04-server-cloudimg-amd64.img"
    arch: "x86_64"
    digest: "sha256:d5b419272e01cd69bfc15cbbbc5700d2196242478a54b9f19746da3a1269b7c8"
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release-20230729/ubuntu-22.04-server-cloudimg-arm64.img"
    arch: "aarch64"
    digest: "sha256:5ecab49ff44f8e44954752bc9ef4157584b7bdc9e24f06031e777f60860a9d17"
  # Fallback to the latest release image.
  # Hint: run `limactl prune` to invalidate the cache
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
    arch: "x86_64"
  - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
    arch: "aarch64"
mounts:
  - location: "~"
  - location: "/tmp/lima"
    writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
# SPECIFIC: by default disk is set to 100GiB, let's see is 64GiB is sufficient
disk: 64GiB
# SPECIFIC: this is need to keep running all services in the stack
memory: 8GiB

provision:
  - mode: system
    # This script defines the host.docker.internal hostname when hostResolver is disabled.
    # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
    # Names defined in /etc/hosts inside the VM are not resolved inside containers when
    # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
    script: |
      #!/bin/sh
      sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
  - mode: system
    script: |
      #!/bin/bash
      set -eux -o pipefail
      command -v docker >/dev/null 2>&1 && exit 0
      export DEBIAN_FRONTEND=noninteractive
      curl -fsSL https://get.docker.com | sh
      # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
      systemctl disable --now docker
      apt-get install -y uidmap dbus-user-session
  - mode: user
    script: |
      #!/bin/bash
      set -eux -o pipefail
      systemctl --user start dbus
      dockerd-rootless-setuptool.sh install
      docker context use rootless
  # SPECIFIC: install nix+direnv
  - mode: user
    script: |
      #!/bin/bash
      curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install linux --no-confirm
      . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
      nix profile install nixpkgs#direnv
      if ! grep -q -F "eval \"\$(direnv hook bash)\"" ~/.bashrc; then
        echo "eval \"\$(direnv hook bash)\"" >> ~/.bashrc
      fi
probes:
  - script: |
      #!/bin/bash
      set -eux -o pipefail
      if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
        echo >&2 "docker is not installed yet"
        exit 1
      fi
      if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
        echo >&2 "rootlesskit (used by rootless docker) is not running"
        exit 1
      fi
    hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
  - guestSocket: "/run/user/{{.UID}}/docker.sock"
    hostSocket: "{{.Dir}}/sock/docker.sock"
  # SPECIFIC expose ports from services
  - guestPort: 10350
  - guestPort: 4222
  - guestPort: 8222
  - guestPort: 16686
  - guestPort: 55679
  - guestPort: 8080
message: |-
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create lima-{{.Name}} --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use lima-{{.Name}}
  docker run hello-world
  ------