27 things that broke, or nearly did, while updating an old self-hosted Supabase on Coolify (Kong to Envoy, Postgres 15 to 17), plus a script that checks secret lengths without printing values.

README.md

The maintained version of these files lives in the self-hosted-supabase handbook. This gist is a snapshot of 2026-10-20; check the repo for fixes.

Files:

• things-that-broke.md: the short list.
• check-secret-lengths.sh: prints the NAME and LENGTH of each variable in a compose .env (never the value); fails on empty secrets and on anything interpolated into SECRET_KEY_BASE shorter than 64 characters. Usage: bash check-secret-lengths.sh <.env> [docker-compose.yml] [NAME=MIN ...]. Needs bash 4+.

Longer version of every item: https://github.com/shsunmoonlee/self-hosted-supabase.

---

Maintained by Seunghun Lee. I offer production self-hosted SaaS setup as a service: LinkedIn · X.

check-secret-lengths.sh

#!/usr/bin/env bash
# check-secret-lengths.sh: print the NAME and LENGTH of every variable in a compose .env file, and flag the
# ones that are empty or too short for what the compose file feeds them into. Values are never printed.
#
# Usage:
#   bash check-secret-lengths.sh <path/to/.env> [path/to/docker-compose.yml] [NAME=MIN ...]
#
# Examples:
#   bash check-secret-lengths.sh /data/coolify/services/<service-uuid>/.env \
#        /data/coolify/services/<service-uuid>/docker-compose.yml
#   bash check-secret-lengths.sh ./.env ./docker-compose.yml SERVICE_PASSWORD_JWT=32
#
# Built-in rules:
#   - any variable whose name contains PASSWORD, SECRET, KEY or TOKEN must not be empty;
#   - SERVICE_PASSWORD_64_* must be at least 64 characters;
#   - with a compose file: every variable interpolated into SECRET_KEY_BASE (Realtime, Supavisor) must be at
#     least 64 characters (Phoenix refuses a shorter cookie secret), and every variable the compose file
#     references must exist in the .env.
# Extra NAME=MIN arguments add minimum lengths.
#
# Exit codes: 0 all checks pass, 1 at least one FAIL, 2 usage error.
# The script also warns when the .env has no trailing newline (a later `>>` append would glue two lines).

set -euo pipefail
[ "${BASH_VERSINFO[0]}" -ge 4 ] || { echo "needs bash 4 or newer (associative arrays)" >&2; exit 2; }

[ $# -ge 1 ] || { sed -n '2,22p' "$0"; exit 2; }
ENV_FILE=$1; shift
COMPOSE_FILE=""
if [ $# -ge 1 ] && [[ "$1" != *=* ]]; then COMPOSE_FILE=$1; shift; fi
[ -r "$ENV_FILE" ] || { echo "cannot read $ENV_FILE" >&2; exit 2; }
[ -z "$COMPOSE_FILE" ] || [ -r "$COMPOSE_FILE" ] || { echo "cannot read $COMPOSE_FILE" >&2; exit 2; }

declare -A LEN=()
declare -A MIN=()
declare -A WHY=()
ORDER=()

add_min() {
  local name=$1 min=$2 why=$3
  if [ -z "${MIN[$name]:-}" ] || [ "$min" -gt "${MIN[$name]}" ]; then
    MIN[$name]=$min
    WHY[$name]=$why
  fi
}

# Parse NAME=value lines. Values are only measured, never echoed. Surrounding single or double quotes are
# not counted. The last line is read even without a trailing newline.
while IFS= read -r line || [ -n "$line" ]; do
  line=${line%$'\r'}
  case "$line" in ''|'#'*) continue ;; esac
  [[ "$line" == *=* ]] || continue
  name=${line%%=*}
  name=${name#export }
  name=${name//[[:space:]]/}
  [[ "$name" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || continue
  value=${line#*=}
  if [[ ${#value} -ge 2 && ( "$value" == \"*\" || "$value" == \'*\' ) ]]; then
    value=${value:1:${#value}-2}
  fi
  [ -n "${LEN[$name]+x}" ] || ORDER+=("$name")
  LEN[$name]=${#value}
  unset value
done <"$ENV_FILE"

if [ -s "$ENV_FILE" ] && [ "$(tail -c 1 "$ENV_FILE" | od -An -c | tr -d ' ')" != '\n' ]; then
  printf '%s\n' "WARN  $ENV_FILE has no trailing newline: start any append with a newline (printf '\\nNAME=...')."
fi

for name in "${ORDER[@]}"; do
  case "$name" in
    SERVICE_PASSWORD_64_*) add_min "$name" 64 "SERVICE_PASSWORD_64_* is meant to be 64 characters" ;;
    *PASSWORD*|*SECRET*|*KEY*|*TOKEN*) add_min "$name" 1 "secret-like name" ;;
  esac
done

for arg in "$@"; do
  [[ "$arg" =~ ^([A-Za-z_][A-Za-z0-9_]*)=([0-9]+)$ ]] || { echo "bad rule: $arg (expected NAME=MIN)" >&2; exit 2; }
  add_min "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}" "rule given on the command line"
done

MISSING=()
if [ -n "$COMPOSE_FILE" ]; then
  # SECRET_KEY_BASE=${VAR} / SECRET_KEY_BASE: ${VAR} / with or without quotes and a :-default.
  while IFS= read -r var; do
    [ -n "$var" ] && add_min "$var" 64 "interpolated into SECRET_KEY_BASE in $(basename "$COMPOSE_FILE")"
  done < <(grep -oE 'SECRET_KEY_BASE[[:space:]]*[:=][[:space:]]*["'\'']?\$\{[A-Za-z_][A-Za-z0-9_]*' "$COMPOSE_FILE" \
             | sed -E 's/.*\$\{//' | sort -u)
  while IFS= read -r var; do
    [ -n "$var" ] || continue
    [ -n "${LEN[$var]+x}" ] || MISSING+=("$var")
  done < <(grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*' "$COMPOSE_FILE" | sed 's/^\${//' | sort -u)
fi

fail=0
printf '%-6s %-44s %6s %6s  %s\n' STATUS NAME LENGTH MIN WHY
for name in "${ORDER[@]}"; do
  len=${LEN[$name]}
  min=${MIN[$name]:-0}
  status=ok
  if [ "$len" -lt "$min" ]; then status=FAIL; fail=1; fi
  printf '%-6s %-44s %6s %6s  %s\n' "$status" "$name" "$len" "$min" "${WHY[$name]:-}"
done
for name in "${!MIN[@]}"; do
  if [ -z "${LEN[$name]+x}" ]; then
    printf '%-6s %-44s %6s %6s  %s\n' FAIL "$name" absent "${MIN[$name]}" "${WHY[$name]}"
    fail=1
  fi
done
for name in "${MISSING[@]+"${MISSING[@]}"}"; do
  [ -n "${MIN[$name]+x}" ] && continue
  printf '%-6s %-44s %6s %6s  %s\n' WARN "$name" absent - "referenced in the compose file, not in the .env (fine if it has a :-default)"
done

exit "$fail"

things-that-broke.md

Things that broke while updating self-hosted Supabase on Coolify

From one production window on 2026-10-19 (Coolify 4.4.6; Kong 2.8 to Envoy, PostgREST 12 to 14, Realtime 2.34 to 2.143, Postgres 15 to 17) and its rehearsal on a clone. The longer version, with the fix for each: https://github.com/shsunmoonlee/self-hosted-supabase.

Coolify

1. A host-side docker compose up --force-recreate drops the coolify network. Coolify attaches it outside the compose file. Record networks and aliases before, reconnect after, probe the internal path.
2. The host compose file is the truth. Coolify's stored copy differed from it for every service.
3. Record every change in Coolify's stored compose (PATCH /api/v1/services/<service-uuid>, base64 docker_compose_raw, instant_deploy: false), or the next Save or Deploy reverts it.
4. Redeploy recreates every container. The Stop button removes them. Use docker stop and one-container recreates.
5. "Load from server" has no API. Edited file-storage files need a click in the UI.
6. Keep "delete unused volumes" off while a rollback volume exists.
7. The nightly cleanup prunes pre-pulled and rollback images. docker save them.
8. A new magic variable gets a value only on a Save. Check it is non-empty before a recreate.
9. The rendered .env has no trailing newline. Start every append with one. A glued append left our clone's postgres-meta "healthy" with an empty CRYPTO_KEY.
10. The scheduled backup has no grants (pg_dump --no-acl --no-owner, one database). Take your own dumps.

Supabase

11. Kong 2.8 cached old container IPs (connect() failed (113: Host is unreachable)) until kong reload. Swap to Envoy first; it re-resolves every 5 seconds.
12. PostgREST v14 postgrest --ready needs PGRST_ADMIN_SERVER_HOST=localhost next to the admin port.
13. Realtime's SECRET_KEY_BASE was empty (an older Coolify template referenced a variable Coolify never fills). GET / returned 500: "cookie store expects conn.secret_key_base to be at least 64 bytes". Check the length of every interpolated secret (check-secret-lengths.sh here prints names and lengths only).
14. Realtime migrations only go forward. Hop through versions upstream pinned, dump _realtime before each hop, give it a long start_period.
15. Stop autoheal for the window.
16. Our Vector db_logs transform failed with function call error for "upcase" on Postgres lines without a severity tag. Fixed upstream long ago; our copy was stale. Diff vector.yml against upstream on every upgrade.
17. Envoy answers 400 to header names with underscores; Kong dropped them silently.
18. envoy --mode validate does not load file-based CDS/LDS. Render and fold them into a static bootstrap to validate.

Postgres 15 to 17

19. pg_upgrade failed on event-trigger ownership in older databases ("Superuser owned event trigger must execute a superuser owned function"). Make postgres a superuser for the run, as upstream's initiate.sh does, and take it back after.
20. pgjwt is gone on 17; logical slots are not carried.
21. Run it over every database, including _supabase.
22. Reindex before a collation refresh, never refresh alone.
23. A late no-go loses the writes made on 17. Set a deadline.
24. Point-in-time recovery starts over on the new major.

Rehearsal

25. The clone was not production: _supabase there was several hundred times smaller than on production, and the clone had no cron jobs. Compare before trusting clone timings.
26. Our recreate tool restored config files as 0600 and the gateway could not read them. Test rollbacks, not only the forward path.
27. Read a failed job's event history before blaming the window. The one job in flight resumed by itself after the workers came back; a job that failed just before the window had an unrelated cause.