s3cmd.md

s3cmd

This is a command line tool for syncing with DigitalOcean Spaces or Amazon S3.

I'm using this in the context of .gitignore'ing Assets from a Statamic site.

DigitalOcean Spaces with s3cmd

Install & configure

s3cmd setup

• SSH into server
• At root: sudo apt install s3cmd
• At root: s3cmd --configure (this will initate a config wizard)
• Gather keys from Digitalocean:
  • Digitalocean > API > Spaces access keys
  • Name token with server name (ie: example-prod-1)
  • Access Key: XXX...XXXX
  • Secret Key: XXX...XXXX
• Default Region: ams3
• S3 Endpoint: ams3.digitaloceanspaces.com
• DNS-style bucket+hostname:port template for accessing a bucket: example-assets-01.ams3.digitaloceanspaces.com
• Encryption password: (something-random-_-save-it-somewhere)
• Path to GPG program: (select default) /usr/bin/gpg
• Use HTTPS protocol: (select default) True
• HTTP Proxy server name:
• HTTP Proxy server port: 0
• Select Yes to test connection and save configuration

.sh Scripts files

• Create these files at root
• They can be run manually ad hoc or automatically on a schedule

Sync Up: s3cmd-spaces-sync-up.sh

#!/bin/bash
s3cmd sync -r --skip-existing --verbose --delete-removed public/assets/ s3://example-assets-01/assets/

• Run as cronjob every hour:
  • command: cd ~/example.come/ && nice -n 19 ./s3cmd-main-sync-up.sh > /dev/null
  • schedule: 15 * * * * (crontab.guru)

Sync down: s3cmd-spaces-sync-down.sh

#!/bin/bash
s3cmd sync -r --skip-existing --verbose --delete-removed s3://example-assets-01/assets/ public/assets/

# If RemoteDisconnected error appears, add --no-check-md5 to command
# ie: https://github.com/s3tools/s3cmd/issues/960

Create backup: s3cmd-spaces-backup.sh

• This creates an isolated, timestamped backup as some protection against a ‘bad sync’.
• This works best with a lifecycle.xml file (see below)

#!/bin/bash
s3cmd put -r public/assets/ s3://example-assets-01/backup-`date +%F\-%a`/ --expiry-days=30

• Run as cronjob every few days

Expire backups with a lifecycle

• See this Gist

Git hooks

Git post-commit hook

• Create .git/hooks/post-commit with contents of:

#!/bin/bash
git pull --rebase

git push

npm ci

npm run build

sh s3cmd-spaces-sync-down.sh

sh s3cmd-spaces-sync-up.sh

• Set permissions on the hook: chmod +x .git/hooks/post-commit

Git post-merge hook

• Create .git/hooks/post-merge with contents of:

#!/bin/bash
composer install

npm ci

npm run build

sh s3cmd-spaces-sync-down.sh

• Set permissions on the hook: chmod +x .git/hooks/post-merge