SDPS Enduro + Archivematica

readme.md

SDPS Enduro + Archivematica

Instructions to set up and configure the Enduro SDPS Kubernetes cluster and the Archivematica Docker Compose environment locally.

Archivematica

Follow the documentation to set up a local instance of Archivematica:

https://github.com/artefactual/archivematica/tree/qa/1.x/hack

SFTP service

We'll use sftpgo to set up an SFTP server in the AM Docker Compose env. Modify AM's Docker Compose file to include a new sftpgo service, mounting the archivematica_storage_service_location_data volume:

diff --git a/hack/docker-compose.yml b/hack/docker-compose.yml
index 9e4be90d..042fd49d 100644
--- a/hack/docker-compose.yml
+++ b/hack/docker-compose.yml
@@ -239,3 +239,14 @@ services:
       - "archivematica_storage_service_location_data:/home:rw"
     links:
       - "mysql"
+
+  sftpgo:
+    image: drakkan/sftpgo:v2.5.5-alpine-slim
+    environment:
+      SFTPGO_LOADDATA_FROM: /etc/sftpgo/initial-data.json
+    ports:
+      - 12380:8080
+      - 12322:2022
+    volumes:
+      - ./sftpgo-data.json:/etc/sftpgo/initial-data.json
+      - archivematica_storage_service_location_data:/home:rw

Create the sftpgo-data.json file used for the other volume in the sftp service, alongside the docker-compose.yml file:

{
  "users": [
    {
      "id": 1,
      "status": 1,
      "username": "archivematica",
      "password": "$2a$10$3Y.Qp.8BlNLCI6t0yKnbJuzLcKBgukuQl4Zg9C1jA6D0Xr3FhDoTu",
      "public_keys": [
        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMsk/SqNiQBndlq/CpfBscHvKC4bsZezSv0fDzsYlBTI radda@ubuntu"
      ],
      "has_password": true,
      "home_dir": "/home",
      "permissions": { "/": ["*"] }
    }
  ],
  "admins": [
    {
      "id": 1,
      "status": 1,
      "username": "admin",
      "password": "$2a$10$3OttaDDy1VIJbMTu80zNP.jx5UQQw2gwtN4QZXIcBIW8870j4koDi",
      "permissions": ["*"]
    }
  ]
}

Create a directory inside the archivematica_storage_service_location_data volume called enduro_transfers:

mkdir ~/.am/ss-location-data/enduro_transfers

Restart the environment:

docker compose up -d

Enduro

Follow the documentation to set up the Enduro Kubernetes cluster:

https://github.com/artefactual-sdps/enduro/blob/main/docs/src/dev-manual/devel.md

Make sure you set am as the preservation system, and check the documentation to work with Archivematica:

https://github.com/artefactual-sdps/enduro/blob/main/docs/src/dev-manual/archivematica.md

We need to create three files inside the repository:

hack/kube/overlays/dev-am/.am.secret

address=http://host.k3d.internal:62080
user=test
api_key=test
transfer_source_path=enduro_transfers
sftp_host=host.k3d.internal
sftp_port=12322
sftp_user=archivematica
sftp_remote_dir=/enduro_transfers
sftp_private_key_passphrase=
amss_url=http://host.k3d.internal:62081
amss_user=test
amss_api_key=test
amss_location_id=e0ed8b2a-8ae2-4546-b5d8-f0090919df04

We are using K3d's internal host to connect with the host from within the cluster. If you use Minikube, change that to host.minikube.internal.

hack/kube/overlays/dev-am/.id_ed25519.secret

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACDLJP0qjYkAZ3ZavwqXwbHB7yguG7GXs0r9Hw87GJQUyAAAAJDGar6Exmq+
hAAAAAtzc2gtZWQyNTUxOQAAACDLJP0qjYkAZ3ZavwqXwbHB7yguG7GXs0r9Hw87GJQUyA
AAAECPz7gCx1Ni9VMMX80QEBgfIFnqyA5mDlxBiXpA3+w7fcsk/SqNiQBndlq/CpfBscHv
KC4bsZezSv0fDzsYlBTIAAAADHJhZGRhQHVidW50dQE=
-----END OPENSSH PRIVATE KEY-----

hack/kube/overlays/dev-am/.known_hosts.secret

To get the SFTP server keys and create the known_hosts file we'll first add an entry to /etc/hosts to match the host name used inside the cluster:

127.0.0.1 host.k3d.internal

Then, from the Enduro folder run:

make tilt-am-knownhosts

This Makefile rule uses ssh-keyscan to generate the secret file directly in the repository and restarts the required services in Tilt.

Warning Each time you recreate the sftpgo service in AM you'll need to run make tilt-am-knownhosts again to update the SFTP server keys.