Docker Swarm in LXC Containers.md

Docker Swarm in LXC Containers

Part of collection: Hyper-converged Homelab with Proxmox

Update 09-10-2024: I have moved on to a setup with Docker LXC on Proxmox, with bind-mounts (without VirtioFS) as described in Gist Docker Swarm in Vm's with CephFS.

Update 07-03-2025: The Dockge LXC does this even better, eliminating most issues, with the Docker LXC](https://community-scripts.github.io/ProxmoxVE/scripts?id=docker)!

After struggling for some days, and since I really needed this to work (ignoring the it can't be done vibe everywhere), I managed to get Docker to work reliable in privileged Debian 12 LXC Containers on Proxmox 8

(Unfortunately, I couldn't get anything to work in unprivileged LXC Containers)

There are NO modifications required on the Proxmox host or the /etc/pve/lxc/xxx.conf file; everything is done on the Docker Swarm host. So the only obvious candidate who could break this setup, are future Docker Engine updates!

Host Setup

My host are Debian 12 LXC containers, installed via tteck's Proxmox VE Helper Scripts

Install the LXC via the Proxmox VE Helper Script

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/debian.sh)"

Backing filesystems

Docker info shows i'm using overlay2, this is the recommended storage driver for Debian. This storage driver requires XFS or EXT4 as backing file system.

docker info | grep -A 7 "Storage Driver:"

 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd

Set userns-remap

As Neuer_User pointed out, running the Docker Containers unprivileged on a privileged

LXC seems the best compromise to run the containers in a relative secure way.

To do so, add a daemon.json on the Docker Servers that are part of the Swarm.

mkdir /etc/docker
nano /etc/docker/daemon.json

{
  "userns-remap": "root"
}

And reboot reboot the Docker Host.

(This moves everything below /var/lib/docker/ to the folder /var/lib/docker/0.0/ existing workload disappear, hence it's a step pre Docker installation!)

Install Docker

The get-docker.sh script is the most convenient way to quickly install the latest Docker-CE release!

curl -fsSL https://get.docker.com -o get-docker.sh
chmod +x get-docker.sh
./get-docker.sh

Join Create Docker / Swarm

Without this step, the next step(s) fail!

# Manager Node
docker swarm init

# Add Node
docker swarm join --token <some-very-long-token>

# Display Join token again
docker swarm join-token worker
docker swarm join-token manager

Add ipv4 for Ingress_sbox

For Docker in LXC to work, the only thing needed is to execute:

nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

on the Docker Swarm Servers

Make it permanent

This doesn't survive reboots, so I created an oneshot systemd service for it, to make sure that after each reboot the setting is applied.

Create a Bash Script

First, we need a Bash script to be executed by the service.

nano /usr/local/bin/ipforward.sh

#!/bin/bash
nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

Make it executable

chmod +x /usr/local/bin/ipforward.sh

Create a Systemd Service

This service is of the type oneshot, during startup it waits for the docker.service to be started, and then 10 seconds for run-docker-netns-ingress_sbox.mount to be loaded. Only after that net.ipv4.ip_forward=1 can be applied.

nano /etc/systemd/system/ingress-sbox-ipforward.service

[Unit]
Description = Set net.ipv4.ip_forward for ingress_sbox namespace
After = docker.service
Wants = docker.service

[Service]
Type = oneshot
RemainAfterExit = yes
ExecStartPre = /bin/sleep 10
ExecStart = /usr/local/bin/ipforward.sh

[Install]
WantedBy = multi-user.target

Start the service and check if it's healthy

systemctl daemon-reload
systemctl enable ingress-sbox-ipforward.service
systemctl start ingress-sbox-ipforward.service
systemctl status ingress-sbox-ipforward.service

Final Checks

Without ipv4.ip_forward set to 1, the Ingress Networking to the Docker Swarm is not active. So it's important to verify if the value is applied successfully.

Manual check if ipv4.ip_forward is set to 1

systemctl status ingress-sbox-ipforward.service | grep ipforward.sh

# Or in a script via:

current_value=$(nsenter --net=/run/docker/netns/ingress_sbox sysctl -n net.ipv4.ip_forward)
echo $current_value

(Now, Docker in LXC seems to behave as Docker in a VM.)

Issues

1. Service in docker-compose resolved wrong ip

To fix this, it’s needed to add a hostname entry for each swarm service, to make it more logical I also add a prefix service to the service names.

services:
  service_nginx: # Prefix service_
    image: nginx
    hostname: nginx

Links

• Proxmox Forum Topic about this subject.

Screenshot

Portainer & portainer agent were both installed using the same stack file you pointed to, just changed the /data volume to a bind mount aimed at the proper directory on the shared cephfs mount.

Frustrating thing is that I know this worked in the recent past. I've had this setup running for a while -- docker swarm in privileged lxc with a cephfs mount for persistent container data. Portainer was happily chugging along for the better part of a year, with dozens of host restarts and service relocations just working as expected. At some point in the last 6 weeks I noticed the portainer service failing (not 100% certain when it stopped working, as my attention has been elsewhere, and hadn't actually tried to log on to the portainer interface for a while). Seems entirely unrelated to the docker or portainer versions (I've tried multiple versions of docker and portainer recently, trying to get it to work as expected).

I also tried removing the mount point from the LXC and installing the ceph client inside the container and mounting via fstab. Same behavior.

Gave up hope this morning. Since I already have an NFS server exporting a couple of the cephfs directories, I just used that -- mounted the same directory via NFS on the docker hosts. Portainer now behaves as expected -- service is able to move to any host w/out complaint.

But yeah -- the added (admittedly minor) complexity of using NFS mounts inside the docker containers instead of a cephfs mount on the LXC makes my eye twitch a bit.

I also tried removing the mount point from the LXC and installing the ceph client inside the container and mounting via fstab. Same behavior.

I just mounted the cephfs filesystem using the ceph client within the virtual machines fstab and portainer/adguard are working properly now. I did not try this in an lxc container though. I had to create a local bridge on each hypervisor and nat out traffic over the point-to-point link to get the virtual machine running on host 1 to communicate with monitors on host 2 and 3, but it is working away nicely now.

I lost ~3h debuging my network, and wasn't able to figure why no external connection is reaching my swarm. Especially that i don't remember if ever on other instances i had to redirect connections locally.
Using your nsenter method I was able to finally start that swarm and connect it to other environments.

Thanks!

Thank you very much for writing this guide! I found this when I try to use docker secrets in portainer but turns out it only supported in docker swarm environment.

I am trying to understand the user mapping. I don't think simply making the daemon.json file is enough. See this docker docs on namespace,

Some distributions do not automatically add the new group to the /etc/subuid and /etc/subgid files. If that's the case, you are may have to manually edit these files and assign non-overlapping ranges. This step is covered in Prerequisites.

And this is the case for me on a fresh privileged debian LXC. I don't see any mapping in both /etc/subuid and /etc/subgid after following all your guide, it's just empty. I think this is crucial if you concern about unprivileged/privileged in the first place.

So my question is:

1. Are you aware of this? If so, is this intended (the no mapping in those two files)?
2. Will existing containers break if I add those two files and the mapping inside?
3. This question is specific. Basically if I want to use GPU transcoding in an unprivileged LXC, I can trick the mapping in the LXC configuration file (discussed in many guides, this for example). Then the question: even if I deploy a privileged LXC and use this namespace mapping for docker inside it, the GPU won't be able to be used straight away, right?

This is a great write-up, thank you! I was trying to figure this out for days.

The systemd service has to be restarted if the docker service is restarted. You can use the PartOf directive to trigger that automatically. Additionally, you can removed delay and instead just I made the service file require run-docker-netns-ingress_sbox.mount. I also don't understand why a separate file is needed just to run the required command?

Below is the version of the systemd unit I'm using that I've adapted from yours:

[Unit]
Description = Set net.ipv4.ip_forward for ingress_sbox namespace
After = docker.service run-docker-netns-ingress_sbox.mount
Requires = run-docker-netns-ingress_sbox.mount
PartOf = docker.service run-docker-netns-ingress_sbox.mount

[Service]
Type = oneshot
RemainAfterExit = yes
ExecStartPre = /bin/sleep 10
ExecStart = nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

[Install]
WantedBy = multi-user.target

Edit: Strangely, my swarm manager worked without ExecStartPre = /bin/sleep 10 but the worker doesn't.

This is a great write-up, thank you! I was trying to figure this out for days.

The systemd service has to be restarted if the docker service is restarted. You can use the PartOf directive to trigger that automatically. Additionally, you can removed delay and instead just I made the service file require run-docker-netns-ingress_sbox.mount. I also don't understand why a separate file is needed just to run the required command?

Below is the version of the systemd unit I'm using that I've adapted from yours:

[Unit]
Description = Set net.ipv4.ip_forward for ingress_sbox namespace
After = docker.service run-docker-netns-ingress_sbox.mount
Requires = run-docker-netns-ingress_sbox.mount
PartOf = docker.service run-docker-netns-ingress_sbox.mount

[Service]
Type = oneshot
RemainAfterExit = yes
ExecStartPre = /bin/sleep 10
ExecStart = nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

[Install]
WantedBy = multi-user.target

Edit: Strangely, my swarm manager worked without ExecStartPre = /bin/sleep 10 but the worker doesn't.

Thank you! I'm currently not using LXC Docker Any more, but a great tip for those who will.