traefik: node-red + mosquitto using letsencrypt

docker-compose.yaml

version: "3.3"

services:

  traefik:
    image: "traefik:v2.2"
    container_name: "traefik"
    command:
      - "--api=true"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
        
        # Entrypoints 
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.mqtt.address=:8883"
      
        # Redirect http to https
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      
        # Let's encrypt configuration
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.email=email@host.com"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
      - "8883:8883"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.enable=true"
      
      - "traefik.http.routers.dashboard.rule=Host(`traefik.zoo.ocean.mofa.studio`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls.certresolver=myresolver"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/,test2:$$apr1$$d9hr9HBB$$4HxwgUir3HP4EsggP/QNo0"

  whoami:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.whoami.rule=Host(`whoami.zoo.ocean.mofa.studio`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=myresolver"
  
  mqtt:
    image: "eclipse-mosquitto"
    container_name: "mosquitto"
    expose:
      - "8883"
      - "9001"
    volumes:
      - "./mosquitto.conf:/mosquitto/config/mosquitto.conf"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.mqtt.rule=Host(`mqtt.zoo.ocean.mofa.studio`)"
      - "traefik.http.routers.mqtt.entrypoints=websecure"
      - "traefik.http.routers.mqtt.tls.certresolver=myresolver"

      - "traefik.tcp.routers.mqtt.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.mqtt.tls.certresolver=myresolver"
      - "traefik.tcp.services.mqtt.loadbalancer.server.port=8883"
      
      - "traefik.tcp.routers.mqtt.entrypoints=mqtt"

      - "traefik.http.services.mqtt.loadbalancer.server.port=9001"

  nodered:
    image: "nodered/node-red"
    container_name: "nodered"
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.nodered.rule=Host(`nodered.zoo.ocean.mofa.studio`)"
      - "traefik.http.routers.nodered.entrypoints=websecure"
      - "traefik.http.routers.nodered.tls.certresolver=myresolver"

      - "traefik.http.services.nodered.loadbalancer.server.port=1880"

mosquitto.conf

port 8883

listener 9001
protocol websockets

Sorry to bother you again.
Could you please share the mosquitto_pub and mosquitto_sub commands you'd use to test it?
If traefik handles it all, you shouldn't need to add any --cafile parameter, isn't it? Ie:

mosquitto_pub -h mqtt.zoo.ocean.mofa.studio -p 8883 -t test -m "hello world"

mosquitto_sub -h mqtt.zoo.ocean.mofa.studio -p 8883 -t secureTest

But this doesn't work for me. The only way I get it working is by downloading the ca from here, rename it to trustid-x3-root.pem and then use the --cafile for both publishing and subscribing:

mosquitto_pub -h mqtt.zoo.ocean.mofa.studio -p 8883 -t test -m "hello world" --cafile "E:\Downloads\trustid-x3-root.pem"

mosquitto_sub -h mqtt.zoo.ocean.mofa.studio -p 8883 -t secureTest --cafile "E:\Downloads\trustid-x3-root.pem"

But then again it looks as if I would need to distribute that certificate, which is public so anyone would be able to decrypt those messages.

Note I'm using mqtt.zoo.ocean.mofa.studio only to be coherent with your code. Obviously I'm doing the tests with the domain I manage.

Thanks for your help with this.

Hmm I didn't test it with mosquitto_pub and mosquitto_sub but distributing the certificate doesn't mean the client can decrypt it, the client certificate is linked to a private certificate hosted by the Certificate authority (let's encrypt), the only way someone could decrypt it is if they have access to both the client and private certs which is not possible.
What client will you be using in the end ?
In most cases, clients will use certificates that are stored at the OS level meaning you do not need to provide a client cert, you just need to specify in the client's configuration that the communication uses SSL/TLS. That doesn't seem possible with mosquitto_pub/sub so it's necessary to hardcode it. You could also point it to your OS's certificate root store if it's in the good format.

Hmm I didn't test it with mosquitto_pub and mosquitto_sub but distributing the certificate doesn't mean the client can decrypt it, the client certificate is linked to a private certificate hosted by the Certificate authority (let's encrypt), the only way someone could decrypt it is if they have access to both the client and private certs which is not possible.

I'll check if running mosquitto_pub and mosquitto_sub from different machines gives a different result.

What client will you be using in the end ?

We are developing a web api with .NET Core. This api acts as mqtt client (only as subscriber) and then resends the data to a web app using SignalR. Our customers are who will have the devices which will do the pub to our broker.
The idea is to allow customers to publish to unencrypted 1883 for testing and to encrypted 8883 for production... or to 8884 encrypted requiring an X.509 client certificate.

I'll check if running mosquitto_pub and mosquitto_sub from different machines gives a different result.

Same result than before. I need to specify the --cafile trustid-x3-root.pem at both sides.

That’s normal behaviour.
You’ll have to check the if the mqtt library you’ll be using uses the OS certificate store or wether you need to include it.
If the goal is to secure your broker so that only your devices can connect you will also need to setup password authentication. Make sure the password is different for ssl vs unencrypted because it’ll pass as clear text.

Is your mqtt websocket connection working using this docker-compose.yaml ? If I understand the traefik info correctly, you don't capture traffic on the outside port 9001, but use the 443 port and redirect to the 9001 port of the mqtt docker ? I can't get this to work .... Thanks for your help

Yes that’s correct, mqtt over websocket is on port 443 in this example so you client needs to use port 443 in secured web socket mode.

After hours of debugging I finally got it ... Your script works perfectly, The error was with the websocket client (hivemq) which for some reason only wants to use the default 9001 ws port (even if a different port is specified). Thanks for sharing !

Nice ! glad to know it worked ! Thanks for the update !

Hi @natcl,

I am trying to use your instructions to get mosquitto working with traefik and for whatever reason I get a connection refused error through MQTT explorer. I've looked at traefik and see the following error with my mqtt tcp service:

I can telnet into that IP/port from my docker server, so the port is working. I have setup things to use slightly different naming, but everything else should be the same. Here is the code that I am using:

mosquitto traefik labels:

  mosquitto:
    image: eclipse-mosquitto
    container_name: mosquitto
    restart: unless-stopped
    ports:
      - 1883:1883
      - 8884:8883
      - 9001:9001
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${DOCKER}/mosquitto/data:/mosquitto/data
      - ${DOCKER}/mosquitto/config:/mosquitto/config
      - ${DOCKER}/mosquitto/log:/mosquitto/log
    environment:
      - TZ=${TZ}
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.mqtt.rule=Host(`mqtt.${DOMAINNAME}`)"
      - "traefik.http.routers.mqtt.entrypoints=https"
      - "traefik.http.routers.mqtt.tls.certresolver=http"

      - "traefik.tcp.routers.mqtt.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.mqtt.tls.certresolver=http"
      - "traefik.tcp.services.mqtt.loadbalancer.server.port=8883"

      - "traefik.tcp.routers.mqtt.entrypoints=mqtt"

      - "traefik.http.services.mqtt.loadbalancer.server.port=9001"
      - "traefik.docker.network=public_facing"

mosquitto conf:

persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
allow_anonymous true
listener 1883
port 8883
listener 9001
protocol websockets

output from my mosquitto.log:

1619451746: mosquitto version 2.0.10 starting
1619451746: Config loaded from /mosquitto/config/mosquitto.conf.
1619451746: Opening ipv4 listen socket on port 1883.
1619451746: Opening ipv6 listen socket on port 1883.
1619451746: Opening websockets listen socket on port 9001.
1619451746: Opening ipv4 listen socket on port 8883.
1619451746: Opening ipv6 listen socket on port 8883.
1619451746: mosquitto version 2.0.10 running

Any ideas on why this would happen?

What port are you trying to connect to from MQTT explorer ?

I am using the following:

I don't have a username/password setup yet, at this point I am just trying to connect anonymously.

Can you show the main traefik config ?

Sure:

docker compose piece:

  traefik:
    image: traefik:v2.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - public_facing
    ports:
      - "80:80"
      - "443:443"
      - "8883:8883"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${DOCKER}/traefik/acme.json:/acme.json
      - ${DOCKER}/traefik/traefik.yml:/traefik.yml:ro
      - ${DOCKER}/traefik/users:/users:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.${DOMAINNAME}`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.usersfile=/users"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.${DOMAINNAME}`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=http"
      - "traefik.http.routers.traefik-secure.service=api@internal"

traefik.yml

api:
  dashboard: true

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"
  mqtt:
    address: ":8883"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false

certificatesResolvers:
  http:
    acme:
      email: fake@email.com
      storage: acme.json
      httpChallenge:
        entryPoint: http

Can you try using my examples to see if it works ? Yours are quite modified so I'm not sure where the issue could be....

Nice docker script. But for using TCP TLS with 8883 port you should change protocol to mqtt instead websocket into mosquitto.conf .

The configuration works for mosquitto 1.x but the ´port’ configuration is deprecated so I will update it.

The goal is to have both TCP and websockets, that’s why both configurations are there.

In this way, I'm guessing some malicious client can also pub/sub to your mqtt broker if somebody knows your server URI.
Am I correct??

I'm trying to setup client certificate type of authentication but with no luck. mqtt broker is behind Traefik as you do.
a reference of client certificate type of authentication is here:
https://primalcortex.wordpress.com/2016/11/08/mqtt-mosquitto-broker-client-authentication-and-client-certificates/
Do you have any experience on this??

I use mosquitto authentication to secure the broker, could this be enough for your use case ?

That may work, I'll give it a try

…

On Thu, May 19, 2022, 6:40 AM Nathanaël Lécaudé ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ I use mosquitto authentication to secure the broker, could this be enough for your use case ? — Reply to this email directly, view it on GitHub <https://gist.github.com/ed8253a34e7b87d879baabeba82cb846#gistcomment-4172110>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMO6L75PZQAOA7NACKWG3TTVKYSDZANCNFSM4T7ORYGQ> . You are receiving this because you commented.Message ID: ***@***.***>

Thank you for your help!
Very short answer is No.
Why I am doing this because I'd like to run eclipse-mosquitto broker (mqqts) on 443/tcp to make some firewalls happy.
My broker is a part of Netmaker (https://github.com/gravitl/netmaker) system that needs client certificate authentication.
I tried tls.passthrough with hostSNI("*") but no luck,,

I believe this could work but I doubt you'll be able to make it work using let's encrypt, you'll probably need you own certificates.

Yes, without reverse proxy, client cert authentication works perfect with own ca. but behind traefik it doesn’t work,,
Thank you

Hi, a doubt..in your docker-compose file why are you using the entrypoint for 8883 if you are using the websecure entrypoint in the mqtt section config?

@iboluda The 8883 entry point is for TCP connections while the 443 is for web sockets, does that answer your question ?

Thanks for clarify me that point. Yes that answer my question