Functional Kafka (KRaft) + UI configuration (SASL_SSL) using Docker

.env.example

# Kafka UI
KAFKA_UI_LOCAL_PORT=5000
KAFKA_UI_USER_NAME=admin
KAFKA_UI_PASSWORD=pass

# SASL
KAFKA_USER=kafka
KAFKA_PASSWORD=password

# Kafka
CLUSTER_ID=123
KAFKA_CLUSTER_NAME=local
HOST=host.docker.internal
BROKER_PORT=9093
CONTROLLER_PORT=9094

# SSL
TRUSTSTORE=broker.truststore.jks
KEYSTORE=broker.keystore.jks
SSL_TRUSTSTORE_PASSWORD=password

readme.md

Introduction

The provided examples of Kafka with Docker are incomplete. It is quite cumbersome to debug the docker compose file since there are a lot of variables involved. Additionally, the thrown exceptions while trying to connect to the Kafka are not very clear. Kafka only accepts SSL Truststore and SSL Keystore which can further the complexity of the configuration.

Dev environments (PLAINTEXT)

Note: this should NOT be used in production environment.

When using Kafka in a local (dev) environment, you might want to use PLAINTEXT as your authentication method. The following command can be used:

docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build

Production environments (SASL_SSL)

docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build

Note: provide the name/password for Kafka set in the .env file. Note: the {{TRUSTSTORE_LOCATION}}, {{YOUR_CERTIFICATES_LOCATION_HERE}}, {{KEYSTORE_FILENAME}}, {{TRUSTSTORE_FILENAME}}, {{YOUR_CERTIFICATES_LOCATION_HERE}} have to be replaced with a hard coded value.

You should now be able to connect to the Kafka UI on localhost with the provided port in the .env.

docker-compose.dev.yml

services:
  kafka:
    volumes:
      - ./secrets:/etc/kafka/secrets:ro

docker-compose.prod.yml

Production docker compose file (SASL_SSL):

services:
  kafka-ui:
    user: "${DOCKER_UID:-1000}:${DOCKER_GID:-1000}"
    extends:
      file: docker-compose.yml
      service: kafka-ui
    environment:
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="${KAFKA_USER}" password="${KAFKA_PASSWORD}";
      KAFKA_CLUSTERS_0_SSL_TRUSTSTORE_LOCATION:{{TRUSTSTORE_LOCATION}}
      KAFKA_CLUSTERS_0_SSL_TRUSTSTORE_PASSWORD: "${SSL_TRUSTSTORE_PASSWORD}"
      KAFKA_CLUSTERS_0_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: '' # DISABLE COMMON NAME VERIFICATION
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: ${HOST}:${BROKER_PORT}
    volumes:
      - {{YOUR_CERTIFICATES_LOCATION_HERE}}:/home/kafkaui/certificates:ro

  kafka:
    user: "${DOCKER_UID:-1000}:${DOCKER_GID:-1000}"
    extends:
      file: docker-compose.yml
      service: kafka
    environment:
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LISTENERS: SASL_SSL://:${BROKER_PORT},CONTROLLER://:${CONTROLLER_PORT}
      KAFKA_ADVERTISED_LISTENERS: SASL_SSL://${HOST}:${BROKER_PORT}
      KAFKA_SSL_KEYSTORE_FILENAME: {{KEYSTORE_FILENAME}}
      KAFKA_SSL_KEYSTORE_CREDENTIALS: keystore_credentials
      KAFKA_SSL_KEY_CREDENTIALS: keystore_credentials
      KAFKA_SSL_TRUSTSTORE_FILENAME: {{TRUSTSTORE_FILENAME}}
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: keystore_credentials
      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SASL_SSL
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,SASL_SSL:SASL_SSL
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""          # Disable hostname verification
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_OPTS: -Djava.security.auth.login.config=/tmp/jaas.conf

    volumes:
      - ./jaas.conf:/home/appuser/jaas.conf
      - {{YOUR_CERTIFICATES_LOCATION_HERE}}:/etc/kafka/secrets:ro

    entrypoint: >
      /bin/sh -c '
         cp /home/appuser/jaas.conf /home/appuser/jaas.conf.copy
         sed -ci "s/{{KAFKA_USER}}/$KAFKA_USER/g" /home/appuser/jaas.conf.copy;
         sed -ci "s/{{KAFKA_PASSWORD}}/$KAFKA_PASSWORD/g" /home/appuser/jaas.conf.copy;
         cp /home/appuser/jaas.conf.copy /tmp/jaas.conf;
         echo "JAAS config file created in /tmp.";
         /etc/confluent/docker/run
       '

docker-compose.yml

services:
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    depends_on:
      - kafka
    ports:
      - "${KAFKA_UI_LOCAL_PORT}:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: ${KAFKA_CLUSTER_NAME}
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:${BROKER_PORT}
      AUTH_TYPE: "LOGIN_FORM"
      SPRING_SECURITY_USER_NAME: ${KAFKA_UI_USER_NAME}
      SPRING_SECURITY_USER_PASSWORD: ${KAFKA_UI_PASSWORD}

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    ports:
      - ${BROKER_PORT}:${BROKER_PORT}
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: PLAINTEXT://:${BROKER_PORT},CONTROLLER://:${CONTROLLER_PORT}
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${HOST}:${BROKER_PORT}
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:${CONTROLLER_PORT}
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      CLUSTER_ID: ${CLUSTER_ID}
    volumes:
      - ./data:/var/lib/kafka/data