Git Repo https://github.com/ProeungChiso/cstad_ite2_apache_kafka_docker_compose_latest.git

1. Configuration

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.6
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181 # Client port
      ZOOKEEPER_TICK_TIME: 2000 # Tick time
      ALLOW_ANONYMOUS_LOGIN: yes # Allow anonymous login
    healthcheck:
      test: nc -z localhost 2181 || exit -1
      interval: 10s
      retries: 3
      start_period: 10s
      timeout: 5s
    networks:
      - kafka-network
    volumes:
      - zookeeper-data:/var/lib/zookeeper/data  

  kafka:
    image: confluentinc/cp-kafka:7.4.6
    hostname: kafka
    container_name: kafka
    depends_on:
      zookeeper:
        condition: service_healthy
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1 # Unique broker ID
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' # Zookeeper service
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://<server-ip>:9092 # Change <server-ip> to your server IP
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 # Listen on all interfaces
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 # Only one node
      KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE: "false" # Disable metrics
      KAFKA_MESSAGE_MAX_BYTES: 2000000000 # 2GB
      KAFKA_REPLICA_FETCH_MAX_BYTES: 2000000000 # 2GB
      KAFKA_REPLICA_MAX_LOG_SIZE: 2000000000 # 2GB
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_LOG_RETENTION_HOURS: 168 # Retain logs for 7 days
      KAFKA_LOG_RETENTION_BYTES: 10737418240 # Retain logs for 10 GB
    healthcheck:
      test: ["CMD-SHELL", "nc -z localhost 9092"]
      interval: 10s
      retries: 3
      start_period: 30s
      timeout: 5s
    networks:
      - kafka-network
    volumes:
      - kafka-data:/var/lib/kafka/data  

networks:
  kafka-network:
    driver: bridge

volumes:
  zookeeper-data:
  kafka-data:  

2. Explanation

💥 Docker Compose Configuration Explanation

This document explains the configuration of your Docker Compose file, which sets up a Kafka ecosystem with Zookeeper and Kafka services. Each service is defined with its respective image, hostname, environment variables, ports, health checks, and volume mappings. The network and volumes used by the services are also outlined.

⭕ Services

1️⃣ Zookeeper Service

👉 The Zookeeper service is responsible for managing the configuration of the Kafka broker(s) and providing synchronization between them.

• Image: Uses confluentinc/cp-zookeeper:7.4.6, which is a Confluent Zookeeper image compatible with the Confluent Kafka setup.
• Hostname & Container Name: The Zookeeper service is named "zookeeper" for both the hostname and container name, ensuring easy communication between services.
• Ports:
  • 2181:2181: The container's port 2181 (Zookeeper client port) is mapped to the host's port 2181.
• Environment Variables:
  • ZOOKEEPER_CLIENT_PORT: Port used by clients to connect to Zookeeper.
  • ZOOKEEPER_TICK_TIME: The basic time unit in milliseconds used by Zookeeper for its heartbeat and session timeouts.
  • ALLOW_ANONYMOUS_LOGIN: Allows anonymous logins to Zookeeper.
• Healthcheck:
  • A health check is configured using nc (netcat) to verify that Zookeeper is listening on port 2181. It checks the connection every 10 seconds, retries 3 times, and times out after 5 seconds if Zookeeper is not ready.
• Volumes:
  • zookeeper-data:/var/lib/zookeeper/data: Maps a persistent volume for storing Zookeeper data.
• Networks:
  • The zookeeper is connected to the kafka-network, ensuring it can communicate with Kafka.

2️⃣ Kafka Service

👉 Kafka is the core messaging system that handles producing and consuming messages.

• Image: Uses confluentinc/cp-kafka:7.4.6, which is a Confluent Kafka image compatible with the specified Zookeeper version.
• Hostname & Container Name: The Kafka service is named "kafka" for both the hostname and container name.