How to Deploy Kafka – Distributed Event Streaming

Deploy Apache Kafka with Docker Compose on Ubuntu 24.04 for real-time streaming and analytics.

Apache Kafka is a high-throughput, distributed event-streaming platform used for building real-time data pipelines and streaming applications. It provides persistent, fault-tolerant message storage and supports large-scale publish-subscribe workloads with low latency. Kafka is widely used for log aggregation, metrics collection, event processing, microservices communication, and real-time analytics.

This article explains how to deploy Apache Kafka using Docker Compose, configure persistent storage, define environment variables, and verify that the Kafka broker is reachable.

Prerequisites

Before you begin, you need to:

Have access to an Ubuntu 24.04-based server as a non-root user with sudo privileges.
Install Docker and Docker Compose.

Set Up the Directory Structure and Environment Variables

In this section, you create a project directory for Kafka and define environment variables in a .env file that Docker Compose loads automatically.

Create the project directory.
console
```
$ mkdir -p ~/kafka-stack/kafka-data
```
Change the data directory permission.
console
```
$ sudo chown 1000:1000 ~/kafka-stack/kafka-data
```
This permission change is required because the Apache Kafka container runs using an internal application user appuser with UID 1000, and it must have write access to the kafka-data directory for logs and persisted messages.
Navigate into the project directory.
console
```
$ cd ~/kafka-stack
```
Generate a UUID for the cluster identifier.
console
```
$ uuidgen | tr -d '-'
```
Copy the generated value for use in the .env file.
Create a .env file to store the environment variables.
console
```
$ nano .env
```
Add the following variables:
ini
```
KAFKA_NODE_ID=1
KAFKA_CLUSTER_ID=YOUR_KAFKA_CLUSTER_ID
KAFKA_PORT=9092
KAFKA_ADVERTISED_HOST=localhost
```
Replace:
- KAFKA_NODE_ID with a unique broker ID (keep 1 for single-node).
- KAFKA_CLUSTER_ID with the unique cluster identifier you generated in the previous step.
- KAFKA_PORT with the port Kafka should listen on.
- KAFKA_ADVERTISED_HOST with the hostname or IP clients should use to connect.
Note

If your applications run on the same server as Kafka, keep KAFKA_ADVERTISED_HOST=localhost. If your applications run on a different server, replace localhost with your public server IP address so external clients can connect.

Save and close the file.

Deploy with Docker Compose

In this section, you create the Docker Compose manifest that deploys Apache Kafka in KRaft mode, enables persistent storage, and exposes the broker for client connections.

Create the Docker Compose manifest.

console

$ nano docker-compose.yaml

Add the following content:

yaml

services:
  kafka:
    image: apache/kafka:4.1.1
    container_name: kafka
    ports:
      - "${KAFKA_PORT}:9092"
    environment:
      KAFKA_NODE_ID: ${KAFKA_NODE_ID}
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${KAFKA_ADVERTISED_HOST}:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LOG_DIRS: /kafka/data
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_CLUSTER_ID: ${KAFKA_CLUSTER_ID}
    volumes:
      - "./kafka-data:/kafka/data"
    restart: unless-stopped

    healthcheck:
      test: ["CMD", "kafka-broker-api-versions", "--bootstrap-server=localhost:9092"]
      interval: 10s
      timeout: 5s
      retries: 5

Save and close the file.

In the manifest:

services: Defines the Kafka service managed by Docker Compose.
- kafka: Runs Apache Kafka in single-node KRaft mode.
image: Uses the official Apache Kafka image.
ports: Publishes Kafka’s default port (9092) on the host.
environment:
- KAFKA_NODE_ID: Unique broker ID.
- KAFKA_PROCESS_ROLES: Runs both broker and controller in one container.
- KAFKA_LISTENERS: Defines internal listener endpoints.
- KAFKA_ADVERTISED_LISTENERS: Defines the external address clients use to connect.
- KAFKA_CONTROLLER_QUORUM_VOTERS: Configures the internal controller quorum.
- KAFKA_LOG_DIRS: Path where Kafka stores log and topic data.
- KAFKA_CLUSTER_ID: Initializes the KRaft cluster.
- Replication and ISR values are set to 1 for single-node operation.
volumes:
- ./kafka-data persists Kafka topic and log data across restarts.
healthcheck: Verifies Kafka availability using the broker API.
restart: unless-stopped: Restarts Kafka automatically unless manually stopped.

Start the Kafka service.
console
```
$ docker compose up -d
```
Verify that Kafka service is running.
console
```
$ docker compose ps
```
Ensure that the output shows the Kafka container is healthy and running.
View logs if needed.
console
```
$ docker compose logs
```
For more information on managing Docker Compose services, see the How To Use Docker Compose article.

Verify Apache Kafka Connectivity

In this section, you verify that your Apache Kafka broker is reachable and accepting client connections by using the Kafka CLI tools.

Install Kafka CLI tools

Update the package index and install the default Java runtime.
console
```
$ sudo apt update && sudo apt install -y default-jdk
```
Download the Kafka CLI tools archive. Refer to the official Apache Kafka downloads page for newer versions.
console
```
$ wget https://downloads.apache.org/kafka/4.1.1/kafka_2.13-4.1.1.tgz
```

Extract the archive to /opt.

console

$ sudo tar -xzf kafka_2.13-4.1.1.tgz -C /opt

Add the Kafka binaries from /opt to your PATH.

console

$ echo 'export PATH=$PATH:/opt/kafka_2.13-4.1.1/bin' | sudo tee /etc/profile.d/kafka.sh

Apply the updated PATH.
console
```
$ source /etc/profile.d/kafka.sh
```

Test Message Production and Consumption

Create a test topic.

console

$ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 3 --replication-factor 1

Start the producer.
console
```
$ kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic
```
This command opens an interactive console where you type messages and press ENTER after each one to publish the messages to the topic.
Open a new terminal and start a consumer to read messages from the beginning.
console
```
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning
```
This command starts the consumer and displays all messages sent to the specified topic.
Verify the topic configuration.
console
```
$ kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic
```
This command displays detailed information about the topic configuration.

Conclusion

You successfully deployed Apache Kafka using Docker Compose in single-node KRaft mode with persistent storage enabled. You configured all required environment variables, prepared the data directory with correct permissions, and exposed the broker for client access. After installing the Kafka CLI tools, you verified end-to-end message flow by producing and consuming test messages.

Cobra Softwares Blog