How to Deploy Redpanda – Kafka-Compatible Streaming Platform

Set up a secure Redpanda streaming platform with TLS and rpk tooling.

Redpanda is a modern, high-performance streaming platform that supports the Kafka API while using a simpler, more efficient architecture. It runs natively in C++ and eliminates JVM components like ZooKeeper, so you avoid common Kafka maintenance difficulties like memory tuning, GC delays, and coordination overhead. This simplified design reduces latency, operating burden, and increases throughput.

This article shows how to set up and configure Redpanda on Ubuntu 24.04, and make its console accessible over a web interface with HTTPS. It covers system preparation, secure configuration practices, and the use of Redpanda tooling such as bootstrap configuration and rpk profiles.

Prerequisites

Before starting, ensure you:

Have access to an Ubuntu 24.04 server as a non-root sudo user. The server should meet the memory and CPU requirements as per your use case.
Configure a domain name, such as redpanda.example.com, to point to your server’s public IP address.

Install Redpanda

Redpanda provides a setup script that detects the operating system and configures the apt repositories automatically. Follow the steps below to update the APT package manager index and install Redpanda on your server.

Update the APT package index.
console
```
$ sudo apt update
```
Download and execute the official Redpanda’s APT setup script as root.
console
```
$ curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash
```
Warning

If the script or its hosting source is compromised, it can run arbitrary commands and fully take over the system. Only run it if you trust the source, and your network.
Install the Redpanda server binary, the Redpanda tuner, and the Redpanda Keeper (rpk) CLI tool.
console
```
$ sudo apt install redpanda -y
```
Verify the installation by checking the rpk version.
console
```
$ rpk --version
```

Configure Firewall Access

By default, Redpanda services communicate on specific TCP ports. To allow external clients, administrators, and other nodes in the cluster to connect, you must explicitly let traffic through the Uncomplicated Firewall (UFW).

The table below outlines the specific ports used by Redpanda and their functions:

Port	Service	Use Case
9092	Kafka API	The primary listener for clients (Producers/Consumers). Used for writing and reading data streams.
8082	HTTP Proxy (Pandaproxy)	Allows access to data via REST API for clients that cannot use the native Kafka TCP protocol.
8081	Schema Registry	Manages versioned schemas (Avro/Protobuf) to ensure data compatibility between apps.
9644	Admin API	Used for monitoring, configuration, and health checks.
33145	Internal RPC	Handles internal communication between nodes

Follow the steps below to open the ports for each specific Redpanda component.

Open all required ports.

console

$ sudo ufw allow 9092,8082,8081,9644,33145/tcp

To generate and renew TLS certificates with Let’s Encrypt, Certbot requires port 80 to be open to perform ACME HTTP-01 challenge validation. Allow port 80 through your firewall.
console
```
$ sudo ufw allow 80/tcp
```
Allow port 443 through your firewall to access Redpanda web console over HTTPS.
console
```
$ sudo ufw allow 443/tcp
```
Reload the firewall.
console
```
$ sudo ufw reload
```

Generate TLS Certificates with Let’s Encrypt

By default, Redpanda data is sent unencrypted. This poses a significant security risk in production environments, as sensitive data streams could be intercepted. To secure your cluster, you must enable TLS encryption.

Follow the steps below to install Certbot, get a certificate, and organize the files for Redpanda.

Install Certbot.
console
```
$ sudo apt install certbot -y
```
Set a shell variable for your domain name. Replace redpanda.example.com with your domain name.
console
```
$ DOMAIN=redpanda.example.com
```
Set a shell variable for your email address. Replace admin@example.com with your email address.
console
```
$ EMAIL=admin@example.com
```
Request a TLS certificate for your domain using Certbot.
console
```
$ sudo certbot certonly --standalone -d $DOMAIN --non-interactive --email $EMAIL
```
- The above command temporarily starts a standalone web server on port 80 to verify domain ownership with Let’s Encrypt.
- Certbot stores certificates under /etc/letsencrypt/live, which is readable only by the root user.
- Since Redpanda runs as the redpanda user, you must copy the certificates to a dedicated directory with appropriate permissions.
Create a directory to store Redpanda TLS certificates.
console
```
$ sudo mkdir /etc/redpanda/certs
```

Copy the public certificate to the Redpanda certificate directory.

console

$ sudo cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt

Copy the private key to the Redpanda certificate directory.

console

$ sudo cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key

Copy the CA certificate to the Redpanda certificate directory.

console

$ sudo cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt

Grant the redpanda user ownership of the certificate directory.
console
```
$ sudo chown -R redpanda:redpanda /etc/redpanda/certs/
```
Set restrictive permissions on the private key so only the owner can read it.
console
```
$ sudo chmod 400 /etc/redpanda/certs/node.key
```

Allow read-only access to the public certificate and CA certificate.

console

$ sudo chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt

Set Up and Verify Automatic Certificate Renewal

Let’s Encrypt certificates expire every 90 days. To avoid service outages, Certbot includes an automated renewal system using certbot.timer. You must add a deploy hook so that whenever certificates are renewed, your Redpanda TLS files are updated and the service reloads securely.

Create the renewal script.

console

$ sudo nano /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh

Add the following contents to the file. Replace redpanda.example.com with your domain.

ini

#!/bin/bash
DOMAIN="redpanda.example.com"

# Copy renewed certificates
cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt
cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key
cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt

# Set ownership and permissions
chown -R redpanda:redpanda /etc/redpanda/certs/
chmod 400 /etc/redpanda/certs/node.key
chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt

# Restart Redpanda to pick up new certificates
systemctl restart redpanda

Make the renewal script executable.

console

$ sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh

Run a dry-run renewal to verify that automatic renewal and the deploy hook work correctly.
console
```
$ sudo certbot renew --dry-run
```

Configure Redpanda for Production

Redpanda runs in development mode by default, which disables hardware optimization and leaves the cluster open. To prepare for production, you will enable hardware tuning, then use the Bootstrap File method to initialize security settings and create the superuser before the cluster starts processing traffic.

Note

Production mode enforces stricter resource checks. Ensure your server meets the memory and CPU requirements, otherwise, Redpanda may fail to start.

Enable Production Mode and Tuning

The autotuner identifies your hardware configuration and optimizes the Linux kernel (I/O schedulers, CPU settings) for performance.

Set Redpanda to production mode.
console
```
$ sudo rpk redpanda mode production
```
Tune the Linux kernel.
console
```
$ sudo rpk redpanda tune all
```
This takes about 30 seconds and dramatically increases performance. Changes to the kernel do not always persist across reboots. Enabling the redpanda-tuner service guarantees that optimizations are automatically reapplied when the node restarts.
Enable the Redpanda tuner service to persist optimizations across reboots.
console
```
$ sudo systemctl start redpanda-tuner
```

Configure Bootstrap File and User

Redpanda allows you to configure cluster settings before the first startup using a .bootstrap.yaml file. This is the cleanest way to set up authentication and security from the beginning.

Create the Redpanda bootstrap configuration file.
console
```
$ sudo nano /etc/redpanda/.bootstrap.yaml
```
Add the following initial configuration to the file.
yaml
```
# Enable SASL authentication globally
enable_sasl: true

# Enable Admin API authentication
admin_api_require_auth: true

# Define superusers (users with full administrative privileges)
superusers:
  - admin
```
Save and close the file. This configuration ensures authentication and administrative access are enforced from the very first startup.

Note

This file is only read on the first startup of Redpanda. Any later configuration changes must be done through the Admin API or rpk cluster config commands.

The bootstrap file defines who the superuser is, but it does not create the password. To create the user credentials on the first boot, define an environment variable in the Redpanda system configuration.
Add the RP_BOOTSTRAP_USER variable to the bottom of the file. Replace STRONG_PASSWORD with your actual password.
console
```
$ echo "RP_BOOTSTRAP_USER=admin:STRONG_PASSWORD:SCRAM-SHA-256" | sudo tee -a /etc/default/redpanda
```
This creates a user named admin with your specified password using the SCRAM-SHA-256 authentication mechanism.

Update Configuration File

Redpanda uses a YAML configuration file located at /etc/redpanda/redpanda.yaml. By default, the service listens on the localhost interface. To allow external connections, configure the advertised addresses. Now configure the main Redpanda settings with your domain name and TLS certificates.

Back up the original configuration.

console

$ sudo cp /etc/redpanda/redpanda.yaml /etc/redpanda/redpanda-backup.yaml

Open the configuration file.
console
```
$ sudo nano /etc/redpanda/redpanda.yaml
```

Replace the entire configuration file with the following content. Replace redpanda.example.com with your actual domain name.

yaml

redpanda:
  data_directory: /var/lib/redpanda/data
  seed_servers: []

  rpc_server:
    address: 0.0.0.0
    port: 33145
  advertised_rpc_api:
    address: redpanda.example.com
    port: 33145

  kafka_api:
    - name: tls_listener
      address: 0.0.0.0
      port: 9092
      authentication_method: sasl
  advertised_kafka_api:
    - name: tls_listener
      address: redpanda.example.com
      port: 9092
  kafka_api_tls:
    - name: tls_listener
      key_file: /etc/redpanda/certs/node.key
      cert_file: /etc/redpanda/certs/node.crt
      truststore_file: /etc/redpanda/certs/ca.crt
      enabled: true
      require_client_auth: false

  admin:
    - address: 0.0.0.0
      port: 9644
  admin_api_tls:
    - enabled: true
      key_file: /etc/redpanda/certs/node.key
      cert_file: /etc/redpanda/certs/node.crt
      truststore_file: /etc/redpanda/certs/ca.crt

rpk:
  tune_network: true
  tune_disk_scheduler: true
  tune_disk_nomerges: true
  tune_disk_write_cache: true
  tune_disk_irq: true
  tune_cpu: true
  tune_aio_events: true
  tune_clocksource: true
  tune_swappiness: true
  coredump_dir: /var/lib/redpanda/coredump
  tune_ballast_file: true

Save and close the file. This configuration defines the core Redpanda node settings:

redpanda

Configures the local data directory and disables seed servers for a single-node cluster.
Defines the RPC API used for internal node communication and advertises it using the public hostname.
Enables the Kafka API with SASL authentication over TLS on port 9092.
Advertises the Kafka endpoint using the public domain so clients can connect securely.
Configures TLS certificates for Kafka traffic using the node certificate and cluster CA.

admin API

Exposes the Redpanda Admin API on port 9644.
Secures the Admin API with TLS using the same certificate authority.
Ensures administrative operations require encrypted connections.

rpk tuning

Applies recommended system tuning for network, disk, CPU, memory, and I/O.
Improves performance and stability for production workloads.
Configures a dedicated directory for Redpanda core dumps.

Start the Redpanda service.
console
```
$ sudo systemctl start redpanda
```
Redpanda initializes the node, applies the bootstrap configuration, creates the configured superuser, and brings the cluster online with authentication and TLS enforced.
Verify the service is active.
console
```
$ sudo systemctl status redpanda
```

Create the Profile Configuration File

Run rpk as your regular sudo-capable user (not via sudo) so it can use your profile configuration.

Create a new rpk profile file to store your Redpanda credentials.
console
```
$ nano ~/rpk-profile.yaml
```

Add the following configuration to the file. Replace redpanda.example.com and STRONG_PASSWORD with your actual broker address and password.

yaml

kafka_api:
  sasl:
    user: admin
    password: STRONG_PASSWORD
    mechanism: SCRAM-SHA-256
  tls:
    enabled: true
    ca_file: /etc/redpanda/certs/ca.crt
  brokers:
    - redpanda.example.com:9092

admin_api:
  tls:
    enabled: true
    ca_file: /etc/redpanda/certs/ca.crt
  addresses:
    - redpanda.example.com:9644

Save and close the file.

Create and switch to the profile from the file.

console

$ rpk profile create production --from-profile ~/rpk-profile.yaml

Now check the cluster health. Thanks to the rpk profile, you don’t need to specify credentials or TLS flags.

console

$ rpk cluster health

Output:

CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          true
Unhealthy reasons:                []
Controller ID:                    0
All nodes:                        [0]
Nodes down:                       []
Nodes in recovery mode:           []
Leaderless partitions (0):        []
Under-replicated partitions (0):  []
Cluster UUID:                     65e28fbb-c1b4-42c3-b0b7-fd82608699f3

Test the Installation

Now that the cluster is secured and running, perform a basic producer-consumer test to ensure it processes data accurately.

Create a test topic named test-topic.

console

$ rpk topic create test-topic

Output should show:

TOPIC       STATUS
test-topic  OK

Produce a message to the topic.

console

$ echo "Hello Redpanda" | rpk topic produce test-topic

Output should show:

Produced to partition 0 at offset 0 with timestamp 1699618845123.

Consume the message from the topic.
console
```
$ rpk topic consume test-topic --num 1
```
The output should display the message payload and metadata in JSON format.

Access Redpanda Console

Redpanda Console is a modern web interface for managing and monitoring your cluster. It provides visibility into topics, messages, consumer groups, and the schema registry.

Having configured the Redpanda repository earlier, install the console package.
console
```
$ sudo apt install redpanda-console -y
```
Configure the Redpanda Console configuration file.
console
```
$ sudo nano /etc/redpanda/redpanda-console-config.yaml
```

Replace the existing content with the following configuration.

yaml

server:
  listenPort: 8080
  listenAddress: 127.0.0.1

kafka:
  brokers:
    - redpanda.example.com:9092
  tls:
    enabled: true
    caFilepath: /etc/redpanda/certs/ca.crt
  sasl:
    enabled: true
    username: admin
    password: STRONG_PASSWORD
    mechanism: SCRAM-SHA-256

redpanda:
  adminApi:
    enabled: true
    urls:
      - https://redpanda.example.com:9644
    tls:
      enabled: true
      caFilepath: /etc/redpanda/certs/ca.crt
    authentication:
      basic:
        username: admin
        password: STRONG_PASSWORD

Save and close the file.

The redpanda-console-config.yaml file controls how the console connects to your Redpanda cluster.

Server: Sets the internal port (8080) where the console listens.
Kafka & Redpanda: Establishes secure TLS connections to your cluster’s Kafka API (messaging) and Admin API (management) using the credentials defined earlier.

Enable the Redpanda Console service to start automatically on boot.
console
```
$ sudo systemctl enable redpanda-console
```
Start the Redpanda Console service.
console
```
$ sudo systemctl start redpanda-console
```

Set Up Nginx Reverse Proxy with Basic Authentication

Since the open-source version of Redpanda Console does not include built-in login functionality, you must secure it using Nginx Basic Authentication. This setup forces users to enter a username and password before they can access the dashboard.

Install Nginx.
console
```
$ sudo apt install nginx -y
```
Install the Apache utilities package.
console
```
$ sudo apt install apache2-utils -y
```
This package provides the htpasswd utility, which is required to generate the password file for Nginx.
Run the following command to create a user named admin. You will be prompted to enter and confirm a password.
console
```
$ sudo htpasswd -c /etc/nginx/.htpasswd admin
```
The above command prompts you to enter and confirm the password for the admin user.
Create a new Nginx configuration file for the console.
console
```
$ sudo nano /etc/nginx/sites-available/redpanda-console
```

Add the following configuration. Replace redpanda.example.com with your domain.

ini

server {
    listen 80;
    server_name redpanda.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name redpanda.example.com;

    # Reuse the existing Let's Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/redpanda.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/redpanda.example.com/privkey.pem;

    location / {
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Save and close the file.

Enable the new site configuration.

console

$ sudo ln -s /etc/nginx/sites-available/redpanda-console /etc/nginx/sites-enabled/

Test the Nginx configuration for syntax errors.
console
```
$ sudo nginx -t
```
Restart Nginx to apply the changes.
console
```
$ sudo systemctl restart nginx
```
Open your web browser and navigate to your domain using HTTPS.
```
https://redpanda.example.com
```
The browser prompts you to enter the username and the password you created earlier. After entering your credentials, you should see the Redpanda Console dashboard secured with a valid TLS certificate.

Cobra Softwares Blog