Introduction
Apache Kafka 4.0 introduces KRaft mode (ZooKeeper-free) as the default. In this guide, we’ll install Kafka 4.0 on a single-node Ubuntu 24.04 server and secure it using HAProxy SSL termination, while keeping Kafka itself running in PLAINTEXT mode internally.
Architecture
| Component | Address | Protocol | Role |
|---|---|---|---|
| Kafka | 0.0.0.0:9092 | PLAINTEXT | Broker listener |
| Kafka | 127.0.0.1:9093 | PLAINTEXT | Controller listener |
| Kafka | kafka.maksonlee.com:9093 | SSL (via HAProxy) | Advertised broker address |
| HAProxy | 192.168.0.127:9093 | SSL | TLS termination |
- Install Java
sudo apt update
sudo apt install openjdk-21-jdk -y
- Download and Extract Kafka
cd /opt
sudo curl -LO https://downloads.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz
sudo tar -xzf kafka_2.13-4.0.0.tgz
sudo mv kafka_2.13-4.0.0 kafka
- Set Kafka Configuration
Edit the Kafka configuration:
sudo vi /opt/kafka/config/server.properties
Replace these 5 lines in the default config:
# 1. Listeners
- listeners=PLAINTEXT://:9092,CONTROLLER://:9093
+ listeners=CONTROLLER://127.0.0.1:9093,SSL://0.0.0.0:9092
# 2. Inter-broker listener
- inter.broker.listener.name=PLAINTEXT
+ inter.broker.listener.name=SSL
# 3. Advertised listeners
- advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
+ advertised.listeners=SSL://kafka.maksonlee.com:9093
# 4. Protocol mapping
- listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
+ listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:PLAINTEXT
# 5. Log directory
- log.dirs=/tmp/kraft-combined-logs
+ log.dirs=/var/lib/kafka/kraft-combined-logs
Create the directory:
sudo mkdir -p /var/lib/kafka/kraft-combined-logs
- Generate a Cluster UUID and Format Log Directories
cd /opt/kafka
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
sudo bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
- Install Certbot and HaProxy
sudo apt update
sudo apt install -y certbot python3-certbot-dns-cloudflare
sudo add-apt-repository ppa:vbernat/haproxy-3.2 -y
sudo apt-get install haproxy=3.2.\*
sudo systemctl enable --now haproxy
- Issue HTTPS Certificate via Cloudflare DNS
Create Cloudflare credentials:
mkdir -p ~/.secrets/certbot
vi ~/.secrets/certbot/cloudflare.ini
Add:
dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN
Secure the file:
chmod 600 ~/.secrets/certbot/cloudflare.ini
Request certificate:
sudo certbot certonly \
--dns-cloudflare \
--dns-cloudflare-credentials ~/.secrets/certbot/cloudflare.ini \
-d kafka.maksonlee.com
- Bundle Certificate for HAProxy
sudo mkdir -p /etc/haproxy/certs/
sudo bash -c "cat /etc/letsencrypt/live/kafka.maksonlee.com/fullchain.pem \
/etc/letsencrypt/live/kafka.maksonlee.com/privkey.pem \
> /etc/haproxy/certs/kafka.maksonlee.com.pem"
sudo chmod 600 /etc/haproxy/certs/kafka.maksonlee.com.pem
- Configure HAProxy for SSL Termination
Edit /etc/haproxy/haproxy.cfg:
frontend kafka_ssl
bind 192.168.0.127:9093 ssl crt /etc/haproxy/certs/kafka.maksonlee.com.pem
mode tcp
default_backend kafka_plaintext
backend kafka_plaintext
mode tcp
server kafka1 127.0.0.1:9092
Restart HAProxy:
sudo systemctl restart haproxy
- Automate Certificate Renewal
Create deploy hook to reload HAProxy on renewal:
sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh > /dev/null <<EOF
#!/bin/bash
cat /etc/letsencrypt/live/kafka.maksonlee.com/fullchain.pem \
/etc/letsencrypt/live/kafka.maksonlee.com/privkey.pem \
> /etc/haproxy/certs/kafka.maksonlee.com.pem
systemctl reload haproxy
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-haproxy.sh
- Run Kafka as a systemd Service
Create the service unit:
sudo vi /etc/systemd/system/kafka.service
Paste:
[Unit]
Description=Apache Kafka 4.0 (KRaft mode)
After=network.target
[Service]
Type=simple
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
# Uncomment if using a non-root user
# User=kafka
[Install]
WantedBy=multi-user.target
Enable and start the service:
sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafka
- Test Kafka via Python
produce_kafka.py
from confluent_kafka import Producer
conf = {
'bootstrap.servers': 'kafka.maksonlee.com:9093',
'security.protocol': 'ssl'
}
producer = Producer(conf)
def delivery_report(err, msg):
if err:
print('Delivery failed:', err)
else:
print(f'Message delivered to {msg.topic()} [{msg.partition()}]')
producer.produce('test-topic', key='key1', value='Hello Kafka!', callback=delivery_report)
producer.flush()
consume_kafka.py
from confluent_kafka import Consumer
conf = {
'bootstrap.servers': 'kafka.maksonlee.com:9093',
'security.protocol': 'ssl',
'group.id': 'test-group',
'auto.offset.reset': 'earliest'
}
consumer = Consumer(conf)
consumer.subscribe(['test-topic'])
print('Waiting for messages...')
msg = consumer.poll(10.0)
if msg is None:
print("No message received.")
elif msg.error():
print("Error:", msg.error())
else:
print(f'Received: {msg.value().decode("utf-8")}')
consumer.close()
Note: mTLS Support?
This setup uses SSL termination at HAProxy, so:
| Aspect | Enabled |
|---|---|
| TLS Encryption | ✅ Yes |
| mTLS | ❌ No |
| Kafka sees client certs | ❌ No |
To enable mTLS, you need to disable HAProxy and let Kafka terminate TLS directly.