Squid Labs

Apache Kafka - Setting up (Part 1)

Kev's photo
Kev
·

2 min read

When we approach Kafka, there are many ways to setup depending on different scenario. In my case which I need to setup for Production, I only use VMs and not Kubernetes. The decision is based on experience handling Apache Kafka on different sites and region, maintenance on VMs would be much easier to debug and have less issue to deal with Kubernetes

In this tutorial, I am using Ubuntu 24.10

Step 1: Install Java

sudo apt install openjdk-21-jdk -y

Next check Java version

java -version

Step 2: Create Kafka user

This is good practice to setup using dedicated user and not root

sudo useradd -m -s /bin/bash kafka
sudo passwd kafka
# switch to kafka user
sudo su - kafka

Step 3: Download and Extract Kafka

wget https://downloads.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz
tar -xzf kafka_2.13-3.9.0.tgz
mv kafka_2.13-3.9.0 kafka

Step 4: Configure Kafka

Kafka requires Zookeeper, which comes bundled with Kafka for development and testing purposes. For production environment, you should set up a dedicated Zookeeper cluster.

Configure Zookeeper

Create a data directory for Zookeeper:

mkdir -p ~/kafka/data/zookeeper

Edit the configuration file for Zookeeper

nano ~/kafka/config/zookeeper.properties

Update the dataDir property to point to the new data directory:

dataDir=/home/kafka/kafka/data/zookeeper

Configure Kafka Broker

Create a data directory for Zookeeper:

mkdir -p ~/kafka/data/kafka

Edit the configuration file

nano ~/kafka/config/server.properties

Update the following properties

log.dirs=/home/kafka/kafka/data/kafka
zookeeper.connect=localhost:2181

Step 5: Setting up Kafka as a systemd service

Setting up Zookeeper systemd

sudo nano /etc/systemd/system/zookeeper.service

add the following

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
After=network.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Create a Kafka systemd

sudo nano /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server
Documentation=http://kafka.apache.org/documentation.html
After=network.target zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Start the services

sudo systemctl daemon-reload
sudo systemctl start zookeeper
sudo systemctl enable zookeeper

sudo systemctl start kafka
sudo systemctl enable kafka

Testing the Kafka service, launch a terminal, this creates a test topic

~/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

List topics

~/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Produce Messages

start a Kafka producer, type a few messages and hit Enter after

~/kafka/bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092

Consume Messages
Open another terminal

~/kafka/bin/kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092
Apache Kafka