Kafka installation on Ubuntu

Kafka message broker is developed by Linkedin. To support their social media platform across the world. Message systems are defined as two Queues, Topics. If a messge send from a producer and received by single consumer then the communication will be point-to-point this is will be allowed in Queues. If Producer produce the message and there coudl be multiple consumers then it is One-to-many or pubf-sub of Publisher/Subscriber model it is implemented by Topic. Kafka supports both message models.

Initial Setup for Kafka

Kafka is build on scale and it runs on Java run time. So prerequisite to run Kafka we need Java as prerequisite. Optional other tools to help for troubleshoot and identify Kafka ports in use with 'netstat' command, topic content to view 'jq' and to have tree view 'tree' need to installed.

apt update
apt install -y net-tools jq tree

On Ubuntu run the following commands to install compatible Java, here I'm using OpenJDK 21 version

 
apt install -y openjdk-21-jdk-headless

Confirm the Java installation get the java version : `java -version`

 

Install Apache Kafka

Let's use kafka's quick start https://kafka.apache.org/quickstart as per the Apache Kafka release model they will have yearly 3 stable releases the last stable release version available 3.9.0 from Nov 2024 with two binary options with Scala version support 2.12 and 2.13 are availble. So we will download kafka at https://www.apache.org/dyn/closer.cgi?path=/kafka/3.9.0/kafka_2.12-3.9.0.tgz

 

wget https://dlcdn.apache.org/kafka/3.9.0/kafka_2.12-3.9.0.tgz

# extract the file and cd into the folder

tar -xzf kafka_2.12-3.9.0.tgz
cd kafka_2.12-3.9.0

start zookeeper

Kafka comes bundled with ZooKeeper. Here’s an example of what your zookeeper.properties might look like:

tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181

start ZooKeeper using the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

To verify Zookeeper working run the below command:

echo "Are you there ZooKeeper" | nc localhost 2181

If it responds with `imok`, then ZooKeeper is operational.

Starting Kafka multiple brokers on single host

Intially try with single broker after understanding how it is working you can move to multi-broker setup as discussed here. So our step here is to Start the Kafka server and then create a topic Now open a new terminal window and start the kafka-server, the start script available in bin directory. To make cluster configuration create server.partitions file to 2 copies to simulate:

cd kafka_2.12-3.9.0/config/
cp server.properties server1.properties # for broker1
cp server.properties server2.properties # for broker2

Most common changes are with the 3 parameters on broker configuration:
Broker 1:

broker.id=1
listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka-logs-1

Broker 2:

broker.id=2
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kafka-logs-2

and then start these three servers in background as follows:

cd ~/kafka_2.12-3.9.0
nohup bin/kafka-server-start.sh config/server1.properties >broker1.out 2>&1 &
tail -f broker1.out

nohup bin/kafka-server-start.sh config/server2.properties >broker2.out 2>&1 &
tail -f broker2.out

#confirm with process check and network stats
jps -lm
netstat -tulpn|grep java

Create Topic

Add another terminal window create topic 'test' with the following commands:

cd ~/kafka_2.12-3.9.0/
bin/kafka-topics.sh --create --bootstrap-server \
localhost:9092 --replication-factor 1 \
--partitions 1 --topic test

Noe, we can confirm the topic is created on the kafka server by listing all topics on it:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Send messages to topic

Producer simulation here, Send some text messages for testing purpose Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default, each line will be sent as a separate message.
Run the producer and then type a few messages into the console to send to the server.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

For example enter the message on topic test as follows:

I love India
Mera Bharat mahan
Viswa guru Bharat
Skill India

Use ctrl-c to exit from producer

Consume messages

Start a consumer script to receive messages from the topic. Kafka also has a command line consumer that will dump out messages to standard output, it should return the lines you typed in the above step.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test 
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test -o 2

same as we did for producer, Consumer can use ctrl-c to exit Hope you enjoyed this learning with me. Keep posting your experiance on this.