Kafka installation on Ubuntu

By -

Apache Kafka is one of the most widely adopted distributed event-streaming platforms used by organizations to process, store, and analyze real-time data at scale.

Originally developed at LinkedIn and later open-sourced through the Apache Software Foundation, Kafka was designed to handle massive volumes of real-time data generated by millions of users worldwide.

In this hands-on tutorial, you will learn how to install Apache Kafka on Ubuntu, configure multiple Kafka brokers on a single host, create topics, publish messages, and consume messages using Kafka command-line tools.

What You'll Learn

By the end of this tutorial, you will be able to:

  • Understand Kafka architecture and messaging concepts
  • Install Apache Kafka on Ubuntu Linux
  • Configure ZooKeeper for Kafka
  • Create a multi-broker Kafka cluster
  • Create and manage Kafka topics
  • Publish messages using Kafka Producers
  • Read messages using Kafka Consumers
  • Verify Kafka services and network ports

What is Apache Kafka?

Apache Kafka is a distributed messaging and event-streaming platform that enables applications to exchange data reliably and in real time.

Kafka is commonly used for:

  • Real-time analytics
  • Log aggregation
  • Event-driven architectures
  • Application monitoring
  • Data pipelines
  • Microservices communication
  • IoT data processing

Core Components

  • Producer – Sends messages to Kafka Topics.
  • Broker – Kafka server responsible for storing messages.
  • Topic – Logical channel used for message categorization.
  • Partition – Enables scalability and parallel processing.
  • Consumer – Reads messages from Topics.
  • Consumer Group – Multiple consumers sharing workload.
  • ZooKeeper – Coordinates broker metadata (used in traditional Kafka deployments).

Prerequisites

Kafka runs on the Java Runtime Environment (JRE), so Java must be installed before Kafka can be started.

Install supporting Tools

We'll also install several useful utilities for troubleshooting and administration.

apt update
apt install -y net-tools jq tree

Why These Tools?

ToolPurpose
netstat   Check network ports used by Kafka
jq      Parse JSON output
tree   Display directory structures

Install Open JDK


For this tutorial, we will use OpenJDK 21.
On Ubuntu run the following commands to install compatible Java, here I'm using OpenJDK 21 version 
 
apt install -y openjdk-21-jdk-headless
Verify and confirm the Java installation get the java version : `java -version`

Expected output should display OpenJDK 21.

OpenJDK21 installed verify
Install Open JDK 21 on Ubuntu


 

Install Apache Kafka on Ubuntu

At the time of writing this post, Apache Kafka provides multiple stable releases. You can always verify the latest version from the official Kafka download page.


Download Kafka: 
 
wget https://dlcdn.apache.org/kafka/3.9.0/kafka_2.12-3.9.0.tgz

# extract the file and cd into the folder

tar -xzf kafka_2.12-3.9.0.tgz
cd kafka_2.12-3.9.0
 Verify the installation directory:
tree -L 1

Install Kafka 3.9.0
Kafka installer download and extract





Start ZooKeeper
Traditional Kafka deployments require ZooKeeper to coordinate broker metadata and cluster state information.
Default ZooKeeper configuration:
tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181

Start ZooKeeper using the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

Verify ZooKeeper Status
Open another terminal and run:
echo "Are you there ZooKeeper" | nc localhost 2181 
Expected response:
imok

If you receive imok, ZooKeeper is running successfully.
Configure Multiple Kafka Brokers on a Single Host
For learning purposes, running multiple brokers on a single server is an excellent way to understand Kafka clustering concepts.
Navigate to the configuration directory::


cd kafka_2.12-3.9.0/config/

#Create separate broker configurations files:
cp server.properties server1.properties # for broker1
cp server.properties server2.properties # for broker2



Most common changes are with the 3 parameters on broker configuration:

Configure Broker 1: 
Edit server1.properties:
broker.id=1
listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka-logs-1



Configure Broker 2
Edit server2.properties:
broker.id=2
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kafka-logs-2



and then start these three servers in background as follows:
Start Kafka Brokers
Broker 1:
cd ~/kafka_2.12-3.9.0
nohup bin/kafka-server-start.sh config/server1.properties >broker1.out 2>&1 &
tail -f broker1.out

Broker 2:

nohup bin/kafka-server-start.sh config/server2.properties >broker2.out 2>&1 &
tail -f broker2.out


Verify Kafka Processes
Check Java processes:
#confirm with process check and network stats
jps -lm

Verify listening ports:
netstat -tulpn|grep java
You should see:
  • ZooKeeper on port 2181
  • Kafka Broker 1 on port 9092
  • Kafka Broker 2 on port 9093



Kafka Processes and Network Ports

Create a Kafka Topic

A Topic is the logical channel through which messages are exchanged. Add another terminal window create topic 'test' with the following commands:


cd ~/kafka_2.12-3.9.0/
bin/kafka-topics.sh --create \
--bootstrap-server \
localhost:9092 --replication-factor 1 \
--partitions 1 --topic test 
Expected output:
Created topic test

List the existing Topics
Now, we can confirm the topic is created on the kafka server by listing all topics on it:

bin/kafka-topics.sh --list \
--bootstrap-server localhost:9092
Output:
test



Send Messages Using a Kafka Producer 


Kafka provides a built-in console producer for testing. Kafka bin comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. 
 

Run the producer and then type some text messages into the console to send to the server.


bin/kafka-console-producer.sh \
--broker-list localhost:9092 \
--topic test

For example enter the message on topic test as follows:

I love India
Mera Bharat mahan
Viswa guru Bharat
Skill India



Each line is treated as an individual Kafka messages.
Press Ctrl+C to exit from producer 


Consume Messages Using Kafka Consumer

Now let's read the messages from the topic.

Start a consumer script to receive messages from the Topic. Kafka also has a command line consumer that will dump out messages to standard output, it should return those lines you typed in the above step.

bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test 

Read Messages from the Beginning
bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test --from-beginning

Read Messages from a Specific Offset
bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test -o 2


same as we did for producer, Consumer can press Ctrl+C to exit.

Challenge for Readers
Try the following exercises:
  1. Create a topic with 3 partitions.
  2. Increase the replication factor.
  3. Start a third broker on port 9094.
  4. Create multiple consumers within a consumer group.
  5. Observe how Kafka distributes messages across partitions.
Share your observations in the comments and let others learn from your experience. 
 Hope you enjoyed this learning with me. Keep posting your experience on this.
What's Next?
In upcoming Kafka tutorials, we'll explore:
  • Kafka Monitoring with Prometheus and Grafana
  • Running Kafka on Kubernetes
  • Kafka with Docker and Docker Compose

    • Stay tuned for more hands-on DevOps, Kubernetes, Platform Engineering, and Apache Kafka tutorials.
    

    

    

    

    

    

    








    

    

    

    


    

    

    

    


    Comments
    

    

    

    

Post a Comment


    

    

    

    




    

    

    

    

    

    
Popular Articles

    

    

    

    

    Ansible URI Module Tutorial: Real-World Application Health Checks, REST API Validation and DevOps Automation
    

    

    


By







-






    

    

    

    

Image


    

    

    
Hello DevSecOps and Automation Engineers! Welcome back to the DevOps Hunter blog. One of the most common requirements in enterprise automation is validating whether an application is actually available after a deployment, patching activity, server reboot, or service restart. Simply starting a service does not guarantee that the application is ready to serve users. This is where the Ansible uri  module  becomes one of the most powerful tools in a DevOps engineer's toolkit. In this hands-on tutorial, we will explore the Ansible URI module, understand its key parameters, and work through several real-world examples that are commonly used in production automation workflows. Production Health Check workflow using URI module What You'll Learn By the end of this article, you will be able to: Understand the purpose of the Ansible URI module Perform HTTP and HTTPS application health checks Validate application response codes Test REST API endpoints Handle multiple acceptable HTTP status...

    


    

    

Read more


    

    

    

    

    DevOps Weapons
    

    

    


By







-






    

    

    

    

Image


    

    

    
       To achieve faster application delivery and stable environments, the right tools must be used in DevOps environments. There is no single tool that fits all your needs such as server provisioning, configuration management, automated builds, code deployments, and monitoring. Many factors determine the use of a particular tool in your infrastructure.     In this article, we will look into core tools which can be used in a typical Devops environment.     Version Control Tools:   Git  - An awesome tool to version your source code and collaborate.   Terraform  - A tool used in building, changing, and versioning the infrastructure across platforms.       Build Orchestration/Continuous Integration Tools:  Jenkins - It is an open-source, lightweight CI tool written in Java, with high extensibility and a fast release cycle.  Buildbot  - An open-source framework for automating software build, test and release process.  CruiseControl  - A CI server written in  # Ruby ...

    


    

    

Read more


    

    

    

    

    Ansible Jinja2 Templates: A Complete Guide with Examples
    

    

    


By







-






    

    

    

    

Image


    

    

    
Here I'm starting this post with famous ARISTOTLE quote. "We are what we repeatedly do. Excellence, then is not an act, but a Habit" Welcome back Ansible Automations  and who are habituated as Ansible automation specialists this post for them to give boosted walk through, In this post, I would like to share my experiments with Jinja2 templates usage on the Ansible playbook. Of-course Jinja is from Japan it is world famous for templatization capabilities badvanced Ansible templating techniquesy integrating with multiple languages such as Python, Ruby, Salt talk etc. In this post we will be cover the following topics: What is Jinja2 template do in Ansible? Template with filters Template with Lists and sets Template module in Ansible Template with Flow Control Template using Looping Template inheritance** What is Jinja2 template do in Ansible? Jinja2 is another Python library created for Flask web framework, that comes part of Ansible installation. It is special ability as i...

    


    

    

Read more