Setting up the Kubernetes namespace for kafka
apiVersion: v1 kind: Namespace metadata: name: "kafka" labels: name: "kafka"
k apply -f kafka-ns.ymlNow let's create the ZooKeeper container inside the kafka namespace
apiVersion: v1 kind: Service metadata: labels: app: zookeeper-service name: zookeeper-service namespace: kafka spec: type: NodePort ports: - name: zookeeper-port port: 2181 nodePort: 30181 targetPort: 2181 selector: app: zookeeper --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: zookeeper name: zookeeper namespace: kafka spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - image: wurstmeister/zookeeper imagePullPolicy: IfNotPresent name: zookeeper ports: - containerPort: 2181image1 - kube-kafka1 From the ZOOKEEPER services get the Cluster IP and use it in the Kafka broker configuration which is next step we are going to perform.
apiVersion: v1 kind: Service metadata: labels: app: kafka-broker name: kafka-service namespace: kafka spec: ports: - port: 9092 selector: app: kafka-broker --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: kafka-broker name: kafka-broker namespace: kafka spec: replicas: 1 selector: matchLabels: app: kafka-broker template: metadata: labels: app: kafka-broker spec: hostname: kafka-broker containers: - env: - name: KAFKA_BROKER_ID value: "1" - name: KAFKA_ZOOKEEPER_CONNECT value: ZOOKEEPER-INTERNAL-IP:2181 - name: KAFKA_LISTENERS value: PLAINTEXT://:9092 - name: KAFKA_ADVERTISED_LISTENERS value: PLAINTEXT://kafka-broker:9092 image: wurstmeister/kafka imagePullPolicy: IfNotPresent name: kafka-broker ports: - containerPort: 9092In the above line number 37 you need to change according to your Zookeeper service NodePort. Now apply
k apply -f kafka-broker.ymlafter apply
watch kubectl get all -n kafkaImage: Kube kafka 2 Step 3: Enable Network communications To ensure that Zookeeper and Kafka can communicate by using this hostname (kafka-broker), we need to add the following entry to the /etc/hosts file on
echo "127.0.0.1 kafka-broker" > /etc/hostsSet the port forwarding as following: kubectl port-forward
Test Kafka Topics using Kafkacat
To easily send and retrieve messages from Kafka, we’ll use a CLI tool named KCat 7Install KCat using the below command:apt install kafkacatImage : Kafkacat installation
Producing message and Consume using Kafcat
Run the below command to create a topic named topic1 and send a test message “hello everyone!” you can enter your own messages.echo "hello everyone!" | kafkacat -P -b 127.0.0.1:9092 -t topic1Now let's consume the message using kafkacat command:
kafkacat -C -b 127.0.0.1:9092 -t topic1Image : Kafkacat Producer Consumer Happy learning Kafka on the Kubernetes, The above experiment I've run on the Killercoda terminal.