DevOps Hunter Security Automation

  Setting up the Kubernetes namespace for kafka apiVersion: v1 kind: Namespace metadata: name: "kafka" labels: name: "kafka" k apply -f kafka-ns.yml Now let's create the ZooKeeper container inside the kafka namespace apiVersion: v1 kind: Service metadata: labels: app: zookeeper-service name: zookeeper-service namespace: kafka spec: type: NodePort ports: - name: zookeeper-port port: 2181 nodePort: 30181 targetPort: 2181 selector: app: zookeeper --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: zookeeper name: zookeeper namespace: kafka spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - image: wurstmeister/zookeeper imagePullPolicy: IfNotPresent name: zookeeper ports: - containerPort: 2181 image1 - kube-kafka1 From th...

There are many monitoring trools available for Kafka brokers. Ive collectiect Various monitoring options: Factor House Kpow Datadog Logit.io Kafka Lag Exporter Confluent CMAK (Cluster Manager for Apache Kafka) Kafdrop Offset Explorer  Let's explore the kafdrop monitoring Prerequisites: To run the Kafdrop monitoring tool Ensure Java installed by checking java -version If you don't find java on your Ubuntu run the following : sudo apt update sudo apt install openjdk-21-jdk -y for other Linux distributions you need to use right package manger to install JDK. Download Kafdrop jar file from github : sudo mkdir kafka-monitor cd kafka-monitor curl -L -o kafdrop.jar https://github.com/obsidiandynamics/kafdrop/releases/download/4.1.0/kafdrop-4.1.0.jar The curl command ensures that Kafdrop may have any release version but the jar file renamed as `kafdrop.jar`. Now all set to go and run the Kafdrop jar file with the --kafka.brokerConnect option where you can give sing...

Kafka message broker is developed by Linkedin. To support their social media platform across the world. Message systems are defined as two Queues, Topics. If a messge send from a producer and received by single consumer then the communication will be point-to-point this is will be allowed in Queues. If Producer produce the message and there coudl be multiple consumers then it is One-to-many or pubf-sub of Publisher/Subscriber model it is implemented by Topic. Kafka supports both message models. Initial Setup for Kafka Kafka is build on scale and it runs on Java run time. So prerequisite to run Kafka we need Java as prerequisite. Optional other tools to help for troubleshoot and identify Kafka ports in use with 'netstat' command, topic content to view 'jq' and to have tree view 'tree' need to installed. apt update apt install -y net-tools jq tree On Ubuntu run the following commands to install compatible Java, here I'm using OpenJDK 21 version apt ...