DevOps Hunter Security Automation

  Setting up the Kubernetes namespace for kafka apiVersion: v1 kind: Namespace metadata: name: "kafka" labels: name: "kafka" k apply -f kafka-ns.yml Now let's create the ZooKeeper container inside the kafka namespace apiVersion: v1 kind: Service metadata: labels: app: zookeeper-service name: zookeeper-service namespace: kafka spec: type: NodePort ports: - name: zookeeper-port port: 2181 nodePort: 30181 targetPort: 2181 selector: app: zookeeper --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: zookeeper name: zookeeper namespace: kafka spec: replicas: 1 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - image: wurstmeister/zookeeper imagePullPolicy: IfNotPresent name: zookeeper ports: - containerPort: 2181 image1 - kube-kafka1 From th...

There are many monitoring trools available for Kafka brokers. Ive collectiect Various monitoring options: Factor House Kpow Datadog Logit.io Kafka Lag Exporter Confluent CMAK (Cluster Manager for Apache Kafka) Kafdrop Offset Explorer  Let's explore the kafdrop monitoring Prerequisites: To run the Kafdrop monitoring tool Ensure Java installed by checking java -version If you don't find java on your Ubuntu run the following : sudo apt update sudo apt install openjdk-21-jdk -y for other Linux distributions you need to use right package manger to install JDK. Download Kafdrop jar file from github : sudo mkdir kafka-monitor cd kafka-monitor curl -L -o kafdrop.jar https://github.com/obsidiandynamics/kafdrop/releases/download/4.1.0/kafdrop-4.1.0.jar The curl command ensures that Kafdrop may have any release version but the jar file renamed as `kafdrop.jar`. Now all set to go and run the Kafdrop jar file with the --kafka.brokerConnect option where you can give sing...

Kafka message broker is developed by Linkedin. To support their social media platform across the world. Message systems are defined as two Queues, Topics. If a messge send from a producer and received by single consumer then the communication will be point-to-point this is will be allowed in Queues. If Producer produce the message and there coudl be multiple consumers then it is One-to-many or pubf-sub of Publisher/Subscriber model it is implemented by Topic. Kafka supports both message models. Initial Setup for Kafka Kafka is build on scale and it runs on Java run time. So prerequisite to run Kafka we need Java as prerequisite. Optional other tools to help for troubleshoot and identify Kafka ports in use with 'netstat' command, topic content to view 'jq' and to have tree view 'tree' need to installed. apt update apt install -y net-tools jq tree On Ubuntu run the following commands to install compatible Java, here I'm using OpenJDK 21 version apt ...

What is Job object in Kubernetes? A Job object will be used to create one or more Pods and the Job ensures that a specified number of Pod instances will be created and terminates after completion of the Job. There could be finite jobs which will run within given certain timeout values. Job tracks for 'Successful' completion of the required task. Jobs can be run in two variants they can be parallel and also non-parallel. Kubernetes Job types There are 3 types of jobs non-parallel jobs [single pod jobs - unless it fails. creates replacement pod when pod goes down] parallel jobs with a fixed completion count parallel jobs with task queue  ##Example type 1: hundred-fibonaccis.yml --- apiVersion: batch/v1 kind: Job metadata: name: fibo-100 spec: template: spec: containers: - name: fib-container image: truek8s/hundred-fibonaccis:1.0 restartPolicy: OnFailure backoffLimit: 3 Create the Job: kubectl create -f hundred-fibonaccis.yml Now let's...

Hello DevSecOps, SRE or Platform Engineer or DevOps Engineers, In this post I want to discuss, Understanding of Kubernetes deployment it's hierarchy of kube objects. Declaratives and imperative ways to make deployment on kube clusters.  How to deploy an application on  Kubernetes pods, just follow these steps as shown in this post.  Here is new learning, I would like to share with you about Kubernetes deployment hierarchy , which internally calls the replication controller to make desired number of replicas of pod temple specified. Kubernetes Deployment hierarchy Let's go to have a deep understanding about Kubernetes deployment hierarchy. 1. Generating Kubernetes Deployment Manifest file We need to create a YAML file to define the deployment of the 'httpd' Apache Webserver. Here we are going to use the '--dry-run' option with client as value and '-o yaml' to generate the YAML file, to redirect the output we can use the g...

Hello DevOps team, in this post I will be exploring the CGroup usage in docker engine. How check CGroup in the Linux file system supporting? To check all types of cgroups that are allowed in Linux system we can check at /sys/fs/cgroup directory What's the default memory limit for docker containers? Interestingly, when you don't define any control group on the memory limits, the docker engine will allocate the full memory of the VM as the maximum memory limit. How do we impose the container memory limits? When you run the container with the 100m upper memory limit for the tomcat image. You can see the limit value docker stats command. Can we control of CPU load per container? How? Yes it is possible to control the CPUs usage to container. Control group helps us in define the limit, this can be decimal values that indicate CPU cycles - example here with 0.1 We can apply both the control group limits on the same container. H...

As of my last update in August 2023, SonarQube 10 already released. However, I can provide you with general steps to install SonarQube on CentOS/Rocky 8. Please note that the steps might need adjustments based on the specific versions you are using. Prerequisites : System requirements: RAM 4GB CPU 1vCPU works, better performance 4 cores Ensure you have the following prerequisites installed on your CentOS 8 server: Create a Vagrant CentOS/8 box for SonarQube installation: Vagrant.configure(2) do |config| config.vm.box = "centos/8" config.vm.boot_timeout=600 config.vm.define "sonarqube" do |sonarqube| sonarqube.vm.network "private_network", ip: "192.168.33.150" sonarqube.vm.hostname = "sonarqube.devopshunter.com" sonarqube.vm.provider "virtualbox" do |vb| vb.cpus = "4" vb.memory = "4096" end end end Bring up the box using `vagrant up`. In the PuTTY / SSH terminal ...