DevOps Hunter Security Automation

 We as DevOps and DevSecOps Engineers working on many microservice based application architectures where we need to manage Kubernetes Cluster  Troubleshot at various levels. You cannot rely on single point of look for failures. While working on Kubernetes Troubleshooting we can make ourselves easy to understand the problem, if we could classify the problem belong to the following categories. Application Failure Master node/ControlPlane Failures Worker node Failures Application Failure - trobleshooting Here I'm listing out these with my understanding and experiance in practice tests provided by Munshad Mohammad on KodeKloud. You should know the architecture how it is deployed what all its dependents, where they have deployed with what endpoints, what names used. Check the service 'name' defined and referring service should match and also check the services 'Endpoints' are correctly defined and in referenceing used correctly. k -n dev-ns get all Better to check that t...

Hey Guys, Welcome to "DevOps Hunter" blog! In this post I would like to share my learnings at different times collected that is about Kubernetes commands and their applied tricks and tips. Initially I've collected few kubectl related alias command tricks Play with the etcd database and then backup and recovery short-cuts Finally worked on the Kubernetes command tools kubectx, kubens for easy switching in CLI. Come on! let's explore about the API resources which we might be frequently use when we prepare the YAML files for each Kubernetes Objects. kubectl api-resources We can get sometime the API version mismatch due to change in API version. This can be examine what is new in the current version How do you identify the certificate file used to authenticate 'apiserver'? cat /etc/kubernetes/manifests/kube-apiserver.yaml|grep tls-cert - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt The tls-cert-file will be Kubernetes apiserver ceri...

  Delegation to a host Here is an example where we can delegate the task to a particular host. This play book is using inventory_hostname from the gather facts. - name: Delegation to localhost hosts: all tasks: - name: create a file on target server file: path: /tmp/i_m_on_target_server.txt state: touch - name: create a file with host named file by delegation to localhost file: state: touch path: "/tmp/{{ inventory_hostname }}.txt" delegate_to: localhost connection paramer We can use this "connection" parameter add to your task level or play level. # Filename: connection_local.yml # To do some task on ansible server # local means without doing ssh command (no need of password and no need of ssh keys) # with the local connection parameter for the play --- - name: This is to determine how the connection parameter works with local hosts: app connection: local gather_facts: false tasks: - name: connecti...

Hello DevOps Automations Engineers!!  Ansible provides us special command 'ansible-vault' that is used to encrypt, decrypt, view an Ansible  playbook, this is also have amazing feature specific to role, vars YAML files, we can apply this to string of text in regular variables.  Why do we need to encrypt our Play books? Our Ansible automation projects, we need to work on multiple tasks and which may have some sensitive data such as database user credentials, any cloud IAM role details or it can be some other applications login credentials that's used to validate URL availability. Or it can be used to store the SSL certificates. At any point of time if the system is using plain text and it  has trouble to your confidential and sensitive data otherwise it could causes huge damage to your organization. Where we need a way to store the sensitive data can be protected by data encryption  tool, and this can be done using the Ansible-vault command.  Le...

Hello DevOps Experts!! let's zoom  into the usage of the Ansible Handlers and notifies   What are Ansible Handlers? The handlers section or the tasks defined under the handlers folder are executed at the end of the play once all tasks are finished. In the handlers tasks we are typically do either start, reload, restart and stop services. Sometimes we may need to execute the task only when a particular change is made that can be notified.  Simple example of Apache web server when we modify httpd.conf file then we want to restart the httpd service.  When we were working on Tomcat, when tomcat service is enabled. then there is a need for the reload firewalld service this is where we need to move this reload task under handlers and the enable tomcat service should have notify the task name 'reload firewalld service'. These are the perfect examples for handlers usage in Ansible play. So here the point is that handler tasks will be performed only when they are notif...

 Hey DevOps or DevSecOps or SRE Guys!! What's up? in the automation journey one more wonderful learning here!  In this post we will be implementing all our Ansible modules one after other to build a complete solution for Java based Application server installation and running using Ansible playbook. At present Tomcat latest version is 10.0.27 so I've used same  Pre-requisites:  To install Apache Tomcat there is separate JDK/JRE compatibility we need to validate before we proceed Create a dedicated user account as 'tomcat' with shell as bash  to manage Tomcat application server Create a separate directory for tomcat server to be installed Execution of multiple tasks in the Playbook will be as follows: Download the Tomcat software from Apache Tomcat 10.0.27.tar.gz Uncompressing the tomcat tar.gz file Change the file permissions and ownership Cleanup after unarchive the tar.gz file Start the Tomcat server Have a task to stop the Tomcat server --- - n...

Hello Guys!! HEAD pointer movement HEAD points to specific commit in the local repo-branch as new commits are made, the pointer changes HEAD always points to the "tip" of the currently checked-out branch in the repo (not the working directory or staging index) last state of repo (what was checkout initially HEAD points to parent of next commit(where writing next commit takes place) HEAD Movement in Git branches Git Reset movements This is most common need of every DevOps team development phase need. There are three options we have but of course two of them are mostly used. Git reset movements at three tree levels soft mixed hard Using --soft reset The soft reset command is to combine many commits into a single one. git reset --soft HEAD (going back to HEAD) git reset --soft HEAD^ (going back to the commit before HEAD) git reset --soft HEAD~1 (equivalent to "^") git reset --soft HEAD~2 (going back to 2 commits before HEAD) Using hard reset mov...

Hey Greetings of the day!! GitHub providing Personal Access Token instead of using username, password for git repository that is on the GitHub. where git subcommands such as git pull, git push, fetch and any remote operations will be depends on this PAT.   There are two choices for PAT  1. fine grain Personal Access Token(PAT)  2. Personal Access Token (Classic)  I understood that it is easy to change permissions, authentication token that use only specific target repositories.  If you want to know ' How PAT works? Where to start PAT ' then this post is absolutely for you! Welcome to this 2 mins read! Fine grained PAT newly introduced in Oct 2022 on the GitHub still at the time this post it is mentioned as [Beta] versioned. Personal Access Token PATS are going to work with commonly defined API on GitHub. Any integration made simplified with this method. How to create PAT on GitHub?  Login to your GitHub account. Click on the profile picture upper-ri...

 Ansible playbook can be a construct of multiple plays or each play may contains multiple tasks. This is where we may have situation where you need to add new task to the existing play or playbook, and we need to test many times that newly added task.  While testing multiple times we many don't want to execute certain tasks such as a task 'Send email notification' when you preparing a 'Reboot of server' or 'Restart of Service' or 'Deployment of a service'. During the testing time you may want to exclude these notification tasks.  There are situations where we might want to run a particular task as per the input at the run time of a playbook. This may be from AWX/Tower UI select them. Ansible tags - to control the tasks of a Playbook I will be explaining in this post, How to run or not to run a particular task in given  playbook.  Important concepts about Ansible tags Ansible tags are keys to identify and control the tasks for execution or exclude fr...

Hey DevOps Team, In this post I would like to give you the knowledge on the special feature about Ansible Facts and we can also customize these facts as global variables to use in multiple playbooks. What is ansible facts?  Ansible facts are just simple variables that are automatically discovered by ansible on a managed nodes. such as system information disk info, os info, package info IP Network and many more ... Why we manage facts? Default these facts will be automatically collected exclusively we need to disable some times. - multiple play in a playbook How can we use facts? we want to run only when enough memory is available on the target machine then install the package. That is the smartest way to do the automation! --- # File : hello.yaml - name: Facts example hosts: "{{targets|default('localhost')}}" tasks: - name: prints details debug: msg: "Hello this machine have {{ ansible_memory_mb['real'] }}" whe...

In this post we are going to learn more  about what is service accounts in Kubernetes and how that is useful. Prerequisites Kubernetes cluster Up and running Let's take the scenario where we get need to connect with the pods, nodes, deployments and other resources in the Kubernetes cluster. you might be working with the automated build with the CICD pipelines to interconnect with each other resources. Pod  is going to work with the planned application deployments. If  you're working in DevSecOps you need to focus on the regular monthly maintenance OS  patching scheduled in this case Kubernetes node maintenance should be done from a pod.  In the above two scenarios there is a need of service account inside the pod. When Kubernetes cluster is created at the same time service account also created and its name is default . We can also create our own service accounts using the following command Every service account is associated with the secret wh...

Hello in this post we will explore about ClusterRoles and ClusterRoleBindings on Kubernetes Cluster. The ClusterRoleBindings are mapping a subjects with ClusterRole. Here Subjects are nothing but rules that can be applicable with an action on the Cluster resources. It deals with Users, Groups and service accounts. In this post we will try to focus with 'User' specific rules. Kubernetes User Access Control with ClusterRoleBindings to ClusterRole   Prerequisite:  1. Kubernetes Cluster up and running  2. Basic understand on RBAC These system related resources such as pods, nodes, storage etcs will be administrated using ClusterRole and ClusterRoleBindings by assigning to a user.   To list the ClusterRoles in the Kubernetes cluster kubectl get clusterrole # Get the Count kubectl get clusterrole --no-headers |wc -l To know about the api-resources that have clusterrole and clusterrolebindings. k api-resources |grep cluster To veiew the clusterrolebindings availabl...

My Understanding about RBAC in Kubernetes RBAC stands for Role based access control in our Kubernetes system we have users that needs to access the kubernetes cluster and it's resources. Here role is that categorize their needs. Let's say our project have developers, admins, presale users. We could define role named as "readers" that allows all users, because its common need to all user to read from the system. We could define a role called "writers" and allow certainer users like "developers" who contribute many things to develop in application end, "Admin" user can have this to control it. We could also define a role called "administrators" to admins users. Administrator role users can have full rights such as delete from the system. Role can be used to define "what can be done?" Role will be given to users, application software. If we need to deal with software then we need to use service account. Service accou...

Kubernetes API Groups What is the Kubernetes API? Kubernetes API means it works with webservice that uses HTTP and REST protocols to enable the access for the API calls.  Let's see this how it works using 'curl' command, where we need to provide the URL then api call object path. Examples To view the Kubernetes version we can use : curl https://controlplane:6443/version -k To get the list of pods in default cluster curl https://controlplane:6443/api/v1/pods -k in this post we will get to know more about the api specifically Kubernetes API Groups. Each group is defined with a specific purpose, such as on api for health check, other for metrics collection logs etc. These metrics, health check will be used for health of the Kubernetes cluster. And the logs will be used for collecting by third party system where all logs will be collected such as ELK stack uses logstash agent.   The API are categorized into two :  1. Core group /api  2. Named group /apis...

Hello all! Welcome to new learning Kubernetes Certificate API in the series of "Kubernetes Security". a. Private key generation  Kubernetes Certificate API We must aware of what does certificate authority CA will do and in Kubernetes how it works. CA server it is a server which is runs certificate API. In your DevOps or DevSecOps team a New Kubernetes Admin joins you. Hhow to handle. Private key, Public key valid pair of CA server sign automated in Kubernetes, it performs following steps: 1. Create CertificateSigningRequest object 2. Review Request 3. Approve Request 4. Share Certs to Users Let's try how it works  A user Maheshwari(Mahi)  want to create certificate files first private key will be generated with RSA algorithm 'mahi.key' the key size could be 2048 bits. openssl genrsa -out mahi.key 2048 b. Certificate Signing request (CSR) object Request can be created by providing key and subject values the result can be stored into a csr file by perfor...

Hello Guys! in this post we are going to explore about the Kubeconfig. This is a special configuration that will be part of Kubernetes Security. We can configure multiple clusters and different users can access these Kubernetes cluster. We can also configure the users can have access to multiple clusters. When we started working on Kubernetes Cluster there is a config file automatically generated for us.  To access a Kube Cluster using the certificate files generated for admin user can be given as follows: kubectl get pods \ --server controlplane:6443 --clisent-key: admin.key --client-certificate admin.crt --certificate-authority ca.crt Every time passing all these TLS details(server,client-key,client-certificate, certificate-authority) including in the kubectl command is tedious process. Instead of this we can move TLS Certificate file set into a config file that is called kubeconfig file. The usage will be as follows kubectl get pods --kubeconfig config Usually this...