Kubernetes Troubleshooting

 We as DevOps and DevSecOps Engineers working on many microservice based application architectures where we need to manage Kubernetes Cluster  Troubleshot at various levels.

You cannot rely on single point of look for failures. While working on Kubernetes Troubleshooting we can make ourselves easy to understand the problem, if we could classify the problem belong to the following categories.

1. Application Failure
2. Master node/ControlPlane Failures
3. Worker node Failures

Application Failure - trobleshooting

Here I'm listing out these with my understanding and experiance in practice tests provided by Munshad Mohammad on KodeKloud.

1. You should know the architecture how it is deployed what all its dependents, where they have deployed with what endpoints, what names used.
2. Check the service 'name' defined and referring service should match and also check the services 'Endpoints' are correctly defined and in referenceing used correctly.

   k -n dev-ns get all
   

3. Better to check that the selector are as properly aligned or not, as per the architecture design definitions. if it is not then you need to change them.

   k -n test-ns edit svc mysql-service
   

4. Identify is there any mismatch for the environment values defined in the deployment cross check with the Kubernetes objects those are integrating.

   k -n test-ns descrube deploy webapp-mysql
   

   If that doesn't match example mysql-user value was mismatched then you can change it, it will automatically redeploy the pods.

   k -n test-ns edit deploy webapp-mysql

5. Check also service NodePort port correctly mentioned or not. If it mismatches then need to replace with correct one as per the design.

   k -n test-ns describe service/web-service
   k -n test-ns edit service/web-service # edit nodePort value correct
   

Controlplane/Kubernetes Master node Failure - Troubleshooting

1. Initial analysis start from nodes, pods
   To troubleshoot the controlplane failure first thing is to check the status of the nodes in the cluster.

   k get nodes 
   

   they all should be healthy then, go for the next step that is status of the pods,deployments,services,replicasets (all) within the namespace on which we have trouble.

   k get po 
   k get all 
   

   Then ensure that pods that belongs to kube-system are 'Running' status.
2. Check the Controlplane services

   # Check kube-apiserver
   service kube-apiserver status 
   or 
   systemctl status kube-apiserver 
   
   # Check kube-controller-manager
   service kube-controller-manager status 
   or 
   systemctl status kube-controller-manager
   
   # Check kube-schduler
   service kube-schduler status 
   or 
   systemctl status kube-schduler
   
   # # Check kubelet service on the worker nodes 
   service kubelet status 
   or 
   systemctl status kubelet 
   
   # # Check kube-proxy service on the worker nodes 
   service kube-proxy status 
   or 
   systemctl status kube-proxy 
   
   # Check the logs of Controlplane components 
   kubectl logs kube-apiserver-master -n kube-system 
   # system level logs 
   journalctl -u kube-apiserver 
   

3. If there is issue with the Kube-scheduler then to correct it we need to change the YAML file preent in default location `vi /etc/kubernetes/manifests/kube-scheduler.yaml`
   You may need to check the file `/etc/kubernetes/manifests/kube-controller-manager.yaml` parameters given for 'command'. Sometime there could be missing or incorrectly entered for the VolumeMounts paths values, if you correct them the kube-systeem pods automatically starts!

Worker Node failure - Troubleshooting

This is mostly around kubelet serivce unable to come up. The bronken Kubernetes cluster can be identified by listing your nodes, where it tells us 'NotReady' state. There could be several reason each one is a case that need to be understood, where Kubelet cannot communicate with the Master node. Identifying the reason is the main thing here.

1. Kubelet service not started: There could be many reasons when worker node fails. One such is if there is a CA certs rotated on the there should be manually you need to start the kubelet service and validated it is running on worker node.

   # To investigate whats going on worker node 
   ssh node01 "service kubelet status"
   ssh node01 "journalctl -u kubelet"
   # To start the kubelet 
   ssh node01 "service kubelet start"
   

   Once started you need to double check that kubelet status again if it shows 'active' then fine.
2. Kubelet Config mismatch : The kubelet service even you start it is failed to come up. There could be some CONFIG related issue. In one of the example practice test given that ca.crt file path wrongly mentioned. You may need to correct the ca.crt file path in the worker node in such case you must know where the kubeconfig resides! so the path is '/var/lib/kubelet/config.yaml' After editing the ca.crt file you need to start the kubelet

   service kubelet start 

   and check the kubelet logs using journalctl.

   journalctl -u kubelet -f 

   And ensure that in the controlplane node list show that node01 status as 'Ready'.
3. Cluster Config mismatching: There could be conffig.yaml file currupted where master ip or port configured wrongly or cluster name, user, context may be wrongly entered that could be reason where kubelet unable to communicate with the master node. Compare the configuration available on the master node and worker node if you found mismatches correct them and restart the kubelet.
4. Finally, check the kubelet status on the worker node and on the master node check the list of nodes.

Enjoy the Kubernetes Administration !!! Have more fun!

Kubernetes Tools Tricks & Tips

Hey Guys, Welcome to "DevOps Hunter" blog! In this post I would like to share my learnings at different times collected that is about Kubernetes commands and their applied tricks and tips.

• Initially I've collected few kubectl related alias command tricks
• Play with the etcd database and then backup and recovery short-cuts
• Finally worked on the Kubernetes command tools kubectx, kubens for easy switching in CLI.

Come on! let's explore about the API resources which we might be frequently use when we prepare the YAML files for each Kubernetes Objects.

kubectl api-resources

We can get sometime the API version mismatch due to change in API version. This can be examine what is new in the current version

How do you identify the certificate file used to authenticate 'apiserver'?

cat /etc/kubernetes/manifests/kube-apiserver.yaml|grep tls-cert
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt

The tls-cert-file will be Kubernetes apiserver cerificate file path .

How do you identify the certificate file used to authenticate 'kube-apiserver' as a client to ETCD server?

You can look into the kube-apiserver manifest file.

cat /etc/kubernetes/manifests/kube-apiserver.yaml 

Do you have any alias tricks for Kubernetes CLI commands?

Yes, I do have many but here I would like to common usable Bash shell alias.

# kubectl can be used with k most common alias 
alias k='kubectl'

# This is to list all available objects, alias will be used with many Kubernetes Objects
alias kg='kubectl get'

# This will be used to describe any kubernetes object 
alias kdp='kubectl describe'

Looking into the logs

Kubernetes collects the logging from all the containers that run in a Pod.

# To look into the logs of any pod 
alias kl='kubectl logs'

# To get into the pod containers 
alias kei='kubectl exec -it'

Realtime scenario: maintenance window on worker node

There can be regular routine maintenance windows on worker nodes may be to have OS patching on the node or any other urgent maintenance then how to handle is important activity as Kubernetes Administrator.

When maintenance starts on node01:

 alias k="kubectl"
 k drain node01 --ignore-daemonsets
 # check pods scheduling on which nodes 
 k get po -o wide
 # check nodes status - observe that node01  STATUS = Ready,SchedulingDisable
 k get nodes 

when maintenance on node01 completes, How to releae that node back to ready state?

First make the node as schedulable using uncordon, then check nodes

 k uncordon node01
 the uncordon sub-command will mark node as schedulable, bring back to ready state for node.
 
 # Check pods, nodes 
 k get nodes,pods -o wide

Existing nodes will not be re-scheduled back to the node01. But if any new pods are created they will be scheduled.

Locking your node for not to perform schedule any new pods

Without effecting existing pods on the node make the node Unschedulable can be done with the cordon

 k cordon node01
 k get nodes -o wide
 

cordon sub-command will mark node as unschedulable.

Kubernetes Ugrade plan

Similar to any OS package managers allow us to upgrade here we can do it for Kubernetes. But we need to be little cautious, If there is any upgrade plan then we need to check that from the kubenetes CLI

 kubeadm upgrade plan
 

How do you find the ETCD cluster address from the controlplane?

From the describe output you can identify the etcd address which is present in the --advertis-client-urls value.

k describe po etcd-controlplane -n kube-system|grep -i advertise-client
Annotations:          kubeadm.kubernetes.io/etcd.advertise-client-urls: https://10.36.169.6:2379
      --advertise-client-urls=https://10.36.169.6:2379

How to get the version of etcd running on the Kubernetes Cluster?

To get the version of the etcd by describe the etcd pod which is present in kube-sytem namespace.

k get po -n kube-system |grep etcd
etcd-controlplane                      1/1     Running   0          22m

k describe po etcd-controlplane -n kube-system|grep -i image:
    Image:         k8s.gcr.io/etcd:3.5.3-0

Where is the ETCD server certificate file located?

To find the server certificate the file location present in '--cert-file' line. To skip -- in the grep use back slash

k describe po etcd-controlplane -n kube-system|grep '\--cert-file'
      --cert-file=/etc/kubernetes/pki/etcd/server.crt

Alternative: another way is to get certifiate files and key files of etcd. You know that etcd is a static pod and which will have the definitions and configuration details as manifest file at /etc/kubernetes/manifests/etcd.yaml. To run the the etcd backup we must pass certfiles, key files. Let's find those from the manifest file.

 cat /etc/kubernetes/manifests/etcd.yaml |grep "\-file"

Where is the ETCD CA Certificate file located?

Generally CA certificates file will be saved as ca.crt.

k describe po etcd-controlplane -n kube-system|grep -i ca.crt
      --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
      --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

Backup and Recovery of ETCD database

ETCD database BACKUP to a snapshot using following command

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/snapshot-pre-boot.db

# validate snapshot created in the /opt directory.
ls -l /opt

How to restore the etcd cluster database?

Same command only in place of save use restore option.

ETCDCTL_API=3 etcdctl  --data-dir /var/lib/etcd-from-backup \
snapshot restore /opt/snapshot-pre-boot.db

To know nmber of clusters configured on the node you can use the following :

k config view
# Be specific to cluster listing you can use get-clusters 
k config get-clusters

Kubernetes Tools

Your life will be easy if you know these two tools as your tricks! kubectx, kubens two customized commandline tools.

Using kubectx

kubectx examples

sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx
kubectx -h
kubectx -c
kubectx 

Download and Setup the kubectx

kubens

Setup the kubens and using it for switching between namespaces.

sudo ln -s /opt/kubectx/kubens /usr/local/bin/kubens
kubens
kubens -h
kubens kube-system
k get po
kubens -
k get po

Kubernetes namespace switching tool kubens setup and executions

Network Tricks

To find the weave-net running on which node

k get po -n kube-system -l name=weave-net -o wide

What is the DNS implementation in your Kubernetes Cluster?

To know dns details the label used 'k8s-app-kube' we can run on pods, deployments we can get the complete implementation of DNS on the Kube:

k -n kube-system get po,deploy -l k8s-app=kube-dns

The execution sample output

Finding Node info using jsonpath

To work on the jsonpath you must know what is the output in json format first. then we can narrow-down to the required field data to be extracted.

k get nodes -o jsonp 
k get nodes -o jsonp | jq
k get nodes -o jsonp | jq -c 'paths' |grep InternalIP

To get the InternalIP address of each node can be retrived first we need to give a try for first node than we can change to all nodes using '*'.

k get no -o jsonpath='{.items[0].status.addresses}'|jq
k get no -o jsonpath='{.items[*].status.addresses[0]}'|jq
k get no -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")]}'
k get no -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}'

Kubectl autocomplete

Set up autocomplete enable in bash shell if that is your current shell, bash-completion package should be installed first.

source <(kubectl completion bash)

Let's add this above line for autocomplete permanently in the .bashrc

> ~/.bashrc

Reference

1. Official cheat sheet

Hope you have enjoyed this post.

Ansible powerful parameters - delegate_to, connection

 

Delegation to a host

Here is an example where we can delegate the task to a particular host. This play book is using inventory_hostname from the gather facts.

- name: Delegation to localhost
  hosts: all
  tasks:
  - name: create a file on target server
    file:
      path: /tmp/i_m_on_target_server.txt
      state: touch
  - name: create a file with host named file by delegation to localhost
    file:
      state: touch
      path: "/tmp/{{ inventory_hostname }}.txt"
    delegate_to: localhost

connection paramer

We can use this "connection" parameter add to your task level or play level.

# Filename: connection_local.yml 
# To do some task on ansible server
# local means  without doing ssh command (no need of  password and no need of ssh keys)
# with the local connection parameter for the play
---
- name: This is to determine how the connection parameter works  with local
  hosts: app
  connection: local
  gather_facts: false
  tasks:
  - name: connection local test
    file:
      state: touch
      path: /tmp/srirama.txt

local action

Now in the following example we will see how local_action will be working. The local_action is an Ansible module which will by pass the ssh connection to do command locally.

- name: Using local acion in this  playbook
  hosts: localhost
  vars: 
    test_file: /tmp/omnamahshivaya.txt
  tasks:
  - name: local action
    local_action: "command touch {{ test_file }}"
    

The better option to do it in local is using connection: local parameter at play level.

Ansible Vault - To save Secrets

Hello DevOps Automations Engineers!!

 Ansible provides us special command 'ansible-vault' that is used to encrypt, decrypt, view an Ansible  playbook, this is also have amazing feature specific to role, vars YAML files, we can apply this to string of text in regular variables. 

Why do we need to encrypt our Play books?

Our Ansible automation projects, we need to work on multiple tasks and which may have some sensitive data such as database user credentials, any cloud IAM role details or it can be some other applications login credentials that's used to validate URL availability. Or it can be used to store the SSL certificates. At any point of time if the system is using plain text and it  has trouble to your confidential and sensitive data otherwise it could causes huge damage to your organization. Where we need a way to store the sensitive data can be protected by data encryption tool, and this can be done using the Ansible-vault command. 

Let's see the ansible-vault command help, in this we will experiment with what all the options we have to play with encryption and decryption of plain text in a file, string, entire YAML file also can be locked with this.

ansible-vault --help
usage: ansible-vault [-h] [--version] [-v] {create,decrypt,edit,view,encrypt,encrypt_string,rekey} ...

encryption/decryption utility for Ansible data files

positional arguments:
  {create,decrypt,edit,view,encrypt,encrypt_string,rekey}
    create              Create new vault encrypted file
    decrypt             Decrypt vault encrypted file
    edit                Edit vault encrypted file
    view                View vault encrypted file
    encrypt             Encrypt YAML file
    encrypt_string      Encrypt a string
    rekey               Re-key a vault encrypted file

ansible-vault with create option

The create option will help us to create new encrypted file, when you execute this it will prompt for a password, confirm password for the Vaultify YAML file, once you entered the credentials it will be opens the default editor  and you may enter the text in that and save the file. When you view the content of the file it will be encrypted 

ansible-vault create vault.yml

Here in place of vault.yml you can use your confidential file.

ansible-vault create 

Encrypt

The encrypt option will enable us to do encrypt any file content, that can be a plain text file or our ansible playbooks.

echo "unencrypted stuff"> encrypt_me.txt
cat encrypt_me.txt
ansible-vault encrypt encrypt_me.txt
cat encrypt_me.txt

Ansible-vault for encryption of a file

Decrypt

ansible-vault decrypt vault.yml
  

View

You can view the content of a encrypted file it can be any playbook or vars which you only owner and  aware of the key to open it.

ansible-vault view valut.yml

Edit

When we have encrypted file that may be created using 'create' or 'encrypt' option of ansible-vault command we can use the following way to edit the file. where the Ansible will prompt for password.

Recreate the password with rekey

Using --ask-vault-pass

ansible-playbook vault_demo.yml --ask-vault-pass

Using --vault-password-file 

ansible-playbook vault_demo.yml --vault-password-file my_pass

Using --vault-id

ansible-playbook vault_demo.yml --valult-id my_pass

Courtesy by Krishna Tatepally[DevOPs Operations Expert]

Ansible handlers

Hello DevOps Experts!! let's zoom  into the usage of the Ansible Handlers and notifies 

What are Ansible Handlers?

The handlers section or the tasks defined under the handlers folder are executed at the end of the play once all tasks are finished. In the handlers tasks we are typically do either start, reload, restart and stop services.

Sometimes we may need to execute the task only when a particular change is made that can be notified.  Simple example of Apache web server when we modify httpd.conf file then we want to restart the httpd service. 

When we were working on Tomcat, when tomcat service is enabled. then there is a need for the reload firewalld service this is where we need to move this reload task under handlers and the enable tomcat service should have notify the task name 'reload firewalld service'. These are the perfect examples for handlers usage in Ansible play.

So here the point is that handler tasks will be performed only when they are notified.

Ensure that handler task name should have a globally unique name.

 Let's see the example of notify and handlers

---
- name: install php
  yum:  name=((item))  state=latest
  with_items:
    - php 
	- php-gd
	- php-pear
	- php-mysql
  notify: restart httpd
  handler
	handler/main.yml

 - name: rstart apache
   service: name=httpd state=restarted

Now you can see how to include the role php-webserver inot the main playbook

# Filename: test-handler.yml
---
 - hosts: web
   user: root
   become: yes
   roles:
     - php-webserver  

Execution of the test-handler will be like this:

ansible-playbook test-handler.yml

Ansible real-time project - Installing and configure Tomcat 10

 Hey DevOps or DevSecOps or SRE Guys!!

What's up? in the automation journey one more wonderful learning here! 

In this post we will be implementing all our Ansible modules one after other to build a complete solution for Java based Application server installation and running using Ansible playbook.

At present Tomcat latest version is 10.0.27 so I've used same 

Pre-requisites: 

• To install Apache Tomcat there is separate JDK/JRE compatibility we need to validate before we proceed
• Create a dedicated user account as 'tomcat' with shell as bash to manage Tomcat application server
• Create a separate directory for tomcat server to be installed

Execution of multiple tasks in the Playbook will be as follows:

• Download the Tomcat software from Apache Tomcat 10.0.27.tar.gz
• Uncompressing the tomcat tar.gz file
• Change the file permissions and ownership
• Cleanup after unarchive the tar.gz file
• Start the Tomcat server
• Have a task to stop the Tomcat server

---
- name: Installation and setup Apache Tomcat 10
  hosts: "{{ targets | default('localhost') }}"
  become: yes
  gather_facts: no
  tasks:
  
    - name: Install openjdk
      yum:
        name: java-1.8.0-openjdk
        state: present
      tags: install_openjdk

    - name: Create user as tomcat
      user:
        name: tomcat
        state: present
        home: /home/tomcat
        shell: /bin/bash
      tags: pre-install

    - name: Create a directory - /opt/tomcat
      file:
        path: /opt/tomcat
        state: directory
        mode: 0755
        owner: tomcat
        group: tomcat
      tags: pre-install

    - name: Download tomcat - apache-tomcat-10.0.27.tar.gz
      get_url:
        url:  https://dlcdn.apache.org/tomcat/tomcat-10/v10.0.27/bin/apache-tomcat-10.0.27.tar.gz
        dest: /opt/tomcat
      tags: pre-install 

    - name: Extract tomcat inside directory and set ownership, permissions
      unarchive:
        src: /opt/tomcat/apache-tomcat-10.0.27.tar.gz
        dest: /opt/tomcat
        extra_opts: [--strip-components=1]
        remote_src: yes
        owner: tomcat
        group: tomcat
        mode: "u+rwx,g+rx,o=rx"
      tags: tomcat-install

    - name: Remove gz file apache-tomcat-10.0.27.tar.gz
      file:
        path: /opt/tomcat/apache-tomcat-10.0.27.tar.gz
        state: absent
      tags: post-install

    - name: Start the Tomcat server
      become_user: tomcat
      shell:
        cmd: nohup ./startup.sh
        chdir: /opt/tomcat/bin
      tags: start-tomcat

    - name: Stop the Tomcat server
      become_user: tomcat
      shell:
        cmd: nohup ./shutdown.sh
        chdir: /opt/tomcat/bin
      tags: stop-tomcat, never
      

You can run the playbook in multiple ways

1. Use the target as webserver group to install Tomcat only on them.   

ansible-playbook tomcat-solutoin.yaml -e targets=webservers

2. Use the tags to stop the Tomcat servers.   

ansible-playbook tomcat-solutoin.yaml -t stop-tomcat -e targets=webservers

3. Use the tags to start the Tomcat server.   

ansible-playbook tomcat-solutoin.yaml -e targets=webservers

Finding & Terminating a process on remote VM

The play can be having two tasks where you need to find the running process by process name or command used to run it. If there is any error ignore them. Try to catch the result into a variable to pass it to next task.
Example - find process which is started with java
Second task is to terminating a process which is identified in the first task. Here we can use 'shell' module where we can use 'kill' running process.

---
 - name: Find Java process and Terminate
   hosts: "{{ targets | default ('localhost') }}"
   gather_facts: false

   tasks:
     - name: Get the running Java process
       shell: "ps -ef|grep tomcat |grep -v grep | awk {'print $2'}"
       register: JavaProcessID

     - name: Print Java Process
       debug:
         msg: "{{ JavaProcessID.stdout }}"

     - name: Terminating Java Process
       become: yes
       become_user: tomcat
       shell: "kill -9 {{ JavaProcessID.stdout }}"

Execute the playbook as follows:

ansible-playbook find_killtomcat.yaml -e targets=webservers

The output Image:

Using shell module to find and kill tomcat process

Undoing changes - git reset

Hello Guys!!

HEAD pointer movement

HEAD points to specific commit in the local repo-branch as new commits are made, the pointer changes

HEAD always points to the "tip" of the currently checked-out branch in the repo (not the working directory or staging index)

last state of repo (what was checkout initially HEAD points to parent of next commit(where writing next commit takes place)

HEAD Movement in Git branches

Git Reset movements

This is most common need of every DevOps team development phase need. There are three options we have but of course two of them are mostly used.

Git reset movements at three tree levels

1. soft
2. mixed
3. hard

Using --soft reset

The soft reset command is to combine many commits into a single one.

git reset --soft HEAD       (going back to HEAD)
git reset --soft HEAD^    (going back to the commit before HEAD)
git reset --soft HEAD~1     (equivalent to "^")
git reset --soft HEAD~2     (going back to 2 commits before HEAD)

Using hard reset

move the HEAD pointer update the Staging Area (with the content that the HEAD is pointing to) update the Working Directory to match the Staging Area

git reset --hard HEAD       (going back to HEAD)
git reset --hard HEAD^      (going back to the commit before HEAD)
git reset --hard HEAD~1     (equivalent to "^")
git reset --hard HEAD~2     (going back two commits before

Backing out changes to commit ids. So refer to the git show, git log commands in between to identify that change which you wish to reset to.

 

Observe the git reset with soft and hard example

GitHub Personal Access Token (PAT) for Linux Users

Hey Greetings of the day!!

GitHub providing Personal Access Token instead of using username, password for git repository that is on the GitHub. where git subcommands such as git pull, git push, fetch and any remote operations will be depends on this PAT. 

 There are two choices for PAT 

1. fine grain Personal Access Token(PAT) 

2. Personal Access Token (Classic) 

I understood that it is easy to change permissions, authentication token that use only specific target repositories. 

If you want to know 'How PAT works? Where to start PAT' then this post is absolutely for you! Welcome to this 2 mins read!

• Fine grained PAT newly introduced in Oct 2022 on the GitHub still at the time this post it is mentioned as [Beta] versioned.
• Personal Access Token PATS are going to work with commonly defined API on GitHub. Any integration made simplified with this method.

How to create PAT on GitHub? 

1. Login to your GitHub account. Click on the profile picture upper-right corner, from there select the 'Settings' menu item. 
2. The left bottom, PAT will be defined under the 'Developer Settings' there you will get new page with 'Personal Access Tokens' into the work area select your choice. There are many actions related to each row tells about a granular choices that GitHub could provide. 
3. Go to right side 'Generate Token' will be a button to hit. Then it will prompt you for login credentials to be entered. Then New personal access token parameters will need to be filled up. 1. Note - alias a authentication methods let's enter the note as "vt_demo_token" 
4. Select the scope - 'repo' is enough, where it will be containing repo:status, repo_deployment, public_repo, repo:invite, security_events.
5. Click on the "Generate Token" button new page will have the token. Make sure to copy your new personal access token now. Remember that you won't be able to see it again eventhough you have created it. As soon as click away of this page you will not get the value any more.

Preserving the TOKEN on your Linux system Create an environment variable store it inside .bash_profile.

MY_GITHUB_TOKEN=GENERATED-TOKEN-VALUE-COPIED FROM GitHub PAT page.

How do I access remote GitHub repository?
Use the following command to set the remote repository:

git remote origin https://github-remote-repo-url 

Use the following command to check the remote repository

git remote -v 

Here -v option indicates that verbose of the remote url and fetch and pull commands could connect with the URLS shown as above command output. push the changes to remote GitHub repository.

git push -u origin https://git-url 

It will prompt for the Password then enter the TOKEN value which is already copied that you can paste here. To store the password you don't want to see the prompting every push, pull activities. To avoice that you can set the remote url as follows:

git remote set-url https://api:$MY_GITHUB_TOKEN@github.com/REMOTE-USER/REMOTE-REPO.git 

Example:

git remote set-url https://api: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@github.com/SanjayDevOps2022/football.git

After setting this all remote operations push, pull are no user access prompted.
If you find this post is useful please share this to your technical communities :)

Ansible Tags - Controls Tasks

 Ansible playbook can be a construct of multiple plays or each play may contains multiple tasks. This is where we may have situation where you need to add new task to the existing play or playbook, and we need to test many times that newly added task. 

While testing multiple times we many don't want to execute certain tasks such as a task 'Send email notification' when you preparing a 'Reboot of server' or 'Restart of Service' or 'Deployment of a service'. During the testing time you may want to exclude these notification tasks. 

There are situations where we might want to run a particular task as per the input at the run time of a playbook. This may be from AWX/Tower UI select them.

Ansible tags - to control the tasks of a Playbook

I will be explaining in this post, How to run or not to run a particular task in given  playbook. 

Important concepts about Ansible tags

• Ansible tags are keys to identify and control the tasks for execution or exclude from the playbook that could have multiple tasks
• Ansible tags are references or aliases to a task inside a play selection can be done with -t or --tags options in the command line 
• Selecting a tag for exclude can be defined with --skip-tags option
• A task may have multiple tags such as sending email task could have tags as email_notify, notifications
• There can be same tag that can be associated with multiple tasks such as notifications  can be associated with sending email and sending slack notices as well

Prerequisites

I've Ansible Controller machine and wanted to install the 'Apache2' webserver on the managed nodes. I've a sample playbook for it. 

How to be specific during the execution of Ansible Playbook?

Now let me create a playbook for install and control a Apache webserver. All my managed nodes are with CentOS so I'm using yum package module in this.

The logic for the below playbook is build with two modules: yum, service

staate attribute of yum package module

service module parameter state and values

You can see the YAML using tags which is our main objective of this post:

---
 - name: Install and control Apache Webservers 
   hosts: "{{ targets | default('localhost') }}" 
   become: yes
   tasks: 
     - name: Install httpd server
       yum:
         name: httpd
         state: present 
       tags: install_httpd, boot_web

     - name: start httpd server
       service:
         name: httpd
         state: started 
       tags: start_httpd, boot_web

     - name: stop httpd server
       service: 
         name: httpd
         state: stopped
       tags: stop_httpd, destroy_web

     - name: uninstall httpd server
       yum:
         name: httpd 
         state: absent
       tags: uninstall_httpd, destroy_web

The Package state – parameter values can be found from the ansible documentation The execution can be specific to start the webserver, that means there should be Apache webserver must be installed and then start the service. To install webserver on two nodes.

ansible-playbook httpd-tags.yml -t install_httpd -e targets=ansible-node-1,ansible-nodes-2

To start the webserver on two remote hosts

ansible-playbook httpd-tags.yml -t start_httpd -e targets=ansible-node-1,ansible-nodes-2

Ansible tags passed in the command line with -t option

To install and start webserver if no targets mentioned means it will be localhost where Ansible controller running there this webserver will be installed and started with the following command

ansible-playbook httpd-tags.yml --tags boot_web 

To stop the webserver on two remote hosts

ansible-playbook httpd-tags.yml -t stop_httpd -e targets=ansible-node-1,ansible-nodes-2

To uninstall the webserver on two remote hosts

ansible-playbook httpd-tags.yml -t uninstall_httpd -e targets=ansible-node-1,ansible-nodes-2

We can also select multiple tags to run the playbook. To stop the webserver on two remote hosts

ansible-playbook httpd-tags.yml -t stop_httpd,uninstall_httpd -e targets=ansible-node-1,ansible-nodes-2

Ansible tags using multiple tags

Same thing can be executed with excluding tasks which we can tell by using --skip-tags options for the install_httpd, start_httd means remaining tasks need to be executed.

ansible-playbook httpd-tags.yml --skip-tag install_httpd,start_httpd -e targets=ansible-node-1,ansible-nodes-2

We can also use common tag instead of using two tags used for stop and uninstall the webserver as destroy_web single tag.

ansible-playbook httpd-tags.yml --tags destroy_web 

Hey is there any other alternative way without tags can I say from here onwards execute the playbook?

Yes, We can also use task-name as an input for a special option –start-at-task for the ansible-playbook command. See this example that tells 'stop httpd server' onwards means two tasks will be executed that is stop_httpd, uninstall_httpd tasks.

ansible-playbooks httpd_tags.yml --start-at-task "stop httpd server"

There are two newly introduced from Ansible 2.5 version onwards they are important and very special tags in Ansible playbooks are always and never.

Always: If you assign the always to tag to a task or play then Ansible Controller will always run that task or play unless you specify that need to be skip using --skip--tags

Never: If you assign the never tag to a task or play then Ansible controller will skip that particular task or play unless you specify with '--tags never' it looks odd! But it works as it meant to do in that way.

 

---
- name: Understanding always, never tags
  hosts: localhost
  become: true
  tasks:
    - name: Tasks execute always 
      command: echo "this is from always tag"
      tags:
        - always

    - name: Install nginx server
      apt:
        name: "nginx"
        state: "present"
      tags:
        - install
        - never

    - name: Deploy webpage to website
      copy:
        src: index.html
        dest: /var/www/html/
      tags:
        - deploy
  

When you run this playbook it will always task will be executed always! The install task will be never executed. What is the command to list all tags present in a Playbook?

ansible-playbook mytags.yml --list-tags
  

Terminology: AWX is free open-source project that enables us to manage Ansible from a web interface.

All the examples executed thanks to Krishna

Courtesy by   Krishna Tatipally

Document References:

1. yum module
2. service module
3. tags management in Ansible

Ansible Facts - Customizations

Hey DevOps Team, In this post I would like to give you the knowledge on the special feature about Ansible Facts and we can also customize these facts as global variables to use in multiple playbooks.

What is ansible facts? 

Ansible facts are just simple variables that are automatically discovered by ansible on a managed nodes. such as system information disk info, os info, package info IP Network and many more ...

Why we manage facts?

Default these facts will be automatically collected exclusively we need to disable some times. - multiple play in a playbook

How can we use facts?

we want to run only when enough memory is available on the target machine then install the package. That is the smartest way to do the automation!

---
# File : hello.yaml
- name: Facts example 
  hosts: "{{targets|default('localhost')}}"
  tasks:
    - name: prints details
      debug:
        msg: "Hello this machine have {{ ansible_memory_mb['real'] }}"

when you run the above playbook it will automatically do this "Gathering Facts". This is done by a special built-in module called 'setup' module. What is this setup module does? Let's examine by executing the following ad-hoc command:

ansible node1 -m setup # To know about node1
ansible localhost -m setup # To know about Ansible controller machine

the above will be printing lot of information in the JSON format thrown on to your monitor, look this is all under "ansible_facts"

To collect all these and displaying on the stdout will be time consuming process. We can skip this by saying I don't want to see! To control this, disable collecting facts process you can inform in the playbook by saying 'gather_facts: false' this will not do automatic collection so that playbook will be executed faster.
Facts are like ansible_distribution ansible_hostname ansible_default_ipv4['address']

---
# File : facts_ex.yaml
- name: Facts example   
  hosts: nodes
  gather_facts: false
  tasks:
    - name: collect facts
	  setup:
    - name: prints details
	  debug:
	    msg: 
		- "{{ ansible_distribution }}" 
		- "{{ ansible_hostname }}"
		- "{{ ansible_default_ipv4['address'] }}"

Custom Facts

Let's go to the node1 and set the facts manually. ofcourse we can also set this with ansible file operation as well.

sudo mkdir -p /etc/ansible/facts.d 
cd /etc/ansible/facts.d

Let's create our facts file under the above created directory structure where ansible controller can look for and fetch the data1 `sudo vi local.fact`

[db_server]
db_installer= Oracle 21c
db_port= 1521

[bu]
project=Banking

Now let's run the fact collection from the ansible controller machine following command:

ansible node1 -m setup -a 'filter=ansible_local'

Use Case 2: Accessing the local facts from the playbook as follows:

---
# File : localfacts_ex.yaml
- name: Custom Facts example   
  hosts: node1
  tasks:
    - name: prints local facts details
	  debug:
	    msg: 
		- "{{ ansible_local }}" 

execute the above example observe the output.

ansible-playbook localfacts_ex.yaml

Now let's try to fetch those internal elements from custom facts, which are defined in the remote managed node 'node1' as dictionary of dictionaries. 

---
# File : localfacts_ex1.yaml
- name: Custom Facts in details example   
  hosts: node1
  tasks:
    - name: prints local facts details
	  debug:
	    msg: 
		- "{{ ansible_facts['ansible_local']['local']['db_server'] }}" 
        - "{{ ansible_facts['ansible_local']['local']['db_server']['db_installer'] }}" 
        
        

Run the above customized facts and retrieving is little complex as it requires little knowledge on the JSON form of data accessing method, but it's okay this is simple example used in the playbook.

ansible-playbook localfacts_ex1.yaml

Now we can control the custom facts collection based on the availability as we know these facts are defined in node1 manage server only, when we pass hosts value as nodes it will be getting failed for node1 or localhost to suppress such cases we have option to use 'when' conditional check as shown below:

---
# File : localfacts_ex2.yaml
- name: Custom Facts example   
  hosts: nodes
  tasks:
    - name: prints local facts details
	  debug:
	    msg: 
		- "{{ ansible_facts['ansible_local']['local']['db_server']['db_installer'] }}" 
	  when: ansible_local.local.db_server.db_installer is defined

The execution will be takes place as follows:

ansible-playbook localfacts_ex2.yaml

Keep smiling 🤣 with rocking automation ideas you are learning with me!!

References:

Official Document

Managing Ansible Facts

Everything about set_facts

Kubernetes security - Service accounts

In this post we are going to learn more  about what is service accounts in Kubernetes and how that is useful.

Prerequisites

Kubernetes cluster Up and running

Let's take the scenario where we get need to connect with the pods, nodes, deployments and other resources in the Kubernetes cluster. you might be working with the automated build with the CICD pipelines to interconnect with each other resources. Pod  is going to work with the planned application deployments.

If  you're working in DevSecOps you need to focus on the regular monthly maintenance OS  patching scheduled in this case Kubernetes node maintenance should be done from a pod. 

In the above two scenarios there is a need of service account inside the pod. When Kubernetes cluster is created at the same time service account also created and its name is default. We can also create our own service accounts using the following command

Every service account is associated with the secret where service account name is  as first part for the secret name followed by token word will be makes it. Example default account has secret as default-token-****.

Here I am going to work in a pod which needs authentication using the service account which created by me. To make this happen need to add a line in the pod definition, under spec section add line as serviceAccount followed by its value. Proceed to create a testing pod.

kubectl create -f mypod-sa.yaml.

To know what Pods are running in this cluster run 

kubectl get pods

Let's go head and see description of pod. Inside the Pod there is a volumeMount configured and it is accessible in specific path of container.

kubectl exec -it mypod-sa -- /bin/bash

Inside the pod 

ls -lrt /var/run/secret/kubernetes.io/serviceaccount 

Here we can see the namespace, token, and certificates file.

TOKEN=`cat var/run/secret/kubernetes.io/serviceaccount/token`
  curl https://Kubernetescluster -k --header "Authorization: Bearer $TOKEN"
  

Just this may be failed. This particular service account which is present in the pod. This cannot have the permissions so the other message saying reason as forbidden.

Now what I am going to do investigate why that is having the permission issue. We need to create a role and rolebinding that associate with service account.

Now all I want to do is same commnd previously executed command from the pod as run early.

Every Service account will be associated with a secret which is the older Kubernetes model it was automatic but now that will be hidden in the latest Kubernetes 1.22 onwards. I'm working on the Kubernetes 1.25 version let's see how it will be now. here I am working on the KillerCoda builtin Kubernetes cluster. I would like to create a ServiceAccount and for it Secret object then that will be used to run a deployment, which is the common requirment in most of the real time projects.
Now I will create a custom serviceaccount, which will be more privileaged to work with deployments.

  kubectl create serviceaccount vt-deploy-sa --dry-run=client -o yaml
kubectl create serviceaccount vt-deploy-sa --dry-run=client -o yaml > vt-deploy-sa.yaml

#Before running creating sa 
kubectl get serviceaccounts
kubectl create -f vt-deploy-sa.yaml 

#Confirm it 
 kubectl get serviceaccounts
 kubectl describe sa vt-deploy-sa
 

Here important rule you need to understand is, One Pod can have one serviceaccount. But in reverse to this, One serviceaccount can be attached with multiple Pods. Let's examine the default serviceaccount what authorized to do?

 kubectl auth can-i create  pods --as=system:serviceaccount:default:default

to understand depth of the above command check about default serviceaccount used --as option with system:serviceaccount:NAMESPACE:SERVICEACCOUNTNAME

When we create our custom serviceaccount we can define our own policy that could be telling what could be done such as list, create, delete etc actions. And that is needs a mapping which is done by Role and RoleBindings. Role is where I am allowed to define the policies for user,group,serviceaccount which it is about to bind. Then I am going to create the RoleBinding which will actually bind the serviceaccount with the Role which have policies.

 kubectl create role deploy-admin --verb=create,list,get,delete --resource=deployments 

Here deploy-admin role is defined with create,list,get, delete actions on the deployment objects.

kubectl create rolebinding deploy-rb --role=deploy-admin --serviceaccount=default:vt-deploy-sa
 

here serviceaccont is defined with the namespace followed by custom serviceaccont.

Now let's try the deployment with serviceaccount.

  kubectl create deploy webapp --image=tomcat \
 --replicas=4 --dry-run=client -o yaml > webapp-deploy.yaml
 
 
 vi webapp-deploy.yaml
 # removed null word containing lines
 apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 4
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      serviceAccountName: vt-deploy-sa
      containers:
      - image: tomcat
        name: tomcat

# Create deployment 
kubectl create -f webapp-deploy.yaml

# list the deploy with default and then custom serviceaccount
kubectl get deploy --as=system:serviceaccount:default:default
kubectl get deploy --as=system:serviceaccount:default:vt-deploy-sa

Now you ccan observe the difference between default serviceaccount vs custom serviceaccount capabilities.

Kubernetes Security - ClusterRoles and ClusterRoleBindings

Hello in this post we will explore about ClusterRoles and ClusterRoleBindings on Kubernetes Cluster. The ClusterRoleBindings are mapping a subjects with ClusterRole. Here Subjects are nothing but rules that can be applicable with an action on the Cluster resources. It deals with Users, Groups and service accounts. In this post we will try to focus with 'User' specific rules.

Kubernetes User Access Control with ClusterRoleBindings to ClusterRole

 

Prerequisite: 

1. Kubernetes Cluster up and running 

2. Basic understand on RBAC

These system related resources such as pods, nodes, storage etcs will be administrated using ClusterRole and ClusterRoleBindings by assigning to a user.

 
To list the ClusterRoles in the Kubernetes cluster

kubectl get clusterrole
# Get the Count 
kubectl get clusterrole --no-headers |wc -l

To know about the api-resources that have clusterrole and clusterrolebindings.

k api-resources |grep cluster 

To veiew the clusterrolebindings available in this Kubernetes Cluster

kubectl get clusterrolebindings 
# Get the Count 
kubectl get clusterrolebindings --no-headers |wc -l

Imperative way

You can have single verb to used to create clusterrole. Here is an example, Create a role which should have access to list the deamonsets.

# Initial check 
kubectl get ds --as krish 

kubectl create clusterrole list-ds-role --resource=daemonsets --verb=list
kubectl describe clusterrole list-ds-role

Create clusterrolebinding list-ds-rb for user 'krish' to map that clusterrole list-ds which created above.

kubectl create clusterrolebinding list-ds-rb --clusterrole=list-ds-role --user=krish 

After ClusterRoleBinding assigned to krish

kubectl get ds --as krish 

Create ClusterRole, ClusterRoleBinding imperative way

Cleanup for ClusterRoles

Cleanup activity can be in the reverse order. First delete the ClusterRoleBinding then clusterrole

kubectl delete clusterrolebinding list-ds-rb 

kubectl delete clusterrole list-ds 

Cleanup ClusterRole and ClusterRoleBindings

 
ClusterRole are Kubernetes Cluster wide and they are not part of any namespace. To know about user or groups are associated with cluster-admin role, use ClusterRoleBindings and describe it. Where we can see in the subject section that will reveals you about user/groups.

kubectl describe clusterrolebinding cluster-admin

To inspect the clusterrole 'cluster-admin' privileges describe will show the PolicyRules where what resources can be used? and what you can do? The '*' astriek is to indicate that 'all'. If you want to get all resources access then '*.*' should be given. And same way to indicate all actions such as create, delete, list, watch, get use '*'. A new user mahi joined the Kubernetes Administrtors team. She will be focusing on the nodes in the cluster. Let's create a ClusterRole and ClusterRoleBindings so that she gets access to the nodes .

 
Initially we will check that she is able to access the nodes or not.

kubectl create clusterrole node-admin 
 --verb=get,list,watch --resource=nodes --dry-run=client -o yaml > node-admin.yaml
kubectl apply -f node-admin.yaml
kubectl describe clusterrole node-admin

Let's bind the node-admin clusterrole to mahi user using clusterrolebinding.

kubectl create clusterrolebinding mahi-rb --clusterrole=node-admin --user=mahi --dry-run=client -o yaml > mahi-rb.yaml

kubectl create -f node-admin-rb.yaml 
kubectl describe clusterrolebindings node-admin-rb

# Check michelle have the access to nodes 
kubectl --as mahi get nodes

If a user responsibilities are growing as they are into the organization for atime being. Here Maheshwari(mahi) user got more responsibilities for maintaining storge that used for Kubernetes cluster. Create the required ClusterRole and ClusterRoleBindings to allow her access Storage. Requirements:

ClusterRole: storage-admin
Resource: persistentvolumes
Resource: storageclasses
ClusterRoleBinding: mahi-storage-admin
ClusterRoleBinding Subject: mahi
ClusterRoleBinding Role: storage-admin

Now you know all the steps how to proceed on the clusterrole, clusterrolebindings

 kubectl create clusterrole storage-admin \
  --verb=* --resource=persistentvolumes --resource=storageclasses \
  --dry-run=client -o yaml > storage-admin.yaml
  
kubectl apply -f storage-admin.yaml
kubectl describe clusterrole storage-admin

kubectl create clusterrolebinding mahi-storage-admin-rb \
 --clusterrole=storage-admin --user=mahi --dry-run=client -o yaml > mahi-storage-admin-rb.yaml  
 
 kubectl create -f mahi-storage-admin-rb.yaml
 kubectl describe clusterrolebinding mahi-storage-admin-rb
 
# Validate that authentication given for mahi user to access storage
kubectl get pv --as mahi
kubectl get sc --as mahi

Here the last execution of fetching the storageclasses using 'mahi' is successful.

Reference:

1. Official Documentation

Kubernetes Security - RBAC

My Understanding about RBAC in Kubernetes

RBAC stands for Role based access control in our Kubernetes system we have users that needs to access the kubernetes cluster and it's resources. Here role is that categorize their needs. Let's say our project have developers, admins, presale users. We could define role named as "readers" that allows all users, because its common need to all user to read from the system. We could define a role called "writers" and allow certainer users like "developers" who contribute many things to develop in application end, "Admin" user can have this to control it. We could also define a role called "administrators" to admins users. Administrator role users can have full rights such as delete from the system.

Role can be used to define "what can be done?"

Role will be given to users, application software. If we need to deal with software then we need to use service account. Service accounts manages to having access control for services that runs softwares. Users can be created to have user access controls.

RoleBindings - who can do it?

In Kubernetes we have RoleBindings as an object. It allows us to users or groups to use roles by mapping that can be defined with role-bindings. RoleBinding is simple concept, role, rolebindings lives at namespace level. For example an ecommerce applicaiton, developers lives in shopping-cart namespace and presale namespace where all the presale systems live and presle team members will be using it. Administrator roles is design to to provide the entire kubernetes cluster level access permissions. That means all namespaces will be accessable to the admin role users. If you have 100 developers working for a project of micro-service based application, you cannot create 100 users and giving the access to each one. here it comes the solution with RBAC where you Kubernetes admin need to create Role and RoleBinding at once and that can be used to 100 users if more developers added still it works without any new ocnfigurations. Roles will lives in namespace constrained, ClusterRole will lives in cluster-wide kubernetes resources. let's see how it works with different users under roles with rolebindings. To check authorization-mode for kube-apiserver-controlplane in the kube-syste namespace.

kubectl get po kube-apiserver-controlplane \
  -n kube-system -o yaml |grep authoriz

How to get the roles present in a namespace?

Let's say here we have created ecom as namespace and application will be ecom-app.

apiVersion: v1
kind: List
metadata:
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: developer
    namespace: ecom
  rules:
  - apiGroups:
    resourceNames:
    - ecom-app
    resources:
    - pods
    verbs:
    - get
    - watch
    - create
    - delete

Role can be changed as per the project requirements that means initially a role may only have access to work with pods later we can add one more resource such as 'deployments'. You could also work on authorization permissions for a user role where you need to create new set of rule for 'deployments' and apiGroups can be defined with "apps" so that we could get access to the users who have this role.

Kubernetes Security - Group API

Kubernetes API Groups What is the Kubernetes API? Kubernetes API means it works with webservice that uses HTTP and REST protocols to enable the access for the API calls. 

Let's see this how it works using 'curl' command, where we need to provide the URL then api call object path.

Examples To view the Kubernetes version we can use :

curl https://controlplane:6443/version -k

To get the list of pods in default cluster

curl https://controlplane:6443/api/v1/pods -k

in this post we will get to know more about the api specifically Kubernetes API Groups. Each group is defined with a specific purpose, such as on api for health check, other for metrics collection logs etc. These metrics, health check will be used for health of the Kubernetes cluster. And the logs will be used for collecting by third party system where all logs will be collected such as ELK stack uses logstash agent. 

 The API are categorized into two : 

1. Core group /api 

2. Named group /apis 

Here all core group will be associated with the core functionality such as namespaces, pod, replication controllers, nodes, endpoints, bindings, events, pv, pvc, configmaps, services, secrets etc. Where as named group API are more organized and going forward all the newer features are going to be made available to these named groups. apps, extensions, networking.k8s.io, storage.k8s.io, certificates.k8s.io etc comes under named groups. 

 

To list out the

curl https://localhost:6443 

To list all apis names

curl https://localhost:6443/apis -k |grep "name" 

Kubernetes Security - Certificates API

Hello all! Welcome to new learning Kubernetes Certificate API in the series of "Kubernetes Security". a. Private key generation 

Kubernetes Certificate API

We must aware of what does certificate authority CA will do and in Kubernetes how it works.

CA server it is a server which is runs certificate API.

In your DevOps or DevSecOps team a New Kubernetes Admin joins you. Hhow to handle.

Private key, Public key valid pair of CA server sign automated in Kubernetes, it performs following steps:

1. Create CertificateSigningRequest object

2. Review Request

3. Approve Request

4. Share Certs to Users

Let's try how it works

 A user Maheshwari(Mahi) want to create certificate files first private key will be generated with RSA algorithm 'mahi.key' the key size could be 2048 bits.

openssl genrsa -out mahi.key 2048

b. Certificate Signing request (CSR) object Request can be created by providing key and subject values the result can be stored into a csr file by performing the following command:

openssl req -new -key mahi.key -subj "/CN=mahi" -out mahi.csr

c. Certificate Manifestation file can be created as any other Kubernetes object using YAML as mahi-csr.yaml where kind can be used as 'CertificateSigningRequest', under the request section we can add the csr content which can be encrypted with 'base64' Linux command along with the removal of newline chars.

cat mahi.csr | base64 |tr -d "\n"

Now prepare the CSR request manifestation using above outcome.
Filename: mahi-csr.yaml

  ---
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: mahi 
spec:
  groups:
  - system:authenticated
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZEQ0NBVHdDQVFBd0R6RU5NQXNHQTFVRUF3d0ViV0ZvYVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRApnZ0VQQURDQ0FRb0NnZ0VCQUs0eDJyd3QzU2F0ZDhxSThEUzJzVDlTcngydHMrQm5Ic202d2lCdkt0Vk5IeXdKCis3Q2VHR09JdlpWN3hOQ08vRkRpT3FXY05HMzhseFR6R2pwWkdqUDNoR2RZdHU1WFNnRjlBbkFGTVZENHBnOVIKOVQzVFBjbU1Rem9ZVllMUE44c2Y5Z3pWdGIrRHV5YTRPd0dVYUNuOUdvaW0yYUV0MTYxOWpmUzRSeEJPVXpjagpFaS9DWlAvY1VUd2dLZWNpTHRKWHhvSGYxRDVuVUhVUFFPQ1JWbGtJSDNkRmZYVWZHSjg3bmpNMzJyRXBqY3gxCkNVZWgzRktLNVA3ZC8rdFB2TUFuNEQ5MzgvLzlvZjBuLzZDa0pSMnZNUStIbkgyK000azArcGVpaWNwSUxQRS8KZVZuNk41UXpUSk5sWldHdmVTMU9ZYzdBczhGa2Q2OXZKanJHcHZjQ0F3RUFBYUFBTUEwR0NTcUdTSWIzRFFFQgpDd1VBQTRJQkFRQXV0ZlpQTTlVODlqaFR5ZzhXSkdsRThlOStuWXZ2MjBJQ08wTVV3bVB6dWNvWU1NMlNiK0x5CmhiS0lod3pUVW9QTk91RGk5aEEwaElVS0tmdmxiNENTOHI1UmJneWdheDNMUHpENXlZS1ZZTGR5NmNYRW9EWmoKbUx5d1VpTHY4UnNKdjI3TUF4bEZoZnBrZXFodTZPVTJBaGlWR
signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
  

Now let's create it with CertificateSigningRequest

kubectl create -f mahi-csr.yaml

You can see the CSR status using following command

kubectl get csr 

The CSR status can be any one of these values 'Approved', 'Issued', or 'Pending'

Kubernetes Certificates

Using 'kubectl certificate' object we Kubernetes Administrators can review the CertificateSigningREquest and then decide wheather to 'approve' or 'deny' the CSR. Before this we must recheck the status of the CSR from above 'kubectl get csr' command.
To aprove the CSR request which we have prepared in the steps for mahi user. When you do a review of the CSR file content

kubectl certificate approve mahi 

If you thing the request doesn't look good you can reject by denying it.

kubectl certificate deny agent-xyz

To get rid of the inappropriate user csr request we can delete the csr.

kubectl delete csr agent-xyz 

kubectl get csr # To confirm it is deleted 

This approved certificate can be viewed in YAML format

kubectl get csr mahi -o yaml 

copy the certificate from the above output it is in base64 encrypted format so need to decode it.

echo "copy paste the certificate value from above yaml output" | base64 --decode 

you could see the first and last lines mentioned with BEGIN and END CERTIFICATE All the certificate operations carried out by Controller Manager, If you look inside this ControllerManager it is having CSR-APPROVING, CSR-SIGNING they are responsible for carrying out these specific tasks. If anyone has sign certifciates they need Root Certificate and key of CA that we can see details with:

cat /etc/kubernetes/manifests/kube-controller-manager.yaml 

Official Document reference: 

1. CertificateSigningRequest object 

Kubernetes Security - Multiple Cluster with Multiple User Config

Hello Guys! in this post we are going to explore about the Kubeconfig. This is a special configuration that will be part of Kubernetes Security. We can configure multiple clusters and different users can access these Kubernetes cluster. We can also configure the users can have access to multiple clusters.

When we started working on Kubernetes Cluster there is a config file automatically generated for us. 

To access a Kube Cluster using the certificate files generated for admin user can be given as follows:

kubectl get pods \
 --server controlplane:6443
 --clisent-key: admin.key
 --client-certificate admin.crt 
 --certificate-authority ca.crt 
 

Every time passing all these TLS details(server,client-key,client-certificate, certificate-authority) including in the kubectl command is tedious process. Instead of this we can move TLS Certificate file set into a config file that is called kubeconfig file. The usage will be as follows

kubectl get pods 
  --kubeconfig config 

Usually this config file will be stored under .kube inside the home directory. If the config file present in this $HOME/.kube/ location and file name as config is automatically detected by the kubectl command while executing.

 

What is KubeConfig contains?

The kubeconfig file have three sections clusters, users, contexts

Cluster section used to define the multiple sets of Kubernetes clusters such as development, testing, preprod and prod environment wise cluster or different organizations integrations use separate clsuter or different cloud providers can have clusters example google-cluster or azure-cluster etc.

And in the Users section we can have admin user, developer user etc. These users may have different privileges on different cluster resources.

Finally contexts manages the above two sections together mapping to form a context. here we will get to know that which user account will be used to access which cluster.

Remember, we are not going to create any new users or configuring any kind of user or authorization in this kubeconfig. We will be using only the existing users with their existing privileges and defining what user acces what cluster mapping. This way we don't have to specify tht user certifcates and server URL in each and every kubectl command to run.

The kubeconfig is in yaml format which basically have above mentioned three sections.

Kubernetes Configuration with different clusters map to Users

Filename: vybhava-config

apiVersion: v1
kind: Config

clusters:
- name: vybhava-prod-cluster
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443

- name: vybhava-dev-cluster
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443

- name: vybhava-gcp-cluster
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443

- name: vybhava-qa-cluster
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443

contexts:
- name: operations
  context:
    cluster: vybhava-prod-cluster
    user: kube-admin
    
- name: test-user@vybhava-dev-cluster
  context:
    cluster: vybhava-dev-cluster
    user: test-user

- name: gcp-user@vybhava-gcp-cluster
  context:
    cluster: vybhava-gcp-cluster
    user: gcp-user

- name: test-user@vybhava-prod-cluster
  context:
    cluster: vybhava-prod-cluster
    user: test-user

- name: research
  context:
    cluster: vybhava-qa-cluster
    user: dev-user

users:
- name: kube-admin
  user:
    client-certificate: /etc/kubernetes/pki/users/kube-admin/kube-admin.crt
    client-key: /etc/kubernetes/pki/users/kube-admin/kube-admin.key
- name: test-user
  user:
    client-certificate: /etc/kubernetes/pki/users/test-user/test-user.crt
    client-key: /etc/kubernetes/pki/users/test-user/test-user.key
- name: dev-user
  user:
    client-certificate: /etc/kubernetes/pki/users/dev-user/dev-user.crt
    client-key: /etc/kubernetes/pki/users/dev-user/dev-user.key
- name: gcp-user
  user:
    client-certificate: /etc/kubernetes/pki/users/gcp-user/gcp-user.crt
    client-key: /etc/kubernetes/pki/users/gcp-user/gcp-user.key

current-context: operations

To view the configuration of current cluster you must have the config in $HOME/.kube/config 

The content of kubernetes configuration can be view with the following command: :

kubectl config view 

Kubernetes Cluster view when default config used

To view newly created customized configurations, we need to specify the file path "vybhava-config" file. Note here the "vybhava-config" is available in the current directory.

kubectl config view --kubeconfig=vybhava-config

Know your Kubernetes cluster 

To check the list of cluster(s) exist in the default kubernetes cluster config

kubectl config get-clusters

Work with your customized config file vybhava-config to know clusters list

kubectl config get-clusters --kubeconfig=vybhava-config

Knowing about Kubernetes cluster from Kube Config

KuberConfig user details

To check the list of user(s) exist in the default kubernetes cluster config

kubectl config get-users

Work with your customized config file vybhava-config to know user list

kubectl config get-users --kubeconfig=vybhava-config

KubeConfig getting the users list

KubeConfig Context

Here the context will be using users, clusters  and each context is identified with a name define also here we can sees at the end of the configuration  current context. 

To find how many contexts in vybhava-config To know for default cluster contexts :

kubectl config get-contexts

To identify what user configured in the 'operator' context we need to use the 'get-contexts' option the mapping output is displayed as a table where 'CURRENT' context will be pointed with '*' in the column.

kubectl config --kubeconfig=vybhava-config get-contexts

Kubernetes Config getting Contexts using kubectl

Here in the Context section we could add a field namespace that can be specific to project module such as production cluster can be mapped to HR application that runs with hr-finance,hr-hirings namespce.

Here we have executed all possible choices for fetching the Users, Clusters, Context from KubeConfig object. Now let's try to set the context

delete user

kubectl config --kubeconfig=vybhava-config get-users
kubectl config --kubeconfig=vybhava-config delete-user test-user
kubectl config --kubeconfig=vybhava-config get-users

Deletion of Users from Config

delete cluster

kubectl config --kubeconfig=vybhava-config get-clusters 
kubectl config --kubeconfig=vybhava-config delete-cluster vybhava-gcp-cluster
kubectl config --kubeconfig=vybhava-config get-clusters 

Kubernetes Cluster deletion from KubeConfig

delete context

kubectl config --kubeconfig=vybhava-config get-contexts 
kubectl config --kubeconfig=vybhava-config delete-context gcp-user@vybhava-gcp-cluster
kubectl config --kubeconfig=vybhava-config get-contexts 

Deletion of Context from Kube Config

The certificate files can be part in cluster certificate-authority user certificates, The best tells that instead of using admin.crt we must use absolute path to the certificate files such as here /etc/kubernetes/pki/ca.crt and one more way to use certificate-authority-data field value can be used with certificate file content base64 encoded data. As we learnt about the content is sensitive then we need to use base64 command to encode run command as  "base64 ca.crt" and that can be understood by Kubernetes automatically.