Scheduling Pods 1: Taint and Tolerance

 Node in the Kubernetes cluster are schedule the pods as per the Node level Taints will control the Pod creation on the Node. 

We can update the taints on one or more nodes with single command. The following are instructions from the kubectl label --help

• A taint consists of a key, value, and effect. As an argument here, it is expressed as key=value:effect. 
• The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to 253 characters. 
• Optionally, the key can begin with a DNS subdomain prefix and a single '/', like example.com/my-app. 
• The value is optional. If given, it must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to 63 characters.  
• The effect must be NoSchedule, PreferNoSchedule or NoExecute. 
• Currently taint can only apply to node.

How does Taint - Tolerance works in Kubernetes?

• Taints allow a node to repel a set of pods.
• Tolerations are applied to pods, and allow the pods to schedule onto nodes with matching taints.
• Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes.
• One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints.
• 
  

How do you know how many worker nodes participating in the Kubernetes cluster?

kubectl get no

This list may includes the controlplane as well. 

How to check do any taints exist on a particular node say node01?

kubectl describe nodes/node01 |grep -i taint

Creating a taint on node01 with key of 'spray', value of 'mortein'and effect of NoSchedule.

kubectl taint nodes node01 spray=mortein:NoSchedule
# Verify the above
kubectl describe nodes/node01 |grep -i taint

Let's create a new pod with the nginx image and pod name as mosquito.

kubectl run mosquito --image=nginx
kubectl get po -w

Observe that pod name  mosquito status is in "Pending" continuously when pod trying to get inside, unable to enter into the node01 due to its spray taint. Here you can taint with any name that will stop pod to schedule it on a particular node.

What is the reason Pod is in pending state?

The main reason here is Pod mosquito connot tolerate taint 'Mortein' spray.

Let's create another pod named 'bamblebee' with the nginx image, which has a toleration set to the taint 'mortein'.

apiVersion: v1
kind: Pod
metadata:
  name: bamblebee
spec:
  containers:
  - image: nginx
    name: bee
  tolerations:
  - key: spray
    value: mortein
    effect: NoSchedule
    operator: Equal
  

Create with

kubectl apply -f toleration.yaml
kubectl get po -w

Noticed that the bee pod was scheduled on node node01 despite the taint. 

How do you check the ControlPlane is having any taints on it? 

It is simple, You can run the describe sub-command on the node/controlPlane

kubectl describe node controlplane

deafult following taint was used Taints: node-role.kubernetes.io/master:NoSchedule 

Remove from node 'controlplane' the taint with key 'node-role.kubernetes.io/master' and effect 'NoSchedule' if one of these key-value pair exists we can control the scheduling the pods.

kubectl taint nodes controlplane node-role.kubernetes.io/master:NoSchedule-

Most of the Pods in the real-time will be created under 'deployment' to have replication controllers the pods creation on the nodes. So most of the toleration declared in a section will be helpful.

Best practice is to take a back up of the original $HOME/.kube/config file then go for the overwrite the 

Kubernetes Labels - Selectors, Annotations

What this post covers?

In this post we will explore all possible options that can be used for Kubernetes Labels and selectors.

• Adding labels
• Show labels 
• Replace labels
• Deleting labels 

 Adding Labels to Pods

Adding labels while you creating bare pods imperatively using 'kubectl run' command, here I'm using three different images httpd, redis, rabbitmq.

 

kubectl run web --image=httpd:2.4.54-alpine \
  --labels="env=prod,author=pavandeverakonda,component=customer,tier=frontend"
kubectl run db --image=redis:alpine \
 --labels="env=prod,author=pavandeverakonda,component=customer,tier=backend"

kubectl run web2 --image=httpd:2.4.54-alpine \
 --labels="env=dev,author=pavandeverakonda,component=customer,tier=frontend"
kubectl run db2 --image=redis:alpine \
--labels="env=dev,author=pavandeverakonda,component=customer,tier=backend"

kubectl run msg-pod --image=rabbitmq \
 --labels="env=prod,author=ranjan,component=customer,tier=integrat"

#Validate the labels with the pods 
kubectl get po --show-labels

Create Pod along with Labels

Show labels for Kubernetes Objects

To see what all labels associated with a kubernetes object use --show-labels option

# for Pods
 kubectl get pods --show-labels 

# for nodes
kubectl get nodes --show-labels

To filter only the prod environment

 kubectl get pods -l env=prod --show-labels 
 

To get without header use --no-headers=true

 kubectl get pods -l env=dev --show-labels  --no-headers=true
 

To get the count of the dev environemnt pods we can use wc -l to count lines

 kubectl get pods -l env=dev --show-labels  --no-headers=true|wc -l
 

Using --selector option 

 To get how many pods present in the business unit(bu) as finance, here we can use --selector option

 kubectl get pods --selector bu=finance --no-headers=true --show-labels
 

To get the count of the pods in finance bu

 kubectl get pods --selector bu=finance --no-headers=true|wc -l
 

To find all objects are present in the prod environment including Pods, ReplicaSet and any other

kubectl get all --selector env=prod --show-labels  --no-headers=true

To get the count

 kubectl get all --selector env=prod  --no-headers=true|wc -l

To Identify the Pod which is part of the "prod" environment, the "finance" BU and of "frontend" tier?

kubectl get all --selector env=prod,bu=finance,tier=frontend  --no-headers=true

more specific to pods only display

kubectl get pod --selector env=prod,bu=finance,tier=frontend

# Only Pod names
kubectl get pod --selector env=prod,bu=finance,tier=frontend  --no-headers=true  -o NAME

Deleting labels

a. Edit the Kubernetes Object b. Using - operator to remove labels

Using - operator to remove labels

Update pod 'web' by removing a label named 'author' (if that label exists with that pod)

kubectl label pods web author-

Deleting a label from a Pod

You can also remove two or more labels at one go, removing env, tier labesl from the web pod:

kubectl label pods web env- tier- 

Deleting multiple labels from a Pod

Kubernetes Annotations Another intresting fact about Kubernetes Objects, they are like labels but they are just metadata not used in the selectors section.

kubectl annotate pod/web contact="admin@vybhavasolutions"

You can also add more number of annotationsto to your a Kubernetes objects anytime like this:

kubectl annotate pod/web whatsapp="9618715457" email="vybhavatechnologies@gmail.com"

viewing Annotations of kubernetes objects The first option is always using describe kubernetes object, where you could see a section for Annotations is available.

kubectl describe pods web -o yaml |grep -c 3 'annotations'

Keep writing to us for more intresting facts on Kubernetes related errors which you have encountered.

Kubernetes Deployment

Hello SRE or Platform Engineer  or DevOps Engineers, In this post I want to discuss, Understanding of Kubernetes deployment it's hierarchy of kube objects. Declaratives and imperative ways to make deployment on kube clusters. 

How to deploy an application on  Kubernetes pods, just follow these steps as shown in this post. 

Here I would like to share with you about Kubernetes deployment hierarchy, which internal call to replication controller to make desired number of replicas of pod temple specified.

Kubernetes Deployment hierarchy

Let's goto have a deep understanding about Kubernetes deployment hierarchy

Step 1: We need to create a YAML file to define the deployment of the Nginx Webserver. Here we are taking a file name as Nginx-deployment.

Command : 

k create deploy httpd-deploy --image=httpd:alpine --dry-run=client -o yaml
k create deploy httpd-deploy --image=httpd:alpine --dry-run=client -o yaml >httpd-deploy.yaml

vi httpd-deploy.yaml
# Insert the following manifestation for deployment using httpd:apline image

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: httpd-deploy
  name: httpd-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd-deploy
  strategy: {}
  template:
    metadata:
      labels:
        app: httpd-deploy
    spec:
      containers:
      - image: httpd:alpine
        name: httpd

k create -f httpd-deploy.yaml #create
k get deploy,po #confirmation
    

1. List all the Deployments of default namespace 

kubectl get deployments
# or use alias name  
kubectl get deploy
# or use alternative name  
kubectl get deployment

2. Describe the Deployment details

kubectl describe deploy web-deployment  

Deployments contains Pods and its Replica information. 

3. To Show Rollout History of the given Deployment

kubectl rollout history deployment web-deployment 

4. Updates the existing deployment using update application new version. Do some R&D work here. Goto the Docker Hub find the Niginx tags consider the version tags to deploy. Currently I could see 1.22.0 and 1.23.1 versioned tags. So start with the new deployment with nginx:1.22.0 image after the pods up and application looks good we can update to the latest version that is nginx:1.23.1 versioned image.

k create deploy web-deploy --image=nginx:1.22 --replicas=2
k get deploy,po -o wide
# upgrade to new version 1.24
k set image deploy/web-deploy nginx=nginx:1.24 
k get deploy,po -o wide

5. Rolls back or roll Forward to a specific version

kubectl create –f app-deploy.yml 
kubectl get deploy app-deploy
kubectl apply –f app-deploy.yml --record 
kubectl rollout undo deployment app-deploy - -to-revision=1 
kubectl rollout undo deployment app-deploy - -to-revision=2

Understanding the Kubernetes Pods

Kubernetes pod 

The Kubernetes pod is going to represent a running process on the Kubernetes cluster.

It encapsulates an application container, storage resources, a unique network IP Address. Inside a pod, we can have one or more containers. The containers in a Pod are automatically co-located and co-scheduled on the same physical or virtual machine in the Kubernetes cluster.

A Multi container pod is going to be very tightly coupled inside. The containers share access to their memory space. They can be connected to each other using localhost. That is if a web application running on a container can be accessed in a busybox container running in the same pod. Containers in the pod can share storage(volumes) disk space as well.

What is a Pod in Kubernetes?

A pod is an atomic unit of scheduling in the Kubernetes ecosystem. Pods are ephemeral they live in a short time span.

On the master node, the different Controllers are running for monitoring and self-healing of the pods. Containers in different pods have distinct IP addresses. And if the pod regenerated every time new IP address will be assigned to the Pod.

What does a pod do?

K8s Pods represent the processes running on a cluster. By limiting pods to a single process, Kubernetes can report on the health of each process running in the cluster. 

Pods have:

• a unique IP address (which allows them to communicate with each other)
• persistent storage volumes (as required)
• configuration information that determines how a container should run.

Although most pods contain a single container, many will have a few containers that work closely together to execute the desired function

What are the benefits of a Pod?

=>When pods contain multiple containers, communications, and data sharing between them is simplified. Since all containers in a pod share the same network namespace, they can locate each other and communicate via localhost. Pods can communicate with each other by using another pods IP address or by referencing a resource that resides in another pod.

Pods can include containers that run when the pod is started, say to perform initiation required before the application containers run. Additionally, pods simplify scalability, enabling replica pods to be created and shut down automatically based on changes in demand

How does a pod work?

=>Pods are created by workload resources called controllers, which manage the rollout, replication, and health of pods in the cluster. For example, if a node in the cluster fails, a controller detects that the pods on that node are unresponsive and creates replacement pod(s) on other nodes.

The three most common types of controllers are:

• Jobs for batch-type jobs that are ephemeral, and will run a task to completion
• Deployments for applications that are stateless and persistent, such as web servers (HTTP servers)
• StatefulSets for applications that are both stateful and persistent such as databases

If a pod has multiple containers, they are all scheduled together on the same server in the cluster, whether VM or physical server. All containers in the pod share their resources and dependencies and can coordinate their execution and termination. For example, pods can contain ‘init’ containers that run before the application containers run, setting up the environment for the applications that follow.

Pods are almost always created by controllers which then can automatically manage the pod lifecycle, including replacing failed pods, replicating pods when necessary, and evicting the pod from cluster nodes when they are complete or no longer needed.

Controllers use information in a pod template to create the pods, and controllers ensure that running pods match the deployment defined in the pod template, for example by creating replicas to match the number defined in the deployment.

How do pods communicate with each other?

=>When a pod is created it is assigned its own unique IP address. If there are multiple containers within the pod, they can communicate with each other simply by using localhost. Communications outside of the pod is achieved by exposing a port. Communications between pods in a cluster takes advantage of the fact that Kubernetes assigns a cluster-private IP address to every pid in a cluster, eliminating the need to either explicitly create links between pods or to map container ports to host ports. In this way, every pod in a cluster can ‘see’ each other without the need for NAT

6)what are the basic kubectl commands used for pods?

1) get

2) create pods

3) delete pods

4) get pods

What are Pod lifecycle states?

The pod life cycle states will be represented with the docker action as shown:

Kubernetes Pod Life Cycle

=>pending

=>running

=>succeeded

=>failed

=>unknown

downloading=>     blocked=>          pending=>      starting=>         ready=>       failed

   (docker image) (docker create)    (created)     (docker start)    (container up)    (exited)

--------------------------------------------------------------------------------------------------------------------

Kubernetes multi-container pods

=>Sidecar containers "help" the main container (may be web application or database). 

For example, log or data change watchers, monitoring adapters, and so on. 

A log watcher, for example, can be built once by a different team and reused across different applications. 

Another example of a sidecar container is a file or data loader that generates data for the main container.

We can define a fresh pod-config in a YAML file sample file is given below:

#File: nginx-pod.yaml
apiVersion: v1
kind: pod
metadata:
  name: nginx-pod
  labels:
    app: nginx
    tier: dev
spec:
  container:
  -name: nginx-container
   image: nginx

Pod Create & Display commands 

1) To create a new pod we can use the kubectl create and then pass the pod definition YAML file. 

kubectl create -f nginx-pod.yaml 
  pod/nginx-pod created
  

Other options to Create a new pod with the nginx image using deployment or using run sub-command:

  kubectl create deployment mynginx -- image=nginx
  kubectl run nginx --image=nginx
  

2) How to check how many pods exists on the system?

To know how many pods running in the current default namespace. To check pod is RUNNING or failed  we can use the kubectl with  get sub-command on Kubernetes cluster running system to get pods status

kubectl get pods

* The READY column in the output indicates Running containers in Pod/Total containers in Pod. 3) To check IP address of the pod we can use the -o wide option

kubectl get pod -o wide 

4) To get the Pod definition in JSON or  YAML file we have to mention the output format as shown:

kubectl get pod nginx-pod -o yaml 

5) Display all details and events of a pod

kubectl describe pod nignx-pod 

6) Validating the pod network by pinging the container IP from the Kubernetes master node

ping 10.240.1.26 

7) To enter into the Pod/container we need to use exec sub-command like in docker we do. We can use sh or bash as per the image used for creation of container, following is the command to getting a shell to a RUNNING container

kubectl exec -it nginx-pod -- /bin/sh

8) To delete the pod use the delete sub-command

kubectl delete pod nginx-pod

9) We can use the get sub-command for many resources together only thing here we need to use comma to separate them. See the following command list one or more resources

kubectl get pods,deploy

10) Display detailed state of one single pod

kubectl describe pods [pod-name] 

11) To delete pod and services which have the label provided

kubectl delete pods,service -l name=[label-name]

When you delete the pod - wait till that termination completes. 12) To delete all pod resources from the default namespace

kubectl delete pods --all

13) Execute a command against a container in a pod.

kubectl exec [pod-name] date

14) If there are multiple container is running inside a Pod, we need to communicate or connect with single container the command is note (this command execute in master node)

kubectl exec [pod-name] -c [container-name] date 

15) The exec command with -i indicates interactive and -t indicates terminal for the container inside the container

kubectl1 exec -ti [pod-name] /bin/bash 

How to Look into the Pod logs

Kubernetes maintains a logger in the ecosystem. To accessing the Kubernetes pod logs  there are two log levels are available:

• Node-level
• Cluster-level

print the logs for a container in a pod
 

kubectl logs [pod-name] 

for example: using nginx pod logs.