Kubernetes Secrets - Securely Manage Passwords, Tokens, and Certificates in Kubernetes

By -

Hello DevOps Engineers, DevSecOps Engineers, SREs, and Kubernetes Administrators!

As organizations continue adopting microservices and cloud-native architectures, Kubernetes has become the preferred platform for container orchestration. However, deploying applications is only part of the journey. One of the most critical responsibilities is protecting sensitive information such as passwords, API keys, tokens, certificates, and database credentials.

Hardcoding secrets inside application code, container images, or deployment manifests is a serious security risk. Kubernetes addresses this challenge through Secret Objects, which are specifically designed to store and manage sensitive information securely.

In this article, we will explore Kubernetes Secrets, understand how they work, create them using different methods, and learn how to consume them inside Pods using environment variables and mounted volumes.

What are Kubernetes Secrets?

A Kubernetes Secret is an object that stores sensitive information separately from application code and Pod definitions.

Typical use cases include:

  • Database passwords
  • API tokens
  • TLS certificates
  • SSH keys
  • Authentication credentials
  • Service account tokens

Why use Kubernetes Secrets?

Kubernetes Secrets provides several benefits:

  • We can store Password, keys, tokens, certificates etc
  • Secrets will reduce the risk of exposing sensitive data
  • Access secrets using volumes and environment variables
  • Secrets object will be created outside pod/containers 
  • When it is created there is NO clues where it will be injected
  • All secrets resides in ETCD database on the K8s master

Important Limitation

A Kubernetes Secret had a limitation of 1MB size data only to store. More that size of sensitive data can be stored on dedicated secret management solutions such as Hashicorp Vault, AWS Secret Manager, Azure Key-Vault.

How does Kubernetes Secret Works?

Kubernetes Secret are created independent of Pod and can be referred by Applications.

Key points about Secrets:

  • Secrets exists as Kubernetes Objects
  • Secrets are referred by application when required.
  • Secrets are stored and controlled by Control Plan
  • Secret data persists in ETCD database

This Kubernetes Secret Objects are similar to ConfigMaps Objects 

Kubernetes Secret objects Using Volume, ENVIRONMENT variables

Pre-check first we will check the Kubernetes Cluster is up and running by checking node list. 
kubectl get nodes
All the Kubernetes master and slave nodes are in Ready status.

NAME STATUS 
  master Ready 
  worker01 Ready 
  worker02 Ready

Methods to access the Secrets

There are two ways to access these Secrets inside the Pod.
Method 1: Environment variables 
Secrets are exposed as Environment variables inside the container:
SECRET_USERNAME=admin
  SECRET_PASSWORD=*****
Method 2: Assign to Path inside Pod 
Secrets are mounted as files within container filesystem
/etc/secrets/username 
  /etc/secrets/password

Creating Kubernetes Generic Secrets

We are talking about Secrets, Let's create a text that can be converted to encrypted format, if we want we can decode that into plain text. In Linux CLI we have base64 command that will be used to do this encode or decode text from file or from standard input to display on standard output that is on our terminal. 

Secrets can be created in Kubernetes following ways: 

  1. from file or directory
  2. from literals
  3. Using YAML Declarative approach

echo "admin" | base64 > username.txt 
cat username.txt 
cat username.txt | base64 -d

# Now password
echo "abhiteja" |base64 > password.txt
cat password.txt | base64 -d

# Create secret from file
kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt
kubectl get secrets
kubectl describe secrets db-user-pass

Validate

Kubernetes secret creation with username, password

The Pod  with Redis  image will be using the secrets as environment variables


FileName: secretenv-pod.yml

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: db-user-pass
            key: username.txt
      - name: SECRET_PASSWORD
        valueFrom:
          secretKeyRef:
            name: db-user-pass
            key: password.txt
  restartPolicy: Never
You can create the Pod using the following command: 
kubectl create -f secretenv-pod.yml
kubectl get pods


Secret environment variables in Redis Pod

Get inside the Redis Pod into the container and check with the 'env' command which will show the SECRET_USERNAME and SECRET_PASSWORD 
kubectl exec -it secret-env-pod -- bash 
env | grep SECRET


Secret as environment variables inside Pod Container

From the literal

Now let's see the simple option that is using --from-literal we can have as many as you wish to store as secret here I'm using three variables stored int the 'mysqldb-secret' object.
  
k create secret generic mysqldb-secret \
 --from-literal=DB_Host=mysql01.vybhava.com \
 --from-literal=DB_User=root \
 --from-literal=DB_Password=Welcome123
 
  k describe secret mysqldb-secret
  k get secret -o yaml
Execution output as follows:
Kubernetes Secret creation from literal example

 
Pod implementation the secret object you can see in the first option we have already discussed.

Creating the Secret declarative way

We can create a secret using the YAML where we can set the data fields as encrypted values

Filename : mysecret.yaml 
apiVersion: v1
data:
  username: a3ViZWFkbQo=
  password: a3ViZXBhc3MK
kind: Secret
metadata:
  name: mysecret
Let's create the secret object with kubectl command 
kubectl create -f mysecret.yaml
kubectl get secrets
kubectl describe secrets mysecret

Creating secret object in Kubernetes using kubectl


Note that secret never shows the data present in the secret section, instead it will be showing the size of the data. It is like masking the sensitive data. And this secrete can be used inside the Pod defination as follows: Filename: mypod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: redis-db
      mountPath: "/etc/redis-db"
      readOnly: true
  volumes:
  - name: redis-db
    secret:
      secretName: mysecret
Creating the mypod defined as Pod and that using the volume defined with the secret section with mysecret which will referred to the above secret that we have created earlier.

Create Pod that uses secrets as volume

Encryption at rest configuration

The secrets which are created in the Kubernetes are NOT really encrypted so not secrets! So we don't share these declarative YAML files in source code repositories. 

How this secret is stored in the ETCD Database? 

To know this we must have etcd-client installed on your machine, on my system it is Ubuntu so let me install it.

apt-get install etcd-client
# Validate installation 
etcdctl
Kubernetes ETCD DB Client installation on Ubuntu

Best practices for Kubernetes Secrets

As an Automation Architect and DevSecOps Engineer, I recommend the following:

1. Never store the sensitive data/ password on Git repo, instead use:
  • Sealed Secret
  • Azure Key-Value
  • AWS Secret Manager
  • Hashicorp Vault
2. Enable encryption for REST 
protect the secrets stored on ETCD database. 

3. RBAC
Restrict access to secret object through Kubernetes Role based Access Control.

4. Rotate Secret data regular internvals
implement Password and certificate rotation policies

5. Use least privileged access 
The application should access only secrets they required.

6. Auditable Secret access
Monitor who is accessed secret within the Kubernetes Cluster

Common Mistakes

1. Storing Secrets in Deployment YAML better to Avoid:
env:
  PASSWORD: Welcome123
2. Committing Secret Manifests to Git
Never store sensitive values in source control repositories.

3. Assuming Base64 Equals Encryption
Base64 encoding only transforms data representation.

4. Overusing Environment Variables
For highly sensitive data, mounted volumes are generally preferred.

Key Takeaways

  • Kubernetes Secrets provide a secure mechanism for storing sensitive application data.
  • Secrets can be created from files, literals, or YAML manifests.
  • Applications can consume secrets through environment variables or mounted volumes.
  • Secret data is stored in ETCD and should be protected using Encryption at Rest.
  • Base64 encoding is not encryption.
  • RBAC, auditing, and secret rotation are critical security practices.
  • Hope you enjoyed this post!!
  • Please share with your friends and comment if you find any issues here.
Can you build a fully secure secret management workflow for your Kubernetes applications?

Comments

Post a Comment

Popular Articles

DevOps Weapons

By -
Image
To achieve faster application delivery and stable environments, the right tools must be used in DevOps environments. There is no single tool that fits all your needs such as server provisioning, configuration management, automated builds, code deployments, and monitoring. Many factors determine the use of a particular tool in your infrastructure.   In this article, we will look into core tools which can be used in a typical Devops environment. Version Control Tools: Git - An awesome tool to version your source code and collaborate. Terraform  - A tool used in building, changing, and versioning the infrastructure across platforms. Build Orchestration/Continuous Integration Tools: Jenkins - It is an open-source, lightweight CI tool written in Java, with high extensibility and a fast release cycle. Buildbot  - An open-source framework for automating software build, test and release process. CruiseControl  - A CI server written in  # Ruby ...
Read more

Ansible URI Module Tutorial: Real-World Application Health Checks, REST API Validation and DevOps Automation

By -
Image
Hello DevSecOps and Automation Engineers! Welcome back to the DevOps Hunter blog. One of the most common requirements in enterprise automation is validating whether an application is actually available after a deployment, patching activity, server reboot, or service restart. Simply starting a service does not guarantee that the application is ready to serve users. This is where the Ansible uri module becomes one of the most powerful tools in a DevOps engineer's toolkit. In this hands-on tutorial, we will explore the Ansible URI module, understand its key parameters, and work through several real-world examples that are commonly used in production automation workflows. Production Health Check workflow using URI module What You'll Learn By the end of this article, you will be able to: Understand the purpose of the Ansible URI module Perform HTTP and HTTPS application health checks Validate application response codes Test REST API endpoints Handle multiple acceptable HTTP status...
Read more

Ansible Jinja2 Templates: A Complete Guide with Examples

By -
Image
Here I'm starting this post with famous ARISTOTLE quote. "We are what we repeatedly do. Excellence, then is not an act, but a Habit" Welcome back Ansible Automations and who are habituated as Ansible automation specialists this post for them to give boosted walk through, In this post, I would like to share my experiments with Jinja2 templates usage on the Ansible playbook. Of-course Jinja is from Japan it is world famous for templatization capabilities badvanced Ansible templating techniquesy integrating with multiple languages such as Python, Ruby, Salt talk etc. In this post we will be cover the following topics: What is Jinja2 template do in Ansible? Template with filters Template with Lists and sets Template module in Ansible Template with Flow Control Template using Looping Template inheritance** What is Jinja2 template do in Ansible? Jinja2 is another Python library created for Flask web framework, that comes part of Ansible installation. It is special ability as i...
Read more