Kubernetes Security - TLS Keys and Certificates

Transport Layer Security  (TLS) Basics 

 Early days it was called Secure Socket Layer (SSL) now it is renamed as TLS. In this post we will explore more TLS  certificate files. and their usage in different Kubernetes Cluster components.  TLS Certificates for Kubernetes Cluster components is basic thing to do a High Availability(HA) in Production configurations and troubleshoot various security for users, applications, and administration level issues.

Public Key Infrastructure used in the Kubernetes Cluster Security

Certificate files

The certificate files are nothing but key pair that have private key, public key. And Public Key which we can consider as a Lock visible to public. The example certificate file can be having extension as .crt or .pem the files like server.crt, server.pem client.crt or client.pem. Whereas Private key it will be owned by the person who generated it. This file cannot be distributed, instead it will be used when Lock is visible to it uses this key to unlock and get the data or send the data in a encrypted format. Most of the private key file extensions can be .key or .pem sample files are server.key or server-key.pem (to differ with server side public key filename changed) client.key or client-key.pem.

There could be three types of certificates

1. Server Certificates - Server side 
2. Root Certificates - are reside at CA 
3. Client Certificates - at browser or client side in Kubernetes component client side

Why we need TLS certificates?

Let's take a online payment transaction example. Customer 'X' wants to transfer the money to a ecommerce vendor online. When user send user details such as username=xxx,password=yyy,trnamount=100. Hacker is going to do an attack in the middle customer - vendor try to get the user details and able to get the control on the transactions. This is called middle-man attack.

To avoid this kind of attacks on your e-business, we need end-to-end secure communication for this use TLS certificates between web-server and web-clients.

Where can I get the Certificates? 

We need to submit the certificate request to Certificate Authority (CA). There ae many CA which are internationally available.

1. VeriSign
2. GeoTrust
3. Let's Encrypt
4. DigiCert

Other than this we can use self-sign certificates in the Kubernetes. When we installed Kubernetes using kubeadm it is built-in for us certs are created automatically for us. Each component uses different Issuer to make it more secure. 

All the certificate are configured with 2048 bit size encrypted with RSA algorithm. All the Kubernetes certificate files will be stored in a common location on the master "/etc/kubernetes/pki" Every where we have public key and private key files private keys are always having the extension as .key and public key may have .pem or .crt file extensions. we can navigate to the /etc/kubernetes/manifests/ path where we can see all the definitions required to run the Kubernetes Cluster. 

1. ETCD database related definitions are in etcd.yaml 
2. Controller-manager - The definitions related to different controller-managers are available in kube-controller-manager.yaml 
3. kube-apiserver - Whole cluster work entry and exits are happen through apiserver its definitions are in kube-apiserver.yaml 
4. kube-scheduler - Pods, Network node are controlled using schedulers and their related definitions are in kube-scheduler.yaml

How do you Identify the certificate files used for the kube-apiserver?

To identify certificate files used for the kube-apiserver can be viewed

view  /etc/kubernetes/manifests/kube-apiserver.yaml 

In this file under the containers section we can see there are several - command having different certificate path where apiserver related we have --tls-cert-file, --tls-private-key-file values are used for apiserver. 

Kube-apiserver certificate details from the manifest file

In Kubernetes master kube-apiserver is a client to the ETCD Database server. To authenticate kube-apiserver authenticate ETCD client certificate used are apiserver-etcd-client.crt defined with the option --etcd-certfile

To identify the key used to authenticate kube-apiserver to the kubelet server uses /etc/kubernetes/pki/apiserver-kubelet-client.key 

 To identify the certificate file used to authenticate kube-apiserver to the kubelet server uses /etc/kubernetes/pki/apiserver-kubelet-client.crt

ETCD Server CA Root Certificate used to serve ETCD Server. It's a best practice to have different CA (Certificate Authority) for ETCD server than kube-apiserver. Trusted CA provide certifcate file available at /etc/kubernetes/pki/etcd/ca.crt

How do you find the Common Name (CN) configured on the Kube API server certificate?

Kubernetes apiserver certificate data configured to view we can use 'openssl' command.

opsnssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

Viewing the Kube-apiserver certificate file using openssl command

There will be lengthy output on your screen but important fields which we can look for are: Issuer having CN which is related to CA, where we are looking for which server it is defined then look for "Subject" line contains the CN value here it is "kube-apiserver". 

By default the name of the CA who is issued the Kube APIserver certificate is mentioned as  'kubernetes'. 

The below are the alternate names configured on the kube-apiserver certificate: [look at the X509v3 Subject Alternative Name: under this DNS values] 

• controlplane 
• kubernetes 
• kubernetes.default 
• kubernetes.default.svc 
• kubernetes.default.svc.cluster.local 

 ETCD Server certificates configuration to view use the following 'openssl' command:

openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout 

Viewing the ETCD Server certificate content

Here we need to observe that the "Subject:" is the Common Name (CN) configured on the ETCD Server cert-file. Server certificate validity We can find the certificate how long it is valid from the same 'openssl' command check under the 'Validity' section. Root CA Certificate validity will be approximately 10 years which we can see in the /etc/kubernetes/pki/ca.crt

Most of the troubles will be misplaced certificate paths. Validate the crt file paths and inside the apiserver.yaml or etcd.yaml file contain paths