How to setup the Docker Private Registry on Ubuntu 19.04 with Docker 19.03.2?

What is Docker registry? There are various situations where a team could work on the microservices. Every time you connect to the internet and pull the images will be cost & time-consuming process. In an organization level if we set up the docker registry, where we can pull the images and all team members will reuse it by using the internal network.

Docker  public registry

A central place to store all organizational required docker images were distributed. There are several implementations of docker registries where it is build up with the following:

• A simple webserver to make access to docker images available
• A complete web application with user access
• Docker builds are available in two services: DockerHub, Docker Trusted Registry

DockerHub is a public repository where you publish your contribution to any image that you made with tag [versions] which can be downloaded by anyone from the internet. It should be searchable. It all depends on the stars, the number of downloads, trust in official category of images. People will consider the comments. 



To create the Docker registry we have two options:

• without security
• with security

Here I would like to share the secure option as follows:

Docker container Registry 

Let's use the docker registry container which is officially published by Docker team on the docker Hub. The latest version of registry image is v2 now. This edocker registry supporst the following features: 

• layer oriented
• layer ID are randomly assigned
• JSON object corresponding to each layer reflecting a parent
• naming accomplished through tags

Docker private registry with Security setting up 

Steps to configure local private docker registry:
1. Create a certificate directory"/docker_data/certs" to hold the TLS certs
check openssl exists, if not run the following installation command:

mkdir -p "/docker_data/certs"
yum -y install openssl

On Ubuntu instance:

apt install -y openssl

2. Generate an SSL/TLS certificate to secure our private docker registry

openssl req -newkey rsa:4096 -nodes -sha256 -keyout /docker_data/certs/domain.key -x509 -days 365 -out /docker_data/certs/domain.crt

# Enter the hostname fqdn 3. Create a directory to store docker images "/docker_data/images"

mkdir -p /docker_data/images

4. Run a docker registry container in docker host "docker-registry"

docker search registry
docker login 

# login with docker user [you must signup for this if not yet had]

docker run -d -p 5000:5000 \
-v /docker_data/images:/var/lib/registry \
 -v /docker_data/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  --restart on-failure \
  --name devops-registry \
  docker.io/registry

# check for the registry container in the list

docker container ls

# rename images which you would like to push to your private registry this is a recommendation

docker pull nginx
docker tag docker.io/nginx mydev.devopshunter.com:5000/my-nginx

# Now list of images to see the above image tag was created out of Nginx image.

docker image ls

5. Pull required docker images to docker host "docker-registry" from docker hub, rename (tag) with our local registry

docker tag docker.io/nginx mydev.devopshunter.com:5000/myweb
docker tag python mydev.devopshunter.com:5000/mypython

6. Push those pulled docker images to our docker private registry container

docker push mydev.devopshunter.com:5000/mypython
docker push mydev.devopshunter.com:5000/myweb

Let's confirm this by looking into the following path

ls -l /docker_data/images/docker/registry/v2/repositories/

# You can see two folders were created for each image, that indicates clearly what we had pushed are successful.
7. Remove old docker images from local docker-host.

 docker rmi ubuntu python nginx  

8. Configure all docker clients to use our certificates
9. docker clients can pull and push docker images into our private docker registry in registry server

curl -X GET http://localhost:5000/v2/_catalog

This is the end of the story of hosting your own docker private registry.

 References:

1. Docker Host and Docker client communication on Ubuntu 19 
2. Docker installation on CentOS 
3. Docker Concepts

Docker EE installation on Windows 2019 on AWS instance

Learning Docker for Certification preparation it is must to understand various types of platforms
Windows Server later versions of 2016 will be suitable to run the docker enterprise engine. and also remember that the latest Docker will run on 64bit platforms only. If you like to test this on AWS AMI then it is a must that you must allocate 30G disk space for the instance. So better stop all other instances first when you run the AWS Windows instance starts.

Choose your Cloud wisely!

Note: Windows instance on AWS is going to consume more disk space, they are fat like elephants!

Docker installation on Windows 2019 AWS AMI

Step1: Login to the AWS console, navigate to your Services-> EC2 instances click on the 'Launch Instance'.

The AWS instance Windows 2019 AMI Selection

Step 2: Choose the instance type from the list:t2micro free-tier

aws instance type selection

Step 3: Go with the default options no changes required.

Step 4: Select the disk storage space for the Windows instance default size is 30GB. we cannot go below this size.

Step 5: Proceed with defaults
Step 6: Security Group for Windows would be RDP for Windows, where this can be compared with the SSH and other protocols for Linux AMIs.

Step 7: Finally review the instance configurations in a single page and confirm it, if everything looks fine for you. To access the Windows RDP need public host IP.

Create a new Windows_2019keys (AWS Keypairs)

After downloading the Keypair, accept the agreement and Launch Instance of Windows 2019. Within a couple of mins, we can see the Windows instance in running state.

Select 'Connect' button, which will show the ec2-<windows-instance-ip>.rdp file.

Click on the file that is downloaded, Open the RDP file then enter the credentials as suggested on the AWS connect pop-up.

Install-Module DockerMsftProvider -force

Install-module in PowerShell command for dockerprovider

Now let us run the nuGet module.

Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -force

Install-PackageProvider for Docker

let us install the Docker EE 19.03.02 using Install-Package PS command:

Install-Package Docker -ProviderName DockerMsftProvider -Force

Wow Docker installation on Windows Server completed successfully.

Let's confirm by saying version

docker version

Please restart the AWS Windows instance from the AWS Console, Go to 'Actions' ->'instance State' then select the 'Reboot instances' option.

You might be interested to see Docker concepts:

1. Install Docker on Ubuntu 19.04
2. Install Docker on CentOS
3. Install Docker on Windows

Jenkins Configure build-server environment on AWS AMI

In this post, I would like to share my experiment with AWS instances using IAM role for authenticating from the remote machine that is Jenkins Master instance. make sure that Jenkins Master should not have any executors on the master so that no job run on the master server, that will be making the Jenkins run in a distributed environment. Jenkins Master-Slave implementation within the AWS instances, and to do this. We need to follow a flow.

Jenkins Master -> key pair -> Build server ->IAM->ec2 or EBS or S3 instance

Create IAM role for the build server

Login to your AWS console, search for IAM on the AWS console

In the left pane select 'Roles' menu item. In the right side, click on the "Create role " blue button.
select the type of trusted entity as "AWS service" that is default one.

Now choose the service that will use this role: click on "EC2" Link.
Click on the 'Permissions' bottom right side button, proceed further to attach the policy.

AWS Create Role - Attach permission policies

Under Create Role you could see the button Create a policy
Filter policies with elasticbeanstalk
Let us search result select the 'AWSElasticBeanstalkFullAccess', click that to get the details of that policy which is already defined. this will have 18 services shown in the summary.
Ignore tag page, Click on review button at the bottom.

Review page enter the following values
Role name: build-server-role
Des: Elastic beanstalk full access

In the bottom click on create

AWS Instance Create Role - Review page

Create a key-pair for a build-server

In the AWS Dashboard -> Under Network and security -> Keypair
Choose the button Create key pair for build-server with the name as: build-server-keys.pem
It will automatically download the key-pair that you have created. Let's open the 'build-server-keys.pem' file in notepad++

Create AWS Linux 2 instance

AWS services search for EC2 dashboard, click on the button 'Launch Instance', which opens the 7 steps wizard. Choose the Amazon Linux 2, free-tier (t2-micro) configure the instance.

AWS instance type selection

Create a Security Group  

Enter the following values for the Security group that will be having only SSH that allows us to connect from the Jenkins Master.

Name: build-server-sg
Des: SSH only for Jenkins
same desc in the inbound rule
no need to changes for an outbound rule

Click on create button

Selecting the security group for build-server

All set to review and launch the instance. Click on the 'Launch' button.

Select an existing key pair for Instance which was already created.

Select an existing key pair

Configuration of Jenkins Credential

Now the Jenkins Credential configuration, go to the Jenkins admin console

Click on the Credentials
-> System
Global Credentials

Jenkins Credentials configuration

Left pane click on the 'Add Credentials' that will open a form fill with the following values:

Kind: SSH Username with the private key
Scope: Global (Jenkins Nodes, items and all child items etc)
Enter the values for Username: ec2-user
Password -> private key
Enter it from the build-sever-keys.pem file content which was already opened in the Notepad++.

ignore password, ID, Des optional.

Jenkins Credentials RSA key authentication

click on 'OK' button.

Remember that the Private DNS name/private IP for the build-server required, it will save the cost for internal network access.

Now select the "Manage nodes" link

Node creation on Jenkins master

Slave server configuration on Jenkins master

You may be interested to view our other knowledgable technical blog articles:

• Jenkins Master

Continuous Integration Automation: Webhook on GitHub notify Jenkins

Hello Guys, Jenkins CI/CD enthusiast,

In this post, I would like to discuss the implementation of Jenkins Continuous Integration automatic build triggers. Where Jenkins master will be stay connected with the GitHub server with an API token based authentication. To connect with GitHub server we need a personal access token.

Webhook for GitHub and Jenkins integration

1. Create token credentials
2. GitHub server configuration

Webhook is a wonderful solution that will keep track of changes that made to the GitHub repo and notify the Jenkins Master. If we create the item or freestyle project will be triggered automatically.

Create token credentials 

Login into the Jenkins console, Click on the left pane - "Manage jenkins" link. Select the "System Configure" navigates to the new page where you need scroll down till you found the GitHub section.
First we will create the token, look for the "Advanced" button on the rightside. Additional actions: 'Convert login and password to token'.
Select the radio button: "From login and password" which prompt for entering the values for the Login, password that used for GitHub where you have repo. Then use the 'Create token creentials' button at the right end.

Create Token for GitHub on Jenkins

Now the tokens are generated we are ready to connect with the GitHub server. move upwards and enter the following values:

• Your GitHub profile used for connecting
• API URL don't change this 
• Credentials select from the dropdown, where you can find the recently created Token for GitHub - Selected
• Click on the Test connection button will produce the result of connectivity 
• Tick mark the 'manage hooks'

GitHub Server configuration on Jenkins

GitHub Configuration for Webhook

The following simple steps will be enables you to get the webhook on the GitHub that connected to Jenkins Master

1. Log in to your GitHub
2. The top right corner you can find your profile link from that menu select 'Settings'.
3. In the left pane at the bottom, Select the 'Developer Settings'
4. The new page for 'Developer Settings' will have three options, Select 'Personal access tokens' from the menu.

Personal access tokens on GitHub

Now we all set to go for testing the Webhook connectivity with Jenkins

Create a freestyle project for Webhook

On the Jenkins console create new freestyle project
Name it as GitHub-webhook
Project type: Freestyle project
Click on Ok button

Jenkins Project for GitHub Webhook

Source Code Management

Select the Git

Jenkins Source Code management

Build Triggers

Select GitHub hook trigger for GITScm polling

Jenkins Build Trigger

In the Build tab
Select the Execute shell

enter the stuff to know that build triggered in my example used the cat the README.md file which is targeted to modify.

Jenkins build section for CI automation

Save the Jenkins Project.

Go to the GitHub page

Modify the code in the Git repository,

added a line in the README.md file

Automatic Build Triggers

In the Jenkins now the job will be triggered automatically

Jenkins Build automatic triggered and the console output 

Here is the conclusion that Webhook is simple mechanism that will helps Jenkins get notified with code changed in the GitHub.

Jenkins Master setup on AWS Instance Ubuntu 18.04

Hello, dear DevOps Enthusiast!

The Jenkins master setup on AWS Cloud instance - Ubuntu 18.04 detailed execution steps that performed are collected and posted in this post.

Jenkins install on Ubuntu instance of AWS

Login to your AWS console, go to EC2 Dashboard and perform the following steps

• Create a Security Group
• Key pair for instance
• Create Ubuntu AMI instance
• Elastic IPs Configure and associate
• Install Java, Nginx, and Jenkins
• Configure Nginx proxy

1. Create a security group

Security Group which will allow the protocol: SSH, TCP ICMP, HTTP, HTTPS
Using security group we can specify the network allow to access. Go to the EC2 dashboard. "Create a security group" button. The Security Group name it as 'my_sg01'. Outbound tab keeps as it is, goto the inbound tab and define for incoming traffic.
Secure Shell ->22-> My IP
HTTP web traffic ->80 -> anywhere
HTTPS web secure traffic -> 443 -> anywhere

Tag it "my_sg01", which is a best pratice to tag our configuration for the security group.

2. Keypair for the Ubuntu instance

A cryptographic system for encrypt and decrypt. which will be used to not to enter password while connecting to the instance. AWS only have a way to connect with keypair. Same keypair can be used for multiple instances. when you download a keypair .pem file which is private potion of the ec2 instances.
In Windows, if you wish to use PuTTY then we need to generate the ppk file from the download.em file.  which can be used to SSH authentication for the PuTTY. If you are connect from git bash you can change the pem file permission 600.

example:

 
chmod 600 jenkins-master.pem

3. Create Ubuntu instance for Jenkins Master

In the EC2 dashboard choose the "Instances", click on the 'Launch Instance' button, and search for 'Ubuntu 18'

Click on the select button, here we choose free tier eligible for Ubuntu instance that is available so proceed with 'continue' button. Select the 't2.micro' from the list of instances. Select the Security Group that we created earlier use it and go with the default disk space,

Configure instance details and review it and launch it. At the time of launch key pair details choose 'from the existing key pair, which we configured earlier.

4. Elastic IP config and associate 

Most restarting instances will have different Public DNS the solution is you can use an Elastic IP address. In the EC2 dashboard in the left pane, you could find the link 'Elastic IPs' click on the 'allocate new address' button. and then click on the 'allocate' button. After 'IP' configure, go to the 'actions' -> Associate address'. Choose the 'jenkins-master' instance.

AWS resources required for Jenkins Master setup

Now go to the instance check the public DNS name and connect with the ppk, user as ubuntu using PuTTY.

5. Install Java, NGINX, Jenkins

All the installations we will do with root user, let's switch then

 sudo -i

Let's install Jenkins first, get the GPG keys of the Jenkins repository :

 
wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list
apt update
apt upgrade

after all upgrade changes applied, we can install the prerequisite software JDK 8 for Jenkins Master and Jenkins run behind the reverse proxy with NGINX.

First, we will install the Open-Jdk8

apt install -y openjdk-8-jdk

Install NGINX

 
apt install -y nginx

Last but not least, Install Jenkins

 
apt install -y jenkins

To confirm that nginx web server is up and running with systemctl command:

 
systemctl status nginx |grep Active

Similarly check for the Jenkins

 
systemctl status jenkins |grep Active

Note that Jenkins process Active but exited, we can ignore for now and proceed how to make this accessible with the webserver.

Configure NGINX webserver
The reason for using NGINX web server in Jenkins master installation, act as reverse proxy:

USER-> NGINX reverse proxy ->Jenkins and vice versa also works

Why do we need NGINX in Jenkins Master setup?

• Application server security
• Reverse proxy generates logs we get much more information.
• Simple logging
• Simple SSL termination for Jenkins is made from NGINX

How NGINX works?

Use the public DNS name in the browser will load the NGINX web page.

Nginx reverse proxy for Jenkins Master

6. Configure Jenkins on NGINX proxy

Make sure you are on root user
 sudo -i

First, disable the links of Nginx server's documents path, it doesn't remove anything!

 unlink /etc/nginx/sites-enabled/default

 Now let's configure for Jenkins server on NGINX server.

vi /etc/nginx/config.d/jenkins.conf

 
upstream jenkins {
 server 127.0.0.1:8080;
 }
 
 server {
 listen 80 default_server;
 listen [::]:80 default_server;
 location / {
  proxy_pass http://jenkins;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
 }
}

root@ip-172-31-12-187:~# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

That's what expected to see 'successful'!

Now reload the nginx service:

 systemctl reload nginx

Open the NGINX url either using public DNS or Elastic IP in the browser again it will be redirected to Jenkins 'Unlock Jenkins' page. Wow!!, proceed further steps as you know regular as we did in the last Jenkins installation post.

Welcome to Jenkins!

Reference:

• Jenkins Installation on CentOS