Ansible Automations Designing & Implementation | Best Practices | Tricks and Tips

Hey DevOps, DevSecOps Engineers, SRE new bees here I am going to share the Learnings which I've executed every day found that this the best and new tip to improve the performance of ansible playbook executions sorted out and listed here.

Planning and designing automation with Ansible

• Most common DevOps tool used for Planning and Designing is Confluence page
• Design document must contain a  clear "Objective" - where you will be describe why you wish to do automation on what area
• Tracking purpose always use a ticketing tool entry preferred tool Jira
• The design can be breakdown into two levels
• High level design where we will detail about what each task need to be covered
• Low level design where we discuss in-depth ideology on each task along with the possible constraints
•   Usage of global variables (AWX UI use extra vars, host_vars, group_vars etc) discuss their necessity
• AWX/Tower Job template construct possible options as input to handle overall objective, if not sufficient then chain with Workflow constructs
• The execution of Job template every possible option that is a valid to consider it as Test Case
• References Each design document may have some researching requirements that may include your internal company confluence pages or external specific Ansible/AWX technical knowledge articles to help the overall objective.

Playbook Tricks and tips

1. Ansible playbook directory the directory layout always have the playbooks in the Project directory preferable because Ansible/AWX tower can read the vars from that path to parallel directories such as group_vars, vars etc. 
2. Playbook description : First thing fist always add a description comment and in that please mention how this playbook works. Other SRE/DevOps engineer should be able to run without asking you how to run this. 
• In the description also include extra variable in order to execute the playbook in happy path also give clarity on mandatory or optional variables.
• While defining the extra variables first list all mandatory variables and then go for optional
• Better to include tags used in the playbook and their purpose so that end user can easily select for task execution or skip the tasks under certain conditions
• Please add comment before every newly introduced task, that should be highlight the detail process task will do.
3. Writing your Task always have named task, that should give brief understanding about what you will be doing in that task, 
• Task name must have title cased
• Sometime you might be copying the task from other playbooks re-check that name is appropriate or not. 
• If a task have some critical logic must add comment on top of the task the purpose should be described
4. Manage Facts : Ansible's gather_facts directive implies time consuming operations so it is a good practice to disable it, when it is not specifically needed in your play.
5. Don'ts during execution: Do not use LIMIT option as much as possible there could be some plays which will skipped if you use LIMIT option on AWX templates. To resolve this, better option is to use hosts: value assigned to a define variable like targets and have a default value as 'all' or 'localhost as per your play need. When you use the hostvars in a play this limit value could be causes an issue to collect facts from the hostvars.
6. I also recommend to specifically set "any_errors_fatal: true" in all the plays where we can expect/catch the ERRORs during the execution. 
7. If you already defined a play with any_errors_fatal: false then DON'T define the ignore_errors to the same play.
8. When you build the email notification logic in a play for successful flow,  always ensure that you must have failure email as well. You can limit the target audience in case of failure
9. Encrypt the sensitive data content inside a playbook with ansible-vault command
10. Always double check your inventory list with ansible-inventory command with list, graph options
11. In production if all possible use cases tested in non-production then better to reduce the verbose level or suppress the logs this can be done from the task level by using `no_log: "{{ log_suppression| default(true) }}"` at the end of the task definition. Here log_suppression is an Ansible variable  this can be changed at the time of execution the value can be either 'true' or 'false'.
12. While dealing with the when condition in Ansible number validation use the int filter, do not use character value comparisons ( x == '0' ) instead provide the numerical value( x == 0 ).
13. While working with the lineinfile module, if same playbook is triggered from multiple AWX Consoles there could be a race condition and it can be fixed with throttle option.  use Ansible's setting "throttle" set to 1 in the task where lineinfile is executed "The throttle keyword limits the number of workers for a particular task. It can be set at the block and task level. Use throttle to restrict tasks that may be CPU-intensive or interact with a rate-limiting API" Example:

              - name: Updating hostname and uptime in days in file CSV
              lineinfile:
                path: /tmp/uptime-report.csv
                line: "{{ inventory_hostname }},{{ box_uptime }}"
              throttle: 1
              delegate_to: localhost

AWX Admins Tricks

• When you install AWX BETTER version always prefer to use latest -1 version that could be stable and consistent during your installation process.
• If you ae using existing AWX/Ansible job templates to test various Automations requirements/enhancements that we work on:
• We usually need to alter the Job Templates temporarily to performing testing (i.e. we change the Projects, and besides we may add some Limits or Skip tags, or we may use some EXTRA Variables like 'block_reboot' or 'log_suppression') 
• AWX Smart Inventory creation is simple to use only thing you need to know how to use the regex (regular expression) that will bring the combination that satisfies to the existing host list and then it will create the new inventory out of existing inventory hosts. To get all hosts

  name.regex:.*

  To get hosts that start with ec2

  name.regex:^ec2

  To get hosts that contains 'prod'

  name.regex:prod

• Problem: there are good chances, that we may forget to undo all/some of these temporary changes, resulting in broke jobs or incomplete runs 
• Solution(Best Practice): Always create a copy of the job you want to use for testing and alter that copy according to the needs, then just delete the temporary job once test finished 
• making a copy of a job is trivial and will ensure we are not breaking anything in the existing system.
• The latest AWX versions have different weird behavior! AWX 17.1.0 inventory creation from the source control is allowed only when the inventory yaml file-permissions have executable! 

  ## To set the permission in git
  git ls-files --stage
  git update-index --chmod=+x 'name-of-shell-script'
  git commit -m "made a file executable"
  git push
      

AWX Workflow flow control better to use "On Success" because we usually want to stop remaining actions/(in my scenario reboots) when there is a failure encountered (this may cause the affect to multiple/all remaining servers).

Best References:

1. Change file permissions while running 
2. Multi-line Strings in YAML
3. In-memory inventory

Ansible Jinja2 Templates - ansible template module

Here I'm starting this post with famous ARISTOTLE quote.

"We are what we repeatedly do. Excellence, then is not an act, but a Habit"

Welcome back Ansible Automations and who are habituated as Ansible automation specialists this post for them to give boosted walk through, In this post, I would like to share my experiments with Jinja2 templates usage on the Ansible playbook. Of-course Jinja is from Japan it is world famous for templatization capabilities by integrating with multiple languages such as Python, Ruby, Salt talk etc.

In this post we will be cover the following topics:

1. What is Jinja2 template do?
2. Template with filters
3. Template with Lists and sets
4. Template module in Ansible
5. Template with Flow Control
6. Template using Looping
7. Template inheritance**

What is Jinja template do?

Jinja2 is another Python library created for Flask web framework, that comes part of Ansible installation. It is special ability as interpolate or put stuff into YAML Variables(other strings). Those variables can be part of HTML document putting stuff at run-time.

A template is any string of text that contains placeholders {{ }} for the template language, to be able to replace those placeholders by some values. The syntax used for the placeholders and the whole syntax of the template as a whole is known as a template language and the underlying code that evaluates the template and puts the new values in is called a template engine Ansible comes with powerful Jinja2 template language.

 

Let's explore more about Ansible Jinja templates experiments with examples.

Prerequisites

Firstly you must have Ansible installed and the ssh-key authorized for all the managed nodes those are configured inside your targeted environment in the Ansible inventory host list.

How to use string filters in a playbook?

This is common requirement where we can use the different string filters such as: upper, lower, title, replace, default value in the following example.

---
- name: test string values
  hosts: localhost
  vars:
    my_name: 'Pavan Deverakonda the great'
  tasks:
    - name: String filter examples
      debug:
        msg: 
        - "The name is {{ my_name }}"
        - "The name is {{ my_name |upper }}"
        - "The name is {{ my_name |lower }}"
        - "The name is {{ my_name |title }}"
        - "The name is {{ my_name |replace('Pavan','Raghav') }}"
        - "The name is {{ title |default('Mr.') }}{{ my_name }}"
  

After the execution of the string filters the output :

Applying string filters on text in a Ansible playbook

How to play with Jinja2 lists and sets in Ansible?

It is very simple to understand about the Lists and Sets in Jinja2 they can be operated similar to Python Lists and Sets here in the Ansible we need to use them in the YAML as follows:

---
- name: test aggregate values
  hosts: localhost
  vars:
    mylist: [10,20,30]
    x: [8,3,9,5,1]
    y: [44, 5,5]
    words: ["Learning","by","doing"]
  tasks:
    - name: aggregate filter examples
      debug:
        msg: 
        - "The max {{ mylist|max }}"
        - "The min {{ mylist|min }}"
        - "The last {{ mylist|last}}"

    - name: set filters 
      debug:
        msg:
        - "The unique: {{ [2,5,2,9,7,5]|unique }}"
        - "The union: {{ x|union(y) }}"
        - "The another union: {{ [10,3.4]|union([4,2])}}"
        - "The intersect: {{ x|intersect(y) }}"
        - "The intersect: {{ [10,20,30,10]| intersect ([40,30])}}"
        - "The joining words: {{ words|join(' ') }}"

The execution on the lists and sets goes here as output:

Jinja2 Lists and Sets in Ansible playbooks

How to write an Ansible playbook using templates?

It's so simple to use the Jinja2 templates in the Ansible playbooks. To know all the available default ansible variables at the runtime, you can inspect with the following command:

ansible -m setup localhost

It will give you the facts about the localhost. To filtering out to get all the facts which starts with 'ansible_'

ansible -m setup db |grep ansible_

ansible_user details using ansible command setup module

Now, Let's move on to our main goal to playing with the templates in playbook and experiment to know how the template engine helps our examples.
 

Writing Simple Ansible playbook using templates module

Ansible uses Jinja2 templates where it can be lookup inside 'templates' folder, where your playbook runs. All the values of variables will replaced at the time of running the playbook. Runtime value replaces at the time of remote execution.
cat templates/sample-conf.j2

env = {{ env }}
remote_ip = {{ ansible_host }}
remote_user = {{ ansible_user_id}}
remote_hostname = {{ (ansible_fqdn|default(ansible_host)) }}

To test the template values in the playbook write it as :

# File: mytemplate-test.yaml
---
- name: test template values
  hosts: web
  vars:
    env: dev
  tasks:
    - name: template module runs on remote
      template:
        src: sample-conf.j2
        dest: /etc/sample.conf
      become: true

Advanced settings for templates adding the user, group, file permissions with mode attributes.

 ansible-playbook mytemplate-test.yaml -b
 

setup the permissions on the templated files.

Ansible Jinja template experiment

Now connect to the Node1 or Node2 where this hosts are in the play, and check that file content of /etc/sample.conf

Jinja2 template execution with ansible playbook

How to specify "if else" statements in Ansible Jinja2 (.j2) templates? 

Assuming that you are assigning somewhere in the playbook or at the time of launching the Ansible in the extra_vars, in AWX here we go with the "test_var" value with some bool value and you want to test the following is the Jinja syntax to test

- debug:
	msg: "{% if test_var == true %} Good to go!{% else %} Not operational!{% endif %}"

You can also include the ansible facts in the message content.

controlplane $ cat condition1.yml 
- name: Condition test
  hosts: localhost
  gather_facts: no
  vars:
    test_var: yes
  tasks:
    - name: checking if-condition
      debug:
        msg: "{% if test_var == true %} Good to go!{% else %} Not operational!{% endif %}"
	

Simple if-else Jinja template syntax in Ansible playbook

Looping with Jinja templates

Here is a fun looping playbook which will be using a list variable 'myex' that stored all my experiances in the IT industry.

---
- name: Testing Jinja templates with loops
  hosts: localhost
  vars:
    myex: ['Teleparadigm','IBM Global Services','BirlaSoft','Virtusa','Rubicon Red', 'Oracle Corporation','Garmin']
  tasks:
    - name: extracting my experiance
      template:
        src: myex.j2
        dest: /tmp/experiance.txt
  

Jinja template you can store in a file called myex.j2

This is my experience list:

{% for e in myex %}
 
{{loop.index}}. {{ e }}

{% endfor %}
 

When we run the playbook we could see a nice output as:

Loops in Jinja templates in Ansible playbook

Nested loops in Jinja template 

Many Thanks! to Mr. Tim Fairweather RedHat Solution Architect, who shared crazy and funny post "Mastering loops with Jinja templates in Ansible" with this inspiration I've created following nested loop section, where I would like to experiment with the template usage and that too the role created with the ansible-galaxy command as shown:

ansible-galaxy init jinja-templates
  tree jinja-templates/
  jinja-templates/
|-- README.md
|-- defaults
|   `-- main.yml
|-- files
|-- handlers
|   `-- main.yml
|-- meta
|   `-- main.yml
|-- tasks
|   `-- main.yml
|-- templates
|-- tests
|   |-- inventory
|   `-- test.yml
`-- vars
    `-- main.yml

Here is template using playbook my flow of developing code as follows:

1. Create variables which will contain a map/dictionary
2. Define a task that uses template module and generates output
3. Create template that will referring to the variables defined in the step 1
4. Add a configuration to work with the do statement in template
5. Create main playbook to test the nested loop in Jinja Template

Create variables 

Let's create the variables which have dictonaries in a yaml format `vi jinja-templates/vars/main.yml`

---
    inspiring_people:
    - name: Nielgogte
      fav_colour: Blue
    - name: Siva
      fav_colour: Blue
    - name: RanjanShee
      fav_colour: Orange
    - name: Sanjeet
      fav_colour: Orange
    - name: Pankaj
      fav_colour: Yellow
    - name: Prateek
      fav_colour: Blue

  colours:
    - name: Blue
      thinks:
        - Sky is The Limit
        - See as sea into the Depth
    - name: Yellow
      thinks:
        - Expressive
        - Task Oriented
        - Brainstroming
    - name: Orange
      thinks:
        - Proactive
        - Extreme
        - Cares a lot
  

Create task using template module 

Now what we want to perform with this role contained task, we will define under the tasks folder where we can define multiple task `vi jinja-templates/tasks/main.yml`

  - name: Create the Jinja2 based template
	template:
	  src: inspiring.j2 
	  dest: /tmp/impactingMe.out
  

Create template with nested for-loop

See here is the logic that goes fantastic for the nested loops in a Jinja Template `jinja-templates/templates/inspiring.j2`

• Some variable names modified to test how they are working
• Inside the template I've used Jinja filter ' | upper'.

 

---
{% for colour in colours %}
Colour number {{ loop.index }} is {{ colour.name }}.
  {% set colour_count = 0 %}
{% for person in inspiring_people if person.fav_colour == colour.name %}
{{ person.name | upper }}
{% set colour_count = colour_count + 1 %}
     {% do colour.update({'inspiring_people_count':colour_count}) %}
{% endfor %}
Currently {{ colour.inspiring_people_count }} inspiring_people choose color {{ colour.name }} as their favourite.
And the following are their thoughts for {{ colour.name }} :
  {% for item in colour.thinks %}
  -> {{ item }}
  {% endfor %}
{% endfor %}
  

All set to test our new learning that is nested loop within Jinja2 templates in Ansible

---
- name: Demonstrating variables in Jinja2 Loops
  hosts: localhost
  connection: local
  gather_facts: no
  roles:
        - jinja-templates

  tasks:
    - name: display file
      shell: cat /tmp/impactingMe.out
      register: output

    - debug:
        msg: "{{ output.stdout.split('\n') }}"	
  

Here we go... that nice output of-course I've worked on it to get it working almost half day :)

Ansible nested loops in Jinja Templates

Template inherence

Ansible Jinja template inheritance allows you to create the building blocks that you can combine use them. Here is an example of an Ansible playbook that uses Jinja2 templates with inheritance to create an index.html page. As The head template implementation can be 'included' and the body template is extendes in the final index template to generate the complete index.html page.

---
- name: Create index.html page
  hosts: localhost
  gather_facts: false
  vars:
    title: My Ansible experiment Website
    description: A sample website created with Ansible and Jinja2 templates
    keywords: sample, website, Ansible, Jinja2
  tasks:
    - name: Create head template
      template:
        src: templates/head.j2
        dest: head.html

    - name: Create body template
      template:
        src: templates/body.j2
        dest: body.html

    - name: Create index.html
      template:
        src: templates/index.j2
        dest: index.html  
  

The head template, created with templates/head.j2, sample might look like this:

The body template, created with templates/body.j2, sample might look like this:

  
  
Welcome to {{ title }}
  
{{ description }}

 

Finally, here's an example of the index.j2 template, which extends the body.j2 and includes head.html that was generated with `head.j2` templates:

  
{% include "head.html" %}
{% extends "body.j2" %}
  

H A P P Y !!    Template using A U TO M A T I O N S !!

Good Document References:

• KillerCoda Interactive 
• RedHat post on 
• Jinja 2.11.x
• Jinja 3.0.x

Hope you enjoyed this post, Keep sharing your comments on this post.