Mastering Ansible JSON Query for Effective Data Filtering

JSON has become the lingua franca for data exchange on the web. Its simplicity, readability, and universality make JSON an ideal format for transmitting structured data between web services, servers, and applications.

Ansible, being a prolific automation tool, often needs to parse, process, and filter JSON data from various sources like APIs, file contents, command outputs, etc. This is where Ansible‘s json_query filter comes into play.

In this comprehensive guide, we will explore various practical examples of how to leverage Ansible json_query to slice and dice JSON data with precision for your automation tasks.

Understanding JSON in Ansible

Let‘s first quickly go through some JSON basics:

• JSON stands for JavaScript Object Notation
• Lightweight text-based data format
• Easy for humans to read/write and machines to parse/generate
• Ideal for transmitting data across networks
• Uses JavaScript syntax to structure data
• Common data types in JSON:
  • Number (integers, floats)
  • String (text within quotes)
  • Boolean (true/false)
  • Array (ordered list within square brackets)
  • Object (collection of key-value pairs within curly braces)
  • Null

Here is an example JSON document:

{
  "name": "John",
  "age": 30,
  "skills": ["Python", "Ansible", "Linux"],
  "active": true
}

As you can see, JSON maps directly to common data structures. This makes it a great fit for data exchange.

Now let‘s see how Ansible handles JSON.

Ansible provides various plugins to deal with JSON data:

• from_json filter – Converts JSON string to Ansible data
• to_json filter – Converts Ansible data to JSON string
• from_yaml filter – Converts YAML string to Ansible data
• to_yaml filter – Converts Ansible data to YAML string
• json_query filter – Query/filter JSON data using JMESPath notation

Among these, json_query provides the most flexible capabilities for filtering JSON data.

Introducing JMESPath for JSON Querying

The Ansible json_query filter uses JMESPath under the hood for querying and filtering JSON documents.

JMESPath allows you to select and transform elements from a JSON document. It provides Ansible the power to query JSON data with expressions similar to how you would query a database.

For example, consider the following simple inventory data in JSON format:

[
  {
    "name": "host1",
    "ip": "10.20.30.40",
    "active": true,
    "services": ["http", "smtp", "ssh"]
  },
  {
    "name": "host2", 
    "ip": "10.20.30.41",
    "active": false,
    "services": ["http", "ftp"]  
  }  
]

Here are some example JMESPath queries on this data:

hosts[].name               # Get names of all hosts
hosts[?active==true].ip    # Get IP of active hosts 
hosts[].services[1]        # Get second service of all hosts

As you can see, JMESPath allows:

• Selecting JSON elements using dot notation and square brackets
• Filtering arrays using comparison operators, functions and wildcards
• Output formatting using built-in functions

This provides immense power for querying JSON documents.

Note: JMESPath syntax may remind you of MongoDB queries. But JMESPath is its own API optimized for data filtering use cases.

We have only scratched the surface of what‘s possible with JMESPath. Visit the JMESPath tutorial to explore its full functionality.

Now let‘s see how we can leverage this within Ansible.

Using json_query Filter in Ansible

The json_query filter allows querying JSON data using JMESPath expressions. This works regardless of the JSON source – hardcoded data, vars, register outputs, command/shell outputs etc.

Let‘s start with some examples to see it in action.

Example 1: Query Simple JSON Array

For our first example, we will query a simple JSON array hardcoded in a variable:

- hosts: localhost
  gather_facts: no

  vars:
    my_hosts:  
      [
         {"name": "host1", "group": "web"},
         {"name": "host2", "group": "db"},
         {"name": "host3", "group": "storage"}
      ]

  tasks:

    - name: Fetch all host names
      debug:
        msg: "{{ my_hosts | json_query(‘[].name‘) }}"

This JMESPath query *[].name selects the name key from all objects within the array.

The output will be:

ok: [localhost] => {
    "msg": [
        "host1", 
        "host2", 
        "host3"
    ]
}

Let‘s try another example.

Example 2: Filter Hosts Belonging to a Group

Here we will filter the hosts belonging to the web group:

- hosts: localhost
  gather_facts: no

  vars:
    my_hosts:  
     [ 
        {"name": "host1", "group": "web"},
        {"name": "host2", "group": "db"}
     ]

  tasks:

    - name: Fetch all web hosts 
      debug:
       msg: "{{ my_hosts | json_query(‘[?group==`web`].name‘) }}"

This uses the JMESPath expression [?group==web].name to filter the array.

The ? operator allows filtering the elements in an array. Here we are selecting elements whose group key matches web.

The output filters host1 successfully:

ok: [localhost] => {
    "msg": [
        "host1"
    ]
}

As you can see, combining JMESPath filtering and projections allows extracting precise data from JSON documents.

Example 3: Query Command Output

In addition to hardcoded data, the json_query filter can query the output of any Ansible command/shell module:

- hosts: localhost

  tasks:

    - name: Run ip command
      shell: ip addr show
      register: ip_output

    - name: Fetch public IP 
      debug:        
        msg: "{{ ip_output.stdout | from_json | json_query(‘[?family==`inet`].[addr]‘) }}"

Here we use the shell ip addr show command whose output is JSON formatted. The stdout is registered to parse using json_query.

We first convert the raw text to JSON using the from_json filter. This allows subsequently applying the JMESPath query to fetch the public IP address.

Example 4: Chain Multiple json_query Calls

The json_query filter can be chained to perform complex multi-step querying:

- hosts: localhost

  vars:
    hosts:
      [ 
        {"name": "host1", "services": [{"name": "http", "port": 80}]},
        {"name": "host2", "services": [{"name": "smtp", "port": 25}]}
       ]

  tasks:

    - name: Get service ports
      set_fact: 
        ports: "{{ hosts | 
                    json_query(‘[].services[].port‘) | 
                    json_query(‘[*][]‘) }}"  

    - name: Print ports          
      debug:
        var: ports

Here we first extract all service ports and then further flatten the output into a simple array. Chaining json_query allows constructing flexible transformation pipelines.

The debug output will be:

ok: [localhost] => {
    "ports": [
        80, 
        25
    ]
}

Example 5: Ansible Playbook to Monitor Websites

To conclude, let‘s go through a real-world example playbook showcasing the strengths of json_query:

playbook.yml

---
- hosts: localhost

  vars:

    sites:
      - name: example1.com
        url: https://example1.com
        status_codes: [200, 301] 

      - name: example2.com
        url: https://example2.com
        status_codes: [200]

  tasks:

    - name: Get site status
      uri:
        url: "{{ item.url }}"
        return_content: yes 
      loop: "{{ sites }}"
      register: result

    - name: Set site status
      set_fact:
        site_status: "{{ result.results | 
                           json_query(‘[].{ name: item.name, ok: item.status in codes }‘, {codes: item.item.status_codes}) }}"

    - name: Print status
      debug:
       var: site_status

This playbook performs the following steps:

1. Fetch site pages using uri module
2. Process output and set availability status for every site
3. Print the availability report

Here json_query constructs the desired availability data by:

1. Transforming uri output into required shape
2. Checking if returned status is expected

The output will be:

ok: [localhost] => {
    "site_status": [
        {
            "name": "example1.com", 
            "ok": true
        }, 
        {
            "name": "example2.com",
            "ok": false  
        }
    ]
}

This demonstrates converting raw JSON output into business data using json_query.

Tips for Effective Data Filtering

Here are some tips for unlocking the true power of Ansible‘s json_query:

1. Structure and Register Command Outputs

Many Ansible modules like uri, command, shell etc. output unstructured text data.

Always register these outputs and convert to JSON using from_json before querying:

- command: <...>
  register: result

- set_fact:
    data: "{{ result.stdout | from_json }}"

- debug: 
   msg: "{{ data | json_query(...) }}"

This structures the output and enables flexible JSON-based filtering.

2. Use Dedicated Filter Vars

Avoid cluttering your playbooks with complex json_query filters.

Set the filtered data to a clean var instead:

- set_fact:

    names: "{{ data | json_query(‘[].name‘) }}"

- debug:
    var: names

This improves readability by separating query logic.

3. Reuse Common Query Snippets

Certain filter snippets like [*].ip_address tend to repeat across tasks.

Store them in vars for better reusability:

ip_address_path: "[*].ip_address"

- debug:
    msg: "{{ data | json_query(ip_address_path) }}" 

Bonus tip: Set vars in group_vars/host_vars for global reuse!

4. Use JMESPath Functions

JMESPath provides builtin functions extending filtering capabilities:

• String functions: length(), starts_with(), ends_with() etc.
• Math functions: max(), min(), avg() etc.
• Date functions: date(), time manipulation etc.
• Encoding: base64(), csv() etc.
• Sorting: sort(), sort_by() etc.

Leverage these functions to reduce playbook complexity.

Conclusion

Ansible json_query is an invaluable asset when working with JSON data. It makes easy work of even highly complex JSON processing tasks using the power of JMESPath querying.

We explored various examples like:

• Filtering simple JSON arrays
• Processing command outputs
• Chaining for flexible transformations
• Constructing business data

The key takeaway is structuring unorganized JSON data enables simpler playbooks through precise filtering. Fix the data, not the playbooks!

As a bonus, following JSON querying best practices improves playbook flexibility, modularity and reusability.

JSON is now an integral part of modern web APIs and data pipelines. I hope this guide helps you become a JSON querying ninja boosting your Ansible skills!