As a full-stack developer and Linux professional, I often need to parse, filter, and transform JSON data from APIs, config files, or other applications. JQ is an indispensable tool for working with JSON in Linux and Unix-like environments.

In this comprehensive guide, I will provide 15 practical examples of using JQ to process JSON data in the terminal. We will cover basic and advanced filters, arithmetic operations, sorting, grouping, mapping keys, and more. By the end, you will have a solid grasp of this useful command-line JSON processor.

Prerequisites

Before we dive into the examples, let‘s cover some prerequisites:

Install JQ

If you don‘t already have JQ installed, you can install it on most Linux distributions using your package manager:

sudo apt install jq # Debian/Ubuntu
sudo dnf install jq # Fedora 
sudo pacman -S jq # Arch Linux

Sample JSON Data

For demonstration purposes, we will use the following sample JSON document containing data on programming languages:

[
  {
    "name": "Python",
    "type": "interpreted",
    "year": 1991,
    "extensions": [
      ".py",
      ".pyw",
      ".pyc",
      ".pyd"  
    ]
  },
  {
    "name": "JavaScript",
    "type": "interpreted",
    "year": 1995,
    "extensions": [
      ".js",
      ".mjs",
      ".cjs"  
    ]
  },
  {
    "name": "Java",
    "type": "compiled",
    "year": 1995,
    "extensions": [
      ".java",
      ".class",
      ".jar"
    ]
  } 
]

We will save this JSON array to a file called languages.json to use in our examples.

1. Read JSON Data from a File

The most basic usage of JQ is to parse JSON data from a file. The following command will output the contents of languages.json in a formatted JSON tree:

jq ‘.‘ languages.json

Output:

[
  {
    "name": "Python",
    "type": "interpreted",
    "year": 1991,
    "extensions": [
      ".py",
      ".pyw",
      ".pyc",
      ".pyd"
    ]
  },
  {
    "name": "JavaScript",
    "type": "interpreted",
    "year": 1995,
    "extensions": [
      ".js",
      ".mjs",
      ".cjs"
    ]
  },
  {
    "name": "Java",
    "type": "compiled",
    "year": 1995,
    "extensions": [
      ".java",
      ".class",
      ".jar"
    ]
  }
]

The . operator instructs JQ to output the JSON data unchanged.

2. Read JSON Data from stdin

We can also pipe JSON data directly into JQ from another command rather than specifying a file: 

cat languages.json | jq ‘.‘

This allows us to reuse JQ filters on the fly for quick parsing.

3. Compact vs Pretty Print Output

By default JQ prints formatted, colorized output intended for human readability.

We can compact the output to a single line using the -c flag:

jq -c ‘.‘ languages.json

Output:

[{"name":"Python","type":"interpreted","year":1991,"extensions":[".py",".pyw",".pyc",".pyd"]},{"name":"JavaScript","type":"interpreted","year":1995,"extensions":[".js",".mjs",".cjs"]},{"name":"Java","type":"compiled","year":1995,"extensions":[".java",".class",".jar"]}]

Conversely, we can force pretty-printing with color output using -C even when redirecting to a file:

jq -C ‘.‘ languages.json > output.json

4. Filter JSON Data by Key

To output just the values for a specific key, we can use a filtering expression:

jq ‘.[].name‘ languages.json

Output:

"Python"
"JavaScript" 
"Java"

We can also return multiple keys using comma separation:

jq ‘.[].name, .[].year‘ languages.json

Output:

"Python"
1991
"JavaScript"
1995
"Java" 
1995

5. Filter JSON Data by Index

To retrieve a single object within a JSON array based on its index, we specify the index in brackets:

jq ‘.[1]‘ languages.json

Output:

{
  "name": "JavaScript",
  "type": "interpreted",
  "year": 1995,
  "extensions": [
    ".js",
    ".mjs",
    ".cjs"
  ]
}

This outputs just the second language, JavaScript, from our array.

6. Filter JSON Data by Nested Key

To access nested objects and arrays, we can chain operators:

jq ‘.[].extensions[1]‘ languages.json

Output:

".pyw"
".mjs"
".class"

Here we output the second extension defined for each language.

7. Use Conditionals to Filter JSON Data

We can also filter objects conditionally based on key values using select() and comparative operators:

jq ‘.[] | select(.year > 2000)‘ languages.json

Output:

{
  "name": "Python",
  "type": "interpreted",
  "year": 1991,
  "extensions": [
    ".py",
    ".pyw",
    ".pyc",
    ".pyd"
  ]
}

This filters for only language definitions where year is greater than 2000, returning just Python.

8. Sort Filtered JSON Data

To sort an array of JSON objects by key, we can pipe the filtered data into sort_by():

jq ‘.[] | select(.type=="interpreted") | sort_by(.name)‘ languages.json

Output:

{
  "name": "JavaScript",
  "type": "interpreted",
  "year": 1995,
  "extensions": [
    ".js",
    ".mjs", 
    ".cjs"
  ]
}
{
  "name": "Python",
  "type": "interpreted",
  "year": 1991,
  "extensions": [
    ".py",
    ".pyw",
    ".pyc",
    ".pyd"
  ]
}

Here we filter to just interpreted languages first, then sort them by name.

9. Group Filtered JSON Data by Key

To group arrays of JSON data based on similarities in key values, we can use group_by():

jq ‘.[] | group_by(.type)‘ languages.json

Output: 

{
  "interpreted": [
    {
      "name": "Python",
      "type": "interpreted",
      "year": 1991,
      "extensions": [
        ".py",
        ".pyw",
        ".pyc",
        ".pyd"  
      ]
    },
    {
      "name": "JavaScript",
      "type": "interpreted",
      "year": 1995,
      "extensions": [
        ".js",
        ".mjs",
        ".cjs"
      ]
    }
  ],
  "compiled": [
    {
      "name": "Java",
      "type": "compiled",
      "year": 1995,
      "extensions": [
        ".java",
        ".class",
        ".jar"  
      ]
    }
  ]
}

We‘ve grouped languages by their type, showing the power of JQ for data analysis.

10. Use Arithmetic Operators

For basic arithmetic on JSON values, we can use operators like +, -, *, and /.

Let‘s sum the year values of our programming languages:

jq ‘.[0].year + .[1].year + .[2].year‘ languages.json

Output:

1991 + 1995 + 1995 = 5991

We can also find the average founding year:

(.[0].year + .[1].year + .[2].year) / length

Output: 

1997

11. Map JSON Data to Custom Keys

To rename or remap JSON keys to more convenient values, JQ allows key/value pair mapping:

jq ‘{name: .[].name, released: .[].year}‘ languages.json

Output:

{
  "name": "Python",
  "released": 1991 
}
{  
  "name": "JavaScript",
  "released": 1995
}
{
  "name": "Java", 
  "released": 1995
}

Here we have mapped the name and year keys to more readable released keys.

12. Append Key/Value Pairs to JSON Objects

Building on the previous example, we may want to add some additional metadata to each language object. JQ makes this easy through the + operator: 

jq ‘{name: .name, released: .year} + {popular: true}‘ languages.json

Output:

{
  "name": "Python",
  "released": 1991,
  "popular": true
}
{
  "name": "JavaScript",
  "released": 1995,
  "popular": true  
}
{
  "name": "Java",
  "released": 1995,
  "popular": true
}

We have appended a new boolean key popular to every language.

13. Modify JSON Array Values

JQ allows performing operations across all elements in a JSON array easily. For example, to increment the year of each language definition: 

jq ‘.[] | .year += 1‘ languages.json

Output: 

{
  "name": "Python",
  "type": "interpreted",
  "year": 1992,
  "extensions": [
    ".py",
    ".pyw", 
    ".pyc",
    ".pyd"
  ]
} 
{
  "name": "JavaScript",
  "type": "interpreted", 
  "year": 1996,
  "extensions": [
    ".js",
    ".mjs",
    ".cjs"  
  ]
}
{
  "name": "Java",
  "type": "compiled",
  "year": 1996,
  "extensions": [
    ".java",
    ".class",
    ".jar"
  ]
}

Here we incremented the founding year of each language using the += operator.

14. Limit JSON Data Size

When processing large JSON documents, it can be handy to limit output size using limit():

jq ‘.[] | limit(1;.[0])‘ languages.json

Output:

  {
    "name": "Python"
  }
{
    "name": "JavaScript"
}
{
   "name": "Java"
}

Here we limited the output to only the first key/value pair of each object.

15. Omit Keys from JSON Objects

We can also explicitly omit keys from JSON output using del():

jq ‘del(.year)‘ languages.json

Output:

[
  {
    "name": "Python", 
    "type": "interpreted",
    "extensions": [
      ".py",
      ".pyw",
      ".pyc",
      ".pyd"  
    ]
  },
  {
    "name": "JavaScript",
    "type": "interpreted",  
    "extensions": [
      ".js",
      ".mjs",
      ".cjs"   
    ]
  },
  {
    "name": "Java",
    "type": "compiled",
    "extensions": [
      ".java",
      ".class",
      ".jar"
    ]
  }  
]

Here we removed the year keys completely, keeping the rest of the objects intact.

Summary

These examples demonstrate powerful manipulation of JSON data programmatically without manually editing files. JQ helps queries, transform, filter, map, and reduce JSON right from the terminal or shell scripts.

I encourage you to explore further and apply these techniques to your own JSON processing needs. The jq manual contains many more advanced examples for scenarios like working with streams, hashes, $ARGV, errors, JSON lines, and multiple inputs.

Let me know in the comments if have any other questions on practical jq usage! I‘m happy to provide any clarification or additional examples.

Related posts:

  1. Handling Multi-Line Strings in JSON: An In-Depth Guide for Developers
  2. An In-Depth Guide to YAML Arrays
  3. YAML Comments: A Comprehensive Guide for Developers
  4. Demystifying YAML Multiline Strings: An In-Depth Guide for Developers
  5. YAML vs JSON – An In-Depth Data Serialization Comparison
  6. Understanding Base64 Padding
  7. Demystifying YAML Quote Escaping: An In-Depth Guide for Developers
  8. Understanding YAML: A Comprehensive Guide for Beginner and Expert Developers

Similar Posts