Unlocking the Power of jq for Bash Scripting

As a Linux system administrator, working with JSON data is a common task. Whether fetching data from an API or parsing log files, jq is an invaluable tool for manipulating JSON in bash scripts. In this comprehensive guide, we‘ll explore the ins and outs of jq, including installation, basic and advanced usage, and real-world examples.

Introduction to jq

jq is a lightweight command-line JSON processor that allows you to filter, map, and transform structured data with the same ease as sed or awk work on text. Using a simple, declarative language, jq lets you query, read, update, and delete JSON elements without needing to write loops or iterate through data structures.

Some key capabilities of jq include:

• Querying JSON documents to return specific fields/elements
• Transforming JSON data by mapping elements to different values
• Filtering JSON arrays to return only matching elements
• String interpolation for injecting JSON values into text
• Converting JSON into other formats like YAML or CSV

In short, jq gives you the power to manipulate JSON data, which is critical when working with modern APIs, microservices, log files, and more.

Installing jq

Since jq is not part of most Linux distributions, you‘ll need to install it manually. Here are a few common install methods:

Ubuntu/Debian

sudo apt install jq

RHEL/CentOS

sudo yum install jq

macOS (Homebrew)

brew install jq

Verify jq is installed properly:

jq --version

With jq installed, let‘s walk through how to use it.

Reading JSON Data

One of jq‘s simplest uses is to format JSON data into readable output. For example:

{"name": "John", "age": 30, "city": "New York"}

To pretty print this JSON, pipe it into jq:

echo ‘{"name": "John", "age": 30, "city": "New York"}‘ | jq

{
  "name": "John",
  "age": 30,
  "city": "New York"
}

By default, jq pretty prints JSON with indentation and new lines to enhance readability. You can also pass the -c flag to show compact output instead:

echo ‘{"name": "John", "age": 30, "city": "New York"}‘ | jq -c

{"name":"John","age":30,"city":"New York"}

In addition to basic formatting, jq becomes more useful when querying JSON:

cat test.json | jq ‘.name‘

This reads test.json, and outputs only the name value. Later we‘ll dive deeper into filtering.

Querying JSON Data

One of jq‘s killer features is querying JSON documents to return only matching data.

Say we have the following data stored in users.json:

[
  {"name": "Alice", "age": 25, "id": 100},
  {"name": "Bob", "age": 30, "id": 101 },
  {"name": "Claire", "age": 28, "id": 102} 
]

We can query this file for the user Alice like so:

cat users.json | jq ‘.[] | select(.name == "Alice")‘

{
  "name": "Alice",
  "age": 25, 
  "id": 100
}

Breaking down this command:

• jq ‘.[]‘ – Loop through the JSON array
• select(.name == "Alice") – Return only objects where name equals Alice

The select() function is incredibly powerful – it filters JSON objects based on conditions you specify, similar to WHERE clauses in SQL.

Some examples of select() with different operators:

# Users over age 27
jq ‘.[] | select(.age > 27)‘

# Users named Alice or Bob
jq ‘.[] | select(.name == "Alice" or .name == "Bob")‘ 

# Users with ID greater than 101 
jq ‘.[] | select(.id > 101)‘

You can also filter nested JSON fields:

{"person": 
  {"name": "John",
   "address": {
     "city": "New York",
     "state": "NY"
   }
 }
}

To return the city for this person:

cat test.json | jq ‘.person.address.city‘

The filtering power of jq makes it easy to target the exact JSON data you need.

Transforming JSON Data

In addition to querying JSON, jq also lets you transform data by mapping values to different ones.

The map() function is perfect for this:

# users.json
[
  {"name": "Alice", "age": 25}, 
  {"name": "Bob", "age": 30}
]  

For example, to transform user ages:

cat users.json | jq ‘map(.age += 10)‘ 

[
  {"name": "Alice", "age": 35},
  {"name": "Bob", "age": 40}
]

This bumped every user‘s age up by 10 years. Here we used map() to transform the original age data.

Some other examples of data transformation with map():

# Uppercase all names 
jq ‘map(.name |= uppercase)‘

# Set everyone‘s city to London
jq ‘map(.city = "London")‘

# Double all ids 
jq ‘map(.id *= 2)‘

These examples demonstrate how map() can dynamically change JSON values on the fly.

Updating JSON Files

So far we‘ve shown reading, querying, and transforming JSON data with jq. But how about updating JSON files?

# users.json 
[
  {"name": "Alice", "age": 25},
  {"name": "Bob", "age": 30}
]

To update Bob‘s age in-place:

jq ‘.[1].age = 35‘ users.json > tmp && mv tmp users.json

Here‘s how it works:

1. .[1] – Select 2nd JSON object
2. .age = 35 – Set age to 35
3. Redirect output to temp file
4. Overwrite original with updated data

Now users.json contains:

[
  {"name": "Alice", "age": 25},
  {"name": "Bob", "age": 35}  
]

While a bit cumbersome, this process enables modifying JSON without having to parse and rewrite entire files.

Deleting JSON Data

We‘ve shown how jq can update JSON elements, but what about deleting them?

The del() function comes to the rescue:

# users.json
[
  {"name": "Alice", "age": 25}, 
  {"name": "Bob", "age": 30}
]

To remove Bob from the JSON array:

jq ‘del(.[] | select(.name == "Bob"))‘ users.json > tmp && mv tmp users.json

Breaking this command down:

1. select() – Find object where name equals Bob
2. del() – Delete matching element
3. Redirect output & overwrite file

The resulting users.json now looks like:

[
  {"name": "Alice", "age": 25}
]

And Bob was deleted!

Working with JSON APIs

A common way developers use jq is interacting with JSON APIs. jq makes it easy to send API requests and manipulate the responses.

For example, to GET data from https://api.github.com/users:

curl https://api.github.com/users | jq

We can also pipe the response into jq filters:

curl https://api.github.com/users | jq ‘.[] | {name, blog}‘

This returns only the name and blog keys from the JSON response.

To POST data:

# Create JSON payload 
data=‘{"name": "John"}‘

curl -H "Content-Type: application/json" \
     -X POST https://api.example.com/users \
     -d "$data" | jq

This demonstrates jq‘s usefulness in not only fetching, but also creating and updating API resources.

Reducing JSON Data

When dealing with massive JSON documents, it becomes important to condense them down to smaller payloads.

jq‘s reduce function lets you aggregate JSON arrays down to simplified objects.

For example:

[
  {"name": "Alice", "age": 25},
  {"name": "Bob", "age": 30}, 
  {"name": "Claire", "age": 28}
]

To get total and average age with reduce:

jq ‘reduce .[] as $item ({}; .total_age += $item.age | .average_age = ($item.age / length))‘

Giving:

{
  "total_age": 83,
  "average_age": 27  
}

Here reduce() iterates through the array, aggregating age data into a simplified object.

This is useful for generating metrics and summaries from large JSON datasets.

Conditionals Based on JSON Values

jq allows bash-style conditionals based on JSON values using if-then-else:

# user.json 
{"name": "John", "age": 30, "admin": false}

To check if a user is an admin:

jq ‘if .admin then "Is Admin" else "Not Admin" end‘ user.json

"Not Admin"

The if-then-else construct outputs different text strings based on the value of .admin.

Some other examples of conditionals:

# Set role field based on admin 
jq ‘if .admin then .role="admin" else .role="guest" end‘

# Category users by age 
jq ‘if .age < 18 then "Under 18" elif .age < 30 then "Young Adult" else "Over 30" end‘ 

# Check null values
jq ‘if .middle_name == null then "No middle name" else .middle_name end‘

Conditionals give jq logic to make decisions on JSON data.

Reading Environment Variables

A neat trick with jq is injecting shell environment variables into JSON:

export NAME="John"
export AGE=30

echo ‘{"name": $NAME, "age": $AGE}‘ | jq

{
  "name": "John",
  "age": 30 
}

This parses the $NAME and $AGE variables into the JSON document.

Environment variables enable dynamic JSON generation based on bash values.

Some other examples:

export PATH=$PATH:/opt/bin

echo ‘{"path": $PATH}‘ | jq 

export HOSTNAME=$(hostname) 

jq ‘{"server": $HOSTNAME}‘

This technique is perfect for building JSON configs from environments.

Formatting JSON as CSV

jq can also convert JSON into other formats like CSV:

[
  {"name": "Alice", "age": 25},
  {"name": "Bob", "age": 30}
]

To format as CSV:

jq -r ‘.[] | [.name, .age] | @csv‘ data.json  

"Alice",25 
"Bob",30

Here we:

1. Process each JSON row
2. Output only name and age
3. Pass data to @csv function

This pipes the array slices into CSV text.

Adding headers:

echo "name,age" ; jq -r ‘.[] | [.name, .age] | @csv‘ data.json

Gives:

name,age
"Alice",25
"Bob",30 

CSV conversion allows passing JSON to legacy systems.

Advanced Filtering

Up until now, we‘ve shown basic jq filtering with select() and simple comparisons. But what about more complex logic?

jq has a robust set of boolean/array/math/string operations:

startswith(text)
endswith(text) 
contains(text) - check if JSON contains value 

IN(array) - match values in array

floor, ceil - round numbers

ltrimstr(str), rtrimstr(str) - trim whitespace 

and, or, not - boolean logic

And many more!

Here are some examples combine advanced filters:

# Names starting with A or B 
select(startswith(.name; "A") or startswith(.name; "B"))

# Ages between 20 and 30
select(.age >= 20 and .age <= 30)  

# Ids not in given list 
select(.id | not IN([205, 206, 207]))

# Trim last name whitespace
.last_name |= rtrimstr(" ") 

These demonstrate more complex filters than basic == comparisons.

When working with real-world JSON, being able to filter, transform, and manipulate data is critical. jq empowers bash users to tackle these tasks with ease.

Conclusion

jq is an indispensable tool for manipulating JSON data in the shell. With its simple yet powerful querying, filtering, and data transformation capabilities, jq can eliminate massive amounts of parsing/formatting code in bash scripts.

We‘ve really only scratched the surface of what jq can do – be sure to check out the jq manual and GitHub page for even more example use cases and advanced features.

As JSON usage grows across modern APIs, services, databases and log files, having jq skills will be hugely valuable for any Linux admin or developer‘s toolbox. The ability to easily slice and dice through structured data ultimately saves time and unlocks new possibilities.

Give jq shot on your JSON-powered projects – you may be surprised just how many tasks it can help simplify!