Iterate Through Arrays with Bash For Loops for Automation Magic

As a Linux system administrator, being able to efficiently process data sets is an invaluable skill for automating tasks. The bash shell offers simple, yet powerful array functionality that unlocks this capability. In this expert guide, we will thoroughly cover bash for loops for iterating through array data, with practical examples of usage along the way.

A Quick Intro to Bash Arrays and Loops

First, what exactly are bash arrays and loops?

Arrays allow you to store multiple data elements under a single variable name. Instead of creating a separate variable for each item, you can use a single array with indexed positions:

fruits=(apple orange banana)  

We can now access each element by its index. For example, to print the second fruit:

echo ${fruits[1]} # orange

Loops let you repeat a section of code multiple times. The bash for loop iterates over a list of values, running once per value:

for i in 1 2 3; do
    echo "Loop iteration $i" 
done

Output:

Loop iteration 1
Loop iteration 2 
Loop iteration 3

When we combine arrays and loops, we get the ability to process every element of an array automatically. Let‘s see it in action.

Displaying Array Contents with a Loop

Let‘s store a list of names in an array, then loop through and print each one:

#!/bin/bash

names=("John" "James" "Sarah" "Mary")

for name in "${names[@]}"; do
    echo $name
done 

When run, this outputs:

John  
James  
Sarah
Mary

The loop iterates through each array index, with $name set to the element value for easy access. No matter how many items are added to the array, this code scales.

Some key points:

• "${names[@]}" expands to all array elements
• The quotes prevent issues with spaces in values
• ${names[index]} also works if you need direct access

Now we can operate on each name inside the loop!

Transforming Array Elements Within the Loop

Building on the previous example, let‘s append a greeting to names:

#!/bin/bash  

names=("Amy" "Brian" "Cathy")  

for name in "${names[@]}"; do
    greeting="Hello $name!"
    echo $greeting  
done

Prints:

Hello Amy!  
Hello Brian!
Hello Cathy!

You can perform any tasks like formatting transformations, writing to files, updating databases, sending API calls, and anything else needed to process array data.

The loop body runs once per element, automating these tasks at scale.

Conditional Logic

Beyond basic operations, we can introduce control flow like if statements for more advanced array processing:

names=("Jim" "Pam" "Dwight")  

for name in "${names[@]}"; do
    if [[ "$name" == "Dwight" ]]; then
        echo "Beets, Bears, Battlestar Galactica" 
    else
        echo "Hello $name"
    fi  
done

Prints:

Hello Jim
Hello Pam  
Beets, Bears, Battlestar Galactica

Now code execution can change dynamically based on the current element value.

Writing Array Elements to a File

Let‘s log our array data to a CSV file for further analysis:

numbers=(12 55 32 71 24)

for n in "${numbers[@]}"; do
    echo $n >> numbers.csv  
done   

This populates numbers.csv with:

12
55  
32
71
24

The redirect >> appends rather than overwrites the file. We have quickly exported array data without needing temporary holders or a database.

Looping Through Associative Array Keys/Values

Associative arrays contain key-value pairs. We can loop through just keys or just values if needed.

Given this array:

declare -A sounds  

sounds[dog]="bark"
sounds[cow]="moo"   
sounds[bird]="chirp"

To print all keys:

for sound in "${!sounds[@]}"; do
   echo $sound
done

# Prints:
dog
cow
bird

The ! provides the array keys rather than the values.

To print all values:

for sound in "${sounds[@]}"; do
   echo $sound
done 

# Prints: 
bark
moo
chirp 

This provides more flexibility when needing to operate on just one or the other.

Multidimensional Arrays

Bash allows arrays containing other arrays. Consider this 2D matrix:

matrix=(
    (1 2 3)
    (4 5 6)
    (7 8 9) 
)

We can access elements like matrix[1][2] = 5.

To loop through all values:

for row in "${matrix[@]}"; do
    for col in "${row[@]}"; do
       echo $col 
    done
done

Prints out each number:

1
2
3
4
5  
6
7
8
9

The nested loop iterates through the outer rows, then inner columns.

This extends well to processing tabular data for reports and analytics.

Associative Arrays

Associative arrays contain key-value pairs for efficient lookups, like dictionaries in other languages.

Define with:

declare -A sounds

sounds[dog]="bark" 
sounds[cow]="moo"
sounds[bird]="chirp"  

We access values via:

echo ${sounds[bird]} # chirp

Checking keys:

animal="frog"

if [[ -z ${sounds[$animal]+x} ]]; then
   echo "$animal sound not found" 
fi

Prints frog sound not found since that key does not exist.

These data structures open many new scripting possibilities!

Practical Examples

While we have covered basic array iteration, you may still wonder how these concepts apply in real shell scripts. Let‘s go through some practical examples.

Read Command Line Arguments into Array

Bash scripts often accept runtime arguments:

#!/bin/bash

files=()

for arg in "$@"; do
    files+=("$arg")  
done

# Operate on files array 

Now the provided filenames get stored for future processing!

Recursive Directory Reading

To recursively traverse a directory structure:

#!/bin/bash

shopt -s globstar nullglob

all_files=() 

for file in **/*; do   
    all_files+=("$file") 
done

# all_files contains full paths recursively

This provides all files under the current directory in an array for easy iteration.

Splitting Strings into Arrays

We can split strings into arrays on a delimiter using IFS and read:

string="apple,orange,banana"

IFS=‘,‘ read -ra items <<< "$string"

echo "${items[1]}" # orange

This splits the string on commas into an array. Great for parsing!

Executing SSH Commands with Arrays

To run SSH commands stored in arrays:

hosts=(
  "server1.example.com"
  "server2.example.com"
)

commands=(
  "uname -a"
  "uptime"
)


for host in "${hosts[@]}"; do
  for cmd in "${commands[@]}"; do 
    ssh $host "$cmd"
  done  
done

This executes each command on each host automatically using native bash process substitution.

Parsing Log Files

To parse a log file into arrays per line:

lines=() 

while read -r line; do
  lines+=("$line")    
done < "app.log"

# Process log data line-by-line  

The while loop populates the array efficiently, avoiding memory issues for large logs.

Randomizing Array Order

To randomize array contents:

nums=(1 2 3 4 5 6 7 8) 

for ((i=0; i<${#nums[@]}; i++)); do
    j=$((RANDOM % ${#nums[@]})) 
    tmp=${nums[i]} 
    nums[i]=${nums[j]}
    nums[j]=$tmp
done

echo ${nums[@]} # Random order!

This fisher yates shuffle algorithm uses swapping to randomize order.

Array Memory Performance

Bash arrays have fast insertion and low memory overhead compared to strings:

Strings:

names=""  
for i in John James Sarah; do
  names="$names $i"  
done

echo $names
# John James Sarah 

Arrays: 

names=()
for i in John James Sarah; do
  names+=("$i")  
done

echo ${names[@]}
# John James Sarah

The string append requires creating new string allocations constantly, becoming inefficient at scale. Arrays only allocate once.

Thus for large data applications, arrays have superior performance.

In Closing

Bash arrays, loops, and their combinations serve as integral tools for nearly any automation task. Whether it is processing command line arguments, traversing directories, transforming data, or exporting to files, arrays simplify the workflow.

I encourage you to explore arrays as fundamental data structures when writing your next shell scripts. Feel free to leave any questions below on creative ways to leverage arrays and loops together in your environment!