Replacing Strings in Files with Bash Scripting

Searching and replacing text patterns is a ubiquitous task that arises across many phases of the data lifecycle. From software development and IT operations to data engineering and analytics, processing text is fundamental. Industry surveys indicate unstructured data doubling yearly – much of it human-generated formats like log files, documents, and emails containing strings in need of standardization, anonymization, or correction [1]. This data abundance containing vital embedded information makes text manipulation skills ever more relevant.

Let‘s explore the primary techniques for replacing strings in files using Bash scripts – a critical tool for the processing, transformation, and enhancement of unstructured data.

Why Replace Strings in Files?

Here are some common use cases where substring replacement in files comes in handy:

• Changing configuration values like database credentials or API keys
• Anonymizing log files by removing personal information to meet GDPR compliance
• Fixing invalid or corrupt data by standardizing formats
• Translating text files by substituting words/phrases
• Trimming logs by filtering unnecessary entries to optimize analytics
• Rewriting file paths referenced within a codebase during CI/CD migrations

Industry research shows average enterprises generating over 15 terabytes daily across thousands of log files and databases [2]. Manual inspection and modification isn‘t pragmatic. Bash scripting allows search/replace transformations to be automated at scale.

Sample File

To demonstrate, we will start with a sample text file data.txt:

Date,Sales,Agent
01/05/2023,10000,John
01/15/2023,8000,Jane  
02/28/2023,12000,Alice

Our script will:

• Replace the ‘Sales‘ header with ‘Revenue‘
• Standardize the date format from MM/DD/YYYY to YYYY-MM-DD
• Swap the Agent name ‘John‘ to ‘Mary‘

Replacing with sed

The sed utility is ideal for simple substring replacement. Its substitution command has the syntax:

sed ‘s/find/replace/‘ file

Let‘s make our first edit:

sed ‘s/Sales/Revenue/‘ data.txt

The change prints to standard output rather than altering the file. To overwrite data.txt, use the -i flag:

sed -i ‘s/Sales/Revenue/‘ data.txt  

The g option makes the substitution global, replacing all instances instead of just the first match per line:

sed -i ‘s/01\//2023-/g‘ data.txt 

Our regular expression matches the date format, which is then replaced with the reformatted YYYY-MM-DD version.

Case insensitivity can be enabled with I:

sed -i ‘s/John/Mary/I‘ data.txt

**For large log files, wrapping sed in a Bash loop can speed up substitutions:

for file in /var/logs/* 
  do
    sed -i ‘s/foo/bar/g‘ "$file" 
  done

This translates to significant time savings compared to running repeatedly by hand.
**

Replacing with awk

The venerable awk also offers excellent string manipulation:

awk ‘{sub(/John/,"Mary")}1‘ data.txt > output.txt

This makes use of awk‘s sub() function to replace "John" with "Mary", before printing each updated line with the 1.

Since awk works with standard streams rather than files, we write the changes to a separate output.txt. To make edits directly in-place, we‘ll redirect back to our original:

awk ‘{sub(/\$[0-9]+/, ""); print }1‘ log.txt > tmp.txt && mv tmp.txt log.txt

Here we scrub monetary values from a financial log by removing $numbers with empty strings.

Additional handy awk capabilities:

awk ‘{gsub(/[aeiou]/,"x")}1‘ file > file.txt # Replace multiple letters 

awk ‘NR % 2 == 0‘ log.txt > even.txt # Filter to even lines

awk -F, ‘{print $1","$3}‘ file.csv > newfile.csv # Reorder columns

Pure Bash Substitution

Bash itself can accomplish search/replace through parameter expansion:

while read line; do
  edited=${line/Jane/Susan}  
  echo $edited
done < data.txt > tmp.txt && mv tmp.txt data.txt  

The format ${var/find/replace} substitutes the first match of "find" with "replace" inside var. Our while loop applies this to each $line, printing the output before renaming tmp.txt back to the original file.

**We can also iterate across files:

for file in *.txt; do
  # Back up original
  cp "$file" "${file}.bak" 

  # Modify file
  while read line; do  
    edited=${line/foo/bar}
    echo $edited >> "$file"
  done < "$file"

done

Here we create .bak versions before substituting and overwriting the content.
**

One-Line Perl Magic

Perl packs powerful text processing into compact scripts and one-liners:

perl -i.bak -pe ‘s/Sales/Revenue/g‘ data.txt

Here -i.bak performs the edits in-place while saving originals to file.bak. The -p wraps the code in a printing loop, while -e allows Perl inline. This replaces all Sales with Revenue using familiar syntax.

More Perl examples:

perl -pe ‘s/JAN|FEB|MAR/Q1/g‘ log.txt > q1_lookup.txt

perl -i -pe‘s/localhost/127.0.0.1/g‘ my_conf_dir/

perl -0777ne ‘s/foo/bar/gs‘ one_big_file 

The first maps shorthand months to quarters for analyis. The second handles mass configuration change. The last operates paragraph-wise on a giant log thanks to Perl‘s powerful regex.

Special Cases and Best Practices

Escaping – Special characters like . [] ^ $ | * + ? { } ( ) \ may need escaping for literal matches.

Binary Files – These techniques are geared for text substitutions. Binary replacement requires bit-level utilities.

At-Scale Changes – Large updates should utilize version control, backups, and skepticism. Verifying a representative sample of changes in staging can instill confidence.

Optimizing – For big files or resource constraints, try streaming line-by-line rather than memory-intensive bulk editing.

Validation – Once strings are replaced, what checks will we run to ensure correctness and prevent data corruption? Post-processing validation is key.

In Closing

Bash scripting boasts versatile facilities for finding and replacing text in files, while interacting easily with an ecosystem of Linux utilities. Keeping your toolbox stocked with this range commands widens your vantage for tackling text processing tasks. Know when to apply brute force regular expressions, or reach for subtler instruments. With practice, shell scripting allows you to conduct an orchestra of data transformations and mutations attuned to your specific needs. Master the art, avoid the tedium, and may your find/replaces run frictionless as a hot knife through butter!