Mastering "set -e" for Bulletproof Bash Scripts

As a professional Linux engineer, building robust and secure Bash scripts that fail fast is a critical priority. The built-in set -e option is an invaluable tool for script error handling. When combined with other best practices, it enables catching failures before they turn to catastrophes.

This comprehensive 4-part guide takes an expert look at set -e, when to use it, how it works internally, pro tips for avoiding pitfalls, and real-world use case examples you can apply right away. Follow along to level up your Bash fu!

Part 1: Understanding Bash Script Failures

63% of Bash scripts contain at least one potential error source per 50 lines of code, per analysis by EMC CodeCare. Bugs usually stem from unchecked assumptions, race conditions, unexpected inputs, and unsupported operations.

Common causes include:

• Uninitialized variables
• Empty inputs
• Missing commands
• Incorrect paths
• Full disks
• Endless loops
• Overflow issues

These can lead to:

• Inconsistent side effects
• Insecure exposures
• Resource leaks
• System instabilities
• Data loss
• Crashes

Without rigorously handling errors, scripts may fail silently or continue running in invalid ways. This leads to scrambled output, incorrect logic, hidden issues, and cascading problems.

Clearly, production Bash scripts require mechanisms to exit early when failures occur before real damage takes place.

Part 2: Employing "set -e" for Better Error Handling

This is where set -e comes in – it directs the shell to immediately exit when any command returns a non-zero exit code.

The default Bash behavior is to continue executing after a command fails. But set -e overrides this by telling Bash to halt execution right after a failure code is detected.

Let‘s contrast the default and set -e behaviors:

#!/bin/bash

# Default behavior 
false
echo "Continues with invalid state"

# With set -e
set -e 
false
echo "Will not reach here" 

Output:

Continues with invalid state 

# Exits directly after false

set -e transforms error handling to quit directly rather than march ahead blindly. This prevents unvalidated operations from accumulating issues below the surface.

According to Google‘s Shell Style Guide, using set -e leads to cleaner code by reducing error checking boilerplate throughout a script.

The Bash Hackers Wiki recommends always enabling set -e in scripts destined for production.

Overall, set -e encourages developing strict Bash code that fails fast. Let‘s explore even more benefits.

Why Rigorously Handle Errors?

Beyond the statistics, there are 3 key motivators for rigorous error handling practices like set -e:

1. Encourages Fixing Bugs

Errors handled early on prompt developers to fix the underlying bugs before release rather than allow them to slip through.

2. Results in Resilient Software

Baking in error handling practices leads to software that anticipates mistakes and withstands unforeseen situations through built-in redundancy and controls.

3. Maximizes Uptime

The fastest way to resolve an issue is to have the software proactively halt and alert you something is wrong. This minimizes damage and downtime.

In short, rigorously handling errors with set -e leads to more secure and resilient software essential for production environments.

Part 3: Advanced "set -e" Usage and Insights

While set -e provides a baseline of protection, truly bulletproof Bash scripts require additional considerations:

Working With Subshells

set -e only applies to the current shell instance. When launching subshells, ensure to propagate the setting:

set -e

(
  # set -e not inherited by default
  false
)

# Fails correctly  
false

Use set -e explicitly in subshells for consistency:

set -e

(
  set -e # Propagate to subshell
  false  
)

false # Also fails correctly  

Alternatively, make set -e inherited with set -E.

Handling Pipelines Properly

With pipelines, set -e only detects the final stage failure:

set -e
false | true # Will not exit!

This is because Bash allows pipelines to return error codes from the last command only.

To fail the whole pipeline immediately, enable set -o pipefail:

set -e
set -o pipefail
false | true # Exits properly now  

Employing Error Traps

While set -e halts execution, we may want to run logic beforehand. The trap command lets us "catch" the failure:

trap ‘echo "Error on line $LINENO"‘ ERR 

bad_command
echo "Never reached"

This prints the exact line number of the failure – very handy!

Traps let us log, clean up resources, send alerts, etc. without the user needing to see the gory details.

Combining With OtherOptions

set -u is commonly paired with set -e to also fail on referencing unset variables:

set -eu

# Both trigger exits:
false
echo $UNDEFINED

We can further chain on set -x to print executed commands or set -v to print raw input lines as read by the parser. This unlocks debugging capabilities.

Part 4: Coding Best Practices

While set -e improves error handling, beware of several gotchas:

1. May Exit on Informational Messages

Tools like grep or ls sometimes report non-zero codes even when not really failing. Use option flags like grep -q to ignore informational codes.

2. Doesn‘t Work Inside Logical OR

set -e inside a logical OR shortcut will NOT exit early as expected:

false || true # Keeps running!

Rely instead on && and explicit checking.

3. Traps Won‘t Halt Without Explicit Exit

Trapped errors don‘t actually abort execution – so ensure to explicitly call exit within trap handlers.

4. Subshells Lose State on Exit

Subshells and co-processes fork a separate process state which gets lost on exit. Manage expectations when handling their errors.

So while set -e helps make scripts strict, also:

• Validate inputs with assertions
• Leverage static analysis
• Prefix commands that expect failure with -
• Use status checks for expected non-zero codes
• Employ timeouts to break endless loops

This "defense in depth" approach prevents you from having to untangle downstream issues triggered by a crash further in the code.

And here are some example patterns pulling together everything we‘ve covered:

Robust Function

do_something() {
  set -Eeuo pipefail
  trap ‘echo "Failed due to $BASH_COMMAND" >&2‘ ERR

  # Main logic...

  return 0  
}

Self-Testing Script

for method in "foo" "bar"; do
  assert_"$method" || exit 1
done  

main "$@"

Resilient Pipeline

set -Eeuxo pipefail

generator | 
  processor |
  ${analyser} | > output.txt

echo "Finished successfully"

Conclusion

Like any sharp tool, set -e can cause damage if used carelessly in scripts.

However, combining set -e with other practices outlined creates an indispensable level of early failure detection. This prevents ignored errors from snowballing into systemic outages further into production.

As Linux professionals and expert coders, we have an obligation to build resilient bash that idles stable and responds gracefully. Set -e is the cornerstone supporting this industry-grade level of software reliability.

So use set -e judiciously, trap before exiting, validate assumptions, and default to crashing early. This commitment to robustness protects our users, customers, and business from dangerous bash script compromises lurking within seemingly innocuous bugs.