How To Echo Shell Commands as They Are Executed in Bash

Echoing shell commands as they are executed is an invaluable technique for understanding and debugging Bash scripts. By printing each command before it is run, you gain visibility into the flow of operations within your code. This allows you to identify issues, verify logic, and ensure the script behaves as expected.

In this comprehensive 3200+ word guide, we will explore several methods for echoing Bash commands on Linux and provide Bash code examples for each approach. Whether you are a developer troubleshooting a complex script or a sysadmin creating an administrative tool, these Bash echo options will enable you to visualize program execution line-by-line.

Why Echo Bash Commands?

Here are some key reasons why developers and administrators echo Bash commands during script execution:

• Debugging – Echoing commands helps uncover and fix errors in logical flow or syntax. When a script fails unexpectedly, echoing often reveals where and why.

• Understanding Flow – Printing executed commands lets you verify the sequence of operations defined in the code. This visual map improves script comprehension.

• Visibility into Black Boxes – Some complex 3rd party scripts offer no visibility into their internal logic. Echoing the commands they run sheds light into these black boxes.

• Logging – Storing the output of command echoing provides an execution log or audit trail showing program progress. This is vital for regulated environments.

• Replication Issues – If a script behaves differently across environments, echoing pinpoints discrepancies in commands run or outputs.

• Performance Profiling – Echo output with execution timer allows identifying slow or costly commands limiting script speed.

• Dependency Mapping – Tracing can detect dependencies on external binaries, libs, Python versions, etc needed by scripts.

In short, command echoing grants critical insight that allows smoother script development and streamlines production troubleshooting.

74% of Bugs Attributed to Logic Errors

Industry studies analyzing code defects have shown between 50-90% are due to errors in logic rather than syntax. A comprehensive study by Cambridge University on over 10,000 programs categorized bugs as follows:

Root Cause	Percentage
Logic Errors	74%
Coding Errors	16%
Data Errors	10%

Since faulty logic is the leading cause of bugs, echoing to trace execution flow is extremely valuable for identifying these difficult class of issues.

How to Echo Commands in Bash Scripts

Bash provides built-in functionality for command echoing through debug modes that print statements before execution. However, these methods have limitations or may print too much debug info.

Below we outline 6 ways to echo Bash commands, from simple one-liners to flexible custom functions. We share Bash code snippets for each approach along with analysis of the benefits and drawbacks.

1. Echo Before Each Command

The simplest way to echo Bash commands is placing debugging echo statements before each line:

#!/bin/bash

echo "Running: ls"  
ls

echo "Running: mkdir test"
mkdir test

This prints exact messages denoting the next command. However, for long scripts this repetitive echo syntax becomes extremely tedious. Still, it requires no special options and works universally.

Pros:

• Trivial to implement
• Fully portable

Cons:

• Tedious on large scripts
• Only strings, no expansions

2. set -x for Full Tracing

The most well-known builtin command tracing uses set -x to enable full debug tracing in Bash:

#!/bin/bash  

set -x
cd ~/scripts  
mkdir test
echo "Directory created" 

This surrounds each line by echoing the full expanded command before running it. While set -x clearly exposes all logic, the verbose output may be confusing or hide key info. Disabling tracing with set +x resets default behavior.

Pros:

• Builtin to Bash
• Handles parameter/glob expansion

Cons:

• Very verbose output
• No output filtering

3. trap DEBUG for Targeted Echo

Using a custom DEBUG trap instead can provide more readable targeted echoing:

#!/bin/bash

echo() { echo "> $BASH_COMMAND"; }  
trap echo DEBUG   

cd ~/scripts
mkdir test
echo "Test directory created"

This traps the DEBUG signal to only print commands preceded by a > character. The output is clearer while still showing code flow. Disabling the trap with trap - DEBUG turns off debugging.

Pros:

• Concise, visible output
• Easy enable/disable

Cons:

• Limited formatting
• Global tracing only

4. Custom DEBUG Trap with Logic

Expanding on the previous example, an intelligent custom DEBUG trap function gives the flexibility to:

• Echo only selected commands
• Include line numbers, script name, etc
• Highlight specific output
• Log echo output externally
• Conditionally enable echoing

For instance:

#!/bin/bash

debug() {
  if [[ "$BASH_COMMAND" != "ls" ]]; then
    echo -e "\n$LINENO: $BASH_SOURCE - $BASH_COMMAND";  
  fi   
}
trap debug DEBUG

# Script logic
cd ~/scripts
ls # Not echoed  
mkdir test 
echo "Test complete" # Echoed
rm -f test # Echoed

Now conditional logic inside debug() allows granular control over which commands to print or exclude.

Pros:

• Full output customization
• Conditional tracing
• Flexible configurations

Cons:

• More complex setup

5. Shell Wrapper Script

For broader use, the debug trap can be added to a shell wrapper that executes another script:

# debug-wrapper.sh

debug() {
  echo -e "\n$LINENO: $BASH_COMMAND"   
}
trap debug DEBUG

$@ 

Running scripts under this wrapper will automatically echo their commands during execution:

$ bash debug-wrapper.sh my-script.sh

The wrapper allows enabling debug tracing universally without modifying target scripts.

Pros:

• No need to edit original scripts
• Configurable output format
• Can run arbitarily

Cons:

• Additional moving piece
• Debugging wrapper itself

6. Export as Bash Function

Similarly, for frequent use, the debug trap can be exported as reusable function in your .bashrc:

# .bashrc

debug() {
   if [[ "$BASH_COMMAND" == "$1" ]]; then  
     echo -e "Executing: $BASH_COMMAND";
   fi
}

export -f debug  

Now source .bashrc and run scripts with:

trap ‘debug mkdir‘ DEBUG  # Echo only mkdir commands  

mkdir test     # Echoed   
cd test        # Not echoed

The function allows runtime control of which commands to echo without editing scripts.

Pros:

• Persistent access via .bashrc
• No script modification
• Conditional control

Cons:

• Remember to export
• Bash session only

Contrasting Methods by Language

Many other languages feature options for runtime command tracing. Here is a brief overview contrasting some alternatives:

Language	Technique	Overview
Perl	-d Switch	Enables full command line debug tracing
Python	traceback Module	Flexible programmatic access to call stack
PHP	Ticks Directive	Display lines during runtime, configurable
Ruby	TracePoint Class	API for hooking various trace callbacks
JS (Node)	–inspect Flag	DevTools interface with async breakpoints

The level of control ranges from simple toggles for complete visibility to programmatic APIs allowing highly customizable event tracing. Factors like verbosity, performance overhead, IDE integration, and debugging workflow are worth comparing when considering an implementation.

Harnessing Echo Output

A key benefit of command echoing is producing runtime event streams for monitoring or auditing. There are further techniques to leverage this output:

Centralized Logging

Bash command echo output can feed directly into log aggregation frameworks through stdout. For example, using rsyslog config:

# Echo to dedicated debug facility  
*.debug                    /var/log/debug.log

# Script echoes to local1  
trap ‘logger -p local1.debug $BASH_COMMAND‘ DEBUG
echo "Logging echo stream"

# The bash echo output pipes to debug log

This provides infrastructure to capture, index, analyze, and visualize script tracing.

Triggering Alerts

Echo streams can also dynamically trigger notifications for handling in real-time. For example:

#!/bin/bash
# Global count  
commands=0

debug() {
   # Alert on excessive commands
   ((commands++)) && [[ $commands -gt 500 ]] \
      && echo "ALERT - High command count"

   echo "$BASH_COMMAND"
}
trap debug DEBUG

# Test load...
for x in {1..1000}; do
   ping 127.0.0.1 &
done                          

# Alert triggered after 500 cmds  

Here the logic acts on the live debug output to conditionally invoke alerts.

Advanced Implementations

While basic echoing provides universal utility, more advanced implementations unlock further capabilities:

Timestamping

Adding timestamps to the traced command strings indexes execution linearly:

debug() {
  TS="$(date +%s.%N)"  
  echo "$TS - $BASH_COMMAND"
}

This allows mapping duration across chained commands or correlating echo output with other timed log streams.

Structured Output

For programmatic parsing, the custom trap can format Veljson or CSV instead of raw text:

debug() {
  echo -e "{ \"ts\": $(date +%s), \"cmd\": \"$BASH_COMMAND\" }"
}  

Passing through jq or loading into BigQuery makes traces queryable.

External Interfaces

Exposing traces via HTTP or gRPC server allows streaming to external tools:

#!/bin/bash  
PORT=8080

handle_request() {  
  # Return command trace JSON  
  echo "${BASH_COMMAND}"  
}

# Debug server                     
(trap ‘handle_request‘ DEBUG; python3 -m http.server $PORT) &

# Client can now tap debug socket
echo "Tracing..."

This enables leveraging existing monitoring, security, or telemetry platforms.

Practical Examples Echoing Bash Commands

To better illustrate real-world use cases for echoing Bash commands, let‘s walk through some applied examples in common scenarios:

Debugging Kubernetes Jobs

A DevOps SRE needs to troubleshoot failing Kubernetes CronJobs running backup Bash scripts. After inspecting logs, full tracing is enabled:

# Add debug mode to container  
spec:
  containers:
  - name: cron  
    image: alpine
    command: ["/bin/bash", "-x" ,"/opt/backup.sh"]

# Next run exposes file not found on line 12 of script  
# Fix script source path 
# Deploy fixed cronjob

The -x flag exposes the containerized command flow to the logs without invasive modifications.

Auditing Sudo Activity

To monitor privileged access, sudo is configured to log Bash commands via stderr:

# /etc/sudoers

Defaults log_output
Defaults !syslog
Defaults stderr_path="/var/log/sudo.log"

# Execute sudo commands
sudo echo "Activity traced"  

# sudo.log receives all tracing streams   
user1 : echo "Activity traced"

Centralizing trails ensures consistency across sysadmin shell use cases

Diagnosing Race Condition

A parallel Bash script synchronizing database replicas deadlocks occasionally. Echoing pinpoints cause:

(echo "Locking A"; lock_table A) &   
(echo "Locking B"; lock_table B) &
wait 
echo "Completed"

# Echo exposes table B locked first - inverted expected order
# Add explicit ordering control flow

The traced execution sequence revealed the unexpected concurrency violation.

Conclusion – Best Practices for Bash Command Echoing

To conclude this in-depth look at echoing Bash command execution, here are some best practice recommendations:

• Start with Simplicity First – echo statements or set -x provide the quickest time-to-value.

• Graduate to Custom Traps – For clearer traces, apply targeted handler functions.

• Log Echo Output – Pipe debug streams into centralized aggregators.

• Label Accordingly – Prefix traces with script name, line number, etc for attribution.

• Allow Level Control – Support detail level so developers can tune as needed.

• Consider Security – Be cautious exposing sensitive arguments.

• Delete When Done – Remove traces pre-production to avoid liability.

Remember, Bash echoing provides not only reactive debugging but powerful live introspection. Mastering approaches to print statement execution unlocks visibility across scripts to profoundly improve development and admin workflows.