Pipes in Linux Explained

Pipes are syntactical glue that allows one command's STDOUT (standard output) to work as STDIN (standard input) of the following command. In simple words, pipes connect two or more commands, scripts, utilities, or programs. Shell uses a vertical bar (|) symbol to represent a pipe.

Commands, scripts, utilities, programs, and processes work in three stages. These stages are taking an input, processing the input, and returning the output (processed input). A pipe connects a command's third stage to the following command's first stage.

Let us take an example.

Suppose we want to process data through three commands in such a way that the output of the first command works as the input of the second command, and the output of the second command works as the input of the third command. In such a case, we have to go through the following steps.

• Save the first command's output to a temporary file.
• Instruct the second command to use the temporary file saved in the first step as the input.
• Save the second command's output to another temporary file.
• Instruct the third command to use the temporary file containing the second command's output as the input.
• Specify a custom or use the default output device for the third command.

The more commands we add to data processing, the more complex it becomes. Each added command requires a separate temporary file. Pipes make this process easier. Pipes connect the output from one command to the input of another command. In other words, instead of sending the output of a command to a destination file or device, pipes send that output to another command as input.

Examples of pipes

Following are examples of how administrators use pipes.

• Alphabetically sorting all contents of the current directory.
• Counting the number of files in a directory.

Alphabetically sorting all contents of the current directory

The ls command lists all contents of the specified directory. The sort command sorts the contents of the specified directory. We can use both commands separately or together by using a pipe sign. To use both commands separately, execute the ls command, save its output in a temporary file, and then execute the sort command. Specify the temporary file as the input file to the sort command.

$ls > tempfile
$sort tempfile

Now, let's do the same task using pipes. Using pipes is simple. Just use the pipe sign (|) between two commands you want to connect in such a way that the first command's output works as the second command's input. In our example, we want to use the output of the ls command as the input of the sort command. To do this, place a pipe sign (vertical bar character) between both commands to connect them.

$ls | sort

The pipe operator receives output from the ls command placed before the pipe and sends this data as input to the sort command placed after the pipe. The following image shows the output of the above command.

Counting the number of files in a directory

The wc command counts the specified input's lines, words, and characters. If we redirect the ls command's output as the input of this command, we get the total number of files in the given directory.

$ls | wc -l

Pipeline

Pipes are not limited to two commands. We can add as many commands as we want in a pipeline. A pipeline is a group of commands connecting in a chain using pipes and providing a single output. When building a complex pipeline, it is best practice to write or add one command at a time and check its output before adding the following command to the pipeline. This approach helps us debug the pipeline in case of an error.

Let us take an example.

The following pipeline displays an alphabetically sorted list of users whose username starts with the word 'user', and their default shell is the /bin /bash.

$cat /etc/passwd | grep user |grep /bin/bash | sort

To build this pipeline, we will add commands in the following manner.

$cat /etc/passwd
$cat /etc/passwd | grep user
$cat /etc/passwd | grep user | /bin/bash
$cat /etc/passwd | grep user |grep /bin/bash | sort

The /etc/passwd file stores local users' database. We used the cat command to read all data of this file, and instead of displaying that data on the monitor screen, we instructed the cat command to redirect that data to the grep command as the input.

We used the grep command to search all lines that contain the word 'user' in the output of the first command. Again, instead of displaying the output on the screen, we redirected the output to the following grep command. We used the following grep command to search all lines that contain the /bin/bash in the output of the second command. Finally, we redirected the filtered output to the sort command. The sort command sorts the input data alphabetically and displays it on the monitor screen.

Conclusion

A pipe is a shell feature that connects two commands. It redirects the first command's output to the second command as the input. Adding multiple commands in a sequence using a pipe creates a pipeline. A pipeline helps administrators process data through various commands.