Shell I/O Redirections and Filters

Introduction

When working with Linux, knowledge of certain commands is very useful. It gives a user superpowers as they are able to manipulate the contents of a file or the files themselves using simple commands. Learning about these helps to increase productivity and saves a lot of time.

Streams

Input and output in a Linux environment is divided into:

1. Standard input (stdin) - represented by 0

2. Standard output (stdout) - represented by 1

3. Standard error (stderr) - represented by 2

Standard input

Standard input conveys data from a user to a program. Programs that expect standard input usually receive input from an input device such as a keyboard.

Example

cat > filename

Standard output

Standard output is generated after a program runs and is displayed on the terminal.

Example

echo "Display on the terminal"

Output

Display on the terminal

Standard error

Standard error display errors generated by a program that has failed to run on the terminal. It informs a user that something is wrong and gives details of what the error is.

Example

ls $

Output

ls: cannot access '$': No such file or directory

Redirection

Linux provides the capability to change where standard input comes from or where the standard output goes using a concept called I/O redirection. This is accomplished using a redirection operator which allows the user to specify the input or output data to be redirected to or from a file.

The output redirection operator is the > (greater than) symbol. This adds content to a file, if it exists, it overrides the content.

Example

ls > file.txt

In this example the ls command is executed and the results are written in a file called file.txt. The results will not displayed as it was redirected.

There is another redirection operator represented using the >> symbol. This will create a file and then append data to it. Notice the difference in between the two, one overwrites data while the other appends data to a file:

Example 2

ls >> file.txt

In the above example the contents of the command ls are redirected to the file.txt and the results are appended to the file. It does not overwrite data in the file.

Pipes

The pipe operator is represented by the | character. It redirects a stream from one program to another. This command is useful in combining multiple commands together to form what are called pipelines. The standard output of one command is redirected into the standard input of another.

Example

ls | less

The above example takes the command ls and pipes it to the less command. The less command displays content one line at a time.

ls displays directory contents across multiple rows. When you pipe it alongside lesseach entry is placed on a new line.

Filters

Filters are commands that change data passing through them, typically via pipes. Although, some filters can be used independently (without filters) the true power to manipulate streams of data to the desired output comes from the combination of pipes and filters.

The commands are:

The cat command

It displays data on the standard output.

Syntax

cat filename

The head command

It is useful when working with large sets of data. It prints the first 10 lines of a file to the standard output. It comes in handy when you want to view the header of a file.

Syntax

head filename

You can also add the -n flag followed by the number of lines that you want to get.

Example

head -n 5 filename

The above example will only print out the first 5 lines of the file.

You could actually exclude the -n flag and use a dash followed by the number of lines directly

Example 2

head -5 filename

The above is going to be the same as head -n 5 filename.

The tail command

It is useful when working with large sets of data. It prints the last 10 lines of a file to the standard output. It comes in handy when you want to retrieve the most recent entry logs.

Syntax

tail filename

Note - It works the same with flags similar to the head command

The wc command

• It stands for word count and it counts lines, words, and characters in a file.

• Some handy flags:

  • -l prints only the number of line

  • -w prints only the number of words

  • -c prints only the number of characters

Syntax:

wc filename

The output follows the order of number of lines, number of words, and number of characters unless you use a flag to specify what you want.

The grep command

It is used to search through files and print lines matching a specified pattern to standard out. By default, the grep command is case-sensitive.

Syntax

grep some_string filename

• i - is used to make grep case insensitive

• w - is used to only match whole words

The sort command

It lets you sort your file in a particular order to the standard output.

Syntax

cat filename | sort

The awk command

This is not just a command but a whole scripting language, mainly used for text processing. It is extremely powerful. The awk command lets you print out specific columns.

awk options 'selection_criteria {action }' input-file > output-file

You can learn more about it here

The uniq command

Lets you filter out and only shows the uniq line records. It removes duplicate lines of data making sure every line is uniq. In most cases, it is used together with the sort command.

Syntax

cat filename | sort | uniq

The sed command

The sed command lets you do a search and replace for a specific string in a text or file. It stands for stream editor.

Syntax

cat filename sed 's/UNIX/Unix/'

In the above example The s on sed implies substituting an occurrence of the first string (UNIX) with the second string (Unix).

The trcommand

It is used to change or delete characters. It is commonly used to translate lowercase characters to uppercase.

Syntax

cat filename | tr "[:lower:]" "[:upper:]"

The fmt command

Reads text from standard input then outputs formatted text on the standard output. It comes in handy when one wants to format a large set of data quickly.

Syntax

fmt [-WIDTH] [OPTION]... [FILE]...

where, the -WIDTH is an abbreviated form of –width=DIGITS and OPTION refers to the options compatible with the fmt command and FILE refers to the file name. If no FILE is specified, or if FILE is a dash(“-“), fmt reads from the standard input.

The pr command

Takes text from standard input and splits the data into page breaks, headers, and footers in preparation for printing.

Syntax

pr [options][filename]

Conclusion

Learning how to use these commands is a crucial skill. Now that you have seen the basics of how redirections, pipes, and filters work, you’ll be able to begin your journey into the world of shell scripting.

Happy scripting, and may your commands always run swiftly and accurately!