Introduction
When working with Linux, knowledge of certain commands is very useful. It gives a user superpowers as they are able to manipulate the contents of a file or the files themselves using simple commands. Learning about these helps to increase productivity and saves a lot of time.
Streams
Input and output in a Linux environment is divided into:
Standard input (
stdin) - represented by0Standard output (
stdout) - represented by1Standard error (
stderr) - represented by2
Standard input
Standard input conveys data from a user to a program. Programs that expect standard input usually receive input from an input device such as a keyboard.
Example
cat > filename
Standard output
Standard output is generated after a program runs and is displayed on the terminal.
Example
echo "Display on the terminal"
Output
Display on the terminal
Standard error
Standard error display errors generated by a program that has failed to run on the terminal. It informs a user that something is wrong and gives details of what the error is.
Example
ls $
Output
ls: cannot access '$': No such file or directory
Redirection
Linux provides the capability to change where standard input comes from or where the standard output goes using a concept called I/O redirection. This is accomplished using a redirection operator which allows the user to specify the input or output data to be redirected to or from a file.
The output redirection operator is the > (greater than) symbol. This adds content to a file, if it exists, it overrides the content.
Example
ls > file.txt
In this example the ls command is executed and the results are written in a file called file.txt. The results will not displayed as it was redirected.
There is another redirection operator represented using the >> symbol. This will create a file and then append data to it. Notice the difference in between the two, one overwrites data while the other appends data to a file:
Example 2
ls >> file.txt
In the above example the contents of the command ls are redirected to the file.txt and the results are appended to the file. It does not overwrite data in the file.
Pipes
The pipe operator is represented by the | character. It redirects a stream from one program to another. This command is useful in combining multiple commands together to form what are called pipelines. The standard output of one command is redirected into the standard input of another.
Example
ls | less
The above example takes the command ls and pipes it to the less command. The less command displays content one line at a time.
ls displays directory contents across multiple rows. When you pipe it alongside lesseach entry is placed on a new line.
Filters
Filters are commands that change data passing through them, typically via pipes. Although, some filters can be used independently (without filters) the true power to manipulate streams of data to the desired output comes from the combination of pipes and filters.
The commands are:
The cat command
It displays data on the standard output.
Syntax
cat filename
The head command
It is useful when working with large sets of data. It prints the first 10 lines of a file to the standard output. It comes in handy when you want to view the header of a file.
Syntax
head filename
You can also add the -n flag followed by the number of lines that you want to get.
Example
head -n 5 filename
The above example will only print out the first 5 lines of the file.
You could actually exclude the -n flag and use a dash followed by the number of lines directly
Example 2
head -5 filename
The above is going to be the same as head -n 5 filename.
The tail command
It is useful when working with large sets of data. It prints the last 10 lines of a file to the standard output. It comes in handy when you want to retrieve the most recent entry logs.
Syntax
tail filename
Note - It works the same with flags similar to the
headcommand
The wc command
It stands for word count and it counts lines, words, and characters in a file.
Some handy flags:
-lprints only the number of line-wprints only the number of words-cprints only the number of characters
Syntax:
wc filename
The output follows the order of number of lines, number of words, and number of characters unless you use a flag to specify what you want.
The grep command
It is used to search through files and print lines matching a specified pattern to standard out. By default, the grep command is case-sensitive.
Syntax
grep some_string filename
i- is used to makegrepcase insensitivew- is used to only match whole words
The sort command
It lets you sort your file in a particular order to the standard output.
Syntax
cat filename | sort
The awk command
This is not just a command but a whole scripting language, mainly used for text processing. It is extremely powerful. The awk command lets you print out specific columns.
awk options 'selection_criteria {action }' input-file > output-file
You can learn more about it here
The uniq command
Lets you filter out and only shows the uniq line records. It removes duplicate lines of data making sure every line is uniq. In most cases, it is used together with the sort command.
Syntax
cat filename | sort | uniq
The sed command
The sed command lets you do a search and replace for a specific string in a text or file. It stands for stream editor.
Syntax
cat filename sed 's/UNIX/Unix/'
In the above example The s on sed implies substituting an occurrence of the first string (UNIX) with the second string (Unix).
The trcommand
It is used to change or delete characters. It is commonly used to translate lowercase characters to uppercase.
Syntax
cat filename | tr "[:lower:]" "[:upper:]"
The fmt command
Reads text from standard input then outputs formatted text on the standard output. It comes in handy when one wants to format a large set of data quickly.
Syntax
fmt [-WIDTH] [OPTION]... [FILE]...
where, the -WIDTH is an abbreviated form of –width=DIGITS and OPTION refers to the options compatible with the fmt command and FILE refers to the file name. If no FILE is specified, or if FILE is a dash(“-“), fmt reads from the standard input.
The pr command
Takes text from standard input and splits the data into page breaks, headers, and footers in preparation for printing.
Syntax
pr [options][filename]
Conclusion
Learning how to use these commands is a crucial skill. Now that you have seen the basics of how redirections, pipes, and filters work, you’ll be able to begin your journey into the world of shell scripting.
Happy scripting, and may your commands always run swiftly and accurately!