Essential Linux Utilities and Shell Programming:

1. Core File Handling Utilities

File handling in Linux is fundamental, and various commands help with creating, reading,
writing, and deleting files, along with setting permissions for security.

Basic File Operations:

●​ Creating Files:​

○​ touch filename – Creates an empty file.

■​ Example: touch newfile.txt
○​ echo "text" > filename – Creates a file with the specified content.
■​ Example: echo "Hello World" > hello.txt
●​ Reading Files:​

○​ cat filename – Reads the content of a file.

■​ Example: cat hello.txt
○​ less filename – Opens the file for viewing in a paginated way.
■​ Example: less largefile.txt
●​ Writing to Files:​

○​ > – Overwrites the content of a file.

■​ Example: echo "New content" > newfile.txt
○​ >> – Appends content to a file.
■​ Example: echo "More content" >> newfile.txt
●​ Deleting Files:​

○​ rm filename – Deletes a file.

■​ Example: rm unwantedfile.txt

Understanding and Setting File Permissions:

In Linux, files have three types of permissions for three types of users: Owner, Group, and
Others. The permissions are:

●​ r – Read (4)
●​ w – Write (2)
●​ x – Execute (1)
You can set permissions using chmod:

●​ chmod 755 filename – Assigns rwx to the owner and rx to group and others.
○​ Example: chmod 755 script.sh

Check file permissions with:

ls -l filename​
-rwxr-xr-x 1 user group 1024 Feb 9 10:00 script.sh

●​

This output shows the permission structure where:

●​ rwx – Owner can read, write, and execute.

●​ r-x – Group can read and execute.
●​ r-x – Others can read and execute.

2. Process Utilities

Linux provides utilities to manage system processes, allowing users to view, start, stop, or
manipulate running processes.

Viewing Processes:

●​ ps aux – Displays all running processes with detailed information.​

○​ Example: ps aux | grep apache (lists processes related to Apache).

●​ top – Displays a real-time view of system processes, CPU usage, and memory
usage.​

○​ Example: top
●​ htop – An interactive process viewer similar to top (requires installation).​

○​ Example: htop

Starting and Stopping Processes:

●​ Start a process:
○​ Running a command normally starts it in the foreground:
■​ Example: ./myprogram
○​ To run it in the background:
■​ Example: ./myprogram &
 ●​ Stopping a process:
○​ kill PID – Stops a process by its process ID.
■​ Example: kill 1234
○​ killall processname – Stops all instances of a specific process.
■​ Example: killall firefox

3. Disk Management

Managing disk space and partitions is an essential administrative task in Linux.

Disk Usage:

●​ df -h – Displays the disk space usage in a human-readable format.

○​ Example: df -h
●​ du -sh directory – Shows the total disk usage of a directory.
○​ Example: du -sh /var/log

Formatting Disks:

●​ mkfs – Creates a filesystem on a partition.

○​ Example: mkfs.ext4 /dev/sda1 formats a partition as ext4.

Partitioning Disks:

●​ fdisk /dev/sda – Opens a disk for partitioning.

○​ Example: fdisk /dev/sda (interactive utility for partitioning).

4. Networking Commands

Networking utilities are essential for testing connectivity, managing network interfaces, and
diagnosing issues.

●​ ping hostname – Sends ICMP echo requests to test connectivity.​

○​ Example: ping google.com

●​ ifconfig – Configures and displays network interface information.​

○​ Example: ifconfig eth0 (displays information for the eth0 interface).

●​ netstat -tuln – Shows open ports and listening services on the system.​

○​ Example: netstat -tuln

5. Text Processing and Filters

Linux provides powerful utilities for working with text files and logs.

grep (Global Regular Expression Print):

grep searches for a specific pattern in text files.

●​ Example: grep "error" /var/log/syslog – Searches for "error" in the system

log.

sed (Stream Editor):

sed is used for transforming text in a file or stream.

●​ Replace a string:​

○​ Example: sed 's/old/new/g' file.txt replaces all instances of "old"

with "new" in file.txt.
●​ Delete lines:​

○​ Example: sed '/pattern/d' file.txt deletes all lines containing

"pattern".

awk (A Text-Processing Language):

awk works with fields and records, making it useful for processing structured text.

●​ Print specific columns:

○​ Example: awk '{print $1, $3}' file.txt prints the first and third
columns of each line.

6. Backup Utilities

Linux offers tools for creating backups and synchronizing files across systems.

●​ tar (Tape Archive):​

○​ Example: tar -czvf backup.tar.gz /home/user/ – Compresses the

/home/user/ directory into a tarball.
●​ rsync (Remote Sync):​

○​ Example: rsync -av /source /destination – Synchronizes files

between directories or systems.
7. sed and awk Programming

Both sed and awk are powerful tools for manipulating and analyzing text data, often used for
scripting.

sed Programming:

●​ Replace text in files:

○​ Example: sed -i 's/old/new/g' file.txt replaces all instances of
"old" with "new" in-place.
●​ Delete lines matching a pattern:
○​ Example: sed '/pattern/d' file.txt deletes all lines containing
"pattern".

awk Programming:

awk is useful for scanning files, filtering data, and generating reports.

●​ Execution of simple awk script:​

○​ Example: awk '{print $1, $2}' file.txt – Prints the first and
second fields (columns) of each line.
●​ Fields and Records:​

○​ awk treats a file as a series of records (lines) and each record is divided into
fields (columns). Fields are referenced as $1, $2, etc.
●​ Patterns and Actions:​

○​ Example: awk '/pattern/ {print $1}' file.txt – If a line matches

"pattern", print the first field.
●​ Using Arrays:​

Example:​
awk '{count[$1]++} END {for (word in count) print word, count[word]}' file.txt

○​ This script counts the occurrences of each unique word in the first column of
file.txt and prints the result.
7. sed and awk Programming

sed (stream editor) is a simple but powerful text processing tool, primarily used for
performing basic text transformations in files and streams. It allows users to make changes
to text files by searching for specific patterns and replacing, deleting, or modifying them.

More Common Operations with sed:

1.​ In-place Editing: By default, sed outputs the changes to standard output. However,
you can use the -i option to modify the file in place:​

○​ Example: sed -i 's/oldword/newword/g' file.txt – Replaces all

occurrences of "oldword" with "newword" in file.txt.
2.​ Substitute and Print Only Changed Lines: To replace a word and print only the
lines where substitutions were made:​

○​ Example: sed -n 's/old/new/p' file.txt

■​ -n suppresses automatic printing.
■​ The p command prints the changed lines.
3.​ Deleting Lines: You can delete lines that match a specific pattern.​

○​ Example: sed '/pattern/d' file.txt – Deletes all lines containing

"pattern".
4.​ Working with Line Numbers: You can target specific lines by using line numbers:​

○​ Example: sed '2d' file.txt – Deletes the second line of file.txt.

○​ Example: sed '1,5d' file.txt – Deletes lines 1 through 5.
5.​ Multiple Substitutions: You can perform multiple substitutions in one go:​

○​ Example: sed -e 's/old1/new1/g' -e 's/old2/new2/g' file.txt

– Replaces both "old1" with "new1" and "old2" with "new2".

More on awk Programming:

awk is a versatile and powerful language for text processing and data extraction. It operates
on text files by processing each line as a record and splitting it into fields (columns). It is
widely used for creating reports, filtering data, and more complex tasks.

Detailed Breakdown of awk Operations:

Basic Structure of an awk Program:​

​
awk 'pattern {action}' filename

1.​
 ○​ pattern – Defines the condition on which the action will be performed (can be
regex or logical).​

○​ action – Specifies what to do when a pattern is found (such as print, modify,

calculate, etc.).​

○​ Example: awk '/error/ {print $0}' log.txt – Prints the entire line
(referred to as $0) for any line containing "error".​

2.​ Field Manipulation: Fields are typically separated by spaces, but you can specify
other delimiters (e.g., commas).​

○​ Print specific fields:​

■​ Example: awk '{print $1, $3}' file.txt – Prints the first and
third fields from each line.
○​ Change the field separator:​

■​ Example: awk -F, '{print $1, $2}' file.csv – Uses , as

the field separator (useful for CSV files).
3.​ Patterns and Actions: awk works by matching patterns and executing actions.
Patterns can be:​

○​ Regular expressions:​

■​ Example: awk '/error/ {print $0}' logfile.txt – Prints

any line containing "error".
○​ Line numbers:​

■​ Example: awk 'NR==5' file.txt – Prints the fifth line only (NR
stands for "number of records", or line number).
○​ Comparison operators:​

■​ Example: awk '$3 > 50 {print $1, $2}' data.txt – Prints

the first two fields if the value of the third field is greater than 50.
4.​ Using awk for Calculations: awk can also be used to perform calculations on
numeric data.​

Example: Summing values in a column:​

awk '{sum += $2} END {print sum}' numbers.txt

○​ This command calculates the sum of the values in the second column.
5.​ Using Arrays in awk: Arrays in awk are associative (similar to dictionaries in other
languages), allowing you to store and manipulate data easily.​
 ○​ Counting Occurrences:

Example:​
awk '{count[$1]++} END {for (word in count) print word, count[word]}' file.txt

■​ This script counts how many times each unique word appears in the
first column of file.txt.
6.​ Begin and End Blocks: awk allows you to execute code before reading any input
(using BEGIN) or after processing all input (using END).​

Example: Adding headers and footers:​

awk 'BEGIN {print "Name, Age"} {print $1, $2} END {print "End of Report"}' file.txt

○​
7.​ Combining awk and sed: You can combine both awk and sed to handle more
complex text-processing tasks.​

Example: Replace text with sed and print specific fields with awk:​
sed 's/oldword/newword/g' file.txt | awk '{print $1, $2}'

○​

8. Practical Applications of sed and awk:

sed Applications:

1.​ Configuration File Edits:​

○​ sed is often used for making automated changes to configuration files. For
example, updating configuration parameters:
■​ Example: sed -i 's/Port 22/Port 2022/'
/etc/ssh/sshd_config – Changes the SSH port from 22 to 2022.
2.​ Cleaning Log Files:​

○​ sed can clean log files by removing unnecessary lines:

■​ Example: sed '/DEBUG/d' logfile.txt – Deletes all lines
containing the word "DEBUG".
3.​ Batch Renaming:​

Use sed in a shell loop to batch rename files:​

for file in *.txt; do
mv "$file" "$(echo $file | sed 's/.txt/.md/')"
done
 ○​

awk Applications:

1.​ Processing CSV Files:​

○​ awk is ideal for extracting and summarizing data from CSV files.

Example: Print specific columns from a CSV:​

awk -F, '{print $1, $3}' data.csv

■​
2.​ Summarizing Data:​

Calculate the total sales from a sales report:​

awk '{sum += $3} END {print "Total Sales:", sum}' sales.txt

○​
3.​ Generating Reports:​

Generate a report from a file:​

awk '{print $1, $2, "has a score of", $3}' scores.txt

○​
4.​ Pattern Matching in System Logs:​

○​ awk can be used to filter system logs by certain criteria:

Example: Print logs from a specific date:​

awk '$1 == "Feb" && $2 == "9" {print $0}' /var/log/syslog