Final Unix & AWK Programming Guide

1. Unix Basics with Examples (From PPT)

LS
Lists files and directories.

Example: ls -l

=> Shows file permissions, ownership, size, and modification time.

CD
Changes the current working directory.

Example: cd /home/user

=> Moves into the /home/user directory.

MKDIR
Creates a new directory.

Example: mkdir myfolder

=> Creates a folder named 'myfolder'.

RM
Removes files or directories.

Example: rm file.txt

=> Deletes 'file.txt'.

TOUCH
Creates empty files.

Example: touch demo.txt

=> Creates a blank file 'demo.txt'.

CAT
Displays file contents.

Example: cat file.txt

=> Outputs all content of 'file.txt'.

NANO/VI
Edits files in terminal.

Example: nano file.txt

=> Opens 'file.txt' in nano editor.

CHMOD
Changes file permissions.
Example: chmod 755 script.sh

=> Full permission for user, read+execute for others.

CHOWN
Changes file ownership.

Example: chown user:group file.txt

=> Changes owner of 'file.txt'.

PS
Shows running processes.

Example: ps aux

=> Detailed list of active processes.

TOP
Displays real-time process list.

Example: top

=> Live CPU and memory usage.

KILL
Terminates a process.

Example: kill 1234

=> Ends process with PID 1234.

GREP
Searches for patterns.

Example: grep 'main' script.sh

=> Finds 'main' in script.sh.

FIND
Searches for files.

Example: find . -name '*.txt'

=> Lists all .txt files.

SORT
Sorts lines in a file.

Example: sort data.txt

=> Sorts lines alphabetically.

UNIQ
Filters repeated lines.

Example: sort data.txt | uniq

=> Removes duplicates.

CUT
Extracts specific fields.

Example: cut -d',' -f2 data.csv

=> Gets 2nd column.

HEAD
Shows beginning lines.

Example: head -n 3 file.txt

=> First 3 lines.

TAIL
Shows ending lines.

Example: tail -n 3 file.txt

=> Last 3 lines.

ECHO
Displays a string.

Example: echo Hello

=> Outputs 'Hello'.

REDIRECTION
Redirects output.

Example: echo Hello > file.txt

=> Writes 'Hello' to file.

PIPE
Connects commands.

Example: ls | grep txt

=> Lists only .txt files.

2. Examples from Unix 1 (1).txt with Explanations
Example 1: To Print the 2nd, 3rd and 5th Columns only

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,int

4,Watermelon,7,9,63

Solution:

awk 'BEGIN{

FS=",";

OFS="|";

print $2,$3,$5;

}' fruits.txt

--------------------------------------------------------------------------------------

Example 2: To Find the total price of an apple (PIPELINING grep & awk)

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,int

4,Watermelon,7,9,63

5,apple,3,15,45

Solution:

grep -i "[Aa]pple" fruits.txt | awk 'BEGIN{FS=",";s=0;}

{

s = s+$5;

END{

print "The total price spent on apple fruit is " s;

}'

--------------------------------------------------------------------------------------

Example 3: Printing the Multiplication of Number 5

Solution:

awk 'BEGIN{

i=1;

while(i<=10)

print "5 * "i" = "i*5;

i++;

}'

--------------------------------------------------------------------------------------

Example 4: Find the number of fields in a record and total no.of records present in the text file

mango

watermelon

pine apple

custard apple

banana

Solution:
awk 'BEGIN{FS=" ";}

print "The Number of fields present in record " NR " is " NF".";

END{

print NR" records are present.";

}' fruits2.txt

---------------------------------------------------------------------------------------------

Input Data:

29,Arun

26,Karthik

28,Kiran

52,Raju

78,Rachel

Example 5:

awk 'BEGIN{FS=",";}

if($1>50)

print "Value is greater than 50.";

else

print "Value is less than 50";

i=1;

while(i<=1)
{

print "Row " NR ", Column 2 (loop "i "): " $2;

i++;

END{

print match("End of Code", /of/);

print RSTART, RLENGTH;

}' example.txt

Example 6:

grep ',A' input3.txt | awk 'BEGIN{FS=",";total_age=0;}

total_age += $1;

count++;

END{

if(count>0)

avg_age = total_age/count;

printf "Average Age for Group A:%.2f\n", avg_age;

else

print "No Data Found";

}'

--------------------------------------------------------------------------------------

ROYAL MAIL HOTEL (CASE STUDY)

employeeDetails.txt (DATA) :

Name,Age,Place,Experience,Salary

Anish,26,Chennai,2,10000

Jai,24,Chennai,2,10000

Kumar,29,Hyderabad,5,32000

John,32,Mumbai,2,11000

Neethu,21,Nagpur,3,13000

Satish,22,Ahmedabad,2,10000

Situation: To Print the complete data

awk 'BEGIN{FS=",";}

print;

}' employeeDetails.txt

Situation: Manager wish to display the employee name and salary working in royal mail hotel

awk 'BEGIN{FS=",";}

print $1,$5;

}' employeeDetails.txt

Situation: Manager wishes to print details of Kumar and Satish

awk '/Kumar|Satish/' employeeDetails.txt

Situation: Manager wishes to find the total expenses of hotel per month in the form of salary

awk -F"," 'BEGIN{

s=0;

}
{

s=s+$5;

END{

print "Total Exprenses per month in form of salary is " s;

}' employeeDetails.txt

Situation: Manager wishes to find the total no.of employees earning 10000 per month

awk '/10000/{

++count;

END{

print "No.of employees earning 10000 per month is " count;

}' employeeDetails.txt

Situation: Manager wishes to find the employees as best performers who completed 2 years of exprerience

and earning more than 10000

awk 'BEGIN{

FS=",";

print "*********************Performance Report*********************";

if(NR!=1)

if($4>=2 && $5>10000)

print $1" is a good performer";

else if($4==2 && $5<=10000)

print $1" Needs to improve";

 }

END{

print "*********************Performance Report*********************";

}' employeeDetails.txt

-----------------------------------------------------------------------------------------

SORT COMMAND

input data file: list

1,John Cena,Title 758,Price $7.30

2,Randy Orton,Title 739,Price $6.20

4,Triple H,Title 893,Price $6.42

5,GoldBerg,Title 392,Price $1.98

input data file: list1

7,Shawn Michales,Title 620,Price $1.64

8,Roman Reigns,Title 920,Price $1.03

3,Brock Lesnar,Title 201,Price $6.71

6,Rey mysterio,Title 109,Price $12.4

Situation: Sort on the 2nd field of file named "list". File list is comma seperated value

Solution: sort -k 2 list;

Situation: Sort the input data file

Solution: sort list;

Situation: Sort can be applied on multiple files as well

Solution: sort -n list list1;

Situation: Sort in reverse order of first numeric column from multiple files

Solution: sort -nr list list1;

Situation: Sort the above input file by removing the duplicate records/lines.

Solution: sort -u list;

Situation: Sort two input files and merge it.

Solution: sort -m list list1;

---------------------------------------------------------------------------------------------------------

UNIQ COMMAND

input data file: list.txt

unix operating system

unix operating system

unix dedicated server

linux dedicated server

Situation: surpass the duplicate records

Solution: uniq list.txt

Situation: Pipelining command using (Sort and Uniq)

Solution: sort list.txt | uniq

Situation: Count the repeated lines

Solution: uniq -c list.txt

Situation: Display only the duplicate lines

Solution: uniq -d list.txt

Situation: Display all duplicate lines

Solution: uniq -D list.txt

Situation: Print only unique lines

Solution: uniq -u list.txt

------------------------------------------------------------------------------------------

GREP COMMAND

demo_file

THIS LINE IS THE 1ST UPPER CASE IN THIS FILE.

this line is the 1st lower case line in the file.

This Line Has All Its First Characters as Upper Case.

Two lines above this line is empty.

And this is the last line.

Situation: Search for a given string and also you can check a string in multiple files

Solution: grep "this" demo_file.txt

Situation: Search for a given string with caseinsensitive approach

Solution: grep -i "this" demo_file.txt

Situation: Search for a line starting with 'Two' and ending with 'empty'

Solution: grep "Two.*empty" example.txt

Situation: To Search for a word and to avoid it to match the substrings -w option is used

Solution: grep -w "is" example.txt

Situation: If you want to display the lines which does not matches the given string/pattern then

Solution: grep -v "Two" example.txt

-----------------------------------------------------------------------------------------------------------

SED COMMAND

file_name

1.Linux - System Admin, Scripting etc.

2.Database - Oracle, mySQL etc.

3.Hardware
4.Security (Firewall, Network etc.)

5.Storage

Operation in Sed

-p = Prints specific number of lines based on line number or pattern

-n = suppress automatic printing of patternspace. It will not print any thing until explicit request to print is

found

-d = delete the line when pattren matches

-i = modifies the text in the input file

SYNTAX:

sed -n '/pattern/'p file_name

Replaces the occurrences of 'Security' with 'security' from file_name.txt

sed 's/Security/security/g' file_name.txt

Prints lines containing 'Security' by duplicating from file_name.txt

sed '/Security/p' file_name.txt

Prints the lines containing 'Security' until explicit request to print is found

sed -n '/Security/p' file_name.txt

Prints lines 2 to 4 from file_name.txt

sed -n '2,4p' file_name.txt

Modifies the text 'Hardware' to 'hardwares' from file_name.txt

sed -i 's/Hardware/hardwares/g' file_name.txt

Prints every 2nd line starts from line 3.

sed -n '3~2p' file_name.txt

------------------------------------------------------------------------------AWK Command-------------------------------------

Regular Expressions

DOT

echo -e "cat\ncut\nfun\nfin\nfan" | awk '/c.t/'

START OF THE LINE

echo -e "This\nThat\nThere\nTheir\nthese" | awk '/^The/'

END OF THE LINE

echo -e "knife\nknow\nfun\nfin\nfan\nnine" | awk '/n$/'

MATCH CHARACTER SET

echo -e "Call\nTall\nBall" | awk '/[CT]all/'

EXCLUSIVE SET

echo -e "Call\nTall\nBall" | awk '/[^CT]all/'

ALTERATION

echo -e "Call\nTBall\nSmall\nShall" | awk '/Call|Ball/'

ZERO OR ONE OCCURRENCE

echo -e "Colour\nColor" | awk '/Colou?r/'

ZERO OR MORE OCCURRENCE

echo -e "ca\ncat\ncatt" | awk '/cat*/'

ONE OR MORE OCCURRENCE

echo -e "111\n22\n123\n234\n456\n222" | awk '/2+/'

GROUPING

echo -e "Apple Juice\nApple Pie\nApple Tart\nApple Cake" | awk '/Apple (Juice|Cake)/'

----------------------------------------------------------------------------------

USER DEFINED FUNCTIONS

# Returns minimum number

function find_min(num1, num2){

if (num1 < num2)

return num1

return num2

# Returns maximum number

function find_max(num1, num2){

if (num1 > num2)

return num1

return num2

# Main function

function main(num1, num2){

# Find minimum number

result = find_min(10, 20)

 print "Minimum =", result

# Find maximum number

result = find_max(10, 20)

print "Maximum =", result

# Script execution starts here

BEGIN {

main(10, 20)

--------------------------------------------------------------------------------

CONTROL FLOW STATEMENTS

IF STATEMENT

awk 'BEGIN {num = 10; if (num % 2 == 0) printf "%d is even number.\n", num }'

IF-ELSE STATEMENT

awk 'BEGIN {

num = 11;

if (num % 2 == 0)

printf "%d is even number.\n", num;

else

printf "%d is odd number.\n", num;

}'

IF-ELSE-IF LADDER

awk 'BEGIN {
 a = 30;

if (a==10)

print "a = 10";

else if (a == 20)

print "a = 20";

else if (a == 30)

print "a = 30";

}'
3. AWK Exercise Solutions with Detailed Explanations
Q1: Remove the header and print names
Command:

awk -F@ 'NR > 1 { print $2 }'

Explanation:

Skips the first line (header) using NR > 1, and prints the 2nd field (NAME).

Q2: Sort names alphabetically

Command:

awk -F@ 'NR > 1 { print $2 }' | sort

Explanation:

Prints the NAME field and pipes it to sort for alphabetical order.

Q3: Sales department employees, descending

Command:

awk -F@ 'NR > 1 && $4=="Sales" { print $2 }' | sort -r

Explanation:

Filters Sales employees and sorts names in reverse order.

Q4: Salary > 60000, print name@salary

Command:

awk -F@ 'NR > 1 && $5 > 60000 { print $2 "@" $5 }'

Explanation:

Selects rows with salary > 60000, prints name and salary separated by '@'.

Q5: Total HR employees

Command:

awk -F@ 'NR > 1 && $4=="HR" { count++ } END { print count }'

Explanation:

Increments a counter for HR department rows and prints count in the END block.

Q6: Sum all salaries

Command:
awk -F@ 'NR > 1 { sum += $5 } END { print sum }'

Explanation:

Adds up all values in the SALARY column.

Q7: Employees older than 40

Command:

awk -F@ 'NR > 1 && $3 > 40 { print $2, $3 }'

Explanation:

Checks AGE column > 40 and prints NAME and AGE.

Q8: Average salary of marketing employees

Command:

awk -F@ 'NR > 1 && $4=="Marketing" { sum+=$5; count++ } END { print sum/count }'

Explanation:

Adds salaries for Marketing and divides by count.

Q9: Minimum salary and name

Command:

awk -F@ 'NR == 2 || $5 < min { min=$5; name=$2 } END { print name, min }'

Explanation:

Tracks lowest salary from second line onwards, stores corresponding name.

Q10: Salary with 10% increment

Command:

awk -F@ 'NR > 1 { inc = $5 * 1.10; print $2, inc }'

Explanation:

Calculates new salary by multiplying with 1.10 and prints with name.

Q11: Sort employees by salary

Command:

awk -F@ 'NR > 1' | sort -t@ -k5,5n

Explanation:
Prints all rows except header, sorted numerically by salary column.

Q12: First 3 data rows

Command:

awk -F@ 'NR > 1 && NR <= 4'

Explanation:

Prints lines 2 to 4 (first 3 data rows after header).

Q13: Names starting with 'S'

Command:

awk -F@ 'NR > 1 && $2 ~ /^S/ { print $2 }'

Explanation:

Uses regex ^S to match names that start with 'S'.

Q14: Names ending in 'a'

Command:

awk -F@ 'NR > 1 && $2 ~ /a$/ { print $2 }'

Explanation:

Regex $ matches names ending in 'a'.

Q15: Names with letter 'y'

Command:

awk -F@ 'NR > 1 && $2 ~ /y/ { print $2 }'

Explanation:

Matches names containing the letter 'y' anywhere.

Q16: Names containing vowels

Command:

awk -F@ 'NR > 1 && $2 ~ /[aeiouAEIOU]/ { print $2 }'

Explanation:

Prints names containing any vowel (case insensitive).