Shree Siddheshwar Women’s College of Engineering, Solapur

Unit 4
Open Source Operating System (SHELL
PROGRAMMING)

Content of Unit 4 :
✓ Bash Shell Scripting
✓ Executing Script
✓ Working with Variables and Input
✓ Using Control Structures
✓ Handling signals, creating functions
✓ Working sed and gawk
✓ Working with web using shell script: Downloading web page, Converting Web page content
to a text file, parsing data, working cURL.

Bash Shell Scripting :

➢ Bash Shell Scripting is a method of writing and executing commands in a sequence to automate
tasks in a Linux environment.
➢ It combines the power of the Linux command-line with programming structures, making it
possible to create scripts that perform repetitive or complex tasks more efficiently.
➢ The shell is a command-line interface (CLI) that allows users to interact with the operating
system.
➢ It interprets and executes user commands, providing an interface to the system’s services and
utilities.
➢ Bash (Bourne Again Shell) is one of the most popular Unix/Linux shells used by default on
most Linux distributions.

Open Source Technology (OST) TY (E&TC) Page 1

 Shree Siddheshwar Women’s College of Engineering, Solapur

➢ In Computer programming, a script is a set of commands for an appropriate run time

environment which is used to automate the execution of tasks.
Bash Script:
➢ A Bash Shell Script is a plain text file containing a set of various commands that we usually
type in the command line.
➢ It is used to automate repetitive tasks on Linux filesystem.
➢ It might include a set of commands, or a single command, or it might contain the hallmarks of
imperative programming like loops, functions, conditional constructs, etc. Effectively, a Bash
script is a computer program written in the Bash programming language.
➢ Bash Scripting refers to writing a series of commands in a file (script) that are executed by the
Bash interpreter.
➢ It allows automation of repetitive tasks, simplifying the management of the system.
➢ Shell scripts are written in plain text, and when executed, the system processes the commands
sequentially.
➢ Bash Scripting is a powerful part of system administration and development used at an extreme
level.
➢ It is used by the System Administrators, Network Engineers, Developers, Scientists, and
everyone who use Linux/Unix operating system.
➢ They use Bash for system administration, data crunching, web application deployment,
automated backups, creating custom scripts for various pages, etc.
Why do we need shell scripts?
There are many reasons to write shell scripts:
1. To avoid repetitive work and automation
2. System admins use shell scripting for routine backups.
3. System monitoring
4. Adding new functionality to the shell etc.
Why Learn Bash Shell Scripting?
1. Automates repetitive tasks: Reduces manual work and time.
2. System administration: Ideal for managing Linux systems, monitoring, and deploying
applications.
3. Text processing: Efficiently manipulate text files and directories.

Open Source Technology (OST) TY (E&TC) Page 2

 Shree Siddheshwar Women’s College of Engineering, Solapur

4. Scripting for DevOps and automation: Often used in continuous integration and deployment
pipelines.
Advantages of Bash Scripting
1. Automation: Eliminates manual effort by automating repetitive tasks.
2. System Administration: Commonly used in managing files, processes, and user accounts.
3. Efficiency: Saves time and reduces errors by simplifying complex processes.
4. Integration: Works seamlessly with other tools, programming
Some Basic Bash Commands
Some commonly used Bash commands include:
• echo: Displays messages or values.
• ls: Lists files and directories.
• pwd: Prints the current working directory.
• cp, mv, rm: Commands to copy, move, and remove files.
• chmod: Changes file permissions.
Basic Structure of a Bash Script
✓ A Bash script is a text file with the extension .sh .
✓ The first line of every script should be:
#!/bin/bash
✓ This is called a shebang, which tells the system which interpreter to use (in this case, Bash).
✓ After the shebang, you can include Bash commands, variables, and logic.
Basic Script Structure
A Bash script typically includes:
1. Shebang (#!/bin/bash): The first line of the script, indicating the interpreter.
2. Commands: Instructions written sequentially to perform tasks.
3. Variables: Store values to use in the script.
4. Control flow: Conditional and looping constructs to create logic.

Executing Bash Shell Scripts in Linux :

➢ A Bash shell script is a sequence of commands written in a text file that is executed by the Bash
interpreter.

Open Source Technology (OST) TY (E&TC) Page 3

 Shree Siddheshwar Women’s College of Engineering, Solapur

➢ The primary purpose of Bash scripting is to automate tasks that would otherwise need to be
executed manually in a Linux terminal.
➢ Executing Bash scripts in Linux is straightforward once the script is written and given proper
permissions.
➢ With the ability to pass arguments, schedule scripts, and use environment variables, Bash
scripting becomes a powerful tool for automating tasks and managing systems efficiently.
Steps to Execute a Bash Script in Linux
1) Step 1: Writing a Script
✓ Before executing, we need to write a Bash script
✓ Open a text editor (e.g., nano, vi, or gedit) and create new file
nano script.sh
✓ Start the script with the shebang line to indicate the interpreter
#!/bin/bash
✓ Add the commands under the shebang line as per requirement
#!/bin/bash
echo "Hello, World!"
2) Step 2: Save the Script
✓ After writing the script, save the file with a meaningful name and a .sh extension
(though it’s not mandatory to use the extension).
3) Step 3: Make the Script Executable
✓ By default, the script might not have execute permissions.
✓ To run the script, we need to change its permissions:
chmod +x script.sh
✓ This command gives the script execute permissions, allowing it to be run like a
program.
4) Step 4: Execute the Script
✓ There are many ways to execute the script. Some of most popular method are listed
below
a) Run the script using its relative or absolute path:
. /script.sh
• The ./ indicates that the script is located in the current directory.

Open Source Technology (OST) TY (E&TC) Page 4

 Shree Siddheshwar Women’s College of Engineering, Solapur

b) Run the script by specifying the full path:

/home/user/scripts/script.sh
• This method is useful when the script is not located in the current
directory..
c) Execute using Bash directly:
• You can also run the script by explicitly invoking Bash:
bash script.sh

Common Issues in Executing Scripts

1. Permission denied: This error occurs if the script lacks execution permissions. Fix it by
running chmod +x script.sh.
2. Command not found: Ensure the script is in the correct directory and you’re using the right
path.
3. Shebang errors: If the script does not run as expected, ensure the correct interpreter is
specified in the shebang (#!/bin/bash).

Working with Variables and Input :

➢ In Bash scripting, variables allow us to store data and reuse it throughout the script.
➢ Unlike traditional programming languages, Bash variables do not require explicit data type
declarations.
➢ Bash treats everything as a string, but can perform basic arithmetic operations on numeric
values.
✓ Variables store information like strings, numbers, filenames, and command outputs.
✓ Bash is loosely typed, meaning there’s no need to declare data types for variables.
a) Declaring Variables
➢ Variables in Bash are assigned values using the = operator. There must be no space between
the variable name, the = symbol, and the value
variable_name=value
Example :
name=“SSWCOE"
age=05

Open Source Technology (OST) TY (E&TC) Page 5

 Shree Siddheshwar Women’s College of Engineering, Solapur

Here, name stores the string “SSWCOE" and age stores the number 05
b) Accessing Variable Values
➢ To access or refer to a variable's value, precede the variable name with a $ symbol.
echo $variable_name
Example
echo $name
This will output: SSWCOE
➢ If we reference a variable without the $, Bash will interpret it as plain text rather than a
variable.
c) Rules for Naming Variables
➢ Variable names can include letters, numbers, and underscores.
➢ Variable names cannot start with a number.
• Correct: my_var, VAR123, user_name
• Incorrect: 123var, -myvar
➢ Variable names are case-sensitive.
• myVar and MYVAR are treated as different variables.
d) Variables can hold different types of values:
✓ Strings: "Hello, World!"
✓ Numbers: 123
✓ Command outputs: Result of a command can be stored in a variable.
Example:
greeting="Welcome to OST Class !“
count=46
today=$(date)
✓ Here greeting holds a string, count holds a number. today holds the result of the date
command.
Variable Types
1) Local Variable:
✓ Variables which are specific to the current instance of shell. They are basically used
within the shell, but not available for the program or other shells that are started from
within the current shell.

Open Source Technology (OST) TY (E&TC) Page 6

 Shree Siddheshwar Women’s College of Engineering, Solapur

For example:
`name=SSWCOE`
✓ In this case the local variable is (name) with the value of SSWCOE . Local variables
is temporary storage of data within a shell script.
2) Environment Variable:
✓ These variables are commonly used to configure the behavior script and programs that
are run by shell. Environment variables are only created once, after which they can be
used by any user.
For example:
`export PATH=/usr/local/bin:$PATH` would add `/usr/local/bin` to the beginning of
the shell’s search path for executable programs.
3) Shell Variables:
✓ Variables that are set by shell itself and help shell to work with functions correctly.
Here some variables are Environment variable, and some are Local Variables.
For example:
• `$PWD` = Stores working directory
• `$HOME` = Stores user’s home directory
• `$SHELL` = Stores the path to the shell program that is being used.
Variable Scope
✓ Global Variables: By default, variables in Bash are global, meaning they can be
accessed throughout the script unless declared within a function.
✓ Local Variables: To restrict a variable's scope to a function, use the local keyword.

Arithmetic with Variables

➢ Bash can perform basic arithmetic operations using the let command or the $(( )) syntax.

Open Source Technology (OST) TY (E&TC) Page 7

 Shree Siddheshwar Women’s College of Engineering, Solapur

Example:

We can also use the let command:

String Manipulation in Variables

✓ Bash allows basic string operations such as concatenation and extraction.
✓ Concatenation: You can combine two or more strings by placing them next to each
other.

String Length: Use ${#variable} to get the length of a string.

Default Values for Variables

✓ Bash provides mechanisms to set default values for variables that are not initialized.
Syntax:

Open Source Technology (OST) TY (E&TC) Page 8

 Shree Siddheshwar Women’s College of Engineering, Solapur

✓ If variable is unset or null, default_value will be used.

Example:
echo ${user_name:-"Guest"}
✓ If user_name is not set, "Guest" will be printed.
Unsetting Variables
➢ We can remove a variable by using the unset command. Once unset, the variable will no longer
hold any value
Example
unset my_variable
echo $my_variable # This will output nothing since `my_variable` is unset
Positional Parameters and Special Variables
Bash also supports special variables that hold command-line arguments passed to the script:
✓ $0: The name of the script.
✓ $1, $2, ...: Positional parameters for arguments passed to the script.
✓ $#: The number of arguments passed to the script.
✓ $@: All the arguments passed to the script.
✓ $?: The exit status of the last command executed.
✓ $$: The process ID of the script.

Working with Input in Bash Shell Scripting

➢ In Bash scripting, working with input is essential for making scripts interactive and versatile.
➢ There are several methods to capture and handle input, including command-line arguments,
prompts (using read), input redirection, and environment variables.
1) Command-Line Arguments
✓ Command-line arguments allow users to pass data to a script when it's executed. These
arguments are accessed using special variables:
✓ $0: The name of the script.
✓ $1, $2, ..., $n: The positional parameters representing the first, second, ..., nth arguments
passed.
✓ $#: The number of command-line arguments passed.
✓ $* or "$@": All arguments passed.

Open Source Technology (OST) TY (E&TC) Page 9

 Shree Siddheshwar Women’s College of Engineering, Solapur

Example:

If executed with ./script.sh arg1 arg2, the output will be

$* treats all arguments as a single word.

$@ treats each argument separately.

2) Using read for User Input
✓ The read command is used to capture user input during the execution of a script. It waits
for user input, stores it in a variable, and then the script proceeds.
Syntax:
read [options] variable_name
Example:

This script will prompt the user for their name and greet them based on the input.
Common read Options:
-p: Display a prompt without using echo

Open Source Technology (OST) TY (E&TC) Page 10

 Shree Siddheshwar Women’s College of Engineering, Solapur

-s: Silent input (useful for passwords)

-n: Limit the number of characters input

3) Input Redirection
✓ Input can also be fed into a script from a file or another command using input
redirection.
✓ This is useful when you have large amounts of data.
Syntax:
command < input_file
Example:

This script reads each line of input.txt and prints it.

Here Documents (<<)

Open Source Technology (OST) TY (E&TC) Page 11

 Shree Siddheshwar Women’s College of Engineering, Solapur

✓ Here Documents allow multiple lines of input to be fed into a command or script,
often used to feed input directly without needing a separate file.
Syntax:

Example:

Piping Input

➢ Piping input (|) passes the output of one command as the input of another. This is especially
useful in chaining commands.
Example:

Environment Variables
✓ You can use environment variables to input data into your script.
✓ Environment variables are accessible to any script or command running within the
shell.
Example:

$USER and $HOME are environment variables available by default.

Open Source Technology (OST) TY (E&TC) Page 12

 Shree Siddheshwar Women’s College of Engineering, Solapur

Special Variables
✓ $?: Exit status of the last command executed (0 for success, non-zero for error).
✓ $$: Process ID of the current script.
✓ $!: Process ID of the last background job.
✓ Always validate user input to prevent errors or unexpected behavior.
✓ Use quotes around variables to prevent word splitting issues, especially when
working with filenames or strings containing spaces.

Using Control Structures for Bash Shell Scripting

➢ Control structures allow Bash scripts to make decisions, repeat tasks, and manage complex
workflows.
➢ Understanding these structures enables the creation of dynamic, efficient scripts that can
automate repetitive tasks and respond to various conditions.
1. Conditional Statements
Conditional statements let a script make decisions based on the evaluation of an expression.
a) The if Statement
✓ The if statement executes commands based on whether a given condition is true or false.
Basic Syntax:

Example

Open Source Technology (OST) TY (E&TC) Page 13

 Shree Siddheshwar Women’s College of Engineering, Solapur

b) The if-else Statement

✓ The if-else statement provides an alternative set of commands if the condition is false.

Example :

c) The if-elif-else Statement

✓ The elif keyword allows multiple conditions to be checked in sequence.

Example

Open Source Technology (OST) TY (E&TC) Page 14

 Shree Siddheshwar Women’s College of Engineering, Solapur

2. The case Statement

➢ The case statement allows multi-way branching, especially useful for handling multiple
possible values of a single variable.

Example

Open Source Technology (OST) TY (E&TC) Page 15

 Shree Siddheshwar Women’s College of Engineering, Solapur

3. Loop Structures
➢ Loops allow repetition of commands, either for a fixed number of times or based on
conditions.
a) The for Loop
➢ The for loop iterates over a list of items.
Syntax

Example

b) The while Loop

➢ The while loop repeats commands as long as a condition is true.
Syntax

Example

Open Source Technology (OST) TY (E&TC) Page 16

 Shree Siddheshwar Women’s College of Engineering, Solapur

c) The until Loop

➢ The until loop is similar to while but runs until the condition becomes true.
Syntax

Example

4. Loop Control Command

➢ Loop control commands like break and continue alter the flow within loops.
a) break Statement
✓ The break command exits a loop when a condition is met.

b) continue Statement
✓ The continue command skips the current iteration and moves to the next one.

Open Source Technology (OST) TY (E&TC) Page 17

 Shree Siddheshwar Women’s College of Engineering, Solapur

Nesting Control Structures

✓ Control structures can be nested within each other to create more complex scripts.

Important Point to remember

1) Control structures in Bash (if, case, for, while, until) enable decision-making and repetition.
2) Conditionals let scripts respond to different user inputs, while loops manage repetitive tasks.
3) Nesting control structures allows complex logical workflows and better control in scripts.
4) Proper use of loop control commands like break and continue can refine loop behavior.

Handling signals
➢ In Linux, Signals are the interrupts that are sent to the program to specify that an important
event has occurred.
➢ Events can vary from user requests to invalid memory access errors.
➢ Various signals, like the interrupt signal, means that the user has asked the program to perform
something which is not present in the user flow of control.
➢ There are two kinds of signals:
A) Maskable
B) Non-Maskable

Open Source Technology (OST) TY (E&TC) Page 18

 Shree Siddheshwar Women’s College of Engineering, Solapur

A) Maskable: - Maskable Signals are the signals that the user can change or ignore.
B) Non-Maskable: - Non-Maskable Signals are the signals that the users cannot change
or ignore. Non-Maskable signals mainly occur if a user is signaled for non-recoverable
hardware errors.

➢ There are various processes in different states in the Linux computer system.
➢ All these processes belong to either the operating system or the user application.
➢ A mechanism is required for the kernel and these processes to coordinate their activities.
➢ One method to perform this, for the process, to inform others if anything vital thing occurs.
That's the reason we have signals.
➢ Basically, the signal means a one-way notification.
➢ The kernel sent the signal to the process, to one process to itself, or another process. Linux
signals trace their origin to Unix signals.
➢ In later Linux versions, real-time signals were added.
➢ Signal inter process is an easy and lightweight form of communication and is therefore
compatible with embedded systems.
What is the Typical Lifecycle of a Signal?
A signal goes through three stages:
1) Generation
2) Delivery

Open Source Technology (OST) TY (E&TC) Page 19

 Shree Siddheshwar Women’s College of Engineering, Solapur

3) Processing

1) Generation :
➢ A signal is generated by a process via the kernel.
➢ Whichever generates the signal addresses process to a particular process.
➢ With the help of the process number, the signal is represented, and it does not contain
any additional data or arguments.
➢ So, signals are lightweight. However, we can pass the additional signal with POSIX
real-time signals.
➢ Functions and system calls that can produce signals contain kill, sigqueue, raise,
pthread_kill, kill, and tgkill.
2) Delivery
➢ We said that the signal remains pending until it is delivered.
➢ Basically, in a small amount of time signal is delivered to the process by the kernel.
➢ If the process has blocked the signal, the process will be pending until unblocked.
3) Processing
➢ When the signal is delivered, it is processed in various ways.
➢ Each signal contains a default action such as terminate the process, ignore the signal,
stop the process, continue the process.

Open Source Technology (OST) TY (E&TC) Page 20

 Shree Siddheshwar Women’s College of Engineering, Solapur

➢ For non-default behavior, handler function might be called. Precisely which of these
happens is stated through sigaction function.
Signals List with Meaning
The below table shows the list of the signals along with their meaning:

S.NO. Signal Name Meaning

HUP is a short form of "hang up." Locate the terminal to be

controlled or hung up on the death of the control process. This
1. SIGHUP
signal is received when the process runs from the terminal,
and that terminal goes abruptly.

This signal is released whenever the user sends an interrupt

2. SIGINT
signal (Ctrl + C).

The SIGQUIT signal is issued when the user sends the quite
3. SIGQUIT
signal (Ctrl + D).

It is an Illegal instruction. There is some machine code that is

4. SIGILL contained in the program, and the CPU cannot understand this
code.

Mostly the SIGTRAP is used from inside debuggers and

5. SIGTRAP
program tracers.

The function named abort () is called by the program. It is an

6. SIGABRT
emergency stop.

Open Source Technology (OST) TY (E&TC) Page 21

 Shree Siddheshwar Women’s College of Engineering, Solapur

SIGBUS is an attempt via which memory is accessed

7. SIGBUS
incorrectly. In memory access, it may cause Alignment errors.

The SIGFPE is the floating-point exception that occurred in

8. SIGFPE
the program.

When the process receives the SIGKILL signal, it has to quit

9. SIGKILL
instantly and cannot perform any of the clean-up operations.

10. SIGUSR1 Left to programmers to do whatever they need.

It was attempted to use memory that had not been allocated in

11. SIGSEGV
the process. This was due to the end readings of arrays etc.

12. SIGUSR2 Left to programmers to do whatever they need.

SIGPIPE signal is used when a process generates output

which is being fed to other process consume it through a
13. SIGPIPE
pipe ("{producer | consumer"), and then the consumer dies
and this signal is then sent to the producer.

In this, the process may eventually request a "wake up

call" from the operating system in the future by calling the
14. SIGALRM
function named alarm (). When that time comes, this signal is
contained in the wake-up calls.

Open Source Technology (OST) TY (E&TC) Page 22

 Shree Siddheshwar Women’s College of Engineering, Solapur

This process was apparently done by someone killing the

15. SIGTERM
program.

This process previously creates one or more child processes

16. SIGCHLD
with the fork () function.

(can be read in conjunction with SIGSTOP)

When a process is interrupted by sending it
17. SIGCONT
the SIGSTOP signal, we must then give it
the SIGCONT signal to restart it.

(can be read in conjunction with SIGCONT.)

If a SIGSTOP signal, is sent to the process, then it is stopped
18. SIGSTOP
by the operating system. All of it, the state is ready to restart it
(by SIGCONT), but until then it gets another CPU cycle.

SIGSTP is a short form for "terminal stop." Basically,

the SIGTSTOP signal is the same as the SIGTSTP signal.
19. SIGTSTP
When the user presses Ctrl +Z on the terminal,
the SIGTSTP signal is sent.

The only difference between SIGTSTP and SIGSTOP is that

pausing is only the default action for SIGTSTP but is the
20. SIGTTIN needed action for SIGSTOP. This process may choose to
handle SIGTSTP differently but is not found options
about SIGSTOP.

Open Source Technology (OST) TY (E&TC) Page 23

 Shree Siddheshwar Women’s College of Engineering, Solapur

When a background process attempts to write output to its

21. SIGTOU terminal. SIGTOU signal is sent by the operating system. The
typical response is based on the SIGTTIN.

With the help of the network connection, the SIGURG signal

22. SIGURG is sent by the operating system whenever the "urgent" out-of-
band data is sent.

The SIGXCPU signal is sent by the operating system to the

process, which crosses its CPU limit. With the shell command
23. SIGXCPU ("ulimit -t unlimited"), we can cancel the CPU limit before
making a run, although there are more chances that something
went wrong if you reach the CPU limit in make.

If there is a process that tried to create a file that is above the

file format, then the operating system sends
a SIGXFSZ signal. We are also able to cancel the limit of the
24. SIGXFSZ
file size by using the shell command ("ulimit -t unlimited")
although it is more likely that something went wrong if we
reached the file size limit before the run.

Both SIGVTALR and SIGALRM signals are the same,

except that the SIGALRM signal is sent after some amount
25. SIGVTALRM
of real-time has passed while SIGVTALRM is sent after
some time has been spent to run the process.

26. SIGPROF SIGPROG is just like SIGVTALRM and SIGALRM,

whereas SIGALRM is sent after some amount of real-time

Open Source Technology (OST) TY (E&TC) Page 24

 Shree Siddheshwar Women’s College of Engineering, Solapur

has passed, SIGPROG is sent after spending some amount of

time to run the process, and the system code run on behalf of
the process.

The SIGWINCH signal is used by the process when any of

27. SIGWINCH
its windows are resized.

(SIGIO is also called SIGPOLL). A process that can arrange

28. SIGIO for this signal to be sent is ready to do this when there is some
input process or an output channel has agreed to write.

A power management service sent this signal to the processes

29. SIGPWR in order to show that power had been changed to a short-term
emergency power supply.

30. SIGSYS Unused

Functions in Shell Scripting

➢ Functions in shell scripting are reusable blocks of code that allow you to organize and structure
complex scripts efficiently.
➢ Using functions, you can avoid redundant code, improve readability, and make debugging
easier.
➢ A function is a collection of statements that execute a specified task. Its main goal is to break
down a complicated procedure into simpler subroutines that can subsequently be used to
accomplish the more complex routine.
Importance of Functions
1. Code Reusability: Functions allow you to write code once and use it multiple times.

Open Source Technology (OST) TY (E&TC) Page 25

 Shree Siddheshwar Women’s College of Engineering, Solapur

2. Modularity: Functions help organize scripts into logical sections, each handling a specific task.
3. Simplified Maintenance: With functions, any code changes only need to be updated in one
place, making maintenance easier.
4. Error Handling: Functions allow specific sections of code to be tested and debugged
individually.
Basic Syntax for Creating Functions
➢ In shell scripting, there are two ways to define a function
Syntax 1:

Syntax 2:

Example of a Basic Function

#!/bin/bash
# Define a function
Imran () {
echo "Hello, welcome to shell scripting!"
}
# Call the function
imran
Functions with Parameters
➢ Shell functions can also accept parameters, just like scripts.
➢ These parameters are passed in the same way as arguments to a script.
✓ $1, $2, …, $N represent the arguments passed to the function.
✓ $# represents the total number of arguments passed.

Open Source Technology (OST) TY (E&TC) Page 26

 Shree Siddheshwar Women’s College of Engineering, Solapur

#!/bin/bash
# Function with parameters
wellcome_user() {
echo "Hello, $1! You are $2 years old College ."
}
# Call the function with arguments
wellcome_user “SSWCOE" 6
✓ $1 refers to the first argument (“SSWCOE"), and $2 refers to the second
argument (6).
✓ This function prints a customized greeting based on the parameters passed.
Returning Values from Functions
➢ Shell functions don’t return values in the same way as in other programming languages, but
you can use return to send back a status code (an integer between 0 and 255), and echo to
print output that can be captured by the calling code.
#!/bin/bash
# Function to check if a number is even
is_even() {
if [ $(( $1 % 2 )) -eq 0 ]; then
return 0 # Success
else
return 1 # Failure
fi
}
# Call the function and check the return value
number=4
is_even $number
if [ $? -eq 0 ]; then
echo "$number is even."
else
echo "$number is odd."
fi

Open Source Technology (OST) TY (E&TC) Page 27

 Shree Siddheshwar Women’s College of Engineering, Solapur

Function Scope and Local Variables

➢ By default, variables within functions are global. To make variables local to a function, use
the local keyword
#!/bin/bash
# Function with a local variable
calculate_area() {
local length=$1
local width=$2
echo $((length * width))
}
# Call the function
area=$(calculate_area 5 10)
echo "The area is: $area"
Nested Functions
➢ Functions in shell scripting can be nested, meaning you can define and call functions within
other functions.
#!/bin/bash
# Define a nested function
outer_function() {
echo "This is the outer function."
inner_function() {
echo "This is the inner function."
}
# Call the inner function
inner_function
}
# Call the outer function
outer_function

Recursion in Shell Functions

➢ Shell functions can call themselves, allowing for recursive behavior.

Open Source Technology (OST) TY (E&TC) Page 28

 Shree Siddheshwar Women’s College of Engineering, Solapur

#!/bin/bash
factorial() {
if [ $1 -le 1 ]; then
echo 1
else
prev=$(factorial $(( $1 - 1 )))
echo $(( $1 * prev ))
fi
}
# Calculate factorial of 5
result=$(factorial 5)
echo "Factorial of 5 is: $result"
Important Points about Functions
✓ Use descriptive names for functions to clarify their purpose (e.g., calculate_area).
✓ Document the function purpose with comments, especially if it accepts parameters
or returns specific values.
✓ Use local for variables inside functions to avoid conflicts with global variables.
✓ Avoid side effects: Functions should ideally return values and not change global
variables directly.
✓ Test functions individually to ensure they perform as expected before integrating
them into the main script.

Working sed and gawk

1) SED (Stream Editor) command
➢ The SED (Stream Editor) command in Unix/Linux is a powerful utility used to process and
manipulate text in files.
➢ It can perform a variety of operations such as searching, find-and-replace, insertion, deletion,
and more, without the need to open the file in an editor.
➢ This makes it a highly efficient tool for managing and editing text, especially when working
with large files or automating tasks in scripts.

Open Source Technology (OST) TY (E&TC) Page 29

 Shree Siddheshwar Women’s College of Engineering, Solapur

➢ SED is a powerful text stream editor. Can do insertion, deletion, search, and
replace(substitution).
➢ SED command in UNIX supports regular expression which allows it to perform complex
pattern matching.
➢ The SED (Stream Editor) is a versatile tool for text manipulation, offering commands to
search, replace, delete, and format text efficiently. Its combination of simple syntax and
powerful regular expressions makes it ideal for automating text transformations in shell scripts,
data pipelines, and system administration tasks.
Basic Syntax of sed
sed [options] 'script' file
• Options: Flags that modify sed behaviour.
• Script: The series of commands or transformations you want to apply.
• File: The input file (or files) to be processed.
Example

sed 's/oldtext/newtext/' file.txt

This command substitutes the first occurrence of oldtext with newtext on each line in file.txt.
Common sed Commands
a) Substitute Command: s
✓ The s command is the most commonly used sed command, which performs
substitutions.
Syntax:
sed 's/pattern/replacement/flags' file
o pattern: Text or regex pattern to find.
o replacement: Text to replace the pattern.
o flags: Optional, specify how many substitutions to perform.
Example
1) Basic Substitute: Replace the first occurrence of "apple" with "orange" in line
sed 's/apple/orange/' file.txt
2) Global Substitute (g): Replace all occurrences of "apple" with "orange“
sed 's/apple/orange/g' file.txt
3) Case-Insensitive Substitute (i): Replace "apple" with "orange" regardless of case

Open Source Technology (OST) TY (E&TC) Page 30

 Shree Siddheshwar Women’s College of Engineering, Solapur

sed 's/apple/orange/gi' file.txt

b) Delete Command: d
✓ The d command deletes lines matching a pattern or specific line numbers.
Examples:
1. Delete Matching Pattern: Delete lines that contain "error“
sed '/error/d' file.txt
2. Delete a Specific Line: Delete the 3rd line
sed '3d' file.txt
3. Delete a Range of Lines: Delete lines 5 to 10
sed '5,10d' file.txt
c) Print Command: p
✓ The p command prints lines matching a pattern, often used with the -n option to
suppress default output.
Examples:
1. Print Matching Pattern: Print lines containing "hello
sed -n ' /hello/p’ file.txt
2. Print Specific Lines: Print the 1st, 3rd, and 5th lines
sed -n '1p; 3p; 5p’ file.txt
d) Insert Command: I
✓ The i command inserts a line before the matched line.
Example:
sed '2i\Inserted Line’ file.txt
This inserts "Inserted Line" before the 2nd line.
e) Append Command: a
✓ The a command appends a line after the matched line.
Example
sed '3a\Appended Line’ file.txt
This appends "Appended Line" after the 3rd line.
2) Introduction to gawk
➢ gawk (GNU Awk) is an implementation of the awk programming language, designed for
text processing, particularly in formatted data (such as CSV or tab-separated files).

Open Source Technology (OST) TY (E&TC) Page 31

 Shree Siddheshwar Women’s College of Engineering, Solapur

➢ gawk is used to:

✓ Select and print specific columns
✓ Filter lines based on patterns
✓ Perform calculations
✓ Generate formatted reports
➢ gawk uses patterns and actions to define what it will look for and what it will do with that
data.
Basic Syntax of gawk

gawk 'pattern { action }’ file

• Pattern: Specifies which lines to operate on.
• Action: Specifies what to do with each line that matches the pattern.
Example
gawk '{ print $1 }’ file.txt
This prints the first field (column) of each line in file.txt
gawk Basics
a) Fields and Records
✓ Records: By default, each line in a text file is treated as a record.
✓ Fields: Each space-separated word in a line is treated as a field.
• $1 represents the first field, $2 the second, and so on.
• $0 represents the entire line.
Example : gawk '{ print $1, $3 }’ file.txt
This prints the first and third fields of each line.
b) Basic Operations :
• Printing Fields: Print specific columns
gawk '{ print $2, $4 }’ file.txt
Prints the second and fourth columns in file.txt
• Calculations: gawk can perform arithmetic operations.
gawk '{ sum = $2 + $3; print sum }' file.txt
Adds the second and third fields and prints the result for each line.
• Pattern Matching: Use patterns to match specific lines.

Open Source Technology (OST) TY (E&TC) Page 32

 Shree Siddheshwar Women’s College of Engineering, Solapur

gawk '/pattern/ { print $0 }’ file.txt

Prints lines that contain "pattern".
Common gawk Commands and Patterns
a) print Statement
✓ The print statement outputs specified fields or text.
Example:
gawk '{ print "Line:", $0 }’ file.txt
This adds "Line:" at the start of each line when printed.
b) BEGIN and END Blocks
✓ BEGIN: Executes commands before processing any records.
✓ END: Executes commands after all records are processed.
Example:
gawk 'BEGIN { print "Start of File" } { print $0 } END { print "End of File" }’ file.txt
This prints "Start of File" before any line processing and "End of File" after all lines.
c) Field Separator (-F Option)
✓ By default, gawk splits fields by spaces. Use -F to specify a custom delimiter, such as
a comma for CSV files.
Example :
gawk -F ',’ '{ print $1, $3 }’ file.csv
This prints the first and third fields from a comma-separated file.
d) Pattern Matching with if Statements
Example:
gawk '{ if ($3 > 50) print $1, $3 }’ file.txt
This prints the first and third fields only if the value in the third field is greater than 50.
Advanced gawk Features
a) Using Regular Expressions in gawk
Example:
gawk ‘/error/ { print $0 }’ file.txt
This prints lines containing the word "error".
b) Built-in Variables
✓ NR: Current record (line) number.

Open Source Technology (OST) TY (E&TC) Page 33

 Shree Siddheshwar Women’s College of Engineering, Solapur

✓ NF: Number of fields in the current record.

✓ FS: Field separator.
✓ OFS: Output field separator.
Example
gawk '{ print "Line:", NR, "- Fields:", NF }’ file.txt
Prints the line number and field count for each line.
c) Modifying Field Separators
• Change Field Separator (FS):
gawk 'BEGIN { FS=":" } { print $1, $2 }’ file.txt
Treats each line as colon-separated.
• Change Output Field Separator (OFS):
gawk 'BEGIN { OFS=" - " } { print $1, $2, $3 }’ file.txt
Outputs fields separated by " - ".
d) Loops in gawk
Example:
gawk ' { for (i=1; i<=NF; i++) print $i }’ file.txt
This prints each field in a line on a new line.
Important Point to remember while using gawk
1) Always run commands without modifying files to confirm expected results.
2) Make use of BEGIN and END to handle pre- and post-processing tasks, like printing headers
or totals.
3) Avoid unexpected results by clearly defining patterns and conditions.
4) Leverage -F for CSV or other structured data formats.
5) For lengthy gawk scripts, write the commands in a separate file with an .awk extension and
execute using -f.

Working with web using shell script:

1) Downloading web page

#!/bin/bash
# URL of the web page to download

Open Source Technology (OST) TY (E&TC) Page 34

 Shree Siddheshwar Women’s College of Engineering, Solapur

URL="https://www.google.com"

# Output file name where the downloaded content will be saved

Downloaded_file="downloaded_page.html"

# Using wget to download the web page

wget -O "$Downloaded_file" "$URL"

# Check if the download was successful

if [ $? -eq 0 ]; then
echo "The web page has been successfully downloaded to $Downloaded_file"
else
echo "There was an error downloading the web page."
Fi

2) Converting Web page content to a text file

sudo apt update

sudo apt install wget
sudo apt install lynx -y

#!/bin/bash

# URL of the web page to download

URL="https://www.sswcoe.edu.in/"

# File name for the downloaded HTML content

HTML="downloaded_page.html"

# File name for the converted text content

TEXT="page_content.txt"

Open Source Technology (OST) TY (E&TC) Page 35

 Shree Siddheshwar Women’s College of Engineering, Solapur

# Using wget to download the web page

wget -O "$HTML" "$URL"

# Check if the download was successful

if [ $? -eq 0 ]; then
echo "Web page downloaded successfully."

# Convert the HTML content to plain text using lynx

lynx -dump -nolist "$HTML" > "$TEXT"

# Check if the conversion was successful

if [ $? -eq 0 ]; then
echo "The content has been successfully converted to text and saved in
$TEXT_FILE."
else
echo "Error: Failed to convert HTML to text."
fi
else
echo "Error: Failed to download the web page."
Fi

3) Parsing data

#!/bin/bash

# File to parse stored at some location in your PC

LOG_FILE="/home/ubuntu/imran/logfile.txt"

# Output files for parsed data

ERRORS_FILE="errors.txt"

Open Source Technology (OST) TY (E&TC) Page 36

 Shree Siddheshwar Women’s College of Engineering, Solapur

WARNINGS_FILE="warnings.txt"

# Check if the log file exists

if [ ! -f "$LOG_FILE" ]; then
echo "Log file not found!"
exit 1
fi

# Clear output files if they already exist

> "$ERRORS_FILE"
> "$WARNINGS_FILE"

# Parse the log file

while IFS= read -r line; do
if [[ "$line" == *"ERROR"* ]]; then
echo "$line" >> "$ERRORS_FILE"
elif [[ "$line" == *"WARNING"* ]]; then
echo "$line" >> "$WARNINGS_FILE"
fi
done < "$LOG_FILE"

# Provide summary
echo "Parsing complete."
echo "Errors saved in $ERRORS_FILE."
echo "Warnings saved in $WARNINGS_FILE."

4) Working cURL.

#!/bin/bash

# URL to fetch HTML content

Open Source Technology (OST) TY (E&TC) Page 37

 Shree Siddheshwar Women’s College of Engineering, Solapur

WEB_URL="https://www.google.com"

# API endpoint for GET and POST requests

API_URL="https://jsonplaceholder.typicode.com/posts"

# File to save web content

WEB_OUTPUT="webpage.html"

# File to save API response

API_GET_RESPONSE="api_get_response.json"

# JSON data for POST request

POST_DATA='{"title": "foo", "body": "bar", "userId": 1}'

# Fetch HTML content from a webpage and save to a file

echo "Fetching HTML content from $WEB_URL..."
curl -o "$WEB_OUTPUT" "$WEB_URL"
if [ $? -eq 0 ]; then
echo "Web page content saved to $WEB_OUTPUT."
else
echo "Failed to fetch web page content."
fi

# Send a GET request to the API and save the response to a file
echo "Sending GET request to $API_URL..."
curl -s -o "$API_GET_RESPONSE" "$API_URL"
if [ $? -eq 0 ]; then
echo "API GET response saved to $API_GET_RESPONSE."
else
echo "Failed to fetch data from API."
fi

Open Source Technology (OST) TY (E&TC) Page 38

 Shree Siddheshwar Women’s College of Engineering, Solapur

# Send a POST request to the API with JSON data

echo "Sending POST request to $API_URL with JSON data..."
curl -s -X POST -H "Content-Type: application/json" -d "$POST_DATA"
"$API_URL"
if [ $? -eq 0 ]; then
echo "POST request sent successfully."
else
echo "Failed to send POST request."
fi

Open Source Technology (OST) TY (E&TC) Page 39