CS3401- ALGORITHMS

UNIT I

INTRODUCTION

Algorithm analysis: Time and space complexity

Generally, there is always more than one way to solve a problem in computer science with
different algorithms. Therefore, it is highly required to use a method to compare the solutions
in order to judge which one is more optimal. The method must be:

 Independent of the machine and its configuration, on which the algorithm is running on.
 Shows a direct correlation with the number of inputs.
 Can distinguish two algorithms clearly without ambiguity.

There are two such methods used, time complexity and space complexity which are discussed
below:

Time Complexity: The time complexity of an algorithm quantifies the amount of time taken
by an algorithm to run as a function of the length of the input. Note that the time to run is a
function of the length of the input and not the actual execution time of the machine on which
the algorithm is running on.

Definition: The valid algorithm takes a finite amount of time for execution. The time required
by the algorithm to solve given problem is called time complexity of the algorithm. Time
complexity is very useful measure in algorithm analysis.

It is the time needed for the completion of an algorithm. To estimate the time complexity, we
need to consider the cost of each fundamental instruction and the number of times the
instruction is executed.

Example 1: Addition of two scalar variables.

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers
//Input: Two scalar variables A and B
//Output: variable C, which holds the addition of A and B
C <- A + B
return C

The addition of two scalar numbers requires one addition operation. the time complexity of this
algorithm is constant, so T(n) = O(1) .

In order to calculate time complexity on an algorithm, it is assumed that a constant time c is

taken to execute one operation, and then the total operations for an input length on N are
calculated. Consider an example to understand the process of calculation: Suppose a problem is
to find whether a pair (X, Y) exists in an array, A of N elements whose sum is Z. The simplest
idea is to consider every pair and check if it satisfies the given condition or not.

The pseudo-code is as follows:

int a[n];
for(int i = 0;i < n;i++)
cin >> a[i]

for(int i = 0;i < n;i++)

for(int j = 0;j < n;j++)
if(i!=j && a[i]+a[j] == z)
return true
return false

Below is the implementation of the above approach:

# Python3 program for the above approach

# Function to find a pair in the given

# array whose sum is equal to z
def findPair(a, n, z) :

# Iterate through all the pairs

for i in range(n) :
for j in range(n) :

# Check if the sum of the pair

# (a[i], a[j]) is equal to z
if (i != j and a[i] + a[j] == z) :
return True

return False
# Driver Code
# Given Input
a = [ 1, -2, 1, 0, 5 ]
z=0
n = len(a)
# Function Call
if (findPair(a, n, z)) :
print("True")
else :
print("False")

# This code is contributed by splevel62.

Output
False
Assuming that each of the operations in the computer takes approximately constant time, let it
be c. The number of lines of code executed actually depends on the value of Z. During
analyses of the algorithm, mostly the worst-case scenario is considered, i.e., when there is no
pair of elements with sum equals Z. In the worst case,
 N*c operations are required for input.
 The outer loop i loop runs N times.
 For each i, the inner loop j loop runs N times.

So total execution time is N*c + N*N*c + c. Now ignore the lower order terms since the lower
order terms are relatively insignificant for large input, therefore only the highest order term is
taken (without constant) which is N*N in this case. Different notations are used to describe the
limiting behavior of a function, but since the worst case is taken so big-O notation will be used
to represent the time complexity. Hence, the time complexity is O(N2) for the above
algorithm.

Space Complexity:

Definition: Problem-solving using computer requires memory to hold temporary data or final
result while the program is in execution. The amount of memory required by the algorithm to
solve given problem is called space complexity of the algorithm.

The space complexity of an algorithm quantifies the amount of space taken by an algorithm to
run as a function of the length of the input. Consider an example: Suppose a problem to find
the frequency of array elements.
It is the amount of memory needed for the completion of an algorithm.

To estimate the memory requirement we need to focus on two parts:

(1) A fixed part: It is independent of the input size. It includes memory for instructions (code),
constants, variables, etc.
(2) A variable part: It is dependent on the input size. It includes memory for recursion stack,
referenced variables, etc.

Example : Addition of two scalar variables

Algorithm ADD SCALAR(A, B)

//Description: Perform arithmetic addition of two numbers
//Input: Two scalar variables A and B
//Output: variable C, which holds the addition of A and B
C <— A+B
return C

The addition of two scalar numbers requires one extra memory location to hold the result. Thus
the space complexity of this algorithm is constant, hence S(n) = O(1).
The pseudo-code is as follows:
int freq[n];
int a[n];
for(int i = 0; i<n; i++)
{
cin>>a[i];
freq[a[i]]++;
}

Below is the implementation of the above approach:

# Python program for the above approach

# Function to count frequencies of array items

def countFreq(arr, n):
freq = dict()

# Traverse through array elements and

# count frequencies
for i in arr:
if i not in freq:
freq[i] = 0
freq[i]+=1

# Traverse through map and print frequencies

for x in freq:
print(x, freq[x])

# Driver Code

# Given array
arr = [10, 20, 20, 10, 10, 20, 5, 20 ]
n = len(arr)

# Function Call
countFreq(arr, n)

# This code is contributed by Shubham Singh

Output
51
20 4
10 3
Asymptotic Notations and its properties Best case, Worst case and average case analysis:

Asymptotic notations are mathematical tools used to describe the growth rates of functions,
particularly in the context of algorithms and data structures.

They are commonly used in the analysis of algorithms to understand how their performance
scales with input size. The three most common asymptotic notations are
 Big O
 Big Omega
 Big Theta.
These notations are often used to describe the best case, worst case, and average case
performance of algorithms.

1. Big O notation (O):

 Definition: Big O notation represents the upper bound of an algorithm's running time or
space complexity in the worst-case scenario.
 Best Case: In the best case, an algorithm may perform better than its worst-case time
complexity, but Big O notation focuses on the worst-case scenario.
 Properties:
 O-notation provides an upper bound on the growth rate of a function.
 It describes the worst-case behavior of an algorithm.
 It's often used for analyzing the upper limit on time or space complexity.
 Example: If an algorithm has a worst-case time complexity of O(n^2), it means the
algorithm will never take more than O(n^2) time to execute, where n is the input size.

2. Big Omega notation (Ω):

  Definition: Big Omega notation represents the lower bound of an algorithm's running
time or space complexity in the best-case scenario.
 Worst Case: While Big Omega focuses on the best-case scenario, it is concerned with
the lower bound of the algorithm's performance.
 Properties:
 Ω-notation provides a lower bound on the growth rate of a function.
 It describes the best-case behavior of an algorithm.
 It's often used for analyzing the lower limit on time or space complexity.
 Example: If an algorithm has a best-case time complexity of Ω(n), it means the
algorithm will always take at least Ω(n) time to execute, where n is the input size.

3. Big Theta notation (Θ):

 Definition: Big Theta notation represents both the upper and lower bounds of an
algorithm's running time or space complexity, providing a tight asymptotic bound.
 Average Case: Big Theta is often used in average-case analysis, where the algorithm's
performance is considered across all possible inputs.
 Properties:
 Θ-notation provides both upper and lower bounds on the growth rate of a
function.
 It describes both the best and worst-case behaviors of an algorithm, providing a
precise characterization of its performance.
 Example: If an algorithm has an average-case time complexity of Θ(n log n), it means
the algorithm's running time grows at a rate proportional to n log n on average.

Recurrence relation: Substitution method - Lower bound

Substitution method:

The substitution method is a technique used to solve recurrence relations by making an initial
guess about the form of the solution and then proving it correct using mathematical induction.
Let's go through an example to illustrate this method.

Consider the following recurrence relation:

[ T(n) = 2T(n/2) + n ]
 This recurrence relation represents the time complexity of a recursive algorithm. We want to
find a closed-form solution for \( T(n) \) using the substitution method.

Step 1: Make a Guess:

We'll make an initial guess that the solution is ( T(n) = O(n > log n) ). This is a common
guess for recurrence relations that arise from divide-and-conquer algorithms like Merge Sort.

Step 2: Prove by Induction:

Base Case: For the base case, let's consider ( T(1) ). Since it doesn't follow the recurrence
relation, let's assume ( T(1) = > c), where ( c ) is some constant. Now, for ( n = 1 ), the guess (
O(n =log n)) holds since ( n \log n = 0), and ( c ) is a constant.

Inductive Step: Assume that T(k) >log k \) for all ( k < n ). Now, we need to prove that ( T(n)
log cn log n ).

T(n)=T(n/2)+1≥clog(n/2)+1=c(logn−1)+1=log n+(1−c)

For ( n > 2 ), we have ( log n > 1 ), so ( n > log n - n + n => log n ).

Step 3: Establish Upper Bound:

Since we've proven that ( T(n) ) is bounded above by ( O(n =log n) ), we've established an
upper bound for the time complexity of the algorithm.

So, the solution to the recurrence relation ( T(n) = 2T(n>2) + n ) using the substitution
method is ( T(n) = O(n > log n) ). This means that the time complexity of the algorithm is
asymptotically bounded above by ( n =log n ).

Lower bound:

Establishing lower bounds for recurrence relations involves proving that the solution to the
recurrence relation is at least a certain function. We typically use techniques like the substitution
method or the recurrence tree method to establish lower bounds. Here's a step-by-step guide to
establishing lower bounds using the substitution method:

1. Make a Guess: Start by guessing the form of the solution to the recurrence relation. This
guess should represent a lower bound on the actual solution.
2. Prove by Induction: Use mathematical induction to prove that the guessed lower bound
is correct. This involves two steps:
 Base Case: Show that the guessed lower bound holds for the base case(s) of the
recurrence relation.
 Inductive Step: Assume that the lower bound holds for all values up to a certain
point (inductive hypothesis) and then prove that it also holds for the next value.
 3. Establish Lower Bound: Once you've proven that your guess is a valid lower bound for
the recurrence relation, you can conclude that the solution to the recurrence relation is
bounded below by the guessed lower bound.
4. Optimize (if necessary): If the lower bound obtained through the substitution method is
not tight enough, refine your guess and repeat the process until you find a tighter lower
bound.

Let's illustrate this with an example:

Consider the recurrence relation for the factorial function:

T(n)=n⋅ T(n−1)

We want to establish a lower bound for T(n).

Step 1: Make a Guess: We'll guess that the solution to this recurrence relation is T(n)=Ω(n!), as
the factorial function typically grows at least as fast as n!.

Step 2: Prove by Induction:

 Base Case: For the base case, (1)=1T(1)=1, and 11 is indeed a lower bound on 1!1!.
 Inductive Step: Assume that T(k)≥c⋅ k! for all k<n. Now, we need to prove that! T(n) ≥c⋅
n!: T(n)=n⋅ T(n−1)≥c⋅ n⋅(n−1)!=c⋅ n!

Step 3: Establish Lower Bound: Since we've proven that T(n) is greater than or equal to c⋅n!,
we've established a lower bound for the recurrence relation.

 In this example, we've shown that the solution to the recurrence relation T(n)=n⋅ T(n−1)
is bounded from below by Ω(n!).

Searching: linear search, binary search and Interpolation Search:

Linear Search:
Linear search, also known as sequential search, examines each element in the collection
one by one until the target element is found or the end of the collection is reached.
It is simple to implement and works well for small collections or unsorted data.

 Algorithm: Linear search scans each element of the array one by one until the target
element is found or the end of the array is reached.
 Time Complexity: In the worst-case scenario, when the target element is at the end of
the array or not present, linear search has a time complexity of O(n), where n is the
number of elements in the array.
 Space Complexity: Linear search requires constant space O(1) because it doesn't use any
extra space.
  Suitability: Linear search is suitable for small arrays or when the elements are not sorted.
It is simple and easy to implement.

Binary Search:
Binary search is an efficient searching algorithm for sorted collections. It works by
repeatedly dividing the search interval in half until the target element is found or the interval is
empty. It requires the collection to be sorted, but it's much faster than linear search for large
collections.

 Algorithm: Binary search works on sorted arrays. It repeatedly divides the search
interval in half until the target element is found or the interval is empty.
 Time Complexity: Binary search has a time complexity of O(log n), where n is the
number of elements in the array. This is because it halves the search space in each
iteration.
 Space Complexity: Binary search requires constant space O(1) because it doesn't use any
extra space.
 Suitability: Binary search is suitable for sorted arrays. It is efficient for large arrays as it
reduces the search space significantly in each step.

Interpolation Search:
Interpolation search is a variation of binary search that works well for uniformly
distributed collections Instead of always dividing the search interval in half, it uses interpolation
to estimate the position of the target element.

 Algorithm: Interpolation search is an improvement over binary search for uniformly

distributed arrays. Instead of always dividing the search space in half, it uses
interpolation to estimate the position of the target element.
 Time Complexity: In the best-case scenario, interpolation search has a time complexity
of O(log log n), which is better than binary search. However, in the worst-case scenario,
it can degrade to O(n) if the array is not uniformly distributed.
 Space Complexity: Interpolation search requires constant space O(1) because it doesn't
use any extra space.
 Suitability: Interpolation search is suitable for uniformly distributed sorted arrays. It can
be faster than binary search when the elements are evenly distributed.

Pattern search:

The Naive String Matching Algorithm

The naive string matching algorithm, also known as the brute-force or simple search algorithm,
is a straightforward pattern search technique used to find occurrences of a pattern within a text. It
works by systematically comparing the pattern with substrings of the text to determine if there is
a match.

Here's how the naïve string matching algorithm works:

 1. Input:
 Text (T): The string in which we want to search for occurrences of a pattern.
 Pattern (P): The string we are searching for within the text.

2. Algorithm:
 Iterate through each position i in the text T where 0≤i≤n−m, where n is the length of the
text and m is the length of the pattern.
 For each position i, check if the substring of length m starting at position i matches the
pattern P.
 If a match is found at position i, record the position or perform any desired action (e.g.,
count occurrences, print position, etc.).

3. Pseudocode:
NaiveStringMatch(T, P):
n = length(T)
m = length(P)
for i from 0 to n - m do:
if T[i:i+m] == P:
output "Pattern found at position", i

4. Complexity:
 Time Complexity: In the worst-case scenario, the naïve string matching algorithm has a
time complexity of O((n−m+1)m), where n is the length of the text and m is the length of
the pattern.
 Space Complexity: The space complexity is O(1) since no additional data structures are
used.

5. Example:
 Text T = "ABABCABABCDABCDABDE", Pattern P = "ABCDABD"
 Iterating through each position in the text, we find a match at position 10:
"ABCDABD" matches the substring starting at position 10 in the text.

Rabin-Karp Algorithm:

The Rabin-Karp Algorithm is a string searching algorithm that uses hashing to find any one of
a set of pattern strings in a text. It was developed by Michael O. Rabin and Richard M. Karp in
1987. The key idea behind the Rabin-Karp algorithm is to hash the pattern and then slide a
window of the same length as the pattern over the text, hashing each substring of the text. This
allows for efficient comparison of the hash values of substrings with the hash value of the
pattern, eliminating the need for exhaustive comparisons.

Here's how the Rabin-Karp algorithm works:

1. Preprocessing:
 Compute the hash value of the pattern.
  Compute the hash values of all substrings of the text of the same length as the pattern
using a rolling hash function.
2. Matching:
 Slide a window of the same length as the pattern over the text.
 Compute the hash value of the current window.
 If the hash value of the current window matches the hash value of the pattern, perform a
full comparison of the substring with the pattern to verify the match.
 If a match is found, record the position of the match or perform any desired action (e.g.,
count occurrences, print position, etc.).
3. Rolling Hash Function:
 A rolling hash function is used to efficiently update the hash value as the window slides
over the text.
 It allows for the hash value of the next window to be computed in constant time, given
the hash value of the previous window and the characters entering and leaving the
window.
4. Collision Handling:
 Since hash values can collide, i.e., different substrings may have the same hash value, it
is necessary to perform a full comparison of the substring with the pattern to verify the
match.
 This is done when the hash values of the current window and the pattern match.
5. Complexity:
 The average-case time complexity of the Rabin-Karp algorithm is O (n+m), where n is
the length of the text and m is the length of the pattern.
 The worst-case time complexity is O(nm), but this is rare in practice and typically occurs
when the rolling hash function has poor distribution.

Knuth-Morris-Pratt algorithm:

The Knuth-Morris-Pratt (KMP) algorithm is a string searching algorithm used to find

occurrences of a pattern within a text. It was developed by Donald Knuth, Vaughan Pratt, and
James H. Morris in 1977. The KMP algorithm efficiently avoids unnecessary character
comparisons by utilizing information about the pattern itself.

Here's an overview of how the Knuth-Morris-Pratt algorithm works:

1. Preprocessing:
 Construct a prefix function (also known as the failure function) for the pattern.
 The prefix function, denoted as π, for a pattern P of length m is an array of length m such
that π[ I ] represents the length of the longest proper prefix of the substring P[0:i] that is
also a suffix of P[0:i].
 The prefix function can be computed in O(m) time using a linear-time algorithm.

2. Searching:
 Iterate through the text T from left to right.
  Maintain a state q representing the length of the longest proper prefix of the substring T
that matches a prefix of the pattern P.
 Compare each character of the text with the corresponding character of the pattern.
 If a mismatch occurs at position i of the text and position j of the pattern, update the state
q using the prefix function until a match is found or the state becomes 0.
 If the state q equals the length of the pattern m, a match is found at position I − m.

3. Complexity:
 The preprocessing step to compute the prefix function takes O(m) time, where m is the
length of the pattern.
 The searching step takes O(n) time, n is the length of the text.
 Therefore, the overall time complexity of the Knuth-Morris-Pratt algorithm is O(m + n).

The Knuth-Morris-Pratt algorithm efficiently avoids unnecessary character comparisons by

utilizing information about the pattern's structure encoded in the prefix function. This makes it
particularly efficient for large texts and patterns, and it outperforms brute-force algorithms like
the naive string matching algorithm in many cases.

Sorting:

Insertion sort:

Insertion Sort is a simple sorting algorithm that builds the final sorted array one element at a
time. It repeatedly takes the next element from the unsorted part of the array and inserts it into its
correct position in the sorted part of the array. Here's a brief overview:

1. Input:
 Array A of n elements to be sorted.

2. Algorithm:
 Start with the second element (i=1) and iterate through the array.
 For each element at index i, compare it with the elements to its left (sorted part) until
finding the correct position for insertion.
 Shift elements to the right to make space for the current element, and insert it into the
correct position.
 Continue this process until all elements are sorted.

3. Complexity:
 Time Complexity:
 Best-case: O(n) when the array is already sorted.
 Average-case: (2)O(n2).
 Worst-case: (2)O(n2) when the array is in reverse sorted order.
 Space Complexity: (1)O(1) - In-place sorting algorithm.
  Stability: Insertion Sort is stable, meaning it preserves the relative order of equal
elements.

Heap Sort:
Heap Sort is a comparison-based sorting algorithm that leverages the structure of a binary heap
to efficiently sort elements in an array. Here's a brief overview:

1. Input:
 Array A of n elements to be sorted.

2. Algorithm:
 Build a max-heap from the input array. This involves rearranging the elements of the
array to satisfy the heap property, which ensures that the parent node is greater than or
equal to its children (for a max-heap).
 Extract the root element (maximum element in a max-heap) and place it at the end of the
array. This operation will decrement the heap size.
 Restore the heap property by heapifying the remaining elements to maintain the heap
structure.
 Repeat the extraction and heapification process until the heap is empty. At each iteration,
the maximum element is placed at the end of the array, effectively sorting the array in
ascending order.
 If a min-heap is used instead of a max-heap, the algorithm will sort the array in
descending order.

3. Complexity:
 Time Complexity:
 Best-case: O(n log n).
 Average-case: O(n log n).
 Worst-case: O(n log n).
 Space Complexity: O(1) - In-place sorting algorithm.
 Stability: Heap Sort is not stable by default, but it can be modified to be stable.

Heap Sort is efficient for sorting large datasets and offers guaranteed O(n log n) time complexity.
It's widely used in practice for its simplicity and versatility. Additionally, it can be used to
efficiently implement priority queues due to its ability to maintain a heap data structure.