Unit 1 Basics of An Algorithm and Its Properties: Muhammad Ibn Musa Al-Khowarizmi.
Unit 1 Basics of An Algorithm and Its Properties: Muhammad Ibn Musa Al-Khowarizmi.
1.0 Introduction 1
1.1 Objectives 2
1.2 Example of an Algorithm 3
1.3 Basics Building Blocks of Algorithms 4
1.3.1 Sequencing, Selection and Iteration 5
1.3.2 Procedure and Recursion 6
1.4 A Survey of Common Running Times 8
1.5 Analysis & Complexity of Algorithms 14
1.6 Types of Computational Problems 17
1.7 Problem Solving Techniques 20
1.8 Deterministic and Stochastic Algorithms 22
1.9 Summary 23
1.10 Solution to Check Your Progress 24
1.11 Further Readings 26
1.0 Introduction
1
Introduction to Algorithm 4. Effectiveness: Every step of the algorithm should be basic and
essential.
In this unit, the basics of the algorithms and its designing process will be
discussed. Section 1.3 will define the algorithm and its uses with suitable
example. An algorithm is designed with three basic building blocks, namely
sequencing, selection, and iteration. A detailed discussion about these building
blocks of an algorithm is presented in Section 1.4.
1.1 Objectives
2
Differentiate between deterministic and stochastic algorithms. Basic of An Algorithm
{
while b ≠ 0 do
{
r ← a mod b;
a ←b;
b← r;
} [end of while loop]
return (b)
// a and b are two positive numbers where a is dividend and b is a divisor
} [end of algorithm]
1. If b=0, return a and exit
2. else go to step 3
3. Divide a by b and assign remainder to r
4. Assign the value of b to a and the value of r to b and go back to step 1
The above algorithm has two inputs and one output. The algorithm is finite as
it terminates in finite steps and produces the desired result. To observe the
same, let us find the GCD of a = 1071 and b = 462 using Euclid’s algorithm.
Iteration 1:
3
Introduction to Algorithm Here, r is not zero, so we will go to Step 3.
3. The integer will get the current value of integer b and the new value of
integer b will be the current value of r.
Here, a=462 and b=147
4. Go back to Step 1.
Iteration 2:
4
1.3.1 Sequencing, Selection and Iteration Basic of An Algorithm
Let’s consider the example of finding GCD of (a, b) with Euclidian method to
understand the basic building blocks of an algorithm(Algorithm 1)
Sequencing: A problem can be solved by performing some actions in a
sequence [called algorithm], and the order of execution of those actions is
important to ensure the correctness of an algorithm.
If the order of steps of algorithm changes and does not follow the steps as
specified, it will not produce the correct output as expected.
Line 5: r ← a mod b;
Line 6: a ←b;
Line 7: b← r;
Line 8: } [end of while loop]
Line 9: return (b)
5
Introduction to Algorithm Line 10: } [end of algorithm]
In above algorithm line 5, line 6 and line 7 are example of sequencing, as these
statements are always executed in sequence as written the text.
Line 3 also acts as iteration or the looping statements. Based on the while loop
condition, the Line 4 to line 8 are executed in repeatedly manner.
(i) Procedure
(ii) Recursion
Procedure
Among a number of terms that are used in stead of procedure are: subprogram
and even function. These terms may have shades of differences in their usage
in different programming languages. However, the basic idea behind these
terms is the same, and is explained next.
It may happen that a sequence of statements frequently occur either in the
same algorithm repeatedly in different parts of the algorithm or may occur in
different algorithms. In such cases, writing repeatedly the same sequence of
statements is a wasteful activity. Procedure is a mechanism to avoid it. For
example we can define GCD(a,b) as a procedure/function only once and can
call it a number of times in a main function with different values of a and b
The general format for defining a procedure might look like this:
Procedure <Name> (<parameter-list>) [: < type>]
<declarations>
<sequence of instructions expected to be occurred repeatedly>
end;
In cases of procedures which pass a value to the calling program another basic
construct (in addition to assignment, read and write) viz., return (x) is used,
where x is a variable used for the value to be passed by the procedure.
Recursion
factorial (1) = 1
6
For those who are familiar with recursive definitions like the one given above Basic of An Algorithm
for factorial, it is easy to understand how the value of (n!) is obtained from the
above definition of factorial of a natural number. However, for those who are
not familiar with recursive definitions, let us compute factorial (4) using the
above definition.
By definition
factorial (4) = 4 * factorial (3).
Again by the definition
factorial (3) = 3 * factorial (2)
Similarly
factorial (2) = 2* factorial (1)
And by definition
factorial (1) = 1
Substituting back values of factorial (1), factorial (2) etc., we get factorial (4) =
4.3.2.1=24, as desired.
In the following procedure factorial (n), let fact be the variable which is used
to pass the value by the procedure factorial to a calling program. The variable
fact is initially assigned value 1, which is the value of factorial (1).
fact: integer;
begin
fact 1
else begin
fact n * factorial (n ─ 1)
return (fact)
end;
end;
7
Introduction to Algorithm (i) There must be in-built mechanism in the computer system that
supports the calling of a procedure by itself, e.g, there may be in-built
stack operations on a set of stack registers.
(ii) There must be conditions within the definition of a recursive procedure
under which, after finite number of calls, the procedure is terminated.
(iii) The arguments in successive calls should be simpler in the sense that
each succeeding argument takes us towards the conditions mentioned in
(ii).
Recursion is an important construct which will be used extensively to solve
sorting algorithms, searching algorithm, matrix multiplications, etc.
For a given problem, more than one algorithm can be designed. However, one
algorithm may be better than the other. To compare two algorithms for a
problem, running time is generally used which is defined as the time taken by
an algorithm in generating the output. An algorithm is better if it takes less
running time. However, this measure should be invariant to any hardware used.
Therefore, the running time of an algorithm can be represented in terms of the
number of operations executed for a given input. More the number of
operations, the larger the running time of an algorithm. So, if we can find the
number of operations required for a given input in an algorithm then we can
measure the running time. This running time of an algorithm for producing the
output is also known as time complexity.
Following are the generalized form of running time for the algorithms:
1. Constant Time (O(k)): If the running time does not depend on the
input size (n) then it is known as constant running time. It can be
represented as
8
where k is a constant Basic of An Algorithm
T(n)=k
k
minimum = a[1]
for ί = 2 to n
if a[ί] < minimum
minimum = a[ί]
end
end if
T(n)=kn
k
T(n)=kn
9
Introduction to Algorithm of input remains half of that of previous iteration, then it is known as
logarithmic time complexity and depicted as O(log n) time. For
example running time of binary search algorithm is O(log n). O(n log
n) is a very common running time for many algorithms which are
solved through divide and conquer technique such as Merge sort ,
Quick sort algorithms, etc., The common operations among all these
problems are in splitting of the array in equal sized sub-arrays and then
solve it recursively.
T(n)=log(n)
T(n)=log(n)
Figure (c) T(n) = log(n)
Quadratic Time: (T(n)= O(n)2)- It occurs when the algorithm is having a pair
of nested loops. The outer loop iterates O(n) time and for each iteration the
inner loop takes O(n) time so we get O(n2) by multiplying these two factors of
n. Practically this is useful for problem for small input size or elementary
sorting algorithms. The worst case time complexity for Bubble sort, Insertion
sort, Selection sort and insertion sort running time complexities are O(n2)
T(n)=n2
10
is n (ie each set is having n elements). The problem is to find whether Basic of An Algorithm
some pairs of these sets are disjoint, i.e there are no common elements
in these pairs and what is the time complexity ?
T(n)=
Time Complexity- The innermost loop takes O(n) time because of n elements.
The second inner loop over 𝑆𝑗 also takes O(n) iterations around the innermost
loop and finally O(n) over 𝑆𝑖 around 𝑆𝑗 iterations. Multiplying all the three
iterations we obtain O(𝑛3 ) time complexity.
11
Introduction to Algorithm example let us consider a problem to find an independent set in a graph
which can be defined as a set of nodes which are not joined by any
edge. Let us formulate the independent set problem in the following
way: given a constant k and a graph G having n nodes (vertices) find
out an independent set of a size k.
The brute force method to solve this problem would require searching
for all subsets of k nodes and for each subset it would examine whether
there is an edge connecting any two nodes for each subset s of a size k .
Below is a pseudo-code for finding an independent set.
Pseudo-code
for each subset s of a size k in a graph G
check whether s is an independent set
if yes, print “ s is an independent set
else stop
In this case the outer loop will iterate O(nk) times and it selects all k-
node subsets of n node of the graph. In the inner loop within each
subset it loops for each pair of nodes to find out whether there is an
edge between the pair which will require O( 2 out of k)
pairs of search i.e.O(k2 ) search. Therefore the total time now is O(k2
nk). Since k is a constant, it can be dropped, finally it is O(nk).
T(n)=nk
12
Pseudo-code : Basic of An Algorithm
Input G(V,E)
{
for each subset s of n number of nodes
verify whether s is an independent set
if s is the largest among all the subsets examined so for
print “s is the largest independent set ”
end if
end for
}end of code fragment
Verification of all pairs of subsets i.e. (2n) whether these subsets are
having edges or not and then selecting the maximum will be O(n2) i.e
the total number of pair of subsets. The total running time would be
O(n2*2n) or O(2n ) .O(2n) running time complexity arises when a search
algorithm considers all subsets of n elements.
The second boy will be left with (n-1) choices among girls for comparison.
There will be only (n-2) options for matching for the third boy, and so forth.
13
Introduction to Algorithm after array girls Multiplying all these options for n boys we obtain n! ie. n(n-
1) (n-2) .......(2) (1)
(ii) O(n!) also occurs where the problem requires arranging n elements into
a particular order (i.e. a permutation of n numbers). A classic example
is travelling salesman problem. Given a n number of cities with
distance between all pairs of cities with the following conditions (i) the
salesman can start the tour with any city but must conclude the tour
with the starting city only (ii) all cities must be visited only once
except the one where from the tour starts. The problem is to find out
the shortest tour covering all n cities. Applying a brute force approach
to find out the solution, a salesman has to explore n! searches which
will take O(n!). Note that a salesman can pick up any city among n
cities to start the tour. Next it will have (n-1) cities to pickup the second
city on the tour. There will be (n-2) cities to pick up the third city at the
next stage and so forth. Multiplying all these choices we get n! i.e. n (n-
1) (n-2) ....(2) (1)
Suppose M is an algorithm with n the input data size. The time and space used
by the algorithm M are the two main measures for the efficiency of M. The
time is measured by counting the number of key operations, for example, in
case of sorting and searching algorithms, the number of comparisons is the
number of key operations. That is because key operations are so defined that
the time for the other operations is much less than or at most proportional to
the time for the key operations. The space is measured by counting the
maximum of memory needed by the algorithm.
The complexity of an algorithm M is the function f(n), which give the running
time and/or storage space requirement of the algorithm in terms of the size n of
the input data. Frequently, the storage space required by an algorithm is simply
a multiple of the data size n. In general the term “complexity” given anywhere
simply refers to the running time of the algorithm. There are 3 cases, in
general, to find the complexity function f(n):
Worst-case − The maximum number of steps taken on any instance of
size a.
Best-case − The minimum number of steps taken on any instance of
size a.
Average case − An average number of steps taken on any instance of
size a.
14
Basic of An Algorithm
Average case: The value of which is in between maximum and minimum
for any possible input. Generally the Average case implies the expected value
of
The analysis of the average case assumes a certain probabilistic distribution
for the input data; one such assumption might be that all possible
permutations of an input data set are equally likely. The Average case also
uses the concept of probability theory. Suppose the numbers ……
occur with respective probabilities 𝑝1 , 𝑝2 , … … 𝑝𝑘, Then the expectation or
average value of E is given by
Best case: Clearly the best case occurs when x is the first element in the array
A. That is . In this case
Worst case: Clearly the worst case occurs when x is the last element in the
array A or is not present in given array A (to ensure this we have to search
15
Introduction to Algorithm entire array A till last element). In this case, we have
.
Average case: Here we assume that searched element appear array A, and it
is equally likely to occur at any position in the array. Here the number of
comparisons can be any of numbers 1,2,3,…,n, and each
Asymptotic notation gives the rate of growth, i.e. performance, of the run time
for “sufficiently large input sizes” and is not a measure of the
particular run time for a specific input size (which should be done
empirically). O-notation is used to express the Upper bound (worst case); Ω-
notation is used to express the Lower bound (Best case) and Θ- Notations is
used to express both upper and lower bound (i.e. Average case) on a function.
We generally want to find either or both an asymptotic lower bound and upper
bound for the growth of our function. The lower bound represents the best
case growth of the algorithm while the upper bound represents the worst case
growth of the algorithm.
16
Basic of An Algorithm
1.6 Types of Computational Problems
Sorting
The sorting is the process to arrange the given set of items in a certain order,
assuming that the nature of the items allow such an ordering. For example,
sorting a set of numbers in increasing or decreasing order and sorting the
character strings, like names, in an alphabetical order.
1. Stability
2. In-place.
A sorting algorithm is called stable if it does not change the relative positions
of any two equal items of input list. Say, in an input list, there are two equal
items at positions i and j where i< j, then the final position of these items in the
sorted list should also be k and l respectively, such that k<l. That is there
should not be any swapping among these equal items and should not
interchange their position with each other.
17
Introduction to Algorithm A sorting algorithm needs extra memory space to store elements during the
swapping process. For small set of items in a list, this constraint is not
observable but, for an input list of large elements the required storage space is
considerable large. An algorithm is said to be in-place if the required extra
memory is not markable.
Searching
Searching is finding an element, referred as search key, in a given set of items
(may have the redundant value). Searching is one of the most important and
frequently performed operation on any dataset/database.
String Processing
Exponential increase in the textual data due to the various applications over
social media and blogs, string-handling algorithms become a current area of
research . Another reason for blooming strings rather text processing is the
kind of data available and the use of the data. Most of the text data is used to
predict the interest of people involving direct or indirect monetary benefits for
commercial organizations specially e-commerce sectors. One of the most
widely used search engine (Google) is also based on string processing.
Graph Problems
It is always favourable for researchers to map a computational problem to a
graph problem. Many computational problems can be solved using graph.
Most of the computer network problems can be solved using graph Algorithms
efficiently. Problems like: visiting all the nodes of a graph (broadcasting in
network), routing in networks (finding the minimum cost path, i.e. the shortest
path, path with minimum delay etc. can be solved efficiently with graph
algorithms.
At the same time some of the graph problems are computationally not easy,
like the travelling salesman and the graph-coloring problems. The Travelling
Salesman Problem (TSP) is used to cover n cities by taking the shortest path
and not visiting any of the city more than once. The graph-coloring
problem seeks to color all the vertices of a graph with minimum number colors
such that, no two adjacent vertices having the same color. While solving TSP
cities can be considered as the vertices of the graph. Event scheduling could be
18
one of the problems which can be solved using graph coloring algorithm. Basic of An Algorithm
Considering events to be represented by the vertices, there exists an edge
between two events only if the corresponding events cannot be scheduled at
the same time.
Combinatorial Problems
These types of problems have a combination of solutions i.e. more than one
solution are possible. The aim of the combinatorial problems is to find
permutations, combinations, or subsets, satisfying the given conditions. The
travelling salesman problem, independent set and the graph-coloring problems
can be categorized as examples of combinatorial problems . From both
theoretical as well as practical point of view, the combinatorial problems are
considered to be one of the most difficult problems in computing. Due to the
combinatorial type of solutions, it becomes very difficult to handle the
problems with big size inputs sets. The number of combinatorial objects (the
output solution) grows rapidly with the problem’s size.
Another problem is the lack of the known algorithms to solve this type of
problems within considerable amount of time. Moreover, it is believed by the
most computer scientists that such algorithms do not exist.
Even though there exist solutions to some combinatorial problems, but these
are considered as fortunate exceptions to the rule. The problem of finding
shortest-path in a network is one of such exceptions.
Geometric Problems
Some of the applications of Geometric algorithms are computer graphics,
robotics and tomography. These algorithms are based upon geometric objects
such as points, lines, and polygons. The geometry procedures are developed to
solve various geometric problems, like construction shapes of geometric
objects, triangles, circles, etc., using ruler and compass.
The closest-pair problem is to find the closest pair out of a given set of points
in the plane.
Numerical Problems
Problems of numerical computing nature are simultaneous linear equations
(linear algebra), differential equations, definite integration, and statistics.
Most of the numerical problems could be solved approximately.
19
Introduction to Algorithm The biggest drawback of numerical algorithms is the accumulation of errors
over the multiple iterations, due to rounding off the approximated result at
each iteration.
Brute force and exhaustive search algorithms are known as blind algorithms.
These algorithms create and evaluate every possible solution and take
exponential and factorial running time. In the previous section we discussed
three such problems: independent set, bipartite matching and travelling
salesman problem. It can be understood by a simple example of finding the
correct four letters word using Brute Force and Exhaustive Search Algorithms.
When the problem size increases, its possible outcomes increases
exponentially which is practically impossible to find.
Step 1. Divide the problem (top level) into a set of sub-problems (lower
level).
Step 2. Solve every sub-problem individually by recursive approach.
Step 3. Merge the solution of the sub-problems into a complete solution of
the problem.
Following are the examples of the problems that can efficiently be solved
using divide and conquer approach.
Binary Search.
Quick Sort.
Merge Sort.
Strassen's Matrix Multiplication.
Closest Pair of Points.
20
Greedy Technique Basic of An Algorithm
Greedy algorithm always picks the best choice (greedy approach) out of
many at a particular moment to optimize a given objective.
The greedy method chooses the local optimum at each step and this
decision may result in overall non-optimum or optimum solution.
The greedy approach doesn't always produce the optimal solution rather
produces very nearby solution to the optimal solution.
Consequently, Greedy algorithms are often very easy to design for the
optimisation problems. Following are some of the examples of the greedy
approach.
Dynamic Programming
21
Introduction to Algorithm Branch and Bound
Branch and bound algorithm efficiently solves the discrete and combinatorial
optimization problems. In branch-and-bound algorithm, a rooted tree is formed
with the full solution set at the root. The algorithm explores the branches of
this tree, representing the subsets of the solution set. A candidate solution of a
root node is considered as a branch only if it is better than the already explored
solution, and is discarded if it cannot produce a better solution than the best
one found so far by the algorithm. Branch and Bound algorithm are methods
for solving global optimization problems. However, it is much slower. Indeed,
it often leads to exponential time complexities in the worst case. On the other
hand, if applied carefully, it can lead to algorithms that run reasonably fast on
average. The general idea of B&B is a BFS-like search for the optimal
solution, but not all nodes get expanded (i.e., their children generated).
Randomized Algorithms
Backtracking Algorithm
S.
Set A S.N. Set B
N.
Geometric
2 B Finding an item in set items.
Problem
1.9 Summary
23
Introduction to Algorithm Problems generally fall into any one of the commonly known categories,
namely sorting, searching, string processing, graph problems, combinatorial
problems, geometric problems, numerical problems, but may be overlapping
also. Many graph theoretic problems can be also combinatorial problems
Similar type of problems can be solved with similar approach. Some of the
commonly used problems solving techniques are Brute Force and Exhaustive
search approach, Divide and Conquer approach, Greedy technique, Dynamic
Programming, Branch and Bound, Randomized algorithms, and Backtracking
algorithm.
24
Ans. Independent set problem can be defined as a set of nodes which are not Basic of An Algorithm
joined by any edge. One way to formulate this problem is that given a constant
k and a graph G having n nodes (vertices) find out an independent set of a size
k.
S.
Set A S.N. Set B
N.
Geometric
2 B Finding an item in set items.
Problem
1 – A, 2 – F, 3 – C, D, 4 – E, 5 – B, 6 – G.
25
Introduction to Algorithm Eight queen puzzle
Map coloring
Sudoku
Following problems can be solved using dynamic programming approach:
26
UNIT- 2 ASYMPTOTIC BOUNDS Asymptotic Bounds
Structure
2.0 Introduction 1
2.1 Objectives 2
2.2 Some Useful Mathematical Functions &Notations 2
2.2.1 Summation & Product 2
2.2.2 Function 3
2.2.3 Logarithms 3
2.2.4 Theorem, Lemmas and Corollary 4
2.3 Mathematical Expectation 5
2.4 Principle of Mathematical Induction 6
2.5 Concept of Efficiency of an Algorithm 7
2.6 Asymptotic Analysis 8
2.6.1 Worst Case and Average Case Analysis 9
2.6.2 Drawbacks of Asymptotic Analysis 10
2.7 Asymptotic Notations & Some Useful Theorems 10
2.7.1 Big-O notation: Upper Bounds 10
2.7.2 Big-Omega() Notation 12
2.7.3 Θ (Theta) notation: Tight Bounds 12
2.7.4 Some Useful Theorems for O, Ω, Θ 13
2.8 Summary 15
2.9 Solutions/Answers 15
2. 0 INTRODUCTION
The above mentioned criterions are used as thebasis of the comparison among
different algorithms. With the help of algorithmic complexity, programmers
improve the quality of their code using relevant data structures. To measure the
efficiency of a code/ algorithm, asymptotic notations are normally used.
Asymptotic notations are the mathematical notations that estimate the time or
space complexity of an algorithm or program as function of the input size. For
example, the best case running time of a function that sorts a list of numbers
using bubble sort will be linear i.e., O(n). On the contrary, the worst case
running time will be O(n2). So, we can say that the bubble sort takes T(n) time,
where, T(n)=O(n2) . The asymptotic behavior of a function f(n) indicate the
the growth of f(n) as n gets very large. The small values of n are generally
1
Introduction to Algorithm ignored as we are interested to know how slow the program or algorithm will
be on large input. It is stated in literature- the slower asymptotic growth rate,
the better the algorithm performance. As per this measurement, a linear
algorithm (i.e., f(n)=d*n+k) is always asymptotically better than a quadratic
one (e.g., f(n)=c*n2+q) for any positive value of c, k, d, and q. To understand
concepts of asymptotic notations, you will be given a idea of lower bound,
upper bound, and an average bound. Mathematical induction plays an
important role in computing the algorithms’ complexity. Using the
mathematical induction, problem is converted in the form mathematical
expression which is solved to find the time complexity of algorithm. Further
to rank algorithms in increasing or decreasing order asymptotic notations such
as big oh, big omega, and big theta are used.
The focus of the unit is to define these bounds mathematically and discuss
some useful theorems related to these bounds.
2. 1 Objectives
Summation:
Sum of sequences
𝑛(𝑛+1)
(i) ∑𝑛𝑖=1 𝑖 = 1 + 2 + ⋯ 𝑛 =
2
𝑛(𝑛+1)(2𝑛+1)
(ii) ∑𝑛𝑖=1 𝑖 2 = 12 + 22 + ⋯ 𝑛2 =
6
𝑛 2 (𝑛+1)2
(iii) ∑𝑛𝑖=1 𝑖 3 = 13 + 23 + 33 + ⋯ 𝑛3 =
4
Product
The expression
1 2 …n
can be denoted in shorthand as
2
𝑛
Asymptotic Bounds
∏𝑖
𝑖=1
2.2.2 Function
For two given sets A and B a rule f which associates with each element of A, a
unique element of B, is called a function from A to B. If f is a function from a
set A to a set B then we denote the fact by f: A B. For example the function
f which associates the cube of a real number with a given real number x , can
be written as 𝑓(𝑥) = 𝑥 3
Suppose the value of x is 2 there f maps 2 to 8
Floor Function: Let x be a real number. The floor function denoted as x
maps each real number x to the integer, which is the greatest of all integers
less than or equal to x.
Ceiling Function: Let x be a real number. The ceiling function denoted as x
maps each real number x to the integer, which is the least of all integers
greater than or equal to x.
x ─ 1 < x x x < x + 1.
2.2.3 Logarithms
Logarithms are important mathematical tools which are widely used in analysis
of algorithms.
3
Introduction to Algorithm Modular Arithmetic/Mod Function
The modular function or mod function returns the remainder after a number
(called dividend) is divided by another number called divisor.
Many programming language has a similar function.
Definition
b mod n = 42 mod 11 = 9.
If b = ─ 42 and n = 11 then
b mod n = ─ 42 mod 11 = 2 ( ─ 42 = (─ 4) 11 + 2)
These terms are often used when writing proofs in algorithms and mathematics
but they are not identical
A statement is a sentence which has objective and logical meaning.
Proof : The proof would be written here. In this case, it would be the induction
proof.
A corollary is a result from a theorem that doesn’t require too much proof to
show. Corollaries are special cases of theorems.
4
2. 3 Some Mathematical Expectation Asymptotic Bounds
Example 2.1: Suppose, the students of MCA, who completed all the courses in
the year 2005, had the following distribution of marks.
Assuming that marks within a class are uniformly scored by the students in the
class, the above table may be approximated by the following more concise
table:
Thus, we assign weight (8/100) to the score 10% ( 8, out of 100 students,
score on the average 10% marks); (20/100) to the score 30% and so on.
Thus
5
Introduction to Algorithm 8 20 57 9
Expected% = of marks 10 × 100 + 30 × 100 + 50 × 100 + 70 × 100 + 90 ×
6
= 47
100
We generalize and formalize these ideas in the form of the following definition.
Mathematical Expectation
For a given set S of items, let to each item, one of the n values, say, v1,
v2,…,vn, be associated. Let the probability of the occurrence of an item with
value vi be pi. If an item is picked up at random, then its expected value E(v) is
given by
n
E(v) = ∑ pi vi = p1 . v1 + p2 . v2 + ⋯ … … pn . vn
i=1
Example 1: Write a proof that the sum of the first n positive integers is
𝐧(𝐧+𝟏)
, that is
𝟐
𝐧(𝐧+𝟏)
𝟏 +𝟐 + ⋯…+ 𝐧 = .
𝟐
Base Step): We must show that the given equation is true for n=1
1(+1)
i.e. 1 = = 1 ⟹ this is true.
2
6
Hence we proved that the first statement is true for n = 1 Asymptotic Bounds
Induction Hypothesis: Let us assume that the given equation is true for any
value of n (n ≥ 1)
n(n+1)
that is 1 + 2 + ⋯ … + n = ;
2
Induction Step: Now we have to prove that it is true for (n+1).
Consider
(n+1)[(n+1)+1]
1 + 2 + 3 + ⋯ … … + n + (n + 1) = 2
7
Introduction to Algorithm consideration. Hence different measures of size of an instance of a problem are
used for different types of problems. Let us take two examples:
(i) In sorting and searching problems, the number of elements, which are to
be sorted or are considered for searching, is taken as the size of the
instance of the problem of sorting/searching.
(ii) In the case of solving polynomial equations or while dealing with the
algebra of polynomials, the degrees of polynomial instances, may be taken
as the sizes of the corresponding instances.
To measure the efficiency of an algorithm, we will consider the theoretical
approach and follow the following steps:
Calculation of time complexity of an algorithm- Mathematically determine
the time needed by the algorithm, for a general instance of size, say, n of
the problem under consideration. In this approach, generally, each of the
basic instructions like assignment, read and write, arithmetic operations,
comparison operations are assigned some constant number of (basic) units
of time for execution. Time for looping statements will depend upon the
number of times the loop executes. Adding basic units of time of all the
instructions of an algorithm will give the total amounts of time of the
algorithm
The approach does not depend on the programming language in which the
algorithm is coded and on how it is coded in the language as well as the
computer system used for executing (a programmed version of) the
algorithm. But different computers have different execution speeds.
However, the speed of one computer is generally some constant multiple of
the speed of the other
In stead of applying the algorithm to many different-sized instances, the
approach can be applied for a general size say n of an arbitrary instance of
the problem but the size n may be arbitrarily large under consideration.
In the next section we will examine the asymptotic approach to analyze the
efficiency of algorithms
8
The asymptotic behavior of a function f(n) (such as f(n)=c*n or f(n)=c*n2, Asymptotic Bounds
etc.) refers to the growth of f(n) as n gets very large. Small values of n are not
considered. The main concern in asymptotic analysis of a function is in
estimating how slow the program will be on large inputs. One should always
remember: the slower the asymptotic growth rate, the better the algorithm. The
Merge sort algorithm is better than the standard sorting algorithms. Binary
search algorithm is better than the linear searching algothm. A linear algorithm
is always asymptotically better than a quadratic one .. Remember to think a
very large input size when working with asymptotic rates of growth. If the
relative behaviors of two functions for smaller values conflict with the relative
behaviors for larger values ,then we may ignore the conflicting behaviors for
smaller values.
𝑇1 (n) = 1000 𝑛2
𝑇2 (n) = 5𝑛4
Despite the fact 𝑇1 (n) ≥ 𝑇2 (n) for n ≤ 14, we would still prefer the solution as
𝑇1 (n) as the time complexity because
O(1) constant
O(log n) logarithmic
O(n) linear
O(n log n) "n log n"
O(n2) quadratic
3
O(n ) Cubic
𝑘 polynomial
O(𝑛 )
O(2𝑛 ) exponential
Consider a linear search algorithm. The worst case of the algorithm is when
the element to be searched for is either not in the list or located at the end of
the list. In this case the algorithm runs for the longest possible time. It will
search the entire list. If an algorithm runs in time T(n), we mean that T(n) is an
upper bound on the running time that holds for all inputs of size n. This is
called worst-case analysis.
A popular alternative to worst-case analysis is average-case analysis which
provides average amount of time to solve a problem. Here we try to calculate
the expected time spent on a randomly chosen input. This kind of analysis is
generally more difficult compared to worst case analysis. Because it involves
9
Introduction to Algorithm probabilistic arguments and often requires assumptions about the distribution
of inputs that may not be very easy to justify. But sometimes it can be more
useful compared to the worst-case analysis of an algorithm. A Quicksort
algorithm, whose worst-case running time on an input sequence of length n is
proportional to n2 but whose expected running time is proportional to n log n.
Let us consider two standard sorting algorithms : The first takes 1000
𝑛2 and the second takes 10 𝑛2 time in the worst case respectively on a
machine. Both of these algorithms are asymptotically same (order of
growth is𝑛2 ). Since we ignore constants in asymptotic analysis, it is
difficult to judge which one is more suitable.
Worst case versus average performance
If an algorithm A has better worst case performance than the algorithm
B, but the average performance of B given the expected input is better,
then B could be a better choice than A.
There are mainly three asymptotic notations if we do not want to get involved
with constant coefficients and less significant terms. These are
1. Big-O notation,
2. Big-Θ ( Theta) notation
3. Big-Ω (Omega) notation
: n ≥ n0
10
Asymptotic Bounds
The above function is still a quadratic algorithm and can be written as:
<= (3 +4 -2) n2
= O(𝑛2 )
2. Show n3 != O(n2).
11
Introduction to Algorithm 2.7.2 Big-Omega() Notation
Big Omega () describes the asymptotic lower bound of an algorithm
whereas a big Oh(O)notation represents an upper bound of an algorithm.
Generally we say that an algorithm takes at least this amount of time without
mentioning the upper bound. In such case, big-() notation is applied. Let's
define it more formally:
f(n) = (g(n)) if and only if there exists some constants C and 𝑛0 such that f(n)
C.g(n) : n ≥ 𝑛0 . The following graph illustrates the growth of f(n) =
(g(n))
As shown in the above graph f(n) is bounded from below by C.g(n). Note that
for all values of f(n) always lies on or above g(n).
If f(n) is Ω(g(n)) which means that the growth of f(n) is asymptotically no
slower than g(n) no matter what value of n is provided.
Example 1
Example1.1: For the function defined by
: show that
In case the running time of an algorithm is Θ(n), it means that once n gets
large enough, the running time is minimum c1⋅n, and maximum c2⋅n, where
c1 and c2 are constants. It provides both upper and lower bounds of an
algorithm. The following figure illustrates the function f(n) = Θ(g(n). As
12
shown in the figure the value of f(n) lies between c1(g(n)) and c2(g(n))for Asymptotic Bounds
sufficiently large value of n.
Now let us define the theta notation: for a given function g(n) and constants
C1, C2 and 𝑛0 where n0>0, C1>0, and C2>0, (g(n)) can be denoted as a set
of functions such that the following condition is satisfied:
0 <= C1g(n) <= f(n) <= C2g(n) for all n >= n0
The above inequalities represent two conditions to be satisfied simultaneously
viz., C1 g(x) f(x) and f(x) C2 g(x))
The following theorem which relates the three functions O, , does not have
proof:
Theorem: For any two functions f(x) and g(x), f(x) = (g(x)) if and only if
f(x) = O (g(x)) and f(x) = (g(x)).
if f(n) is Θ(g(n)) this means that the growth of f(n) is asymptotically at the
same rate as g(n) or we can say the growth f(n) is not asymptotically
The following theorems are quite useful when you are dealing (or solving
problems) with O, and
13
Introduction to Algorithm 𝑚
=∑ 𝑎𝑘 𝑛 𝑘
𝑖=0
𝑚
𝑓 (𝑛 ) ≤ ∑ |𝑎𝑘 |𝑛𝑘
𝑖=0
𝑚 𝑚
≤ 𝑛𝑚 ∑ |𝑎𝑘 |𝑛𝑘−𝑚 ≤ 𝑛𝑚 ∑ |𝑎𝑘 |for 𝑛 ≥ 1
𝑖=0 𝑖=0
Proof: 𝑓(𝑛) = 𝑎𝑚 𝑛𝑚 + ⋯ … … … . 𝑎1 𝑛 + 𝑎0
𝑓 (𝑛 ) = 𝑂 (𝑛 𝑚 ) … … … . (1)
𝑓 (𝑛 ) = Ω (𝑛 𝑚 ) … … … . . (2)
Solution:
(i) Here The degree of a polynomial f(n) is, m = 3, So by Theorem
1, 2 and 3:
𝑓 (𝑛) = 𝑂(𝑛3 ), 𝑓(𝑛) = Ω (n3 )and 𝑓(𝑛) = 𝛩(𝑛3 ),
Let f(n) and g(n) be two asymptotically positive functions. Prove or disprove
the following (using the basic definition of O, Ω andΘ):
a) 4𝑛2 + 7𝑛 + 12 = 𝑂(𝑛2 )
b) 𝑙𝑜𝑔𝑛 + log(𝑙𝑜𝑔𝑛 ) = 𝑂(𝑙𝑜𝑔𝑛)
14
c) 3𝑛2 + 7𝑛 − 5 = 𝛩(𝑛2 ) Asymptotic Bounds
d) 2𝑛+1 = 𝑂(2𝑛 )
e) 22𝑛 = 𝑂(2𝑛 )
f) 𝑓(𝑛) = 𝑂 (𝑔(𝑛))𝑖𝑚𝑝𝑙𝑖𝑒𝑠 𝑔(𝑛) = 𝑂(𝑓 (𝑛))
g) max{f(n), g(n)} = 𝛩(𝑓 (𝑛) + 𝑔(𝑛))
h) 𝑓(𝑛) = 𝑂(𝑔(𝑛))𝑖𝑚𝑝𝑙𝑖𝑒𝑠 2𝑓(𝑛) = 𝑂(2𝑔(𝑛) )
i) 𝑓(𝑛) + 𝑔(𝑛) = 𝛩(min(𝑓(𝑛), 𝑔(𝑛))
j) 33𝑛3 + 4𝑛2 = Ω(𝑛2 )
(
2.8 Summary
To that end,
12 + 22 + ⋯ + (𝑛 + 1)2 = 𝟏𝟐 + 𝟐𝟐 + ⋯ + 𝒏𝟐 + (𝑛 + 1)2
𝒏(𝒏 + 𝟏)(𝟐𝒏 + 𝟏)
= + (𝒏 + 𝟏)𝟐
𝟔
15
Introduction to Algorithm 𝑛(𝑛 + 1)(2𝑛 + 1) + 6(𝑛 + 1)2
=
6
(𝑛 + 1)(2𝑛2 + 𝑛 + 6𝑛 + 6)
=
6
(𝑛 + 1)(2𝑛2 + 7𝑛 + 6)
=
6
(𝑛 + 1)(𝑛 + 2)(2𝑛 + 3)
=
6
(𝑛 + 1)[(𝑛 + 1) + 1][2(𝑛 + 1) + 1]
=
6
∑ 2𝑖 = 2𝑛+1 − 1
𝑖=0
Induction base: For n = 0,
20 = 1 = 20+1 − 1.
Induction hypothesis: Assume, for an arbitrary nonnegative integer n, that
20 + 21 + 22 + ⋯ + 2𝑛 = 2𝑛+1 − 1.
To that end,
20 + 21 + 22 + ⋯ + 2𝑛+1 = 𝟐𝟎 + 𝟐𝟏 + 𝟐𝟐 + ⋯ + 𝟐𝒏 + 2𝑛+1
= 𝟐𝒏+𝟏 − 𝟏 + 2𝑛+1
= 2(2𝑛+1 ) − 1
= 2(𝑛+1)+1 − 1.
Check Your Progress2
a) 4𝑛2 + 7𝑛 + 12 = 𝑂(𝑛2 )
(4𝑛2 + 7𝑛 + 12) ≤ 𝑐, 𝑛2 … … … (1)
for c = 5 and n ≤9; the above inequality (1) is satisfied.
Hence 4𝑛2 + 7𝑛 + 12 = 𝑂(𝑛2 ).
d) 2𝑛+1 = 𝑂(2𝑛 )
2𝑛+1 ≤ 𝐶. 2𝑛 ⟹ 2𝑛+1 ≤ 2. 2𝑛
e) 22𝑛 = 𝑂(2𝑛 )
22𝑛 = ≤ 𝐶. 2𝑛
4𝑛 ≤ 2. 2𝑛 ……(1)
No value of C and n0 Satisfied this in equality (1)
22𝑛 ≠ 𝑂(2𝑛 ).
17
Introduction to Algorithm
1
𝑆𝑖𝑛𝑐𝑒 (𝑓(𝑛) + 𝑔(𝑛) ≤ 𝑓(𝑛)
2
𝑓(𝑛) + 𝑔(𝑛) ≤ 2𝑓(𝑛)
𝑓(𝑛) + 𝑔(𝑛) ≤ 𝑓(𝑛) + 𝑓(𝑛) [ 𝑓(𝑛) > 𝑔(𝑛)]
1
Satisfied for 𝐶1 = 2 𝑎𝑛𝑑 𝑛 ≥ 1
2) R.H.S inequality
max{f(n), g(n)} ≤ C1 . (f(n) +g(n) ………(2)
This inequality (2) is satisfied for C2 = 1 and n ≥ 1
inequality (*) is simultaneously satisfied for
1
𝐶1 = , 𝐶2 = 1 𝑎𝑛𝑑 𝑛 ≥ 1
2
Remark: Let f(n) = n and g(n) =𝑛2 ;
then max{n,𝑛2 } = 𝛩 (𝑛 + 𝑛2 )
𝑛2 = Θ (𝑛2 ); which is TRUE (by definition of Θ)
18
Unit-3 Complexity Analysis of Simple Algorithms Complexity Analysis of
Simple Algorithms
Structure
3.0 Introduction 1
3.1 Objectives 1
3.2 A Brief Review of Asymptotic Notations 2
3.3 Analysis of Simple Constructs 2
3.4 General Rules and Analysis of Simple Algorithms 4
3.4.1 A Summation Algorithm 4
3.4.2 Polynomial Evaluation Algorithm 4
3.4.3 Matrix Multiplication 8
3.4.4 Exponent Evaluation 10
3.4.5 Linear Search 15
3.4.6 Sorting Algorithm 17
3.5 Summary 20
3.6 Solutions/Answers 21
3.0 Introduction
Computational complexity describes the amount of processing time required
by an algorithm to give the desired result. Generally we consider the worst-
case time complexity (big O notation) which is the maximum amount of time
required to execute an algorithm for inputs of a given size, whereas average-
case complexity, which is the average of the time taken on inputs of a given
size is less common. In the previous unit we introduced a concept of efficiency
of an algorithm and discussed three asymptotic notations which are formal
methods for analyzing algorithm efficiency in terms of time and space
complexities. The complexity analysis of algorithm helps to understand the
behavior of the algorithm and compare it with other algorithms for a large
input size. The focus of the unit is to perform complexity analysis of small
problems. The structure of the unit is as follows: section 3.3 makes a brief
review of asymptotic notations followed by analysis of simple constructs such
as looping statement, conditional statement , consecutive looping and if
statements. In the subsequent sections we describe general rules for analysis of
algorithms and take up several examples for complexity analysis
3.1 Objectives
After studying this unit, you should be able to analyze time complexities of:
1
Introduction to Algorithm 3.2 A brief review of Asymptotic Notations
The complexity analysis of algorithm is required to measure the time and space
required to run an algorithm. In this unit we focus on only the time required to
execute an algorithm. Let us quickly review some asymptotic notations (Please
refer to the previous unit for detailed discussion)
The central idea of these notations is to compare the relative rate of growth of
functions.
(i) 𝑇(𝑛) = 𝑂(𝑓(𝑛)) if there are two positive constants C and n0 such
that 𝑇(𝑛) ≤ 𝐶𝑓(𝑛) where n ≥ 𝑛0
(ii) 𝑇(𝑛) = Ω(𝑓 (𝑛)) if there are two positive constants C and n0 such
that 𝑇(𝑛) ≥ CΩ f(n) where n ≥ n0
(iii) 𝑇(𝑛) = 𝜃(𝑓(𝑛)) if and only if 𝑇(𝑛) = O(𝑓(𝑛))𝑎𝑛𝑑𝑇(𝑛) =
Ω(𝑓 (𝑛))
The second definition, 𝑇(𝑛) = Ω (𝑓(𝑛)) says that the growth rate of T(n) is
faster than or equal to (≥) f(n).
Ex. 𝑖𝑛𝑡 𝑥;
𝑥 = 𝑥 + 5
𝑥 = 𝑥 −5
2) O(n): This is running time of a single looping statement which includes
comparing time, increment or decrement by some constant value looping
statement.
// Here c is a positive integer constant
for (i = 1; i <= n; i += c) {
// simple statement(s)
}
for (int i = n; i > 0; i -= c) {
// simple statement(s)
}
2
3) O(nc): This is a running time of nested loops. Time complexity of nested Complexity Analysis of
loops is equal to the number of times the innermost statements is executed. Simple Algorithms
For example the following sample loops have O(n2) time complexity
3
Introduction to Algorithm else
statement 2
1. int i, tempresult;
2. tempresult =0;
3. for (i=1 ; I <=n; i++)
4. tempresult = tempresult + i * i * i
5. return tempresult;
Line# 3- The for loop has several unit costs: initializing i, cost for testing i<=n
(n+1 unit cost) and cost of incrementing i(1 unit of cost). Total cost is 2n +2
Line# 4- 2 units of time for multiplication, 1 unit for addition and one unit of
time for assignment operation in one cycle. Therefore the total cost of this line
is 4n
Line# 5- It will take 1 unit of time. Overall cost will be = 6n+4 which is
written as O(n).
4
Struct polynomial{ Complexity Analysis of
Simple Algorithms
int coefficient;
int exponent;
};
P(x)=15∗x∗x∗x∗x+17∗x∗x∗x−12∗x∗x+13∗x+16
Horner’s method:
P(x)= (((15∗x+17)∗x−12)∗x+13)∗x+16
Please observe the basic operations are: multiplication, addition and
subtraction. Since the number of additions and subtractions are the same in
both the solutions, we will consider the number of multiplications only in
worst case analysis of both the methods.
For the complexity analysis of polynomial expression evaluation we’ll look at
the worst case (maximum number of multiplications) to get an upper bound .
P(x)=anxn+an-1xn-1+…..+a1x1+a0x0
+a2∗x∗x∗+a1∗x+a0
In the first term, it will take n multiplications, in the second term it will take
n-1 multiplications, in the third term it takes n-2 multiplications….. In the
last two terms: a2∗x∗x∗and a1∗x it takes 2 multiplications and 1
5
Introduction to Algorithm multiplication accordingly.
Number of multiplications needed in the worst case is
=n(n+1)/2= O( 𝑛2 )
P(x)=(…(((an∗x+an−1)∗x+an−2)∗x+...+a2)∗x+a1)∗x+a0
In the first term, it takes one multiplication, in the second term one
multiplication, in the third term it takes one multiplication …. . Similarly in
all other terms it will take one multiplication.
T(n) = ∑𝑛𝑖=1 1 = n
T(n)=n
6
Evaluate_Horner(a,n,x) Complexity Analysis of
{ Simple Algorithms
p = A[n];
for (i = n-1; i ≤0;i--)
p = p * x + A[i];
return p;
}
follows
At x=3,
p(x) = (3x+5)x+6
p(3)=(9+5).3+6
= (14).3+6
=42+6
=48
Complexity Analysis
First step is one initial assignment that takes constant time i.e O(1).
For loop in the algorithm runs for n iterations, where each iteration cost
O(1) as it includes one multiplication, one addition and one assignment
which takes constant time.
Hence total time complexity of the algorithm will be O(n) for a polynomial of
degree n.
7
Introduction to Algorithm …………………………………………………………………………………
……………………………………………………………………………….…
3. Write basic algorithm to evaluate a polynomial and find its complexity. Also
compare its complexity with complexity of Horner’s algorithm.
…………………………………………………………………………………
…………………………………………………………………………………
…………………………………………………………………………………
…………………………………………………………………………………
…………………………………………………………………………………
…………………………………………………………………………….……
8
Step I : Pseudo code: For Matrix multiplication problem where we will Complexity Analysis of
multiply two matrices A and B of order 3x3 each and store the result in matrix Simple Algorithms
C of order 3x3
1. Multiply the first element of the first row of the first matrix with first
column, first element of the second matrix.
2. Similarly perform this multiplication for first row of first matrix and first
column of second matrix. Now take the sum of these values.
3. The sum obtained will be the first element of product matrix C
4. Similarly Compute all the remaining elements of
product matrix C.
C= A x B
Step II : Algorithm for multiplying two square matrix of order n x n and find
the product matrix of order n x n
Matrix_Multiply(A,B,C,n)
{
1 2 3
A= 2 3 4
4 5 6
1 1 1
B= 2 3 2
3 2 1
9
Introduction to Algorithm c11 c1 c13 1x1+2x2+3x3 1x1+2x3+3x21x1+2x2+3x1
2
21 c2 c23 = 2x1+3x2+4x3 2x1+3x3+4x22x1+3x2+4x1
2
c31 c3 c33 4x1+5x2+6x3 4x1+5x3+6x24x1+5x2+6x1
2
= 14 13 8
20 19 12
32 31 20
Complexity Analysis
First step is, for loop that will be executed n number of times i.e it will take
O(n) time. The second nested for loop will also run for n number of time and
will take O(n) time.
Assignment statement inside second for loop will take constant time i.e O(1) as
it includes only one assignment.
The third for loop i.e innermost nested loop will also run for n number of
times and will take O(n ) time . Assignment statement inside third for
loop will cost O(1) as it includes one multiplication, one addition and one
assignment which takes constant time.
Hence, total time complexity of the algorithm will be O(n3) for matrix
multiplication of order nxn.
41=4
42=(41)2=42=16
44=(42)2=162=256
48=(44)2=2562=65,536
Therefore the final answer for 411, we only need to multiply three of them
(skipping 44 because the corresponding bit in 𝑛 is set to
zero):411=65,536⋅16⋅4=4,194,304.
A[s]
11
Introduction to Algorithm Output: Final value of an.
1. result=a
2. for i=s-2 to0
3. result = result *result
4. if A[i]= 1then
5. result= result *a
6. return result (i.e an)
Iteration 1:
i=3
result=a *a= a2
A[3] ≠ 1
Iteration 2:
i=2
result= a2 * a2 = a4
A[2] ≠ 1
Iteration 3:
i=1
result= a4 * a4 = a8
A[1] ≠ 1
Iteration 4:
i=0
result= a8 * a8 = a16
A[0] = 1
result = a16 * a = a17
return a17
Hence
1. Set x =a
2. if A[0]= 1 then set result=a
3. else set result=1
4. Initializei=1
5. compute x = x *x
6. if A[i] = 1 then compute result = result *x
7. Increment i by 1 as i=i+1 and if i is less than equal to s-1 then go to
step4.
8. return computed value as result.
1. x=a
2. if A[0]=1then
3. result =a
4. else
5. result=1
6. for i= 1 tos-1
7. x= x * x
8. ifA[i]=1
9. result= result *x
10. return result (i.e an)
Let us take an example to illustrate the above algorithm to compute a17 In this
exponent n=17 which is equivalent to binary string 10001
13
Introduction to Algorithm Iteration 1
i=1
x=a
*a= a2
A[1] ≠
1
Iteration 2
i=2
x= a2 * a2 = a4
A[2] ≠ 1
Iteration 3
i=3
x= a4 * a4 = a8
A[3] ≠ 1
Iteration 4
i=4
x= a8 * a8 = a16
A[4] = 1
result = result * x = a * a16 = a17
return a17
From the above discussion we can conclude that the complexity for left to
right binary exponentiation and right to left binary exponentiation is
logarithmic in terms of exponent n.
1. Compute a283 using left to right and right to left binary exponentiation.
…………………………………………………………………………………
………………………………………………….………………………………
14
………………………………………………………………………………… Complexity Analysis of
………………………………………………………………………………… Simple Algorithms
…………………………………………………………………………………
…………………………………………………………………………………
We are given with a list of items. The following table shows a data set for
linear search:
7 17 3 9 25 18
In the above table of data set, start at the first item/element in the list and
compared with the key. If the key is not at the first position, then we move
from the current item to next item in the list sequentially until we either find
what we are looking for or run out of items i.e the whole list of items is
exhausted. If we run out of items or the list is exhausted, we can conclude
that the item we were searching from the list is not present.
In the given data set key 25 is compared with first element i.e 7 , they are not
equal then move to next element in the list and key is again compared with
17 , key 25 is not equal to 17. Like this key is compared with element in the
list till either element is found in the list or not found till end of the list. In
this case key element is found in the list and search is successful.
Let us write the algorithm for the linear search process first and then analyze
its complexity.
15
Introduction to Algorithm {
if (a[i]==key)
found = true
break;
}
if (i==n)
found =
false
return found
}
For the complexity analysis of this algorithm, we will discuss the following
cases:
Best Case: The best case - we will find the key in the first place we look, at
the beginning of the list i.e the first comparison returns a match or return found
as true. In this case we only require a single comparison and complexity will
be O(1).
Worst Case: In worst case either we will find the key at the end of the list or
we may not find the key until the very last comparison i.e nth comparison.
Since the search requires n comparisons in the worst case, complexity will be
O(n).
Average Case: On average, we will find the key about halfway into the list;
that is, we will compare against n/2 data items. However, that as n gets larger,
the coefficients, no matter what they are, become insignificant in our
approximation, so the complexity of the linear search, is O(n). The average
time depends on the probability that the key will be found in the collection -
this is something that we would not expect to know in the majority of cases.
Thus in this case, as in most others, estimation of the average time is of little
utility.
16
Case Best Case Worst Case Average Complexity Analysis of
Case Simple Algorithms
item is present O(1) O(n) O(n/2) = O(n)
item is not O(n) O(n) O(n)
present
Most of the times an algorithm run for the longest period of time as defined in
worst case. Information provide by best case is not very useful. In average
case, it is difficult to determine probability of occurrence of input data set.
Worst case provides an upper bound on performance i.e the algorithm will
never take more time than computed in worse case. So, the worst-case time
analysis is easier to compute and is useful than average time case.
3.4.6 SORTING
- Internal Sort: - Internal sorts are the sorting algorithms in which the
complete data set to be sorted is available in the computer’s main memory.
- External Sort: - External sorting techniques are used when the collection
of complete data cannot reside in the main memory but must reside in
secondary storage for example on a disk.
In this section we will discuss only internal sorting algorithms. Some of the
internal sorting algorithms are bubble sort, insertion sort and selection sort.
For any sorting algorithm important factors that contribute to measure their
efficiency are the size of the data set and the method/operation to move the
different elements around or exchange the elements. So counting the
number of comparisons and the number of exchanges made by an algorithm
provides useful performance measures. When sorting large set of data, the
number of exchanges made may be the principal performance criterion,
since exchanging two records will involve a lot of time.
Bubble Sort
17
Introduction to Algorithm
A list of numbers is given as input that needs to be sorted. Let us explain
the process of sorting via bubble sort with the help of following Tables:
First Pass
23 18 15 37 8 11
18 23 15 37 8 11
18 15 23 37 8 11
18 15 23 37 8 11
18 15 23 8 37 11
18 15 23 8 11 37
Second Pass
18 15 23 8 11 37
15 18 23 8 11 37
15 18 23 8 11 37
15 18 8 23 11 37
15 18 8 11 23 37
15 18 8 11 23 37
15 18 8 11 23 37
15 8 18 11 23 37
15 8 11 18 23 37
Third Pass
15 8 11 18 23 37
8 15 11 18 23 37
8 11 15 18 23 37
Fourth Pass
8 11 15 18 23 37
8 11 15 18 23 37
Fifth Pass
8 11 15 18 23 37
In this the given list is divided into two sub list sorted and unsorted. The
largest element is bubbled from the unsorted list to the sorted sub list. After
each iteration/pass size of unsorted keep on decreasing and size of sorted
sub list gets on increasing till all element of the list comes in the sorted list.
With the list of n elements, n-1 pass/iteration are required to sort. Let us
discuss the result of iteration shown in above tables.
In pass 1, first and second element of the data set i.e 23 and 18 are compared
and as 23 is greater than 18 so they are swapped. Then second and third
element will be compared i.e 23 and 15, again 23 is greater than 15 so
swapped. Now 23 and 37 is compared and 23 is less than 37 so no swapping
take place. Then 37 and 8 is compared and 37 is greater than 8 so swapping
take place. At the end 37 is compared with 11 and again swapped. As a result
largest element of the given data set i.e 37 is bubbled at the last position in the
array. At each pass the largest element among the remaining elements in the
unsorted array bubbles up towards the sorted part of the array as shown in the
table above. This process will continue till n-1 passes.
The first version of the algorithm for above sorting method is as below:
Bubble Sort Algorithm- Version1
18
// A is the list of n elements to be Complexity Analysis of
sorted function bubble sort (A, n) Simple Algorithms
{
int i,j
for ( i= 1 to n-1
for (j = 0 to n-2
{
if (A[j]>A[j+1])
{
// swapping of two adjacent elements of an array A
exchange (A[j], A[j+1])
}
}
}
T(n) = O(𝑛2 )
Let us reanalyze the above algorithm to improve the running time algorithm
further. From the example it is visible that the Bubble sort algorithm divides
the array into unsorted and sorted sub-arrays. The inner loop rescans the
sorted sub- array in each cycle, although there will not be any exchange of
adjacent elements. The modified version (version 2) of the algorithm
overcomes this problem:
Version -2
19
Introduction to Algorithm
}
}
}
There will be no change in the number of iterations i.e. n-2 iterations in the
first pass, but in the second pass it will be n-3, in the third pass it will be n-4
iterations and so on. In this case too, the complexity remains to be O(𝑛2 ) but
the number of exchange operations will be less. This requires further
improvement of the algorithm.
In some cases there is no need of running n-1 passes in the outer loop. The
array might be sorted in less than that. The following is the modified version
(Version -3) of the algorithm
{
int i,j
for ( i= 1 to n-1)
{
flag = 0;
for (j = 0 to n-i-1)
{
if(A[j]>A[j+1])
{
// swapping of two adjacent elements of an array A
exchange ( A[j], A[j+1])
flag = 1;
}
if (flag = = 0)
exit;
}
}
In case there is no swapping, flag will remain set to 0 and the algorithm will
stop running.
Time Complexity
In the modified algorithm, the inner loop will execute at least once to verify
that the array is sorted but not (n-i-1) times. Therefore the time complexity will
be:
T(n) = C* (n-1)
= O(n)
3. 5 Summary
20
matrices of order n x n with time complexity in the order of O(n3). For Complexity Analysis of
exponent evaluation both approaches i.e left to right binary exponentiation and Simple Algorithms
right to left binary exponentiation is illustrated. Time complexity of these
algorithms to compute xn is O(log n). Different versions of bubble sort
algorithm are presented and its performance analysis is done at the end.
P(x)=anxn+an-1xn-1+…..+a1x1+a0x0
Iteration 1,
poly = x * 0 + a[4] = 3
Iteration 2,
poly = x * 3 + a[3] = 2 * 3 + 2 = 6 +2 = 8
Iteration 3,
poly = x * 8 + a[2]
= 2 * 8 + 0 = 16 + 0 = 16
Iteration 4,
poly = x * 16 + a[1]
= 2 * 16 + (-5) = 32 -5 = 27
Iteration 5,
poly = x * 27 + a[0]
= 2 * 27 + 7 = 54 + 7 = 61
function(a[n], n, x)
{
poly = 0;
21
Introduction to Algorithm
result =1;
for (j=0; j<i; j++)
{
result= result * x;
}
}
return poly.
}
#include<stdio.h>
int main()
{
int a[3][3],b[3][3],c[3][3],i,j,k,sum=0;
{
printf("\n");
for (j=0;j<3;j++)
printf("%d\t",b[i][j]);
22
} Complexity Analysis of
for(i=0;i<3;i++) Simple Algorithms
for(j=0;j<3;j++)
c[i][j]=0;
for(i=0;i<3;i++)
{
for(j=0;j<3;j++)
{
sum=0;
for(k=0;k<3;k++)
sum=sum+a[i][k]*b[k][j];
c[i][j]=sum;
}
}
printf("\nThe multiplication of two matrix is\n");
for(i=0;i<3;i++)
{
printf("\n");
for(j=0;j<3;j++)
printf("%d\t",c[i][j]);
}
return 0;
}
23
Introduction to Algorithm Iteration no. i Bit x result
1 1 1 a2 a *a2 = a3
2 2 0 a4 a3
3 3 1 a8 a3 * a8= a11
4 4 1 (a8 )2 a16 * a11= a27
5 5 0 (a16 )2 (a27 )
6 6 0 (a32 )2 a27
7 7 0 (a64 )2 a27
8 8 1 (a128)2 (a256 )* a27= a283
24
Basics of An Algorithm
UNIT 4 SOLVING RECURRENCES
Structure
4.0 Introduction
4.1 Objectives
4.2 Recurrence Relation
4.3 Methods for Solving Recurrence Relation
4.3.1 Substitution method
4.3.2 Recursion Tree Method
4.3.3 Master Method
4.4 Summary
4.5 Solution to check your progress
4.6 Further Reading
4.0 INTRODUCTION
4.1 OBJECTIVES
Like all recursive functions, a recurrence relation also consists of two steps: (i) one or
more initial conditions and (ii) recursive definition of a problem
0 𝑖𝑓 𝑛 = 0
𝑓𝑛 = {0 𝑖𝑓 𝑛 = 1
𝑓𝑛−1 + 𝑓𝑛−2 𝑖𝑓 𝑛 ≥ 2
This formula says that “by adding two previous sequence (or term) we can get the
next term”.
For example 𝒇𝟐 = 𝒇𝟏 + 𝒇𝟎 = 𝟏 + 𝟎 = 𝟏;
𝑓3 = 𝑓2 + 𝑓1 = 1 + 1 = 2 ; 𝑓4 = 𝑓3 + 𝑓2 = 2 + 1 = 3 and so on
Example 2 Find out the value of n! = n (n-1) (n-2)……. (3) (2) (1) for n ≥ 1
1 𝑖𝑓 𝑛 = 1
𝑛! = {
𝑛. (𝑛 − 1)! 𝑖𝑓 𝑛 > 1
int fact(intn)
1: 𝑖𝑓 𝑛 == 0 𝑡ℎ𝑒𝑛
2: 𝑟𝑒𝑡𝑢𝑟𝑛 1
3: 𝑒𝑙𝑠𝑒
4: 𝑟𝑒𝑡𝑢𝑟𝑛 𝑛 ∗ 𝑓𝑎𝑐𝑡(𝑛 − 1)
5: 𝑒𝑛𝑑𝑖𝑓
Let us try to understand the efficiency of the algorithm in terms of the number of
multiplications operations required for each value of n
Let 𝑇(𝑛) denoted the number of multiplication required to execute the n!.,
2
that is 𝑇(𝑛) denotes the number of times the line 4 is executed in factorial Basics of An Algorithm
algorithm.
We have the initial condition T(0) = 1; since when n = 0, fact simply returns
(i.e. Number of multiplication is0).
1: if(𝑛 == 0)
2: 𝑟𝑒𝑡𝑢𝑟𝑛 1
3: if(𝑛 == 1)
4: 𝑟𝑒𝑡𝑢𝑟𝑛 𝑥
5: 𝒆𝒍𝒔𝒆
7: 𝑒𝑛𝑑𝑖𝑓
The algorithm 3 performs one comparison and one return statement. Therefore,
𝑇(0)𝑎𝑛𝑑 𝑇(1) = O(1) = 𝑎
When n > 1; the algorithm3 performs one recursive call with input parameter (n – 1)
at line 6, and some constant number of basic operations. Thus we obtain the recurrence
relationas:
𝑇(1) = 𝑎 (𝑏𝑎𝑠𝑒 𝑐𝑎𝑠𝑒)
𝑇(𝑛) = {
𝑇(𝑛 − 1) + 𝑏 (𝑅𝑒𝑐𝑢𝑟𝑠𝑖𝑣𝑒 𝑠𝑡𝑒𝑝)
3
Solving Recurrence To find out the recurrence relation when n=1 ( base value) we have to find the value
of 𝑇0 .
Since 𝑇0 refers to the initial amount, 𝑇0 = 5000
With the above definitions we can calculate the value of 𝑇𝑛 for any value of n. For
example:
𝑇3 = (1.15)𝑇2 =(1.15)(1.15)𝑇1 = (1.15)(1.15)(1.15)𝑇0 = (1.15)3 (5000)
The above computation can be extended to any arbitrary value of n.
𝑇𝑛 = (1.15)𝑇𝑛−1
.. ..
= ((1.15)𝑛 (5000)
Three methods are discussed here to solve recurrence relations: Substitution method,
Recursion Tree method and Mater method. .We start with Substitution method
Or 𝑇(𝑛) ≤ 𝑐. 𝑛𝑙𝑜𝑔𝑛
Step2: Now we use mathematical Induction.
Here our guess does not hold for n=1because 𝑇(1) ≤ 𝑐. 1𝑙𝑜𝑔1
𝑖. 𝑒. 𝑇(𝑛) ≤ 0 𝑤ℎ𝑖𝑐ℎ 𝑖𝑠 𝑐𝑜𝑛𝑡𝑟𝑎𝑑𝑖𝑐𝑡𝑖𝑜𝑛 𝑤𝑖𝑡ℎ 𝑇(1) = 1
4
Basics of An Algorithm
(i) Induction step: Now assume it is true for n = n/2
𝑛 𝑛 𝑛
𝑖. 𝑒. 𝑇 ( ) ≤ 𝑐. ( ) log ( ) 𝑖𝑠 𝑡𝑟𝑢𝑒.
2 2 2
Now we have to show that it is true for the value of n
𝑖. 𝑒. 𝑇(2) ≤ 𝑐. 𝑛𝑙𝑜𝑔𝑛
𝑛
We known that 𝑇 (𝑛) ≤ 2𝑇 (⌊ ⌋) + 𝑛
2
𝑛 𝑛
≤ 2(𝑐 ⌊ ⌋ 𝑙𝑜𝑔 ⌊ ⌋ + 𝑛
2 2
𝑛
≤ 𝑐𝑛𝑙𝑜𝑔 ⌊ ⌋ + 𝑛 ≤ 𝑐𝑛𝑙𝑜𝑔𝑛 − 𝑐𝑛𝑙𝑜𝑔2 + 𝑛
2
≤ 𝑐𝑛𝑙𝑜𝑔𝑛 − 𝑐𝑛 + 𝑛
≤ 𝑐𝑛𝑙𝑜𝑔𝑛 ∀𝑐 ≥ 1
Thus 𝑇(𝑛) = 𝑂(𝑛𝑙𝑜𝑔𝑛)
This recurrence (1) describe the running time of any divide-and-conquer algorithm.
𝒏
Method (steps) for solving a recurrence 𝑻(𝒏) = 𝒂𝑻 ( ) + 𝒇(𝒏) using recursion
𝒃
tree:
5
Solving Recurrence 𝒏
(b) Now we have to find the value of 𝑻 ( ) by putting (n/b) in place of n in
𝒃
equation (1).That is
𝑛
𝑛 𝑛
𝑇 (𝑏 ) = 𝑎𝑇 ( 𝑏𝑏 ) + 𝑓 (𝑛⁄𝑏) = 𝑎𝑇 + 𝑓 (𝑏2) + 𝑓(𝑛⁄𝑏) … (2)
From equation (2), now 𝑓(𝑛⁄𝑏 ) will be the value of node having a branch (child
𝑛
nodes) each of size T(n/b). Now each 𝑇 (𝑏 )in figure-a will be replaced as follows:
c) In this way you can expend a tree one more level (i.e. up to (at least) 2 levels).
Step2: (a) Now you have to find per level cost of a tree. Per level cost is the sum of
the cost of each node at that level. For example per level cost at level1 is
𝑛 𝑛 𝑛
( ) + 𝑓 ( ) + ⋯ 𝑓 ( ) (𝑎 𝑡𝑖𝑚𝑒𝑠). This is also calledRow-Sum.
𝑏 𝑏 𝑏
(b) Now the total (final) cost of the tree can be obtained by taking the sum of costs of
all these levels.
𝒏
Example1: Solve the recurrence 𝑻(𝒏) = 𝟐𝑻 ( ) + 𝒏 using recursion tree method.
𝟐
1. To make a recursion tree, you have to write the value of 𝑓 (𝑛) at root node. And
2. The number of child of a Root Node is equal to the value of a. (Here the
value of a = 2).So recursion tree be looks like as:
6
Basics of An Algorithm
𝑛
b) Now we have to find the value of 𝑇 ( 2 ) in figure (a) by putting (n/2) in
place of n in equation (1).That is
𝑛
𝑛 𝑛
𝑇 ( ) = 2𝑇 ( 2 ) + 𝑛 = 2𝑇 ( 2 ) + 𝑛⁄2 … (2)
𝑏 2 2
𝑛
From equation (2), now ( ) will be the value of node having 2 branch (child nodes)
2
𝑛
each of size T(n/2). Now each 𝑇 ( 2 ) in figure-a will be replaced as follows:
7
Solving Recurrence c) In this way, you can extend a tree up to Boundary condition (when problem
size becomes 1). So the final tree will be looks like:
Now we find the per level cost of a tree, Per-level cost is the sum of the costs within
each level (called row sum). Here per level cost is For example: per level cost at depth
2 in figure-c can be obtained as:
𝑛 𝑛 𝑛 𝑛
( 2 ) + ( 2 ) + ( 2 ) + ( 2 ) = 𝑛.
2 2 2 2
Then total cost is the sum of the costs of all levels (called column sum), which gives
the solution of a given Recurrence. The height of the tree is
To find the sum of this series you have to find the total number of terms in this series.
To find a total number of terms, you have to find a height of a tree.
Height of tree can be obtained as follow (see recursion tree of figure c): you start a
𝑛 𝑛
problem of size n, then problem size reduces to( ), then( 2 ), and so on till boundary
2 2
condition (problem size 1) is not reached. That is
𝑛 𝑛 𝑛
𝑛 → ( ) → ( 2) → ⋯ … … … → ( 𝑘)
2 2 2
8
𝑛 2𝑛 Basics of An Algorithm
Example2: Solve the recurrence 𝑇 (𝑛) = 𝑇 (⌊3 ⌋) + 𝑇 (⌊ 3 ⌋) + 𝑛
Solution: We always omit floor & ceiling function while solving recurrence.
Thus given recurrence can be written as:
𝑛 2𝑛
𝑇(𝑛) = 𝑇 ( 3 ) + 𝑇 ( 3 ) + 𝑛 … … … (1)
2 𝑘
( ) = 1 ⟹ 𝑘 = 𝑙𝑜𝑔3⁄2 𝑛 ⟹ 𝐻𝑒𝑖𝑔ℎ𝑡 𝑜𝑓 𝑡ℎ𝑒 𝑡𝑟𝑒𝑒.
3
𝑛 + 𝑛 + ⋯ … … … + 𝑛(𝑙𝑜𝑔3⁄2 𝑛 𝑡𝑖𝑚𝑒𝑠)
𝑛𝑙𝑜𝑔2 𝑛
⟹ 𝑛𝑙𝑜𝑔3⁄2 𝑛 = = 𝑂(𝑛𝑙𝑜𝑔2 𝑛) − − − − − (∗)
3
𝑙𝑜𝑔2
2
9
𝑛 𝑛 𝑛
Solving Recurrence 𝑛 → ( ) → ( 2) → ⋯ … … … → ( 𝑘)
3 3 3
𝑛 + 𝑛 + ⋯ … … … + 𝑛(𝑙𝑜𝑔3 𝑛 𝑡𝑖𝑚𝑒𝑠)
𝑛𝑙𝑜𝑔2 𝑛
⟹ 𝑛𝑙𝑜𝑔3 𝑛 = = 𝛺(𝑛𝑙𝑜𝑔2 𝑛) − − − − − (∗∗)
𝑙𝑜𝑔2 3
𝑇(𝑛) = 𝛩 (𝑛𝑙𝑜𝑔2 𝑛)
Remark: If
𝒇(𝒏) = 𝑶(𝒈(𝒏))𝒂𝒏𝒅 𝒇(𝒏) = 𝜴(𝒈(𝒏))𝒕𝒉𝒆𝒏 𝒇𝑰𝒏) = 𝜣(𝒈(𝒏))
Solution:
Figure-a to figure-c shows a step-by-step derivation of a recursion tree for the given
recurrence 𝑇(𝑛) = 2𝑇 (𝑛 − 1) + 1
10
⟹ 𝑛 → (𝑛 − 1) → (𝑛 − 2) → ⋯ … … . . → 2 → 1 Basics of An Algorithm
a) 𝐹𝑎𝑠𝑡_𝑃𝑜𝑤𝑒𝑟(𝑥, 𝑛)
{𝑖𝑓 (𝑛 == 0)
𝑟𝑒𝑡𝑢𝑟𝑛 1;
𝑒𝑙𝑠𝑒𝑖𝑓 (𝑛 == 1)
𝑟𝑒𝑡𝑢𝑟𝑛 𝑥;
𝒆𝒍𝒔𝒆
11
Solving Recurrence 𝑛 𝑛
𝑟𝑒𝑡𝑢𝑟𝑛 𝑥 ∗ 𝐹𝑎𝑠𝑡_𝑝𝑜𝑤𝑒𝑟 (𝑥, ) ∗ 𝐹𝑎𝑠𝑡_𝑝𝑜𝑤𝑒𝑟 (𝑥, )
2 2
}
b)
Fibnacci (n)
{ if (n = = 0)
return 0;
if (n = = 1)
return 1;
𝑛 𝑛 𝑛
𝑏. 𝑇(𝑛) = 𝑇 ( ) + 𝑇 ( ) + 𝑇 ( ) + 𝑛
2 4 8
12
Basics of An Algorithm
4.3.3 MASTERMETHOD
The master method provides us a straight forward method for solving recurrences of
the form
𝑛
𝑇(𝑛) = 𝑎𝑇 (𝑏 ) + 𝑓 (𝑛), 𝑤ℎ𝑒𝑟𝑒 𝑎 ≥ 1 𝑎𝑛𝑑 𝑏 > 1 are constants and 𝑓 (𝑛) is an
asymptotically positive function. This recurrence gives us the running time of an
𝑛
algorithm that divides a problem of size n into a subproblems of size ( ). 𝑏
𝑛
The a subproblems are solved recursively, each in time 𝑇 (𝑏 ). The cost of dividing
the problem and combining the results of the subproblems is described by the function
𝑓 (𝑛)This recurrence is technically correct only
𝑛 𝑛 𝑛
when (𝑏 ) is an integer, so the assumption will be made that (𝑏 ) is either ⌊𝑏 ⌋
𝑛
or ⌈ ⌉ since such a replacement does not affect the asymptotic behavior of the
𝑏
recurrence. The value of a and b is a positive integer since one can have only a whole
number of subproblems.
Theorem1: Master Theorem
The Master Method requires memorization of the following 3 cases; then the solution
of many recurrences can be determined quite easily, often without using pencil &
paper.
𝑛
Case3: If 𝑓(𝑛) = 𝑂(𝑛𝑙𝑜𝑔𝑏𝑎+∈ )for some ∈> 0 , and if 𝑎𝑓 (𝑏 ) ≤ 𝑐𝑓 (𝑛) for
Remark: To apply Master method you always compare 𝑛𝑙𝑜𝑔𝑏𝑎 and 𝑓 (𝑛)The larger of
the two functions determines solution to the recurrence problems.If the growth rate
of these two functions then it belongs to case 2.In this case we multiply by a
logarithmic factor to get the run time solution (T(n)) of recurrence relation.
13
∈
Solving Recurrence If 𝑓(𝑛) is polynomially smaller than 𝑛𝑙𝑜𝑔𝑏𝑎 (by a factor of 𝑛 then case 1 will be
applicable to find 𝑇(𝑛).
If 𝑓 (𝑛) is polynomially larger than 𝑛 𝑙𝑜𝑔𝑏𝑎 (by a factor of1⁄𝑛∈ then
𝑇(𝑛) = 𝜃(𝑓(𝑛) 𝑤ℎ𝑖𝑐ℎ 𝑖𝑠 𝑖𝑛 𝒄𝒂𝒔𝒆 𝟑.
2𝑛
Example2: Consider the recurrence of 𝑇(𝑛) = 𝑇 ( ) + 1, in which 𝑎 = 1,
3
3 𝑙𝑜𝑔3 2 𝑙𝑜𝑔3 2
𝑏 = 2 , 𝑓 (𝑛) = 𝑛 =1. Since 𝑓 (𝑛) = 𝛩( 𝑛 ). By Master Theorem (case2), we
get 𝑇 (𝑛) = Θ( 𝑛𝑙𝑜𝑔𝑏𝑎 . 𝑙𝑜𝑔𝑛) = Θ(𝑙𝑜𝑔𝑛).
Q.1: Write the first two cases (Case 1 and Case 2) of Master method to solve a
recurrence relation of the form:
𝑛
𝑇(𝑛) = 𝑎𝑇 ( ) + 𝑓 (𝑛)
𝑏
Q.2: Use Master Theorem to give the tight asymptotic bounds of the following
recurrences:
𝑛
a. 𝑇 (𝑛) = 4𝑇 ( ) + 𝑛
2
𝑛
b. 𝑇(𝑛) = 4𝑇 ( 2 ) + 𝑛2
14
Basics of An Algorithm
4.4 SUMMARY
2. Space complexity of and algorithm is the number of elementary objects that this
algorithm needs to store during its execution. The space occupied by an
algorithm is determined by the number and sizes of the variables and data
structures used by the algorithm.
Average case: The value of 𝑓(𝑛) which is in between maximumandminimum for any
possible input. Generally the Average case implies the expected value of 𝑓(𝑛)
7. O-notation: Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there
exist two positive constants c > 0 and 𝑛0 ≥ 1 such that f(n)≤ cg(n)for all n,
𝑛 ≥ 𝑛0 Big-Oh notation gives an upper bound on the growth rate of a function.
8. 𝜴-notation: Given functions f(n) and g(n), we say that f(n)is 𝛺(g(n)) if there
exist two positive constants c > 0 and 𝑛0 ≥ 1 such that f(n) cg(n) for all n, 𝑛 ≥
𝑛0 Big-Omega notation gives a lower bound on the growth rate of a function.
10. When an algorithm contains a recursive call to itself, its running time can often
be described by a recurrence. A recurrence relation is an equation or inequality
that describes a function in terms of its value on smaller inputs.
15
Solving Recurrence
11. There are three basic methods of solving the recurrence relation:
12. Master method provides a “cookbook” method for solving recurrences of the
𝑛
form: 𝑇(𝑛) = 𝑎𝑇 (𝑏 ) + 𝑓 (𝑛)𝑤ℎ𝑒𝑟𝑒 𝑎 ≥ 1 𝑎𝑛𝑑 𝑏 > 1 are constants
13. In master method you have to always compare the value of 𝑓(𝑛) with 𝑛𝑙𝑜𝑔𝑏𝑎
to decide which case is applicable. If 𝑓 (𝑛) is asymptotically smaller than 𝑛𝑙𝑜𝑔𝑏𝑎 ,
then case 1 is applied. If 𝑓(𝑛) is asymptotically same as 𝑛𝑙𝑜𝑔𝑏𝑎 , then case 2 is
𝑛
applied. If 𝑓(𝑛) is asymptotically larger than 𝑛𝑙𝑜𝑔𝑏𝑎 , and if 𝑎𝑓 (𝑏 ) ≤
𝑐. 𝑓(𝑛)𝑓𝑜𝑟 𝑠𝑜𝑚𝑒 𝑐 < 1, then case3 isapplied.
Q 1: a)
At every step the problem size reduces to half the size. When the power is an odd
number, the additional multiplication is involved. To find a time complexity of this
algorithm, let us consider the worst case, that is we assume that at every step
additional multiplication is needed. Thus total number of operations T(n) will reduce
to number of operations for n/2, that is T(n/2) with three additional arithmetic
operations(In odd power case: 2 multiplication and one division). Now we can write:
𝑇(𝑛) = 1 𝑖𝑓 𝑛 = 0 𝑜𝑟 1
𝑛
𝑇(𝑛) = 𝑇 ( ) + 3 𝑖𝑓 𝑛 ≥ 2
2
Instead of writing exact number of operations needed by the algorithm, we can use
some constants. The reason for writing this constant is that we are always interested
to find “asymptotic complexity” instead of finding exact number of operations needed
by algorithm, and also it would not affect our complexity also.
𝑇 (1) = 𝑎 𝑖𝑓 𝑛 = 0 𝑜𝑟 𝑛 = 1 (𝑏𝑎𝑠𝑒 𝑐𝑎𝑠𝑒)
𝑇(𝑛) = { 𝑛
𝑇( )+𝑏 𝑖𝑓 𝑛 ≥ 2 (𝑅𝑒𝑐𝑢𝑟𝑠𝑖𝑣𝑒 𝑠𝑡𝑒𝑝)
2
𝑎 𝑖𝑓 𝑛 = 0 𝑜𝑟 𝑛 = 1 (𝑏𝑎𝑠𝑒 𝑐𝑎𝑠𝑒)
b) 𝑇(𝑛) = {
𝑇(𝑛 − 1) + 𝑇(𝑛 − 2) + 𝑏𝑖𝑓 𝑛 ≥ 2 (𝑅𝑒𝑐𝑢𝑟𝑠𝑖𝑣𝑒 𝑠𝑡𝑒𝑝)
Q2 (a) The recursion tree for the given recurrence relation is:
16
Basics of An Algorithm
Figure a
Figure b
……. n
………n
…….. n
……………… ………………..
𝒏
Figure c: A Recurrence Tree for T(n) = 4T( ) + n
𝟐
(2𝑙𝑜𝑔2 𝑛 − 1) 𝑛(𝑛 − 1)
=𝑛 = = 𝑛2 − 𝑛 = 𝜃(𝑛2 )
2−1 1
17
Solving Recurrence ∴ 𝑇(𝑛) = 𝜃(𝑛2 )
Q2(d)
Case1: If for some 𝑓(𝑛) = 𝑂(𝑛𝑙𝑜𝑔𝑏 𝑎−∈ ) for some ∈ > 0 ,then 𝑇(𝑛) = Θ(𝑛𝑙𝑜𝑔𝑏 𝑎 )
Solution2:
𝑛
a) In a recurrence 𝑇 (𝑛) = 4𝑇 ( 2 ) + 𝑛, 𝑎 = 4, 𝑏 = 2,
𝑓 (𝑛) = 𝑛 𝑙𝑜𝑔𝑏𝑎 = 𝑛2 . 𝑁𝑜𝑤 𝑐𝑜𝑚𝑝𝑎𝑟𝑒 𝑓 (𝑛)𝑤𝑖𝑡ℎ 𝑛𝑙𝑜𝑔𝑏𝑎 .
𝑆𝑖𝑛𝑐𝑒 𝑓 (𝑛) = 𝑂 (𝑛𝑙𝑜𝑔𝑏𝑎 −∈ ).
𝑤ℎ𝑒𝑟𝑒 ∈= 1. 𝐵𝑦 𝑀𝑎𝑠𝑡𝑒𝑟 𝑇ℎ𝑒𝑜𝑟𝑒𝑚 𝑐𝑎𝑠𝑒 1 𝑤𝑒 𝑔𝑒𝑡 𝑇(𝑛) = Θ (n2 ).
𝑛
b) 𝑇(𝑛) = 4𝑇 ( 2 ) + 𝑛2 ; 𝑖𝑛 𝑤ℎ𝑖𝑐ℎ 𝑎 = 4, 𝑏 = 2, 𝑓(𝑛) = 𝑛2 𝑎𝑛𝑑 𝑛𝑙𝑜𝑔𝑏𝑎 = 𝑛2 .
𝑁𝑜𝑤 𝑐𝑜𝑚𝑝𝑎𝑟𝑒 𝑓(𝑛)𝑤𝑡ℎ 𝑛𝑙𝑜𝑔𝑏𝑎 ;
𝑠𝑖𝑛𝑐𝑒 𝑓(𝑛) = 𝑛2 = Θ(𝑛𝑙𝑜𝑔𝑏𝑎 ).
𝑇ℎ𝑢𝑠 𝐵𝑦 𝑀𝑎𝑠𝑡𝑒𝑟 𝑇ℎ𝑒𝑜𝑟𝑒𝑚 (𝑐𝑎𝑠𝑒2),
𝑤𝑒 𝑔𝑒𝑡 𝑇(𝑛) = Θ(𝑛𝑙𝑜𝑔𝑏𝑎 . 𝑙𝑜𝑔𝑛) = Θ(𝑛2 𝑙𝑜𝑔𝑛).
18