UNIT I

INTRODUCTION TO ALGORITM
DESIGN

Session - 1
Syllabus
 Introduction, Fundamentals of algorithm(Line
count, operation count), Algorithm Design Techniques
(Approaches, Design Paradigms), Designing an algorithm
and its Analysis(Best ,Worst & Average case),
Asymptotic Notations(O, Ω, ʘ) based on Orders of
Growth, Mathematical Analysis – Induction, Recurrence
Relation - Substitution method, Recursion method,
Master's Theorem
Algorithms
 A finite set of instructions or logic, written in order, to
accomplish a certain predefined task.

 Algorithm is not the complete code or program.

 It is just the core logic (solution) of a problem.

 Can be expressed either as an informal high level

description as pseudo code or using a flowchart.
 Characteristics of an Algorithm
 Input
An algorithm should have 0 or more well defined inputs.
 Output
An algorithm should have 1 or more well defined outputs
 Unambiguous
Algorithm should be clear and unambiguous.
 Finiteness
Algorithms must terminate after a finite no. of steps.
 Feasibility
Should be feasible with the available resources.
 Independent
An algorithm should have step-by-step directions which should be
independent of any programming code.
Pseudo code
 It is one of the methods that could be used to represent an
algorithm.

 It is not written in a specific syntax

 Cannot be executed

 Can be read and understood by programmers who are familiar

with different programming languages.

 Transformation from pseudo code to the corresponding program

code easier.

 Pseudo code allows to include control structures such as WHILE,

IF-THEN-ELSE, REPEAT-UNTIL, FOR, and CASE, which are available
in many high level languages.
Difference between Algorithm
and Pseudocode
Algorithm Pseudo code
A finite set of instructions or logic, a generic way of describing an algorithm
written in order, to accomplish a without using any specific programming
certain predefined task. language-related notations.
It is just the core logic (solution) of a It is an outline of a program, written in a form
problem which can easily be converted into real
programming statements.
Easy to understand the logic of a Can be read and understood by programmers
problem who are familiar with different programming
languages.

Can be expressed either as an It is not written in a specific syntax. It allows to

informal high level description as include control structures such as WHILE, IF-
pseudo code or using a flowchart. THEN-ELSE, REPEAT-UNTIL, FOR, and CASE,
which are present in many high level languages.
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session - 2
Why to design an algorithm?
 General approaches to the construction of efficient
solutions to problems
◦ They provide templates suited for solving a broad range of
diverse problems.

◦ They can be translated into common control and data structures

provided by most high-level languages.

◦ The temporal and spatial requirements of the algorithms which

result can be precisely analyzed.
Algorithm Design Approaches
Based on the architecture

 Top Down Approach

 Bottom up Approach
Algorithm Design Techniques
1. Brute Force
◦ To solve a problem based on the problem’s statement
and definitions of the concepts involved.
◦ Easiest approach to apply
◦ Useful for solving small – size instances of a problem.
 Some examples of brute force algorithms are:
◦ Computing an (a > 0, n a nonnegative integer) by
multiplying a*a*…*a
◦ Computing n!
◦ Selection sort, Bubble sort
◦ Sequential search
2. Divide-and-Conquer & Decrease-and-Conquer
Step 1
Split the given instance of the problem into several
smaller sub-instances
Step 2
Independently solve each of the sub-instances
Step 3
Combine the sub-instance solutions.
 With the divide-and-conquer method the size of the
problem instance is reduced by a factor (e.g. half
the input size),
 With the decrease-and-conquer method the size is
reduced by a constant.
Examples of divide-and-conquer algorithms:
 Computing an (a > 0, n a nonnegative integer) by
recursion
 Binary search in a sorted array (recursion)
 Mergesort algorithm, Quicksort algorithm recursion)
 The algorithm for solving the fake coin problem
(recursion)
3. Greedy Algorithms "take what you can get now" strategy

 At each step the choice must be locally optimal

 Works well on optimization problems

 Characteristics

1. Greedy-choice property: A global optimum can be arrived at by selecting a

local optimum.

2. Optimal substructure: An optimal solution to the problem contains an

optimal solution to sub problems.

 Examples:
◦ Minimal spanning tree

◦ Shortest distance in graphs

◦ Greedy algorithm for the Knapsack problem

◦ The coin exchange problem

◦ Huffman trees for optimal encoding

4.Dynamic Programming
 Finds solutions to subproblems and stores them in memory for later use.

 Characteristics

1. Optimal substructure:

Optimal solution to problem consists of optimal solutions to subproblems

2. Overlapping subproblems:

Few subproblems in total, many recurring instances of each

3. Bottom up approach:

Solve bottom-up, building a table of solved subproblems that are used to

solve larger ones.
 Examples:
◦ Fibonacci numbers computed by iteration.
◦ Warshall’s algorithm implemented by iterations
5. Backtracking methods
 The method is used for state-space search problems.
What is State-space search problems
- State-space search problems are problems, where the problem
representation consists of:
◦ initial state
◦ goal state(s)
◦ a set of intermediate states
◦ a set of operators that transform one state into another.
◦ a cost function – evaluates the cost of the operations (optional)
◦ a utility function – evaluates how close is a given state to the goal state (optional)
 The solving process solution is based on the construction of a state-
space tree
 The solution is obtained by searching the tree until a goal state is
found.
 Examples:
◦ DFS problem
◦ Maze problems
6. Branch-and-bound
 Branch and bound is used when we can evaluate each node using
the cost and utility functions.
 At each step we choose the best node to proceed further.
 Branch-and bound algorithms are implemented using a priority
queue.
 The state-space tree is built in a breadth-first manner.
 Example:
8-puzzle problem.
N queens problem
Recollect
 Different design Approaches/ Design
Paradigms
◦ Brute force
◦ Divide and Conquer Unit 2
◦ Greedy Algorithms
}
◦ Dynamic Programming Unit 3
◦ Backtracking Unit 4
◦ Branch and Bound Unit 5
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session - 3
Algorithm Analysis
 An algorithm is said to be efficient and fast,
◦ if it takes less time to execute
◦ consumes less memory space.
 The performance of an algorithm is measured on the
basis of
◦ Time Complexity
◦ Space Complexity
 Space Complexity
◦ The amount of memory space required by the algorithm in its
life cycle.

◦ A fixed part For example simple variables & constant used and
program size etc.

◦ A variable part For example dynamic memory allocation,

recursion stacks space etc.

◦ Space complexity S(P) of any algorithm P is

S(P) = C + SP(I)

Where C is the fixed part

S(I) is the variable part of the algorithm

 Time Complexity - T(n)
◦ The amount of time required by the algorithm to run to
completion.

◦ T(n) can be measured as the number of steps, provided each

step consumes constant time.
Algorithm analysis
 The worst-case complexity of the algorithm is the function
defined by the maximum number of steps taken on any
instance of size n.
 The best-case complexity of the algorithm is the function
defined by the minimum number of steps taken on any
instance of size n.
 Finally, the average-case complexity of the algorithm is the
function defined by the average number of steps taken on
any instance of size n.
 Mathematical Analysis
 For Non-recursive Algorithms
◦ There are four rules to count the operations:
 Rule 1: for loops - the size of the loop times the
running time of the body
 Find the running time of statements when executed only once
 Find how many times each statement is executed
 Rule 2 : Nested loops
 The product of the size of the loops times the running time of the
body
 Rule 3: Consecutive program fragments
 The total running time is the maximum of the running time of the
individual fragments
 Rule 4: If statement
 The running time is the maximum of the running times of if stmt
and else stmt.
 Rule 1: for loops
for( i = 0; i < n; i++) // i = 0; executed only once: O(1)
// i < n; n + 1 times O(n)
// i++ n times O(n)
// total time of the loop heading:
// O(1) + O(n) + O(n) = O(n)
sum = sum + i; // executed n times, O(n)
The loop heading plus the loop body will give: O(n) + O(n) = O(n).
IF
1. The size of the loop is n (loop variable runs from 0, or
some fixed constant, to n)
2. The body has constant running time (no nested loops)
Loop running time is: O(n)
 Rule 2 : Nested loops
sum = 0;
for( i = 0; i < n; i++)
for( j = 0; j < n; j++)
sum++;
 Applying Rule 1 for the nested loop (the ‘j’ loop) we get O(n) for the
body of the outer loop. The outer loop runs n times, therefore the
total time for the nested loops will be O(n) * O(n) = O(n*n) = O(n^2)
 for( i = 0; i < n; i++)
for( j = i; j < n; j++)
sum++;
 Here, the number of the times the inner loop is
executed depends on the value of i
i = 0, inner loop runs n times
i = 1, inner loop runs (n-1) times
i = 2, inner loop runs (n-2) times
…
i = n – 2, inner loop runs 2 times
i = n – 1, inner loop runs once.
Thus we get: ( 1 + 2 + … + n) = n*(n+1)/2 = O(n^2)
Running time is the product of the size of the
loops times the running time of the body.
 Rule 3: Consecutive program fragment
sum = 0;
for( i = 0; i < n; i++)
sum = sum + i;
sum = 0;
for( i = 0; i < n; i++)
for( j = 0; j < 2*n; j++)
sum++;

 The first loop runs in O(n) time, the second - O(n^2) time, the
maximum is O(n^2)

The total running time is the maximum of

the running time of the individual
fragments
 Rule 4: If statement

if C
S1;
else
S2;
The running time is the maximum of the running times of S1
and S2.

The running time is the maximum of the

running times of if stmt and else stmt
Exercise
a. c.
sum = 0; sum = 0;
for( i = 0; i < n; i++) for( i = 0; i < n; i++)
for( j = 0; j < n * n; j++) for( j = 0; j < i*i; j++)
sum++; for( k = 0; k < j; k++)
Ans : O(n^3) sum++;
Ans : O(n^5)
d.
sum = 0;
b. for( i = 0; i < n; i++)
sum = 0; sum++;
for( i = 0; i < n; i++)
val = 1;
for( j = 0; j < i; j++)
for( j = 0; j < n*n; j++)
sum++;
val = val * j;
Ans : O(n^2)
Ans : O(n^2)
Order of Growth Function
Linear Search Analysis
linear(a[n], key)
for( i = 0; i < n; i++)
if (a[i] == key)
return i;
else return -1;
 Worst Case : O(n) // Rule 1 for loop explanation
 Average Case :
If the key is in the first array position: 1 comparison
If the key is in the second array position: 2 comparisons
...
If the key is in the ith postion : i comparisons
...
So average all these possibilities: (1+2+3+...+n)/n = [n(n+1)/2] /n = (n+1)/2
comparisons.
The average number of comparisons is (n+1)/2 = Θ(n).
 Best Case : O(1)
Binary Search Analysis
binarysearch(a[n], key, low, high)
while(low<high)
{
mid = (low+high)/2;
if(a[mid]=key)
return mid;
elseif (a[mid] > key)
high=mid-1;
else
low=mid+1;
}
return -1;
Bubble sort analysis
int i, j, temp;
for(i=0; i<n; i++)
{
for(j=0; j<n-i-1; j++)
{
if( a[j] > a[j+1])
{
temp = a[j];
a[j] = a[j+1];
a[j+1] = temp;
}
}
}
Worst Case: In Bubble Sort, n-1
comparisons will be done in 1st pass, n-2 in
2nd pass, n-3 in 3rd pass and so on. So the
total number of comparisons will be
(n-1)+(n-2)+(n-3)+.....+3+2+1
Sum = n(n-1)/2
Hence the complexity of Bubble Sort is O(n2).
Best-case Time Complexity will be O(n), it
is when the list is already sorted.
Insertion Sorting Analysis
int i, j, key;
for(i=1; i<n; i++)
{
key = a[i];
j = i-1;
while(j>=0 && key < a[j])
{
a[j+1] = a[j];
j--;
}
a[j+1] = key;
}
 Worst Case Time Complexity : O(n^2)
 Best Case Time Complexity : O(n)
 Average Time Complexity : O(n^2)
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session - 3
Asymptotic Notations
 Main idea of asymptotic analysis
◦ To have a measure of efficiency of algorithms
◦ That doesn’t depend on machine specific
constants,
◦ That doesn’t require algorithms to be
implemented
◦ Time taken by programs to be compared.
 Asymptotic notations
◦ Asymptotic notations are mathematical tools to
represent time complexity of algorithms for
asymptotic analysis.
Asymptotic Analysis
 Asymptotic analysis refers to computing
the running time of any operation in
mathematical units of computation.
◦ Ο Notation
◦ Ω Notation
◦ θ Notation
 Big Oh Notation, Ο
◦ It measures the worst case time complexity
or longest amount of time an algorithm can
possibly take to complete.
O(g(n)) = { f(n): there exist positive constants c and n0 such that
0 <= f(n) <= cg(n) for all n >= n0}
 0 <= f(n) <= cg(n) for all n >= n0}

If f(n) = 3n+2 If f(n) = 3n+2

g(n)=n g(n)=n2
Then Then
3n+2 <= cn 3n+2 <= cn2
For instance c=4 For instance c=1
3n+2<=4n n=1,2,3,4,5,6…
n>=2 3n+2<=n2
Hence f(n) = O(g(n)) n>=5
Hence f(n) = O(g(n))
 Ω Notation: (Best Case)
◦ Ω notation provides an asymptotic lower
bound
If f(n) = 3n+2 If f(n) = 3n+2
g(n)=n g(n)=n2
Then Then
f(n)>= cg(n) 3n+2 >= cn2
For instance c=1 For instance c=1
3n+2 >= n n=1,2,3,4,5,6…
Hence f(n) = omega(g(n)) 3n+2 >=n2
Hence f(n) = omega(n)
logn
log(logn)
 Θ Notation:
Θ((g(n)) = {f(n): there exist positive constants c1, c2 and n0 such that
0 <= c1*g(n) <= f(n) <= c2*g(n) for all n >= n0}
0 <= c1*g(n) <= f(n) <= c2*g(n) for all n >= n0}

If f(n) = 3n+2
g(n)=n
Then
c1*g(n) <= f(n) <= c2*g(n)
C1,c2>0 and N>n0
For instance c2=4
f(n)<= c2g(n)
3n+2 <= 4n n0=1
f(n)>= c1g(n)
For instance c1=1
3n+2 >= n
Hence f(n) = theta(n)
Order of Growth Function
Worksheet No. 4
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session - 5
Recursion recall
 Functions calls itself
 consists of one or more base cases and
one or more recursive cases.
Mathematical Analysis - Induction
 Consider a recursive algorithm to compute the
maximum element in an array of integers.
 You may assume the existence of a function “max(a,b)”
that returns the maximum of two integers a and b .
Solution
 Let p(n) stand for the proposition that Algorithm finds and returns the maximum
integer in the locations A[1] through A[n]. Accordingly, we have to show that (∀n)
p(n) is true.
 BASIS: When there is only one element in the array , i.e., n=1, then this element is
clearly the maximum element and it is returned on Line 2. We thus see that p(1) is
true.
 INDUCTIVE STEP: Assume that Algorithm finds and returns the maximum
element, when there are exactly k elements in A.
 Now consider the case in which there are k + 1 elements in A. Since (k + 1) > 1,
Line 4 will be executed.
 From the inductive hypothesis, we know that the maximum elements in A[1]
through A[k] is returned. Now the maximum element in A is either A[k+1] or the
maximum element in A[1] through A[k] (say r). Thus, returning the maximum of
A[k+1] and r clearly gives the maximum element in A, thereby proving that p(k) →
p(k+1).
 By applying the principle of mathematical induction, we can conclude that (∀n) p(n)
is true, i.e., Algorithm is correct.
 Find the exact solution to the recurrence
relation using mathematical induction
method

 Solution is T(n)=nlogn
Solution
 Basis: At n = 1, both the closed form and the recurrence
relation agree (0=0) and so the basis is true.
 Inductive step: Assume that T(r) = rlogr for all 1≤ r ≤ k . Now
consider T(k+1). As per the recurrence relation, we have,

 We can therefore apply the principle of mathematical induction

to conclude that the exact solution to the given recurrence is
nlogn.
Worksheet No. 5
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session – 6
Substitution method
 Guess the solution.
 Use induction to find the constants and
show that the solution works.
1. Guess the solution.

Example
2. Use induction to find the constants and show that the solution works.

Example:
1 if n = 1 ,
T (n) =
2T (n/ 2) + n if n > 1 .
1. Guess: T (n) = n lg n + n. [Here, we have a recurrence with an exact func-
tion, rather than asymptotic notation, and the solution is also exact rather than
asymptotic. Weíll have to check boundary conditions and the base case.]
2. Induction:
Basis: n = 1 ⇒ n lg n + n = 1 = T (n)
Inductive step: Inductive hypothesis is that T (k) = k lg k + k for all k < n.
Weíll use this inductive hypothesis for T (n/ 2).
n
T (n) = 2T + n
2
n n n
= 2 lg + + n (by inductive hypothesis)
2 2 2
n
= n lg + n + n
2
= n(lg n − lg 2) + n + n
= n lg n − n + n + n
= n lg n + n .
Lecture Notes for Chapter 4: Recurrences 4-3

Generally, we use asymptotic notation:

• We would write T (n) = 2T (n/ 2) + (n).
• We assume T (n) = O(1) for sufficiently small n.
• We express the solution by asymptotic notation: T (n) = (n lg n).
• We donít worry about boundary cases, nor do we show base cases in the substi-
tution proof.
• T (n) is always constant for any constant n.
• Since we are ultimately interested in an asymptotic solution to a recurrence,
it will always be possible to choose base cases that work.
• When we want an asymptotic solution to a recurrence, we donít worry about
the base cases in our proofs.
• When we want an exact solution, then we have to deal with base cases.
For the substitution method:
• Name the constant in the additive term.
• Show the upper (O) and lower ( ) bounds separately. Might need to use dif-
ferent constants for each.

Example: T (n) = 2T (n/ 2)+ (n). If we want to show an upper bound of T (n) =
2T (n/ 2) + O(n), we write T (n) ≤ 2T (n/ 2) + cn for some positive constant c.
Another Example
 How to prove by the substitution method that
if T(n)=T(n−1)+Θ(n)then T(n)=Θ(n2)
Video Link
https://www.youtube.com/watch?v=Zhhh9qpAVN0
Worksheet No. 6
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session – 7
Recurrence relation - recursion
 solving recurrences
◦ expanding the recurrence into a tree
◦ summing the cost at each level
Example
Expand T(n/4)
Expand T(n/16)
Summing the cost at each level
Adding up the costs
Applying the geometric sum
Polishing the result we get
Verify the guess by applying substitution method
Lets see another example
Lets practice
 Consider T(n) = 3T(n/2) + n. Use a
recursion tree to derive a guess for an
asymptotic upper bound for T(n) and
verify the guess with the substitution
method.
 T(n) = T(n/2) + n2.
 T(n) = 2T(n − 1) + 1.
Worksheet No. 7
UNIT I
INTRODUCTION TO ALGORITM
DESIGN

Session – 8
Master’s theorem method
 Master Method is a direct way to get the solution. The master method
works only for following type of recurrences or for recurrences that
can be transformed to following type.

T(n) = aT(n/b) + f(n) where a >= 1 and b > 1 and f(n) is an asymptotically
positive function.

 There are following three cases:

1. If f(n) < O(nlogba), then T (n) = ϴ (nlogba).

2. If f(n) = ϴ (nlogba) , then T (n) = ϴ (nlogbalogn).

3. If f(n) > Ω (nlogba), and f(n) satisfies the regularity condition, then
T (n) = ϴ (f(n)).
Solve the problems
 T(n) = 3T(n/2) + n2
 T(n) = 7T(n/2) + n2
 T(n) = 4T(n/2) + n2
 T(n) = 3T(n/4) + n lg n
T(n) = 3T(n/2) + n2
T(n) = aT(n/b) + f(n) where a >= 1 and b > 1
a=3 b=2 f(n) = n2
1. If f(n) < O(nlogba), then T (n) = ϴ (nlogba).
2. If f(n) = ϴ (nlogba) , then T (n) = ϴ (nlogbalogn).
3. If f(n) > Ω (nlogba), and f(n) satisfies the regularity condition,
then T (n) = ϴ (f(n)).
Step 1: Calculate nlogba = nlog23 = n1.58
Step 2: Compare with f(n)
Since f(n) > nlogba
i.e. n2 > n1.58
Step 3: Case 3 is satisfied hence complexity is given as
T(n) = Θ(f(n)) = Θ (n2)
T(n) = 7T(n/2) + n2
T(n) = aT(n/b) + f(n) where a >= 1 and b > 1
a=7 b=2 f(n) = n2
1. If f(n) < O(nlogba), then T (n) = ϴ (nlogba).
2. If f(n) = ϴ (nlogba) , then T (n) = ϴ (nlogbalogn).
3. If f(n) > Ω (nlogba), and f(n) satisfies the regularity
condition, then T (n) = ϴ (f(n)).
Step 1: Calculate nlogba = nlog27= n2.80
Step 2: Compare with f(n)
since f(n) < nlogba
Step 3: Case 1 is satisfied hence complexity is given as
T(n) = Θ (nlogba) = Θ (n2.8)
T(n) = 4T(n/2) + n2
T(n) = aT(n/b) + f(n) where a >= 1 and b > 1
a=4 b=2 f(n) = n2

1. If f(n) < O(nlogba), then T (n) = ϴ (nlogba).

2. If f(n) = ϴ (nlogba) , then T (n) = ϴ (nlogbalogn).

3. If f(n) > Ω (nlogba), and f(n) satisfies the regularity condition, then T
(n) = ϴ (f(n)).

Step 1: Calculate nlogba = nlog24= n2

Step 2: Compare with f(n) // Since f(n) = nlogba * log0n

Step 3: Case 2 is satisfied hence complexity is given as

T(n) = Θ (f(n)logn) = Θ (n2logn)

Worksheet No. 8