INSTITUTE OF BUSINESS & MANAGEMENT SCIENCES/CS

THE UNIVERSITY OF AGRICULTURE PESHAWAR

AMIR MUHAMMAD KHAN CAMPUS MARDAN

Program: BS(CS)-VI
Course Title Analysis of Algorithms
Course Code: CS-512
Credit Hours: 03
Course Week: 16
Total Credit Hours: 48
Instructor: Noorul Wahab

Course Objectives:
The subject belongs to applied mathematics, which is very important for the
students of computer science. As the Introduction of Algorithms has already been
studied by the students so in this subject they will study the analysis of those
algorithms as well as some more advanced algorithms. It has broad spectrum
implementations in the area of software design because before writing the software it
is required to check the complexity as well as the performance of the algorithms used
in it.

Week-1
-Introduction to Algorithms: Algorithm is a step by step solution to a particular
problem. The word ‘algorithm’ stems from ‘algoritmi’ which is the Latin form of the
name of a Persian mathematician Abu Abdullah Muhammad ibne Musa al-
Khwarizmi. Algorithm can be represented as a pseudo code or a flowchart or be
implemented in a particular programming language.

-Algorithmic Notations: Here is an algorithm for finding the largest of three integer
values.

Algorithm 1.0: (Largest of three integers) Given three integer values NUM1, NUM2,
NUM3 this algorithm finds the value MAX of the largest of these three.
Step 1. Set MAX = NUM1.
Step 2. [Compare and update] If NUM2 > MAX, then:
Set MAX = NUM2.
[End of if structure]
If NUM3 > MAX, then:
Set MAX = NUM3.
[End of if structure]
Step 3. [Print] Write MAX and Exit.

Each step is given a Step number. The algorithm is completed with Exit. Comments
are put inside brackets. For displaying the output we will use Write and for reading
input we use Read.

Parameter: Those variable in the problem statement which are not assigned specific
values are called parameter.
Instance: When we assign some specific values to these parameter then it becomes
one instance of the problem.

Fall semester 2012 Page 1 of 72

For example in algorithm 1.0 one instance could be: NUM1= 5, NUM2 = 10,
NUM3=4 and solution to this instance of the problem is MAX = 10

-Control Structure of Algorithms: The flow of control in an algorithm can be

sequential, conditional or repetitive. In sequential flow the steps are executed in their
right order. In case of conditional flow the flow is controlled by some conditions. For
example if condition1 is met then it takes one route but if it is false then it goes the
other way. This conditional structure may have one, two or more than two
alternatives. if is the single alternative statement, if-else is double alternative and if-
elseif is multiple alternative structure.

-Flowchart: It is a symbolic representation of an algorithm showing the flow of

control. It starts with the ‘Start’ symbol and terminates with ‘Stop’ symbol. A
diamond is used for decision making steps. A rectangle shows a process while
parallelogram is used for input/output. Below is the flowchart of Algorithm 1.0 given
above.
Single alternative:
If condition, then:
[Block A]
[End of If structure]

Double alternative:
If condition, then:
[Block A]
Else:
[Block B]
[End of If structure]

Multiple alternatives:
If condtion1, then:
[Block A]
Else if condition2, then:
[Block B]
Else if condition3, then:
[Block C]
Else
[Block D]
[End of If structure]

Start

MAX = NUM1

Is NUM2
> MAX

Yes
No
MAX = NUM2

Fall semester 2012 Page 2 of 72

 Is NUM3
> MAX

Yes No

MAX = NUM3

Write
MAX

Stop

-Repetitive flow (Iteration Logic):

Repeat-for loop:
Repeat for C = S to E by I:
[Block]
[End of loop]

Repeat-while loop:
Repeat while condition:
[Block]
[End of loop]

-Complexity of Algorithms: Complexity of an algorithm is a measure of its time and

space. . The time complexity of an algorithm is equal to the number of operations it
performs. To see how fast an algorithm is, we can write a program for it and see how
fast it runs. But there are some factors which make this measurement difficult such as
speed of the CPU, the programming language and compiler used, operating system
etc. Another way to compute the complexity of an algorithm is to determine the
number of time-consuming operations like +, -, *, comparisons, assignments.

The big-Oh notation gives an upper bound on the growth rate of a function.
Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants
c and n0 such that 0 <= f(n) <= cg(n) for n >= n0

Example1: 2n + 10 is O(n)
2n + 10 <= cn
(c - 2) n >= 10
n >= 10/(c - 2)
Pick c = 3 and n0 = 10

Example2: n2 is not O(n)

n2 <= cn
n <= c
This inequality cannot be satisfied because c must be a constant.

Example3: 3n3 + 20 n2 + 5 is O(n3)

For c= 4 and n0 = 21, 3n3 + 20 n2 + 5 <= c n3 for n >= n0

Fall semester 2012 Page 3 of 72

Ascending order of some known functions:
O(log N) < O(N) < O(N log N) < O(N2) < O(Nk) < O(en) < O(n!)

The statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the
growth rate of g(n).
We can use the big-Oh notation to rank functions according to their growth rate.

-Analysis of algorithms:

-Asymptotic Algorithm Analysis: The asymptotic analysis of an algorithm

determines the running time in big-Oh notation. To perform the asymptotic analysis
-we find the worst-case number of primitive operations executed as a function of the
input size
-we express this function with big-Oh notation
Determining how many times the basic operations are done for each value of the input
size is called time complexity analysis.

Algorithm 1.1: (Largest element of an array) Given an integer array ARR[] with size
n this algorithm finds the value MAX of the largest of all the elements of the ARR[].
operations
Step 1. Set MAX = ARR[0]. 2
Step 2. Repeat for C = 1 to n by 1: 2 + 2n
[Compare and update] If ARR[C] > MAX, then: 2(n-1)
Set MAX = ARR[C]. 2(n-1)
[End of if structure]
[End of loop]
Step 3. [Print] Write MAX and Exit. 1
Total: 6n + 1
Example:
-We determine that algorithm 1.1 executes at most 6n + 1 primitive operations.
-We say that algorithm 1.1 “runs in O(n) time”.
Since constant factors and lower-order terms are eventually dropped anyhow, we can
disregard them when counting primitive operations.

Week-2
-Searching Algorithms:
-Sequential Search:
Problem: Check if the value/key x is in the list L of n elements?
Inputs: n, L, x
Outputs: the location of x in L (0 if x is not in L)
integer seqSearch(integer n, T L[], T x)
{
for I = 1 to N step 1
if L[I] == x
return I
end if
end for
}

Worst case: If x is not in the list then the above algorithm will do N comparisons at
maximum. T(n) = n
Best case: If x is the first element in the list then it will do only 1 comparison.

Fall semester 2012 Page 4 of 72

T(n) = 1

If p = 1, which means x is in the list as for the above case, then n(1-p/2) + p/2 give
(n+1)/2
If p = 1/2, which means that for x to be in the list there is fifty-fifty chance, then n(1-
p/2) + p/2 give n[1-(1/2)/2] + (1/2)/2
= n(1-1/4) + ¼
= (4n – n)/4 + ¼
= 3n/4 + ¼
= (3n + 1)/4 (on the average 3/4 of the list is searched)

-Binary Search (divide-and-conquer):

Problem: Check if the value/key x is in the list L of n elements?
Inputs: n, L, x
Outputs: the location of x in L (0 if x is not in L)
integer binSearch(integer n, T L[], T x)
{
Start = 1, End = n

While (Start < or = End)

Mid = floor[(Start + End) / 2]
If L[Mid] == num
return Mid;
else if L[Mid] > num
End = Mid - 1;
else
Start = Mid + 1;
End if
End while
return 0;
}

Worst case: If N is 16 the above algorithm will do 5 comparisons to find out that x is
not in L because every time it is slicing the size to be searched into half.
For example if the list is {4, 9, 13, 14, 16, 18, 23, 28, 30, 31, 33, 38, 40, 42, 44, 50}
and we need to find x = 63 in this list.
T(n) = lg n

Pass 1: First pass it will divide the list at Mid = floor[(1 + 16)/2] = 8 and then
compare the 8th element 28 with x (63). As 63 is greater than 28 therefore we do not
need to compare x with the number at index lower than 8 because the list is already in
order. Pass 2: Mid = floor[(9 + 16)/2] = 12
Again 12th element i-e 38 is less than x therefore we need to compare it with the right
half again.
Pass 3: Mid = floor[(13 + 16)/2] = 14
14th element is again less than x
Pass 4: Mid = floor[(15 + 16)/2] = 15
15th element is again less than x
Pass 5: Mid = floor[(16 + 16)/2] = 16

We can see that the number of comparison the binary search do is lg N + 1

lg 16+1 = 4 + 1 = 5

Fall semester 2012 Page 5 of 72

If the size of list is N = 32 then lg 32 + 1 = 5 + 1 = 6
This means that when the size of the list is doubled the binary search only do one
more comparison.

Best case: If x is the middle element then it will do only 1 comparison.

T(n) = 1

-Log to the base 2:

Log y (X) = Log10 (X) / Log10 (Y) (for any base Y)
Log 2 (X) = Log10 (X) / Log10 (2) (for base 2)
Log 2 (8) = Log10 (8) / Log10 (2) = 0.90309/0.30103 = 3

-Induction: To derive a general principal from specific. Mathematical induction is

used to infer that a given statement is true for all natural numbers.

Example: Show that for all positive integers n,

1 + 2 + 3 +… + n = [n (n+1)] / 2
Induction base: For n = 1
1 = [1(1+1)]/2 = 1
Induction hypothesis: Assume, for an arbitrary positive integer n, that
1 + 2 + 3 +… + n = [n (n+1)] / 2
Induction step: We need to show it for n+1
1 + 2 + 3 +… + (n+1) = [(n+1) [(n+1)+1)] / 2
1 + 2 + 3 +… + n + n+1 [the term before n+1 is n]
By our hypothesis 1 + 2 + 3 +… + n = [n (n+1)] / 2
Therefore 1 + 2 + 3 +… + n + n+1 = [n (n+1)] / 2 + n + 1
= [n (n+1) +2(n + 1)]/2
= [(n+1)(n+2)]/2
= [(n+1) [(n+1)+1)] / 2

-Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ….

f(0) = 0, f(1)= 1, f(2) = f(1) + f(0),
f(n) = f(n-1) + f(n-2)
Below is a recursive algorithm to find out the nth term of this sequence. During the
computation it is usually recalculating many terms.

Algorithm fibo1.0:
Problem: Determine the nth term in the Fibonacci sequence.
Inputs: a nonnegative integer n.
Outputs: fib, the nth term of the Fibonacci sequence.
int fib (int n)
{
if (n <= 1)
return n;
else
return fib (n - 1) + fib (n-2);
}

To compute 0th and 1st term we need 1 computation. For 2nd term 3 computations are
required i-e one each for 0th and 1st term and 1 for the 2nd term itself.
nth term in the T(t) Number of terms computed

Fall semester 2012 Page 6 of 72

sequence
0 1
1 1
2 3
3 5
4 9
5 15
6 25

Notice that for n greater than or equal to 2 the number of terms computed are always
greater than 2n/2
T(n) > 2n/2
We use induction to prove this.
Induction base: To compute a current term we need to consider the previous two
terms.
T(2) = 3 > 22/2
T(3) = 5 > 23/2
Induction hypothesis: Let us suppose T(m) > 2m/2 for all m such that 2 ≤ m < n.
Induction step: We need to prove the above hypothesis for n.
T(n) = T(n-1) + T(n-2) + 1
T(n) > 2(n-1)/2 + 2(n-2)/2 + 1 (by induction hypothesis)
We have changed = to > because we know that 2(n-1)/2 < T(n-1) therefore the whole
expression on the R.H.S must be less than the L.H.S
To substantiate that L.H.S is greater than R.H.S we can diminish the term 2(n-1)/2 to
2(n-2)/2still keeping the > expression intact and similarly we can also drop 1.
T(n) > 2(n-2)/2 + 2(n-2)/2 + 1
T(n) > 2.2(n-2)/2
T(n) > 2(n-2)/2 + 1
T(n) > 2[(n-2) + 2]/2
T(n) > 2n/2

A non-recursive algorithm for the same problem given in fibo1.0:

Algorithm fibo1.1:
int fibo(int n)
{
int fib1 = 0;
int fib2 = 1;
int f;
if (n <= 1)
return n;
else
for (int i = 2; i <= n; i++)
{
f = fib1 + fib2;
fib1 = fib2;
fib2 = f;
}
return f;
}

This algorithm computes fib1, fib2 and then loops from 2 to n to find the nth term i-e
it takes n + 1 computations to find the nth term. Now lets us compare both the
algorithms fibo1.0 and fibo1.1. We assume that both the algorithms run on a

Fall semester 2012 Page 7 of 72

hypothetical machine which can compute one term in the computation in 1
nanosecond.
N n+1 2n/2 fibo1.0 Fibo1.1
10 11 32 32 ns 11 ns
20 21 1024 1 micro s 21 ns
40 41 1048576 1048.576 x 10+3 ns = 41 ns
1048 x 10+3 x 10-3 micro
s = 1048 micro s [1 ns =
10-3 micro s]
60 61 1073741824 1.073741824 x 10+9 ns = 61 ns
1 sec [1 ns = 10-9 s]
80 81 1099511627776 1099.511627776 x 10+9 81 ns
ns = 1099 s = 1099 x
1/60 min = 18 min [1 s
= 1/60 m]
100 101 1125899906842624 1125899.906842624 x 101 ns
10+9 ns = 1125900 s =
1125900 x 1/60 min =
18765 x 1/1440 days =
13 days [1 min =
1/1440 day]
120 121 1152921504606846976 1152921504.606846976 121 ns
x 10+9 ns = 1152921505
s = 1152921505 x 1/60
min = 19215358 x
1/1440 days = 13344
days = 13344 x 1/365
years = 35 years [1 day
= 1/365 year]

Week-3
-Sorting Algorithms: Sorting the input elements in some order like decreasing order
or non-decreasing order.
-Exchange Sort:
Problem: Arrange the elements of a list in ascending order.
Inputs: array A of size n
Outputs: array A with elements sorted in ascending.
void exchangeSort (A[],int n)
{
for (i=1; i<= n; i++)
for (j=i+l; j<= n; j++)
if (A[j] < A[i])
exchange A[i] with A[j];
}

Every-case time complexity of exchange sort:

For the first pass of the outer loop the inner loop executes n-1 times, for the second
pass the inner loop executes n-2 times and so on till it executes just 1 time for the last
pass of the outer loop.

Fall semester 2012 Page 8 of 72

T(n) = (n-1) + (n-2) + (n-3) + ... + 1 = [n (n- 1)]/2 [Proof by induction]

Induction base: For n = 1

(1-1) = [1(1-1)]/2
For n = 2
(2-1) + (2-2) = [2(2-1)]/2
For n = 3
(3-1) + (3-2) + (3-3) = [3(3-1)]/2
Induction hypothesis: Assume, for an arbitrary positive integer n, that
(n-1) + (n-2) + (n-3) + ... + 1 = [n (n- 1)]/2
Induction step: We need to show it for n+1
(n+1-1) + (n+1-2) + (n+1-3) + ... + 1 = [(n+1) (n+1)- 1)]/2 = [(n+1) (n)]/2
(n) + (n-1) + (n-2) + ... + 1
(n) + [n(n-1)]/2 [by induction hypothesis]
2 2
[n +2n-n]/2= (n +n)/2 = n(n+1)/2

-Other sorting algorithms:

-Other considerations: When applying the theory of complexity analysis one should
also be aware of the time it takes to execute the basic operations, the control
instructions, and other overhead instructions like initialization instructions before the
loops.
Suppose Algo1 and Algo2 are two algorithms to solve the same problem with time
complexities as n and n2 respectively. Algo1 appears more efficient if the basic
operations and the control instructions take the same time in both algorithms. But let
us suppose that Algo2 takes t time to execute the basic operation and Algo1 take
1000.t time to execute the basic operation.

For Algo1 T(n) = n x 1000t

For Algo2 T(n) = n2 x t
To find out that Algo1 is more efficient we must solve the following inequality:
n x 1000t < n2 x t
Dividing both sides by tn we get
1000 < n
This means that Algo1 is efficient than Algo2 when n is equal to or more than 1000.
But if n is less than 1000 then Algo2 is more efficient than Algo1.
Algorithms with time complexities such as n or 2n or cn are called linear-time
algorithms because their time complexities are linear in the input size n.
Algorithms with time complexities such as n2 or 10 n2 are called quadratic-time
algorithms. From the above example it can be stated that linear-time algorithm is
eventually more efficient than any quadratic-time algorithm. In the theoretical
analysis of an algorithm the interest lies in their eventual behavior and we can group
algorithms according to their eventual behavior.
All linear functions are grouped under Θ(n). So we can say that f(n) = 2n is order of
Θ(n) or f(n) is Θ(n). Similarly 10 n2 is order of n2

The table below shows that the term n + 300 is of very little effect on the third column
as compared to the second. Therefore we drop the low order terms and the constants
to get the order of a function. For example T(n) for exchange sort is n2/2 - n/2. By

Fall semester 2012 Page 9 of 72

dropping the term n/2 we get n2. We can say that exhange sort is a quadratic-time
algorithm or a Θ(n2).
N 10 n2 10 n2 + n + 300
1 10 311
10 1000 1310
20 4000 4320
30 9000 9330
40 16000 16340
60 36000 36360
100 100000 100400
200 400000 400500
500 2500000 2500800
1000 10000000 10001300

-Complexity order:
-big-Oh: It describes the asymptotic behavior of a function because it is concerned
only with eventual behavior. We say that "big O" puts an asymptotic upper bound on
a function and tells us how fast a function grows or may be declines. The rate of
growth of a function is called its order that is why the letter O is used.

Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants
c and n0 such that 0 <= f(n) <= cg(n) for n >= n0
For example when we say n2 + 10n is O(n2) it means that for some value of n greater
than n0 and a positive real constant c the function n2 + 10n fall beneath n2 and stays
there.

N 2 n ^ 2 n ^ 2 + 10n
1 2 11
2 8 24
3 18 39
4 32 56
5 50 75
6 72 96
7 98 119
8 128 144
9 162 171
10 200 200
11 242 231
12 288 264
13 338 299
14 392 336
15 450 375

Fall semester 2012 Page 10 of 72

n2 + 10n <= cn2
Take c = 2 and n0= 10
10 x 10 + 10 x 10 <= 2 x 10 x 10
200 <= 200
This holds for n0 = 10 but we need to prove that it also holds for n > 10. This we
prove by induction:
We need to show that (n+1)2 + 10(n+1) <= 2(n+1) 2
n2 + 1 + 2n + 10n + 10 <= 2(n2 + 1 +2n)
n2 + 12n + 11 <= 2n2 + 4n + 2
n2 + 8n + 9 <= 2n2 (subtracting 4n and 2 from both sides)
8n + 9 <= n2 (subtracting n2 from both sides)
8 + 9/n <= n (dividing both sides by n)
And this holds for any n>= 10 because 9/n is always less than 1 when n >= 10

Therefore we can say that n2 + 10n grows at the order of n2

We can also take c = 10 and n0= 2

-Show that 7n + 8 = O(n)

7n + 8 <= c.n
Take c= 8 and n0=8
7n + 8 <= 8 x n
Induction base: for n = 8
7 x 8 + 8 <= 8 x 8
64 <= 64
Induction hypothesis: for all n >= 8
7n + 8 <= 8 x n
Induction step: for (n+1)
We need to show that 7(n + 1) + 8 <= 8 x (n + 1)
7n + 7 + 8 <= 8n + 8
7n + 7 + 8 <= 7n + n + 8
7n + 7 <= 7n + n (subtract 8 from both side)
The relationship hold because n >= 8

Fall semester 2012 Page 11 of 72

Therefore 7n + 8 = O(n)

-Show that exchange sort is O(n2):

For exchange sort T(n) = n (n- 1)/2
n(n-1)/2 <= c n2
Take c= ½ and n0 = 1
1(1-1)/2 <= ½ x 1 x 1
0 <= 1/2

-big-Omega notation: It puts an asymptotic lower bound on a function.

Given functions f(n) and g(n), we say that f(n) is Ω(g(n)) if there are positive
constants c and n0 such that 0 <= cg(n) <= f(n) for n >= n0

5 n2 is Ω(n2)
c n2 <= 5 n2
Take c = 5 and n = 1
5 x 1 x 1 <= 5 x 1 x 1

n2 + 10n Ω(n2)
c n2 <= n2 + 10n
Take c = 1 and n0= 1
1 x 1 x 1 <= 1 x 1 + 10 x 1
1 <= 11
N 1 n ^ 2 n ^ 2 + 10n
1 1 11
2 4 24
3 9 39
4 16 56
5 25 75
6 36 96
7 49 119
8 64 144
9 81 171
10 100 200
11 121 231
12 144 264
13 169 299
14 196 336
15 225 375

Fall semester 2012 Page 12 of 72

-Show that exchange sort is Ω(n2):
For exchange sort T(n) = n (n- 1)/2
c n2 <= n(n-1)/2
Take c= 1/4 and n0 = 2
¼ x 2 x2 <= 2/2
1 <= 1

-Θ notation (theta): It sandwiches a function inside two bounds. When we say that
f(n) is Θ(g(n)) we mean that g(n) is both a tight upper-bound and a tight lower-bound
on the growth of f(n). So for all n >= n0 the function f(n) is equal to g(n) to within a
constant factor.

Given functions f(n) and g(n), we say that f(n) is Θ(g(n)) if there are positive constants
c1, c2 and n0 such that 0 <= c1g(n) <= f(n) <= c2g(n) for n >= n0
In other words f(n) is Θ(g(n)) only if f(n) is O(g(n)) and f(n) is Ω(g(n))

1/2 n2- 3n is Θ(n2)

c1 n2 <= 1/2 n2- 3n <= c2 n2
Divide throughout by n2
c1 <= 1/2 – 3/n <= c2
Take c1 = 1/14, c2 = 1/2 and n0 = 7
1/14 <= (7 – 6)/2(7) <= 1/2

-Show that exchange sort is Θ(n2):

For exchange sort T(n) = n (n- 1)/2
c1 n2 <= n(n-1)/2 <= c2 n2
c1 n2 <= (n2/2 - n/2) <= c2 n2
Divide throughout by n2
c1 <= (1/2 - 1/2n) <= c2
Take c1 = 1/4, c2 = 1/2 and n0 = 2
1/4 <= (2 – 1)/2(2) <= 1/2

-Show that 5n + 7 is not Ω(n2) and hence is not Θ(n2)

Fall semester 2012 Page 13 of 72

Proof by contradiction: Let’s assume that 5n+7 is Ω(n2)
cn2 <= 5n+7
Take c=1 and n0=1
1 <= 12
Now increase n0 to 10
1 x 10 x 10 <= 5 x 10 + 7
100 <= 57 (which is not true)
Therefore 5n + 7 is not Ω(n2)

-Little-oh: To state that f(n) is strictly less than g(n) we use little-oh. The big-O may
or may not give an asymptotically tight bound for example 2n2 = O(n2) is a tight
bound but to say that 2n is O(n2) is not a tight bound. But big-O is usually used as a
tight upper-bound while little-o is used to describe an upper-bound that is not tight.
For any postive constant c>0, there exists a constant n0>0 such that 0 <= f(n) < cg(n)
for n >= n0
This definition means that f grows much slower than g.
The difference between the big-O and little-o is that the big-O has to be true for at
least one positive constant but the latter must be true for every positive constant.
Every function f that is little-o of g is also big-O of g, but not every function that is
big-O of g is little-o of g.

-2n is o(n2)
2n < cn2
Divide both sides by cn
2/c < n
That is we can choose any n > 2/c
Take c = 1, then n should be greater than 2. Let n = 3
2x3<1x3x3
6<9
Now take c = 1, n = 100
2 x 100 < 1 x 100 x 100
200 < 10000
If c = 1 and n = 1000000000
2 x 1000000000 < 1 x 1000000000 x 1000000000
2000000000 < 1000000000000000000
The function f(n) becomes insignificant relative to g(n) as n approaches infinity.

If we take c = 1/10, then n should be greater than 20. Let n = 21

2 x 21 < 1/10 x 21 x 21
42 < 44

-2n2 is O(n2) but is not o(n2)

Proof by contradiction:
2n2 < cn2
Divide both sides by n2
2<c
c must be greater than 2 for the above statement to hold which is contrary to the
definition of little-o because it should hold for any positive constant c.

Fall semester 2012 Page 14 of 72

-little-omega notation: It gives a lower bound that is not asymptotically tight.
Given functions f(n) and g(n), we say that f(n) is ω(g(n)) if for any positive constants
c> 0 there exists a constant n0>0 such that 0 <= cg(n) < f(n) for n >= n0

- n2/2 is ω(n)
c.n < n2/2
Divide both sides by n
c < n/2 or 2c < n
Take c = 1, then n should be greater than 2. Let n = 3
1x3<3x3/2
3 < 4.5
Let c = 1, n = 100
1 x 100 < 100 x 100 / 2
100 < 5000
Let c = 1, n = 1000000000
1 x 1000000 < 1000000 x 1000000 / 2
1000000 < 500000000000
The function f(n) becomes arbitrarily large to g(n) as n approaches infinity.

If we take c = 1/10, then n should be greater than 1/5. Let n = 2/5

(1/10 x 2/5) < 2/5 x 2/5 x 2
1/25 < 2/25

-Properties of order:
f(n) = Ω(g(n)) if and only if g(n) = O(f(n))
f(n) = ω(g(n)) if and only if g(n) = o(f(n))
f(n) = Θ(g(n)) if and only if f(n) = O(g(n)) and g(n) = O(f(n))

Fall semester 2012 Page 15 of 72

-Functions of order factorial:
For all a > 0
an=o(n!)
This implies that n! is worse than any exponential complexity function.

-Ordering of complexity categories (from slower growth rate to faster growth

rate): d > c, k > 2
O(1) constant
O(log(n)) logarithmic
O(n) linear
2
O(n ) quadratic
O(nk) polynomial
n
O(c ) exponential
O(dn) exponential
O(n!) factorial

If a function f(n) is in a complexity category that is above the category containing

g(n) then f(n) = o(g(n))

-When to appeal to these notations: If it is possible to obtain the exact time

complexity of an algorithm then we can determine its order by throwing out constant
and lower-order terms. But if this is not possible then we can use the above notations
to determine its order.

Week-4
- Complexity analysis
o Time
o Every
o Worst
o Average
-Every-case time complexity: When the basic operation is always done the same
number of times for every instance of size n then its time complexity is called every-
case. Consider the following algorithm that adds all the number in an array:

Problem: Add all the numbers in the array S of size n.

Inputs: Positive integer n, array of numbers S of size n
Outputs: sum, the sum of the numbers in S.
int sum(int n, S[ ])
{
total = 0;
for (i = 1; i <= n; i++)
total = total + S[i];
return total;
}

The basic operation total = total + S[i]; is performed n number of times

regardless of the values of the elements of the array. Therefore we can say that every-
time complexity of this algorithm is T(n) = n

-Matix multiplication:

Fall semester 2012 Page 16 of 72

Problem: Determine the product of two n x n matrices.
Inputs: a positive integer n, two-dimensional arrays M1 and M2
Outputs: a two-dimensional array M containing the product of M1 and M2.
void matrixMult(int n, A[][], B[][], C[][])
{
for (i=1; i <= n; i++)
for (j=1; j <= n; j++)
for (k=1; k <= n; k++)
M[i][j] = M[i][j] + M1[i][k] * M2[k][j];
}

Every-time complexity of matrix multiplicaiton T(n) = n3 as the first loop is executed

n times, the second loop is executed n x n times and the third loop is executed n x n x
n times.

-Average-case time complexity of sequential search:

Average case (when x is in the list): Let us suppose that all the elements of the list are
distinct then probability of x to be at position k in the list is 1/n i-e all the elements
have the same probability of 1/n which means that they are equally likely to be in any
of the slots of the list.
T(n) = (1 x 1/n) + (2 x 1/n) + (3 x 1/n) + … + (n x 1/n)
= 1/n (1 + 2 + 3 + … + n)
= 1/n [n(n+1)/2]
= (n+1)/2

Average case (when x is not in the list): Let us p be the probability that x is in the
list. The probability that x is at kth position is p/n such that it is equally likely to be in
any of the slots of the list. The probability that x is not in the list is 1-p and to find out
that x is not in the list we need to loop n times the basic operation.
T(n) = (1 x p/n) + (2 x p/n) + (3 x p/n) + … + (n x p/n) + n(1-p)
= p/n (1 + 2 + 3 + … + n) + n(1-p)

Fall semester 2012 Page 17 of 72

 = p/n [n(n+1)/2] + n(1-p)
= p(n+1)/2 + n(1-p)
= {p(n+1) + 2[n(1-p)]}/2
= [p(n+1) + 2(n-pn)]/2
= [p(n+1) + 2n-2pn]/2
= (pn+p + 2n-2pn)/2
= (-pn+p + 2n)/2
= -pn/2 + p/2 + 2n/2
= -pn/2 + p/2 + n
= n - pn/2 + p/2
= n(1 – p/2) + p/2

Week-5.6
-Introduction to Computational Complexity: It is the study of all possible
algorithms that can solve a given problem. A computational complexity analysis tries
to determine a lower bound on the efficiency of all algorithms for a given problem.
For example the lower bound of sorting algorithms is Ω(n log n) which means that
non of the sorting algorithms designed so far can do better than n log n. The table
below shows that an algorithm with complexity n2 (such as exchange sort) takes
almost 31 years but an algorithm with complexity n log n (such as merge sort) takes
29 seconds.

When we analyze a specific algorithm, we determine its time complexity but we do

not analyze the problem itself. In computationl complexity analysis we do analyze the
problem to see how diffenent algorithms solve it and to see if it is possible to design a
better algorithm or to prove that a more efficient algorithm is not possible.

Let us assume that one basic operation takes 10-9 seconds (1 nanosecond)
N f(n) = n f(n) = n log n f(n) = n2
10 0.01 microsec 0.033 microsec 0.1 microsec
9
10 1 second 29.9 seconds 31.7 years

-Some sorting algorithms: We will study different sorting algorithms like insertion,
selection, merge and quick sort and will make a comparison between them.
When the extra space required by a sorting algorithm does not increase with the input
size (i-e it is a constant) then such algorithms are called in-place.

-Insertion sort:
-Begin with the second element of the list (next time start with the third and so on).
-Advance the other elements by one position to the right till the correct position of the
selected element is encountered. Insert the selected element at that position.
-Repeat the above steps until all the elements are done.

Problem: Sort a list of n keys in ascending order.

Inputs: array L of size SIZE, indexed from 1 to SIZE
Outputs: the array L containing the keys in ascending order.
void insertionSort(int L[], int SIZE)
{

Fall semester 2012 Page 18 of 72

 int numToInsert, insertLoc;
for(int i = 2; i <= SIZE; i++)
{
numToInsert = L[i];
insertLoc = i;
for(int j = i - 1; (j > 0 && numToInsert < L[j]); j--)
{
insertLoc = j;
L[j + 1] = L[j];
}
L[insertLoc] = numToInsert;
}
}

-Selection sort:
-Find the minimum value in the list
-Swap this minimum value with the element at first position. Once this minimum
value sits in the first position then it is at its correct expected position. Next repetition
will not consider this first element but will rather start with the second element.
-Repeat the above steps for the remainder of the list

Problem: Sort a list of n keys in ascending order.

Inputs: array L of size n, indexed from 1 to n
Outputs: the array L containing the keys in ascending order.
void selectionSort(int L[], int n)
{
int min = 0;
int temp;
for(int i = 1; i < n; i++)
{
min = i;
for(int j = i + 1; j <= n; j++)
{
if(L[j] < L[min])
min = j;
}
Exchange L[i] and L[min]
}
}

-Comparing Exchange/Insertion/Selection sorts:

In terms of comparison of keys Exchange sort and Selection sort both have the same
time complexity. But in terms of assignments the Exchange sort is quadratic where as
Selection sort is linear because Selection sort only keeps track of the index of the
minimum key and does the exchange after the loop terminates.
If the list is already in ascending order then Exchange sort does better than Selection
sort in terms of assignments. In that case Exchange sort does no assignments of
records.
On average Insertion sort performs better than Selection sort when compared by the
number of comparisons. Selection sort’s time complexity in terms of assignments is
linear whereas that of Insertion sort is quadratic.

Week-7
-Best and Worst time complexity of insertion sort:
1- int numToInsert, insertLoc; cost times
2- for(int i = 2; i <= SIZE; i++) c1 n

Fall semester 2012 Page 19 of 72

 {
3- numToInsert = arr[i]; c2 n-1
4- insertLoc = i; c3 n-1
5- for(int j = i - 1; (j > 0 && c4 Sumi=2 to n ti
numToInsert < arr[j]); j--)
{
6- insertLoc = j; c5 Sumi=2 to n (ti-1)
7- arr[j + 1] = arr[j]; c6 Sumi=2 to n (ti-1)
}
8- arr[insertLoc] = numToInsert; c7 n-1
}

Let ti be the number of times the inner for loop test is executed
T(n) = c1n + c2(n-1) + c3(n-1) + c4(Sumi=2 to n ti) + c5(Sumi=2 to n (ti-1)) + c6(Sumi=2
to n (ti-1)) + c7(n-1)
The best case occurs when the elements are already in ascending order. In that case ti
in step 5 becomes 1 because on every iteration of the outer loop the condition
numToInsert < arr[j]) is false. In that case step 6 and 7 are not executed.
T(n) = c1n + c2(n-1) + c3(n-1) + c4(n-1) + c5(0) + c6(0) + c7(n-1)
= c1n + c2(n-1) + c3(n-1) + c4(n-1) + c7(n-1)
= c1n + c2n – c2 + c3n – c3 + c4n- c4 + c7n – c7
= c1n + c2n + c3n + c4n + c7n – c2 – c3 – c4 – c7
= (c1 + c2 + c3 + c4 + c7) n – (c2 + c3 + c4 + c7)
We can express the above as an-b for constants a and b that depend on the statement
costs ci and that is a linear function.

The worst case occurs when the elements are in descending order. In that case we
must compare each element arr[i] with each element in the entire sorted array arr[1…
i-1] so ti = I for 2, 3, …,n
Sumi=2 to n (ti-1) = n(n-1)/2 [Proof by induction]
(2-1) + (3-1) + (4-1) + … + (n-1)
1 + 2 + 3 + … + n-1

Sumi=2 to n ti = n(n+1)/2 – 1 [Proof by induction]

So T(n) = c1n + c2(n-1) + c3(n-1) + c4[n(n+1)/2-1] + c5[n(n-1)/2] + c6[n(n-1)/2] + c7(n-
1)
= c1n + c2n - c2 + c3n - c3 + c4[(n2+n-2)/2] + c5[n2/2 - n/2] + c6[n2/2 -n/2] + c7n- c7
= c1n + c2n - c2 + c3n - c3 + c4n2/2 + c4n /2 + c4 + c5n2/2 - c5n/2 + c6n2/2 - c6n/2 + c7n-
c7

= c4n2/2 + c5n2/2 + c6n2/2 + c1n + c2n + c3n + c4n/2 - c5n/2 - c6n/2 + c7n - c2 - c3 + c4 -
c7
= (c4/2 + c5/2 + c6/2) n2+ (c1 + c2 + c3 + c4/2 - c5/2 - c6/2 + c7) n - (c2+ c3 + c4 + c7)
We can express the above as a n2+bn-c for constants a, b and c that depend on the
statement costs ci

-Average time complexity of insertion sort:

Week-8
-Selection sort:

Fall semester 2012 Page 20 of 72

-Lower Bounds for Algorithms that Remove at Most One Inversion per
Comparison:
Permutation: n!/(n-r)!
Permutations for {1, 2, 3} are (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)

We use indices for the elements of a permutation for example in the permutation (3, 1,
2) k1= 3, k2= 1, and k3= 2
Inversion: An inversion in a permutation is a pair (ki, kj) such that i < j and ki > kj
For example the permutation {5, 3, 7, 2} contains the following inversions:
(5, 3), (5, 2), (3, 2), (7, 2)
If there are no inversions in a permutation then that permutation is in sorted order.
An algorithm that removes at most one inversion after each comparison and sorts n
distinct keys only by comparisons of keys must do at least n(n-1)/2 comparisons in
worst case. To prove this we need to show that there is a permutation with n(n-1)/2
inversions and when that permutation is given as input to the algorithm it will have to
remove that many inversion and therefore do at least that many comparisons. Such a
permutation is the list in descending order {n, n-1, n-2… 3, 2, 1}

Proof by induction:
The permutation {n, n-1, n-2… 3, 2, 1} has n(n-1)/2 inversions

Induction base: For n = 2

(2, 1) has one inversion, 2(2-1)/2 = 1
For n = 3
(3, 2, 1) has three inversions i-e (3, 2), (3, 1), (2, 1),
3(3-1)/2 = 6/2= 3
For n = 4
(4, 3, 2, 1) has six inversions i-e (4, 3), (4, 2), (4, 1), (3, 2), (3, 1), (2, 1),
4(4-1)/2 = 12/2= 6
Induction hypothesis: Assume, for an arbitrary positive integer n, that the
permutation {n, n-1, n-2… 3, 2, 1} has n(n-1)/2 inversions
Induction step: Show it for n+1
We need to show that the permutation {(n+1), n, n-1, n-2… 3, 2, 1} has (n+1)[(n+1)-
1]/2 or n(n+1)/2 inversions

n, n-1, n-2… 3, 2, 1 has n(n-1)/2 [by induction hypothesis] ------- eq(1)

For base cases we can see that when:
n increases from 2 to 3 there is an increase of 2 inversions i-e (2, 1) has one inversion
while (3, 2, 1) has three inversions
n increases from 3 to 4 there is an increase of 3 inversions i-e (3, 2, 1) has three
inversions while (4, 3, 2, 1) has six inversions
n increases from 4 to 5 there is an increase of 4 inversions i-e (4, 3, 2, 1) has six
inversions while (5, 4, 3, 2, 1) has ten inversions
n increases from n to n+1 there is an increase of n inversions
Therefore the total inversions for {(n+1), n, n-1, n-2… 3, 2, 1} is n + n(n-1)/2
= (2n + n2 – n)/2
= (n2 + n)/2
= n(n+1)/2 [Proved]

For example the list {4, 3, 2, 1} has the following 6 inversions:

Fall semester 2012 Page 21 of 72

(4, 3), (4, 2), (4, 1), (3, 2), (3, 1), (2, 1)
And in this case n=4 and n(n-1)/2 becomes 4(4-1)/2 = 6

Transpose of a permutation {3, 1, 2} is {2, 1, 3}

Transpose of (34, 78) is (78, 34). If the first pair has an inversion then the second
must not have an inversion and vice versa.
Given a permutation, the pair (s, r) is an inversion in either the permutation or its
transpose but not in both. There are n(n-1)/2 such pairs of integers between 1 and n.
This means that a permutation and its transpose have exactly n(n-1)/2 inversions
between them. So the average number of inversions in a permutation and its transpose
is
[n(n-1)/2] / 2
=[n(n-1)/2] x (1/2)
=n(n-1)/4

Therefore, if we consider all permutations equally probable for the input, the average
number of inversions in the input is also n(n-1)/4. Because we assumed that the
algorithm removes at most one inversion after each comparison, on the average it
must do at least this many comparisons to remove all inversions and thereby sort the
input.

Insertion Sort removes at most the inversion consisting of S [j] and x after each
comparison, and therefore this algorithm is in the class of algorithms addressed by
worst case complexity of n(n-1)/ 2 and average case complexity of n(n-1)/4. Similarly
Exchange sort and selection sort are also in this class.
For example the permutation {3, 15, 9, 8} has inversions (15, 9), (15, 8), (9, 8)
If we are comparing 15 and 9 and exchange them then the permutation becomes:
{3, 9, 15, 8} which has inversions (9, 8) and (15, 8). Therefore Exchange sort does at
most one inversion after each comparison.

Week-11
-Introduction to divide and conquer approach: This design strategy involves the
following steps:
-Divide an instance of a problem into one or more smaller instances.
-Solve (conquer) each of the smaller instances using recursion.
-In some cases it is necessary to combine the solutions to the smaller instances in
order to obtain the solution to the original instance.

-Binary search method: Binary search is an example of divide and conquer

approach. In order to find the location of a number in an array whose elements are
already in ascending order we divide the array in the middle. We check if the middle
element is the required number then we return that location. Otherwise we take the
left half of the array if the required number is less than the element at the middle
location or take the right half if it is greater. Below is a recursive version of binary
search algorithm. The recursion clearly shows the divide-and-conquer approach
because each call to the recursion function is trying to find a solution to a specific
smaller instance of the original problem.

int x = 145;

Fall semester 2012 Page 22 of 72

const int n = 8;
int arr[] = {12, 15, 19, 20, 50, 99, 145, 216};

int binarySearch(int start, int end)

{
int mid;
if (start > end)
return 0;
else
{
mid = (start + end)/2;
if (x == arr[mid])
return mid;
else if (x < arr[mid])
return binarySearch(start, mid - 1);
else
return binarySearch(mid + 1, end);
}
}
Analysis of a recursive algorithm involves two steps i) determining the recurrence and
ii) solving the recurrence. i-e runtime of original problem = time taken to reduce
problem + runtime of reduced problem
In recurrence of the form T(n) = aT(n/b) + f(n)
’a‘ is the number of subproblems that the original problem is divided into. For
example in binary search algorithm ‘a‘ is 1 because we search only one of the two
halves. ‘b‘ is 2 because the new subproblem has half the elements of the original
problem. And f(n) = 1 because finding the middle of the list and recuring into one half
is a constant time operation.

The following examples of recursive algorithms and their analysis shows different
techniques for doing this analysis.

-Worst case time complexity of the above algorithm:

The worst case occurs when the number to find is the largest of the array or is not
present in the array. If size n of the array is power of 2 then each recursive call
receives an array half the size of the pervious call. So the recurrence is:
Tn = Tn/2 + 2
Total comparison is: 2 comparisons for the top level plus the number of comparisons
for the recursive calls.
When n is 1 then the numbers of comparisons are two therefore the recurrence
formula is:
Tn = Tn/2 + 2 ---------------------------------------------- Rec 1
for n > 1, n a power of 2
T1 = 2
Here are the first few values of the recurrence:
T2 = T2/2 + 2 = T1 + 2 = 2 + 2 = 4
T4 = T4/2 + 2 = T2 + 2 = 4 + 2 = 6
T8 = T8/2 + 2 = T4 + 2 = 6 + 2 = 8
T16 = T16/2 + 2 = T8 + 2 = 8 + 2 = 10
Guess and Test method:
It appears that Tn = 2 lg n + 2 which is the solution to the recurrence in Rec 1 above.
We prove this solution by induction:

Fall semester 2012 Page 23 of 72

Induction base: for size = 1
T1 = 2 lg 1 + 2 = 0 + 2 = 2
Induction hypothesis: Assume, for an arbitrary n > 0 and n a power of 2, that
Tn = 2 lg n + 2
Induction step: Because the recurrence is only for power of 2, the next value to
consider after n is 2n. Therefore, we need to show that
T2n = 2 lg 2n + 2
Insert n = 2n in the recurrence Rec 1.
T2n = T2n/2 + 2
= Tn + 2
= (2 lg n + 2) + 2 [as Tn = 2 lg n + 2 from our induction hypothesis]
=2 lg n + 1 + 1 + 2
=2 lg n + lg 2 + lg 2 + 2 [lg 2 = 1]
=2 lg n + 2 lg 2 + 2
=2(lg n + lg 2) + 2
=2 lg 2n + 2 [lg n + lg 2 = lg 2n]

Therefore
Tn = 2 lg n + 2
If n is not a power of 2 then
Tn = 2 FLOOR[lg n] + 2

Substitution method:
Tn = Tn/2 + 2 ---------------------------------------------- Rec 1
for n > 1, n a power of 2
T1 = 2
Substitute n by 2k and T2k by Ck
Tn/2 = T2k /2 = T2k-1= Ck-1

C0 = 2
Ck= Ck-1 + 2 ---------------------------------------------- Rec 2
The first few values of the new Rec 2:
C1= C1-1 + 2 = C0 + 2 = 2 + 2 = 4
C2= C2-1 + 2 = C1 + 2 = 4 + 2 = 6
C3= C3-1 + 2 = C2 + 2 = 6 + 2 = 8
C4= C4-1 + 2 = C3 + 2 = 8 + 2 = 10
The solution to the transformed recurrence is:
Ck= 2k + 2
By inverse substitution:
2k = n
 k = lg n
Tn = 2 lg n + 2

Problem: Determine n! = n (n − 1) (n − 2) … (3) (2) (1) when n ≥ 1.

0! = 1
Inputs: a nonnegative integer n.
Outputs: n!
int factorial(int n)
{

Fall semester 2012 Page 24 of 72

 if (n == 0)
return 1;
else
return n * factorial(n - 1);
}

tn = tn-1 + 1
The number of multiplications done at the top level is 1 plus the number of
multiplications done for the recursive calls.
The recursion stops when n = 0 and no multiplication is done in that step. Therefore
t0 = 0
t1 = t1-1 + 1 = t0 + 1 = 0 + 1 = 1
t2 = t2-1 + 1 = t1 + 1 = 1 + 1 = 2
t3 = t3-1 + 1 = t2 + 1 = 2 + 1 = 3
t4 = t4-1 + 1 = t3 + 1 = 3 + 1 = 4
The solution to the recurrence is:
tn = n

Induction base: for n = 0

t0 = 0
Induction hypothesis: Assume, for an arbitrary positive integer n, that
tn = n
Induction step:
We need to show that:
tn+1 = n+1
Insert n+1 in the recurrence:
tn+1 = tn+1-1 + 1
= tn + 1
= n + 1 (as tn = n by our hypothesis)

-Partial fraction decomposition: is a procedure to reduce the degree of either the

numerator or the denominator of a rational function.
f(x) = (x + 2) / (2x2 – x – 1)
Factorize the denominator:
2x2 – 2x + x – 1 = 2x(x – 2) + 1(x – 1)
= (x - 1) (2x + 1)
So we can write the rational functions as:
A/(x – 1) + B/(2x + 1) -------------------------------------------------------------eq 1
(x + 2) / (x – 1) (2x + 1) = A/(x – 1) + B/(2x + 1) ------------------------------eq 2
Multiply both sides by (x – 1) (2x + 1) we get:
(x + 2) = A(2x + 1)+ B(x – 1) ----------------------------------------------------eq 3
To find out the value of A we put (x – 1) = 0 i-e the coefficient of B
So x = 1
Putting x = 1 and (x – 1) = 0 in eq 3:
(1 + 2) = A(2 + 1)+ B(0)
3A = 3 => A = 1
To find out the value of B we put (2x + 1) = 0 which gives x = -1/2
Putting x = -1/2 and (2x + 1) = 0 in eq 3:
(-1/2 + 2) = A(0)+ B(-1/2 – 1)
3/2 = B(-3/2) => B = -1
Putting A = 1 and B = -1 in eq 1 gives

Fall semester 2012 Page 25 of 72

1/(x – 1) -1/(2x + 1)
Therefore
(x + 2) / (2x2 – x – 1) = 1/(x – 1) -1/(2x + 1)

Fall semester 2012 Page 26 of 72

Fall semester 2012 Page 27 of 72
Fall semester 2012 Page 28 of 72
Fall semester 2012 Page 29 of 72
-Merge sort:
-Divide the array into two sub-arrays each with n/2 items.
-Recursively divide each sub-array till the size becomes 1
-Sort and merge the sub-arrays going up the stack to get the original array with sorted
elements.

Fall semester 2012 Page 30 of 72

From the general form of the recurrence T(n) = a T(n/b) + f(n) we get the following
recurrence for merge sort.
T(n) = 2 T(n/2) + O(n)
a = 2 because the merge sort function is called for each of the two halves.
b = 2 because each of the new subproblem has half the elements in the original list.
f(n) = O(n) as combining the results of the subproblems in merge function is O(n)

3 2 8 1
Sort & divide
merge 1 2 3 8

2 3 3 2 8 1 1 8

Sort & divide divide Sort &

merge merge
3 2 8 1

Problem: Sort an array of n keys in ascending order.

Inputs: L = {x1, x2, x3, …, xn} of size n, indexed from 1 to n
Outputs: the array L containing the keys in ascending order.
void mergeSort (int L[], int n)
{
if (n>1)
{
int s1= n/2
int s2 = n – s1;
int L1[] = L{ x1, x2, x3, …, xs1} [copy elements to L1]
int L2[] = L{ xs1+1, x s1+2, x s1+3, …, xn} [copy elements to
L2]
mergeSort(L1, s1);
mergeSort(L2, s2);
merge(L1, s1, L2, s2, L, n);
}
}

Problem: Merge two sorted arrays into one sorted array.

Inputs: positive integers s1 and s2, array of sorted keys L1 indexed from 1 to s1, array
of sorted keys L2 indexed from 1 to s2.
Output: the array L containing the keys in ascending order.
void merge(int L1[], int s1, int L2[], int s2, int L[], int n)
{
int i = 0, j = 0, counter = 0;
for(; i < s1 && j < s2;)
if(L1[i] < L2[j])
L[counter++] = L1[i++];
else

Fall semester 2012 Page 31 of 72

 L[counter++] = L2[j++];

for (; i < s1; i++)

L[counter++] = L1[i];
for (; j < s2; j++)
L[counter++] = L2[j];
}

-Analysis of extra space usage: Merge sort is not an in-place algorithm because the
extra space required increases with the input size. Inside mergeSort()we are creating
two arrays L1 and L2. This extra space is two times the space required for the original
elements because in each recursive call the sum of the size of two arrays is half the
size at the previous level. The following algorithm does much of the manipulation on
the original array and reduces the temporary array size to n.

void merge(int start, int mid, int end)

{
int i = start, j = mid + 1, counter = start;
int tempArr[SIZE];

for(; i <= mid && j <= end;)

if(arr[i] < arr[j])
tempArr[counter++] = arr[i++];
else
tempArr[counter++] = arr[j++];

if(i > mid)

for(int c = j; c <= end; c++)
tempArr[counter++] = arr[c];
else
for(int c = i; c <= mid; c++)
tempArr[counter++] = arr[c];

for(int k = start; k <= end; k++)

arr[k] = tempArr[k];
}

void mergeSort(int start, int end)

{
int mid;

if (start < end)

{
mid = (start + end)/2;
mergeSort(start, mid);
mergeSort(mid + 1, end);
merge(start, mid, end);
}
}

Fall semester 2012 Page 32 of 72

Fall semester 2012 Page 33 of 72
Fall semester 2012 Page 34 of 72
Fall semester 2012 Page 35 of 72
We can check this answer by verifying that it satisfies the recurrence 5g(n-1) – 6g(n-2):
gn=5x2(n-1) – 6x2(n-2)
=5x2(n-1) – 3x2x2(n-2)
=5x2(n-1) – 3x2(n-1)
=2x2(n-1) =2n

Fall semester 2012 Page 36 of 72

Fall semester 2012 Page 37 of 72
-Quick sort:
Worst-case: When all the elements are already in nondecreasing order then the
partition process results in two halves one of which is empty and the other half
contains all the elements greater than the pivot i-e (n-1). The partitioning cost is O(n).
The recurrence is therefore
T(n) = T(n-1) + T(0) + O(n)
T(n-1) is the recursive call of the right half of the subproblem while T(0) is for the
right subproblem which in this case is empty. O(n) is the cost of the partition function
at each level.
The solution for this recurrence is T(n) = [n(n-1)]/2 or O(n2)

Best-case: If the partitioning produce two subarrays of size n/2 and [n/2]-1 then the
recurrence is
T(n) = 2 T(n/2) + O(n)

Fall semester 2012 Page 38 of 72

Fall semester 2012 Page 39 of 72
Fall semester 2012 Page 40 of 72
Week-9.10

A complete binary tree is a binary tree that satisfies the following conditions:

 All internal nodes have two children.

 All leaves have depth d.

An essentially complete binary tree is a binary tree that satisfies the following
conditions:

 It is a complete binary tree down to a depth of d–1.

Fall semester 2012 Page 41 of 72

  The nodes with depth d are as far to the left as possible.

A heap is an essentially complete binary tree such that

 The values stored at the nodes come from an ordered set.

 The value stored at each node is greater than or equal to the values stored at its
children. This is called the heap property.

-Heap sort:

Fall semester 2012 Page 42 of 72

Fall semester 2012 Page 43 of 72
Fall semester 2012 Page 44 of 72
Fall semester 2012 Page 45 of 72
Fall semester 2012 Page 46 of 72
-Comparison of Merge, Quick and Heap sort:
Quick sort extra space usage is less as compared to Merge sort. Heap sort is in-place.
On the average heap sort is worse than quick sort in terms of comparisons and
assignments.

-Decision trees for sorting algorithms: A binary tree can be constructed to represent
all the possible permutations of keys in a list in order to sort them. Such trees are
called decision trees because at each node a decision is taken as to which node to visit
next. Below is an example of a sorting algorithm that sorts three keys:
void sortthree (int n1, int n1, int n3)
{
if (n1 < n2)
{
if (n2 < n3)
cout << n1, n2, n3;
else if (n1 < n3)

Fall semester 2012 Page 47 of 72

 cout << n1, n3, n2;

else
cout << n3, n1, n2;
}
else if (n2 < n3)
{
if (n1 < n3)
cout << n2, n1, n3;
else
cout << n2, n3, n1;
}
else
cout << n3, n2, n1;
}

n1<n2
Yes No

n2<n3 n2<n3

Yes No Yes No

n1,n2,n3 n1<n3 n1<n3 n3,n2,n1

Yes No Yes No

n1,n3,n2 n3,n1,n2 n2,n1,n3 n2,n3,n1

Week-12.13
-Dynamic Programming: It is similar to divide-and-conquer approach i-e a big
problem is divided into smaller problems. But the difference is that in case of
dynamic programming the sub problems are overlapping while in divide-and-conquer
the sub problems are solved separately.
In dynamic programming once a sub-problem is solved its results are saved which can
later on be used to solve other sub problems instead of recomputing them. To use
dynamic programming, the problem must observe the principle of optimality, that
whatever the initial state is, remaining decisions must be optimal with regard the state
following from the first decision. In other word this principle is said to apply if an
optimal solution to an instance of a problem contains optimal solutions to all sub
instances.

Fall semester 2012 Page 48 of 72

An example of dynamic programming: The non-recursive algorithm Algorithm
fibo1.1 uses dynamic programming to save the previous two terms in order to find the
next term.
1
3 V1 V2
9
V5 5 1 2 3
2
3 V3
V4
4
Graph 1

Suppose there is an edge between every vertex in the graph then the possibilities of
paths from each vertex to every other vertex increase to factorial. A subset of all these
path is a set of paths from one vertex to another vertex that passes through all the
other vertices: (n-2)(n-3)…1 = (n-2)!

Fall semester 2012 Page 49 of 72

We can represent a weighted graph containing n vertices by a two dimensional array
W[][] where:
W[i][j] = weight on the edge {if there is an edge from vi to vj}
W[i][j] = ∞ {if there is no edge from vi to vj}
W[i][j] = 0 {if i = j}
If there is an edge from one vertex to the other this means they are adjacent and
therefore such an array is called Adjacency Matrix. Below is the Adjacency Martix
(W) for the given graph 1:
1 2 3 4 5
1 0 1 ∞ 1 5
2 9 0 3 2 ∞
3 ∞ ∞ 0 4 ∞
4 ∞ ∞ 2 0 3
5 3 ∞ ∞ ∞ 0

Suppose we put the shortest paths between any two vertices in a matrix D like this:
1 2 3 4 5
1 0 1 3 1 4
2 9 0 3 2 5
3 10 11 0 4 7
4 6 7 2 0 3
5 3 4 6 4 0

Fall semester 2012 Page 50 of 72

-Floyd’s Algorithm for shortest path: This algorithm computes the array D from
array W and so it is an algorithm for the shortest path problem. Note that it can do
these computions using only one array because the values in kth iteration uses the
values maintained from (k-1)th iteration.
Let Dk[i]1 to n[j] 1 to n represent an array of lengths of the shortest paths from vi to vj
using only vertices in the set {v1, v2, ..., vk} as intermediate vertices and 0 = < k <= n.
So D0[i]1 to n[j] 1 to n will be the adjacency martix for the graph 1 because D0[i][j] is the
lenght of a shortest path from vi to vj that is not allowed to pass through any of the
other vertices. Which is also the weight on the edge between them.
D0 = W and Dn = D
This algorithm uses dynamic programming to determine Dk from D(k-1) using the
following strategy:
1. Establish a recursive property (process) with which we can compute D(k) from
D(k−1).
2. Solve an instance of the problem in a bottom-up fashion by repeating the
process (established in Step 1) for k = 1 to n. This creates the sequence D0, D1,
D2,...,Dn

To get D(k) from D(k−1) we consider two cases:

Case 1: D(k)[i][j] = D(k-1)[i][j]. This means that using only vertices in {v1, v2, …, vk}
there is at least one shortest path from vi to vj that does not use vk.
We can see this in the given graph 1 for D(5)[4][3] = D(4)[4][3] = 2 because even if v5
is included still the shortest path will be {v4, v3}
Case 2: If all the shortest paths from vi to vj do use vk as an intermediate vertice then
any shortest path from vi to vj using only vertices in {v1, v2, ..., vn}can be represented
as:
vi to vk to vj
We can see that vk cannot be an intermediate vertex from vi to vk or from vk to vj.
D(k)[i][j] = D(k-1)[i][k] + D(k-1)[k][j]
For example: D(2)[5][3] = D(1)[5][2] + D(1)[2][3] = 4 + 3
Combining the above two cases:
D(k)[i][j] = minimum(D(k-1)[i][j], D(k-1)[i][k] + D(k-1)[k][j])

Problem: Compute the shortest paths from each vertex in a weighted graph to each of
the other vertices. The weights are nonnegative numbers.

Inputs: A weighted, directed graph and n, the number of vertices in the graph. The
graph is represented by a two-dimensional array W which has both its rows and
columns indexed from 1 to n, where W[i][j] is the weight on the edge from the ith
vertex to the jth vertex.

Outputs: A two-dimensional array D, which has both its rows and columns indexed
from 1 to n, where D[i] [j] is the length of a shortest path from the ith vertex to the jth
vertex.

void floyd (int n, int W[][], int D[][]

{
D = W; // this is optional as the computation can be done
only with W
for (int k = 1; k <= n; k++)

Fall semester 2012 Page 51 of 72

 for (int i = 1; i <= n; i++)
for (int j = 1; j <= n; j++)
D[i][j] = minimum(D[i][j], D[i][k] + D[k][j]);
}

It is O(n3)

Fall semester 2012 Page 52 of 72

Fall semester 2012 Page 53 of 72
Fall semester 2012 Page 54 of 72
Fall semester 2012 Page 55 of 72
The above algorithm computes the length of shortest paths from each vertex to every
other vertex but does not shows the path (i-e the vertices used by the shortest path).
Following is a modified version of the above algorithm which uses another array P of
the same size as W. It initially fills the array P with zeros. It then puts the highest
index of an intermediate vertex on the shortest path from vi to vj. But if there is no
intermediate vertex then its value remains 0.

void Floyd2 (int n, int W[][], int P[][])

{
for(int i = 1; i <= n; i++)
for (int j = 1; j <= n; j++)
P[i] [j] = 0;

for(int k = 1; k <= n; k++)

for(int i = 1; i <= n; i++)
for(int j = 1; j <= n; j++)
if (W[i][k] + W[k][j] < W[i][j])
{
P[i][j] = k;
W[i][j] = W[i[[k] + W[k][j];
}
}

When this algorithm in applied to the graph 1 then P will be:

1 2 3 4 5
1 0 0 4 0 4
2 5 0 0 0 4
3 5 5 0 0 4
4 5 5 0 0 0
5 0 1 4 1 0

Once we have P then we can use the following recursive function to print the shortest
path between vertex q and r:

void path (index q, r)

{
if (P[q][r] ! = 0)
{
path (q, P[q][r]);
cout << "v" < < P[q][r] << ", ";
path (P[q][r], r);
}
}

If P is global then calling this function as path(5, 3) will print V1, V4.

Fall semester 2012 Page 56 of 72

-The traveling salesperson problem: In this problem a shortest route is determined
that starts at node A, visits each of the other nodes once, and ends up at node A. Such
a route/path is called an optimal tour. For example for graph 2, given below, the
optimal tour is tour 3 with length 21.
Because the starting vertex is irrelevant to the length of an optimal tour, we will
consider v1 to be the starting vertex.
2
V1 V2
1
6 3 4 6
7
9
V4 V3
8
Graph 2

Tour 1: [v1, v2, v3, v4, v1] = 22

Tour 2: [v1, v3, v2, v4, v1] = 26
Tour 3: [v1, v3, v4, v2, v1] = 21

If in a graph of n nodes there is an edge from every vertex to every other vertex and
we consider all possible tours, as we did above, then the second vertex on the tour can
be any of (n-1) vertices, the third vertex can be any of (n-2) and so on. The nth vertex
will be only one vertex. This means that the total number of tours is (n-1)! Because
(n-1)(n-2)…1 = (n-1)!
So any algorithm considering all the possible tours will be worse than exponential.

Fall semester 2012 Page 57 of 72

Let us see if this problem observe the principle of optimality. If on an optimal tour vk
is the first vertex following v1 then the subpath of that tour from vk back to v1 must be
a shortest path from vk to v1 that passes through each of the other vertices exactly
once. So as the principle of optimality applies therefore we can use dynamic
programming for this problem.

Below is the Adjacency Martix (W) for the given graph 2:

1 2 3 4
1 0 2 9 ∞
2 1 0 6 4
3 ∞ 7 0 8
4 6 3 ∞ 0

Let V = set of all the vertices, A = a subset of V

and D[vi][A] = length of a shortest path from vi to v1 passing through each vertex in A
exactly once.
So for graph 2 V= {v1, v2, v3, v4} (note that curly braces are used to represent a set
where as square brackets to represent a path)
If A= {v3, v4}then D[v2][A] represents minimum of the paths that starts from v2 and
reaches v1 passing through v3 and v4.
D[v2][{v3, v4}] = minimum(length[v2, v3, v4, v1], length[v2, v4, v3, v1])
= minimum(20, ∞) = 20

The length of an optimal tour = minimum(W[1][j] + D[vj][V-{v1, vj}]) for 2 =< j <= n
W[1][j] means the weight of the edge from vertex v1 to vertex vj.
D[vj][V-{v1, vj}] is the minimum of the paths that starts from vj and reaches v1
passing through all the vertices in V except v1 and vj.
This can be generalize to:
D[vi][A] = minimum(W[i][j] + D[vj][A-{vj}]) where i != 1 and vi is not in A and
whereas vj belongs to A which is a non empty set.
And D[vi][Ø] = W[i][1] (which means a path that starts from vi and reaches v1 and
without passing through any other vertex)

The Dynamic Programming Algorithm for the Traveling Salesperson Problem:

Problem: Determine an optimal tour in a weighted (nonnegative), directed graph.
Inputs: A weighted, directed graph, and n, the number of vertices in the graph. The
graph is represented by a two-dimensional array W, which has both its rows and
columns indexed from 1 to n, where W [i] [j] is the weight on the edge from ith vertex
to the jth vertex.
Outputs: A variable minlength, whose value is the length of an optimal tour, and a
two-dimensional array P from which an optimal tour can be constructed. P has its
rows indexed from 1 to n and its columns indexed by all subsets of V − {v1}. P [i] [A]
is the index of the first vertex after vi on a shortest path from vi to v1 that passes
through all the vertices in A exactly once.
void optimalPath(int n, int W[][], int P[][], int &minlength)
{
int i, j, k;
int D[1 .. n] [subset of V - {v1}];

Fall semester 2012 Page 58 of 72

 for (i = 2; i <= n; i++)
{
// paths starting from vi and terminating on v1 without
passing through any other vertices (i-e a direct edge)
D[i][Ø] = W[i][1];
}

for (k = 1; k <= n - 2; k++)

{
// k <= n-2 because v1 and the current node (i-e the starting
node) are excluded as they cannot pass through themselves
for (all subsets A ⊆ V - {v1} containing k vertices)
{
for (i such that i ≠ 1 and vi is not in A)
{
D[i][A] = minimum(W[i][j] + D[j][A - {vj}]);
//j: vj ∊ A
P[i][A] = value of j that gave the minimum;
}
}
}

D[1][V - {v1}] = minimum (W[1][j] + D[j][V - {v1, vj}]);

2 ≤ j ≤ n
P[1][V - {v1}] = value of j that gave the minimum;
minlength = D[1][V - {v1}];
}

Shortest paths between vertices of graph 2

1 2 3 4
1 0 2 8 12
2 1 0 6 4
3 8 7 0 8
4 4 3 9 0

Table D
Ø v1 v2 v3 v4 v3, v4 v2, v4 v2, v3 v2, v3, v4
1 21
2 1 ∞ 20
3 ∞ 8 14 12
4 6 4 ∞ ∞

Table P
Ø v1 v2 v3 v4 v3, v4 v2, v4 v2, v3 v2, v3, v4
1 3
2 3 2 3
3 2 4 4
4 2 2 2

Fall semester 2012 Page 59 of 72

-Constructing the optimal tour from P: Optimal tour from v1 back to v1 passing
through all the other vertices only once can be obtained from P as follows:
Index of first node (that is next to v1) = P[1][{v2, v3, v4}] = 3
Use the index obtained for the first node as an index for row in P:

Fall semester 2012 Page 60 of 72

Index of second node = P[3][{v2, v4}] = 4
Index of third node = P[4][{v2 }] = 2
So the optimal tour is [v1, v3, v4, v2, v1]

Fall semester 2012 Page 61 of 72

Week-14
- Introduction to greedy approach
o Prim’s Algorithms
o Kruscal’s Algorithm
o Dijkstra’s Algorithm

Fall semester 2012 Page 62 of 72

Fall semester 2012 Page 63 of 72
Fall semester 2012 Page 64 of 72
Week-15
- Introduction to backtracking
o The sum of subset problem
o Graph coloring

Fall semester 2012 Page 65 of 72

Fall semester 2012 Page 66 of 72
Fall semester 2012 Page 67 of 72
-The theory of NP:

Fall semester 2012 Page 68 of 72

Fall semester 2012 Page 69 of 72
Fall semester 2012 Page 70 of 72
Fall semester 2012 Page 71 of 72
Week-16
Revision

Loop invariant
Strategies of algorithms:
1. Greedy
2. Divide & conquer
3. Prune & search
4. Dynamic programming
5. Branch and bound
6. Approximation
7. Heuristics

Total Marks: 100

Recommended Readings:
1. Foundation of Algorithms using C++ Pseudocode Second Edition
By: Richard E. Neapolitan, Kumarass Naimipour
2. Introduction to Algorithms
By: Thomas H. Coremen
3. Websites those contain algorithms and research papers

Fall semester 2012 Page 72 of 72