Order Analysis of Algorithms

07/02/25 Dr.T.Vetriselvi 1
 Sorting problem
• Input: A sequence of n numbers, a1,a2,
…,an
• Output: A permutation (reordering)
(a1’,a2’,…,an’) of the input sequence such
that a1’≤a2’ ≤… ≤an’
– Comment: The number that we wish to sort
are also known as keys

07/02/25 Dr.T.Vetriselvi 2
 Insertion Sort
• Efficient for sorting small numbers

• In place sort: Takes an array A[0..n-1]

(sequence of n elements) and arranges
them in place, so that it is sorted.

07/02/25 Dr.T.Vetriselvi 3
It is always good to start with numbers

5 2 4 6 1 3

j=0
j=5
j=4
j=3
j=2
j=1
1
5
2 2
4
5
1 5
1
4
3 6
1
5
3
4 1
6
3
5 6
3

Invariant property in the loop:

At the start of each iteration of the algorithm, the subarray a[0...j-1]

contains the elements originally in a[0..j-1] but in sorted order
07/02/25 Dr.T.Vetriselvi 4
 Pseudo Code
• Insertion-sort(A)
1. for j=1 to (length(A)-1)
2. do key = A[j]
3. #Insert A[j] into the sorted sequnce A[0...j-1]
4. i=j-1
5. while i>0 and A[i]>key
6. do A[i+1]=A[i]
7. i=i-1
8. A[i+1]=key //as A[i]<=key, so we
place //key on the right side of A[i]

07/02/25 Dr.T.Vetriselvi 5
Loop Invariants and Correctness
of Insertion Sort
• Initialization: Before the first loop starts, j=1. So, A[0]
is an array of single element and so is trivially sorted.

• Maintenance: The outer for loop has its index moving

like j=1,2,…,n-1 (if A has n elements). At the beginning
of the for loop assume that the array is sorted from
A[0..j-1]. The inner while loop of the jth iteration places
A[j] at its correct position. Thus at the end of the jth
iteration, the array is sorted from A[0..j]. Thus, the
invariance is maintained. Then j becomes j+1.
– Also, using the same inductive reasoning the elements are
also the same as in the original array in the locations A[0..j].

07/02/25 Dr.T.Vetriselvi 6
Loop Invariants and Correctness
of Insertion Sort
• Termination: The for loop terminates
when j=n, thus by the previous
observations the array is sorted from
A[0..n-1] and the elements are also the
same as in the original array.

Thus, the algorithm indeed sorts and is thus

correct!

07/02/25 Dr.T.Vetriselvi 7
 Analyzing Algorithms

07/02/25 Dr.T.Vetriselvi 8
 The RAM Model
• A generic one processor Random Access
Machine (RAM) model of computation.
• Instructions are executed sequentially
(and not concurrently)
• We have to use the model so that we do
not go too deep (into the machine
instructions) and yet not abuse the notions
(by say assuming that there exists a
sorting instruction)
07/02/25 Dr.T.Vetriselvi 9
 The RAM Model
• Our model has instructions commonly
found in real computers:
– arithmetic (add, subtract, multiply, divide)
– data movement (load, store, copy)
– control (conditional and unconditional branch,
subroutine call and function)
• Each such instruction takes a constant
time

07/02/25 Dr.T.Vetriselvi 10
 Data types & Storage
• In the RAM model the data types are float and
int.
• Assume the size of each block or word of data is
so that an input of size n can be represented by
word of clog(n) bits, c≥1
• c ≥1, so that each word can hold the value of n.
• c cannot grow arbitrarily, because we cannot
have one word storing huge amount of data and
also which could be operated in constant time.

07/02/25 Dr.T.Vetriselvi 11
 Gray areas in the RAM model
• Is exponentiation a constant time operation? NO
• Is computation of 2n a constant time operation?
Well…
• Many computers have a “shift left” operation by
k positions (in constant time)
• Shift left by 1 position multiplies by 2. So, if I shift
left 2, k times…I obtain 2k in constant time !
– (as long as k is no more than the word length).

07/02/25 Dr.T.Vetriselvi 12
 Some further points on the RAM
Model
• We do not model the memory hierarchy
– No caches, pages etc
– May be necessary for real computers and real
applications. But the discussions are too specialized
and we do use such modeling when required. As they
are very complex and difficult to work with.
– Fortunately, RAM models are excellent predictors!
Still quite challenging. We require knowledge in logic,
inductive reasoning, combinatorics, probability theory,
algebra, and above all observation and intuition!

07/02/25 Dr.T.Vetriselvi 13
 Lets analyze the Insertion sort
• The time taken to sort depends on the fact
that we are sorting how many numbers
• Also, the time to sort may change
depending upon whether the array is
almost sorted (can you see if the array
was sorted we had very little job).
• So, we need to define the meaning of the
input size and running time.
07/02/25 Dr.T.Vetriselvi 14
 Input Size
• Depends on the notion of the problem we are studying.

• Consider sorting of n numbers. The input size is the

cardinal number of the set of the integers we are sorting.

• Consider multiplying two integers. The input size is the

total number of bits required to represent the numbers.

• Sometimes, instead of one numbers we represent the

input by two numbers. E.g. graph algorithms, where the
input size is represented by both the number of edges
(E) and the number of vertices (V)
07/02/25 Dr.T.Vetriselvi 15
 Running Time
• Proportional to the Number of primitive
operations or steps performed.
• Assume, in the pseudo-code a constant
amount of time is required for each line.
• Assume that the ith line requires ci, where
ci is a constant.
• Keep in mind the RAM model which says
that there is no concurrency.
07/02/25 Dr.T.Vetriselvi 16
 Run Time of Insertion Sort
Steps Cost Times
for j=1 to n-1 c1 n
key=A[j] c2 n-1
i=j-1 c3 n-1
n-1
while i>0 and A[i]>key c4 t
j=1
j

n-1
do A[i+1]=A[i] c5  (t
j=1
j  1)

n-1
i=i-1 c6  (t
j=1
j  1)

A[i+1]=key c7 (n-1)

In the RAM model the total time required is the sum of that for each statement:
n-1 n-1 n-1
T(n)=c1n  c2 (n  1)  c3 (n  1)  c4  t j  c5  (t j  1)  c6  (t j  1)  c7 (n-1)
j=1 j=1 j=1
07/02/25 Dr.T.Vetriselvi 17
 Best Case
• If the array is already sorted:
– While loop sees in 1 check that A[i]<key
and so while loop terminates. Thus tj=1
and we have:
n-1 n-1 n-1
T(n)=c1n  c2 (n  1)  c3 (n  1)  c4  1  c5  (1  1)  c6  (1  1)  c7 (n-1)
j=1 j=1 j=1

=(c1  c2  c3  c4  c7 )n  (c2  c3  c4  c7 )

The run time is thus a linear function of n

07/02/25 Dr.T.Vetriselvi 18
 Worst Case: The algorithm cannot run slower!
• If the array is arranged in reverse sorted array:
– While loop requires to perform the comparisons with
A[j-1] to A[0], that is tj=j

07/02/25 Dr.T.Vetriselvi 19
 Recall

07/02/25 Dr.T.Vetriselvi 20
 Average Case
• Instead of an input of a particular type (as in best
case or worst case), all the inputs of the given
size are equally probable in such an analysis.
– E.g. coming back to our insertion sort, if the elements in
the array A[0..j-1] are randomly chosen. We can
assume that half the elements are greater than A[j]
while half are less. On the average, thus tj=j/2. Plugging
this value into T(n) still leaves it quadratic. Thus, in this
case average case is equivalent to a worst case run of
the algorithm.
– Does this always occur? NO. The average case may tilt
towards the best case also.

07/02/25 Dr.T.Vetriselvi 21
DIVIDE AND CONQUER (Recursive Algorithms)

The divide-and-conquer paradigm involves three steps

at each level of the recursion:
Divide
the problem into a number of subproblems.
Conquer the subproblems by solving them recursively.
If the subproblem sizes are small enough, however,
just solve the subproblems in a straightforward
manner. •
Combine the solutions to the subproblems into the
solution for the original problem.
07/02/25 Dr.T.Vetriselvi 22
 Merge sort
• The merge sort algorithm closely follows the
divide-and-conquer paradigm. Intuitively, it
operates as follows.
• Divide: Divide the n-element sequence to be
sorted into two subsequences of n/2 elements
each.
• Conquer: Sort the two subsequences
recursively using merge sort.
• Combine: Merge the two sorted subsequences
to produce the sorted answer.
07/02/25 Dr.T.Vetriselvi 23
• MERGE SORT

07/02/25 Dr.T.Vetriselvi 24
07/02/25 Dr.T.Vetriselvi 25
07/02/25 Dr.T.Vetriselvi 26
07/02/25 Dr.T.Vetriselvi 27
07/02/25 Dr.T.Vetriselvi 28
07/02/25 Dr.T.Vetriselvi 29
 Recurrence

07/02/25 Dr.T.Vetriselvi 30
07/02/25 Dr.T.Vetriselvi 31