AB-

STRACT
These
notes
discuss,
briefly,
topics
listed in
MU syl-
labus of
course
CSC402.

Analy-
sis of
Algo-
rithms
CSC402

ANALYSIS OF ALGORITHMS
Sunil Sitaram Shelke, Prof. Suhas Lawand
 Page |1

MODULE 1:- Introduction

0.0. Algorithm and it’s Analysis:- A Bird’s Eye View Notes

This sub-
• An algorithm is a finite sequence of well-defined instructions to solve a well-defined computational problem.
Section is
• Well-defined:- Means unique interpretation. Having no ambiguity. Ex:- Do x+4, Concatenate strings x and y.
added as a
not well defined:- Add 4 or 5 to x, Get leafy vegetables, If ( x==0) { either sort array A or search an element in A}
preliminary
• Computational Problem:- A solvable problem which can be solved by a series of computations.
note for
• Computation:- Evolving process through a sequence of simple and local steps. (Theory of Computation is all helping
about mathematical theory of defining and working on concept of a computation. So do not worry about this now).
understand
Eq:- Is  a local step? Maybe. May be not. Most of the computer architectures have left shift operation which can
be performed in 1 clock cycle. So, for a 32-bit architecture, doing 13 can be considered as a local step. But
why and how
calculating 3 will still be not considered a local step, as at least 2 clock cycles will be required to do so. So, theoretical
whether or not to consider exponentiation by 2 a local step is entirely up to us.
Eq:- Sorting an array of  elements can almost surely be considered as a non local step. We clearly can analysis is
see lot many comparisons and other operations required to achieve sorted effect. This is so for  in general. done and how
But can we consider sorting an array of  elements a local step? Surely, 5 elements can not be sorted in 1 step.
But then it does not require more than 8 comparisons. If we do not want to pay attention to number of steps re- that may
quired to sort 5 elements then it can well be considered as a local step.
be different
• Basically, given any problem, an algorithm is a sequence of computations which ALWAYS stops with a correct an- from so
swer, for all inputs. But what is a problem?
called practi
cal aspects.
Problem, Description of a Problem
Reasons,
• Example1:- Very informal way:- Consider checking whether a given natural number  is a prime number or
not. Bold sentence is an English description of what is it that we want to solve. why it is
still
• More Formal:- Above problem can be described more formally as below:-
i/p:-  ∈ ℕ o/p:- Yes ,  prime important
No, otherwise is a
Problem can then be defined as i/p – o/p description along with set of all possible values that  can take i.e set of
all possible input instances. So, say, we label above problem as  then problem, more formally, is question
 = {, , , … } , apart from it’s i/p – o/p description. whose ans
• Example2:- Very informal way:- Given  and an  size array  of integers, sort  increasingly. wer is
readily
• More Formal:- First, describe set of all possible inputs i.e input instances. Any particular input instance will be a
sequence  of  integers, for eq. for  = 8 as an example,  = , , ,4, −, −, , . So, mathematically, understood
any such  ∈ ℤ , where ℤ is the set of all integers. ℤ creates a sequence of  integers, by a cartesian product of ℤ
with itself  −  times. once these
details are
Any input instance will be such a sequence. We can get an array of any size  and elements of array can be any
integers. We will say we have solved sorting problem if we can sort any such random array for any value of . So, digested.
we collect all such sequences for all possible values of  ∈ ℕ. Any such sequence will be a particular input at a par-
ticular time. So,
 = { | ∃ ∈ ℕ  ∈ ℤ } …set of input instances
Particular I/P instance :-  … 
O/P:- permutation  of  such that   ≤   ≤   …  

• Why do all this? Later we will be interested mostly in runtime and space complexity of any algorithm which solves a
particular given problem. We will see soon that time complexity surely depends on size of a particular input in-
stance and it’s structure also. So we can write unambiguously about time and space requirements dependent on
instance. Moreover, a mathematical description of an input instance gives us more clear idea of how instance looks
like and how can we go about processing it for an answer.

• Basically, a problem, formally, is a set of all possible input instances. But, how to represent an instance? Does it
even matter if we represent an instance in binary or hexadecimal or some other fancy or cryptic symbol system?

Instance Representation, Size of an Input Instance

• Consider Example1 above of prime checking. An instance of this problem  will be a natural number.
If we represent instance in decimal system then  = {,,4, … }. However, if we choose to represent a
natural number in binary then  = {, , , , , , , , , , , … }.
 Page |2

• Why should we worry about binary representation? We need not, actually, theoretically. But do we really have a
device which understands things in decimal representation? Electronic computers work in binary so they do not
understand decimal. Also, any algorithm which solves prime checking problem will spend time in reading input in-
stance and will read it multiple times, probably, to solve it correctly. Naturally, time taken by algorithm will be de-
pendent on time taken to read input instance, which depends on size of an instance which in turn depends on rep-
resentation of instance. For eg. time to read binary is more than time to read same number in decimal as there are
less spaces required in decimal.

• Consider Example2 above of sorting. An example instance given above  = , , ,4, −, −, ,  can be
written in binary as  =  , , , , , , ,  .. Notice that −, − are repre-
sented in s complement.

• So, time taken by an algorithm will be dependent on size of an input instance and that size depends on how in-
stance is represented. So, which representation should we choose? The answer is that representation will not mat-
ter as long as  different representations are only constantly different from each other. Means, size of an instance
in one representation is only constantly different from size of the same instance in other representation. For eg, it
takes log10  places to represent integer . But requires log   places in binary. log  log10  = log   = .

• We will not discuss technical reasons why above arguments about independence from actual representation. This
is a subject matter of Theory of Computation where these things are discussed in detail to establish theory of
computational complexity. So, size of an input instance is generally, traditionally represented by a parameter .
Notice that this single parameter  talks about total size of instance in 1 single number.

• But wait. Consider problem of adding two matrices × and × , for some  and  i.e adding two matrices having
p rows and q columns. Consider it takes  bits to store a single number. So, total input instance size will be  =
 of both matrices combined together. Generally, we see that number of operations to solve this problem is
given as . So, we generally see it as time requirement of matrix addition algorithm as ,  = . This be-
comes  parameter time equation. This does not look like .

• So why this change? Point is that we maintain that time complexity is a function of input size but when we want get
just an estimate of time requirement then we may relax a bit and calculate time in terms of important parameters
about instance, ignoring everything else. So, as in example above we may relax and write time requirement as
,  as an estimate instead of  as an exact equation. So we are allowed to slightly misuse/abuse con-
cept of size of input instance, as long as we are not abusing/misusing too much. Difference between time
requirement calculation after abusing/misusing/relaxing concept of size must be only constantly different
from exact time requirement.

• Another example of such relaxation is seen in sorting. If one integer requires some constant  bits and we have 
size array of such numbers then total size of an instance is  = , because we have already used  for number
of elements in array. So we should be writing time requirement equation of any sorting algorithm as a function of
size    . But we generally see time equation as  i.e as a function of =number of elements in array.
Why? Assume that we do time analysis of bubble sort and say equation is  =  +  + . Now, if we
2
 ′ 3′
choose to write it as a function of exact input size  then replacing  =  ⁄, we get   = 2 +  + . If you
′ 
see, ratio is in terms of  , which is assumed to be constant. So, if algorithm requires quadratic time then after

relaxing input size, it still requires quadratic time. So, if we are interested in seeing only what kind of function
time function is or growth of time function then ignoring exact size and measuring time in terms of other
important parameters is fine too.

• So, to mention clearly, time requirement/complexity of an algorithm is a function  of input instance
size  or it is , , , . .  if we are being relaxed and interested in estimating time in terms of a few im-
portant parameters of input instance , , , …. . Whatever we choose, it is always better to write down very
clearly which parameters are you using to do time and/or space requirement analysis.

1.1. Performance Analysis of an Algorithm

• If someone asks, what is it that any algorithm requires, apart from input and output, what will you answer? When
someone asks, what do you mean by analysing an algorithm, what will you answer?

• Almost surely, everyone will quote time and space requirement of a proposed algorithm, as analysis of that algo-
rithm. Time and Space are universal resources required by any algorithm whatsoever. So doing time and space
analysis of an algorithm is a necessary activity. But is it true that performance analysis of an algorithm means doing
only time and space analysis?

• In general, NO. Performance analysis of an algorithm means resource requirement analysis of an algorithm, for
every important resource it uses. Below are a few examples of resources other than time and space:-
▪ In Internet routing algorithms, how much network bandwidth is required is one of the important resources to
be considered as a performance analysis exercise.
▪ In parallel computing, a parallel algorithm is run on multiple independent processors/processing units at the
same time, to reduce time, utilizing parallel architecture of system. So, naturally every processor becomes a
resource and analysis of number of processors required simultaneously is an important exercise.
▪ For a machine learning algorithm, more the data, mostly better it’s performance. Not necessarily, if it’s a bad
algorithm. But, having a lot of data is not enough. Interesting and important is to know how much minimal
 Page |3

amount of data is required for ML algorithm to learn automatically. Training data is an important resource.
Training data is called sample. Knowing minimum number of training data required to learn automatically is
an important performance analysis exercise. That is, doing sample complexity analysis of a learning algo-
rithm is another example of analysis other time and space analysis.

• Time and Space are, as said, universal resources required by any algorithm. For this course, we will focus
only on time and space analysis of an algorithm as a performance analysis exercise.

Time and Space Complexity

Time Complexity as measured by Program Execution Time

• For us computer engineering people, every algorithm will be implemented as an electronic computer program.
What if we do time analysis of that program? Program execution time is dependent on the performance of underly-
ing OS, size of executable created by the compiler and most importantly on the quality of high level source code
written by programmer. These factors change from system to system. But then program execution time analysis
gives an estimate near to real run time performance. For situations which are real time, for eq. automated trading in
markets, program execution time analysis is a must because of sensitivity to time.

• A possible drawback of documenting program execution time analysis is that in order to get the trend of time re-
quirement graph the program has to be executed on different instances of different sizes and multiple instances of
same size and characteristic. Such an analysis may be called empirical analysis, as purpose is to generate data
of execution time as a function of input size by choosing input instances carefully and running over several such
instances. We have to do curve fitting exercise/regression exercise on execution time data generated. Also, 2 dif-
ferent algorithms having really different time performance may not be separated out by such an empirical analysis.
It may happen that difference in time performance starts showing out sharply for very large input size which we
might not have considered for doing empirical analysis.

For eq. to emphasize the point, consider an algorithm A whose run-time is say   =   and an algorithm B
with run-time   = . / . Algorithm B is asymptotically slower than algorithm A. But, B is better than A if
2.5
  ≤   i.e   ≤ . After solving, we get B is better than A for  ≤ 6 . That means a slower algo-
10
rithm B is as good or better than a faster algorithm A as long as input size does not cross 6 . For larger input size
i.e ≥ 7 , A is substantially better than B.

Suppose these time functions are hidden from us because we are doing empirical analysis and we have not fit any
curve over our execution time data plot right now. Say we run programs of both B and A for  ≤  ≤ 7 , we will
see that execution time graphs of these 2 programs are not too much different from each other. To see sharp differ-
ence we have to have data points for small and large input sizes. Also, we have to make sure that our empirical
analysis is ok by making sure that systems characteristics like processor load, circuit slow down does not influence
true run time of programs on both systems.
The example is very superficial. But it gives intuition about possibility of us getting mislead by empirical
analysis if we are not careful about choosing input instance sizes, particular input instances having certain struc-
tures, to get true run-time performance,

• Hence, we go towards theoretical run-time analysis also i.e run-time analysis of algorithm. But there we have to
spend additional efforts because in theoretical analysis we really are not talking about physical time i.e we do not
talk in terms of seconds, nano seconds or physical space i.e number of bits or bytes or so. An analogy from physics
may help here.
▪ We have to choose what is meant by time and space and then theoretical unit of time and unit of
space.
▪ Before doing that we have to choose the universe in which we are going to measure time and space.
▪ The universe for us will be a mathematical model of a computer. We have to chalk down compo-
nents of our universe i.e mathematical model of a computer. Such a conceptual universe should not
be too far away from actual universe i.e an actual electronic computer. It should not be too close to
actual electronic computer also. Otherwise, why do theoretical analysis in the first place !!
▪ We have to conceptualise our theoretical model by ignoring enough of things from actual electronic
computer.
▪ Just like we have basic elements in universe, say atoms, we have to choose atoms in our mathemati-
cal model of a computer. This means we have to choose those instructions/operations which can be
considered as simple and very basic, from which complicated operations can be built, just like com-
plex physical objects are made of elementary atoms.

Lets see briefly what those things may be, so that we can start doing time and space complexity analysis of algorithms.

Theoretical Model of a Computer for performance analysis of an Algorithm:- rough overview

Basic Instruction Set:- Set of Basic Steps

• Intel i7 processor has nearly  individual instructions to operate with. In theoretical analysis, we want to
build a model of a computer which is not too far away. But is it really convenient and fruitful to deal with 8
types of basic steps in theoretical analysis? Not really.
• So we say that we have a few basic steps in our model. How many? That number is not that important.
Why? You look at algorithm and decide dynamically, looking at algorithm, which operation you want to call
 Page |4

basic i.e atomic and which not. That way you have instruction set, just like i7. Guidelines are not to ignore
certain operations altogether and always.

• Example basic step set may be {add, subtract, multiply, divide, assignment, comparison, pointer derefer-
encing, reading an arbitrary node, pointer arithmetic, dynamic memory allocation of arbitrary size, freeing
memory, referencing a random array element, reallocating array, dereferencing higher dimensional point-
ers} etc.

• Just like we do not have sorting as a basic instruction in an actual processor, we should not think
of adding sorting arbitrary size array as a basic step in theoretical model too.

Time and Execution time of a basic step

• Consider Intel i7 processor to be of frequency nearly 3 GHz. This means a clock cycle is of . . So, we
can consider .  to be one unit of time for i7. Is it really convenient and insightful to deal with exactly this
scale of time unit? Not really.

• So, in our model we say that we have unit of time as just 1. Not 1 ns, not  , not  . Just . What does
that  stand for? We say that time cost  is added if any basic instruction/step is used by an algorithm.
Simply, we charge  for every execution of every basic step used by an algorithm.

• Now, i7 may require nearly  clock cycles i.e 3 units of time for multiply instruction. That means some basic
instructions in i7 require multiple units. If we want, we can add this feature in our model by saying some-
thing like algorithm costs  ≥  time step cost for add basic step, instead of just . We can say  ≥ 
time step cost for a comparison step, instead of charging just  for its execution . This can be done to show
behaviour that even though basic steps are really atomic, each may not cost same. Cost is time, somehow.

• So, if we have say, for example,  basic steps, theoretically, in our model of a computer, we can
have  ,  , … ,  , all ≥  time costs, all different from each other. All these will potentially show in
time analysis equation. Eventually, you will not like it much, as a student. You will ask whether
tackling all these  constants in time equation has given any good insight.

No matter whether you get bored of such equation, you should not ignore those constants in the first at-
tempt if you want an estimate which is not too bad. In asymptotic analysis, you will have chance to make
one approximation by setting 1 ,  , … , 11 to  i.e assuming that all basic steps have same time cost and
that is . Furthering asymptotic analysis, you will do further approximation by dropping all constants in
equation altogether and just deal with kind of function your time equation is i.e whether it is quadratic equa-
tion, cubic equation, polynomial time equation, more than polynomial, exponential etc.

• If we do not want to do a very tight analysis of time performance then we can ignore time costs of
certain basic steps. This can be done for a few basic steps which are executed rarely or a constant num-
ber of times where of course that constant is not large. In such a case, you can ignore time cost from equa-
tion, to simplify equation. For eq. there is really no noticeable difference between  = 4 +  and  =
4. Additional  in  = 4 +  may be because of an assignment step that has been executed twice.
You can drop that, for convenience.
But of course there is a noticeable difference between  =  +  +  and  =  +  ,
even for certain large values. Dropping a variable term for convenience takes you away from good esti-
mate, if your intention is to get good estimates.
But, in asymptotic analysis, there is no difference between above two equations, as we are inter-
ested in noticing only that both are quadratic equations and hence for larger and very very large values of
, both equations are not much different. Hence asymptotically 2 algorithms corresponding to above 2 time
equations are nearly as good as each other for very very large values of .

• What is the point of above discussion?

Point is that your time equation depends on how much you are ignoring in it. Your willingness to ignore de-
pends on whether you want nearly exact time performance estimate or good enough estimate or you are
doing asymptotic analysis of algorithm’s time performance, where asymptotic analysis means for very very
large values of  i.e analyzing lim .
→∞

Read this entire block again, if you have confusion about time costing.

Finally:- Calculating Time Complexity for an input instance

• For an algorithm, say  , Decide parameters of input instance in which you want to measure time perfor-
mance. i.e you may choose to measure time as a function of input size absolutely or as seen above in case
of sorting, you may not care about exact size but measure time in terms of number of elements in array.
Decide such parameters, say , , , , ….

• Decide basic steps of algorithm. From above subsection, decide time cost of each basic step of algorithm
i.e decide whether you want to charge different time units for different steps or charge  for each execution
of each basic step. Decide that.

• Finally,  , , , …  =   ∗   , where there are say  ,  , … ,  basic steps and   is the
time cost per execution you decided of basic step  and  is the number of times step  is exe-
cuted throughout algorithm execution. You will mostly set   =  for simplifying equation.
 Page |5

• As discussed in previous sub section, once you get time equation, you may approximate that equation de-
pending on whether you are doing asymptotic analysis or not.

Memory i.e Space

Lower Address
0 1 2 …

W …
Random Access Machine memory model

• Electronic computer memory comes as an array of typically 32 or 64 bit size words. We measure on lo-
cation chunk as a word. We can add this feature in theoretical model of a computer, saying one memory
location is of constant size, say,  , for some constant . How large is ? Does not matter as long as it is a
constant. That saves us from fraction calculations.

• Just like von-Neumann stored program-data memory model, we call it as RAM standing for Random Ac-
cess Machine (not random access memory), which stores theoretical algorithm executable in this memory.

• Here comes a bad assumption:- We assume that no matter how large is a particular data element ac-
cessed by a basic step like add, compare etc, it takes only  or a constant time cost to read that element no
matter how large it really is. We basically assume that all the individual data elements which come as a part
of input instance, are constant in size i.e each data elements either fit in  word in RAM requiring  size
and hence require  time step cost to read OR require at most  such locations in RAM requiring at most 
time steps to read a data element, for some constant .

Is this really true? Does  ,  require same amount of time as  , 19 ? NO. Size of
an operand really matters to the time requirement of a basic theoretical instruction also. But lets look at
how much actual read time contributes to time complexity equation.
If in  ,  , data elements require 1 and  words in RAM. Both a and b will require 1and 
time steps to read respectively, assuming  word can be read in  time step. If  ,  is executed 
times then total time step charge for it will 1 +   ×  ×  . Whereas, if we ignore reading time then
time charge will be  ×  . First is accurate, but only constantly different from second. This constant dif-
ference shows only constantly in final run-time equation i.e for example, using first results in cubic equation
then using second also results in a cubic equation.
If we are willing to give up on actual constants then assuming that every data element takes
only  time step every time algorithm reads it. This is not really close to actual performance. But,
again, in this course we will be dealing asymptotic analysis, mostly, where not only constants but
all lower order terms in an equation are also grossly neglected.

•
Page |6
 Page |7

Insertion Sort

1. Strategy Notes

2. Informal Algorithm

3. Initial Algorithm
Swap(x,y)
Insertion Sort (A, n) {
1 For  =  to n x=x+y; y=x-y;
2 i = k − ; // we want to fix A[k] x=x-y;
3 While i >  and Ai > Ai +  // Go backwards starting from current position }
4  + , ; // A[k] will move backwards through swaps, whenever required Swap(x,y)
{
5  =  − ;
x=x^y; y= x^y;
Algo0 Insertion Sort
x= x^y;
}
• Swap is write intensive operation, taking 3 assignment operations. There are at least 3 ways to implement swap.
Swap(x,y)
All require 3 assignments but XOR based swap is fastest, avoiding temporary variable, in terms of number of actual
{
clock cycles required to evaluate rvalue of assignment operation. But independent of actual implementation of
temp=x; x=y;
swap, 3 assignment operations are strictly required to swap.
y= temp;
}
• If  is not at its correct position in  …  then let, correct position of  in  …  is say  ≤  < . Algo-
rithm takes  −  −  swaps and consequently  −  −  assignments to get  at it’s correct position in
 … . Basically, algorithm literally moves element  one place left whenever it finds chance to do so.
Did you notice that
either x or y will be
• Imagine that in a theatre, in a row of  seats, your seat number is say  and everyone from seat number 4 on- A[k] i.e swapping
wards are seated mistakenly to seat numbers  less than their correct seats. You come and you first seat at seat involves element
number . To move to seat number , you are literally swapping seat with people starting with person seating at  A[k] again and
and hence you are kind of sliding towards your actual seat number . Is it really required to do so? How awkward again ?
and frustrating it would be to you and everyone around?

• You can reduce amount of work by asking everyone from seat number  to seat number  (which they have to) to shift
to seat number  more than they are right now. Once they do that, seat number  is vacant and you get there by
making only  move yourself.

• This reduces amount of movements quite much. But you are at seat number  already. So, how can person at
seat number  move to seat number ? This can be done if you get up and stand outside the row.

• Implementing these changes to above algorithm we get the following algorithm.  is playing your role and  … 
are seat numbers.

4. Refined Algorithm After avoiding Swap operations

Insertion Sort (A, n) A Boolean operator
1 For  =  to n is short-circuited
2 currentval= Ak; // Insert A[k] into it’s proper position in A[1…k-1] if second operand
3 i = k − ; is not evaluated, if
4 While i >  and Ai > currentval // Go backwards starting from current position 1st operator decid-
es the result.
5  +  = ; // Shift elements greater than currentval 1 position right, to make place
6  =  − ;
What if and opera-
7  +  = ; // Correct position found…copy currentval to this position
tor in line 4 is not
Algo1 Insertion Sort short-circuited?
 Page |8

Have you already noticed redundant assignment that may happen in some input instance? Spot out the line number and
answer in which input instance redundant assignment operation may occur. Also figure out if you can get rid of that redun-
dant operation without adding other cost to algorithm. If adjustments you suggest introduce other cost to get rid of this re-
dundant operation then adjustment will turn to be useless as overall adjustment may cancel out.

5. Example

6. Correctness

7. Is it a Stable algorithm? YES

s
• For simplicity, without loss of generality, assume that one element of array is duplicated once i.e  =  = ,
for some  ≤  <  ≤ . In line 4, element at index  will be fixed first.

• Coming to  = , algorithm tries to fix  in already sorted array  …  − . Let position of old .   in
 …  −  be index  ≤  < . Due to correctness of the algorithm, sorted position of  will turn out to be  + 
since test condition at line 4 works on > and not on ≥.

• This proves that in sorted order    comes before   . Hence, algorithm is stable.

8. Space and Time Complexity Analysis :- Best and Worst Case

Time Complexity Analysis

Consider the constant cost of each elementary/important operation to be as below:-

 =cost of an assignment/arithmetic operation,  =cost of a comparison

Insertion Sort (A, n) COST

1 For  =  to n   +  +  
2 currentval = Ak;   − 
3 i = k − ;   − 
4 While i >  and Ai > currentval 
   ,  = count of test condition evaluated in ℎ iteration
5  +  = ;   − ,  −  times test condition is TRUE in kth iteration
6  =  − ;   − 
7  +  = ;   − 

CT1.1 Costing of Insertion Sort

As seen in CT1.1, the running time equation of Insertion sort becomes

 

 =   +  +   +   −  +  ∑  +  ∑ −  General

Run Time
 
Function
=  +     +  +   (1.1)
As seen from above time equation,

• Algorithm is sensitive not only to  (which should be true), but also to what instance of size  it is executing on. This is
seen by another variable  . It basically tells, given an instance, to find correct position of  ℎ element of array,
how many steps backwards were taken (and hence how many elements were copied to right by 1 position).

• There are  very important operations. Given  ℎ element, number of comparisons i.e search time and number
of copy to right operations done to find correct position and put it there. We may not see significance of both of
them together in above given algorithm. But, a slight modification of Insertion sort makes these operations clearly
distinct from each other.
• It is true that  ≤  ≤  . Below are the reasons.

• Is it possible that  = , for every  ?. YES. This happens when given instance is already increasingly sorted.
So, only  comparison breaks out of while loop. As a result, no element is copied  place to the right to find correct
position of  ℎ elements. Putting  =  in Equation(1.1), we get
 Page |9

 =  +   ∑  +  +   = 4 +   −  +   Best
Case


(1.2)
=  −  , for constants  = 4 +  and  =  + 

 ∈ Θ
Number of copy to right operations = 
As seen above,  becomes a linear equation in . Hence  ∈ Θ. But this situation being a best case,
time spent is minimum. Hence this is the minimum time requirement for the given algorithm, given any in-
put instance. Hence,
 ∈ .

• Max value for  ? It is  =  . This happens when given instance is decreasingly sorted. So, every given  ℎ ele-
ment finds  position of array as its correct position. As a result,  −  elements are copied  place to the right
to find correct position of  ℎ element. Putting  =  in Equation(1.1), we get


 =  +   ∑  +  +   =  +   +  +   −  +   Worst
Case


(1.3)
=  +  −  , for constants  =  +  and  =  +  and  =  + 

 ∈ Θ 

1
Number of copy to right operations = 
 −  = = 


• As seen above,  becomes a quadratic equation in . Hence  ∈ Θ . But this situation being a
worst case, time spent is maximum. Hence this is the maximum time requirement for the given algorithm,
given any input instance. Hence
 ∈  

Space Complexity Analysis

• As seen from the algorithm,  extra memory places are required to maintain algorithm’s control variables and main-
taining current  ℎ element in currentval. Assuming, every data element requires a constant amount of
memory, say ≤  number of bytes, for some constant , algorithms requires ≤  ∈ Θ bytes, independent
of size of the input i.e  and instance characteristic (no matter whether it is best case i/p or worst case i/p).
• Hence,  ∈ Ο , or most appropriately  ∈ Θ (although  is not mathematically wrong)

9. Average case Time Complexity:- An Attempt

10. Input Instance Characteristic and Run time – Relation between Inversions and Run
time
In an array A,
• What we have analysed till now is time complexity behaviour of algorithm ALGO1 from best, worst and average a pair of indi-
case perspective. We have figured out lower and upper bounds independent of any given input instance. Given an Ces (i,j) is an
input instance  , where of course | | = , we know | | =  ∈ Ω  | | =  ∈  . These up- Inversion if
i<j and A[i]>A[j]
per and lower bounds are different functions. So there is no way we can bound time complexity function by Θ oper-
ator. But we know that algorithm is sensitive to input permutation i.e it takes time depending on how good/bad per-
mutation input is.
If every pair
(i,j) is an inversion
• It is natural to ask whether we can find time function | | =  for algorithm, given input instance  , such that
then there can be
| | ∈ . If so, we will know around exact time for a given input instance, rather than knowing just upper maximum n(n-1)/2
and lower bounds. inversions, since
there are at most
• We already know that given  ℎ element  , search time and hence copy to right operations depend on how n(n-1)/2 pairs (i, j)
many elements in sorted array  …  −  are greater than  and that decides time complexity function.

• In an array  of size , a pair of indices ,  is called as an inversion, if  <    >  i.e higher ele- Which instances
ment is at lower index. Let  ,  be the count of inversions in given input instance  . have 0 or
 − 
 ≤  ,  ≤ n(n-1)/2

inversions?
 P a g e | 10

• In ALGO1, given input instance  , for fixing element  while loop of lines 4 to 6 runs exactly as many times as
many elements are greater than  in array  …  −  i.e depends on how many indices  …  −  are in in-
version with index  . Summing inversions over all values of , we get  , .

• Also, line 1 executes  −  times. Adding this and above point, we get | | ∈ Θ  −  +  ,  .

• Above time equation gives tight bound on run time, given an input instance. If  is best case input instance charac-
terized by  ,  =  we get  ∈ Θ −  +  i.e  ∈ Θ. If  is worst case input instance charac-
1 1
terized by  ,  = 
, we get  ∈ Θ  −  + 
 i.e  ∈ Θ .

• So, in general, given an input instance  with | | = , we have

 = Θ +  ,  (1.4)

• We can alternatively write down time complexity equation as a -parameter equation

,  ,   = Θ +  ,  (1.4)

Input  Number of Com- Copy to Example In- How many Instances?
size=  parisons(among Right Ops stance Charac-
array elements)
teristic
BEST Ω −  Increasingly Exactly  instance requires
Case Sorted Maximum time. But >  in-
stances have time function ∈
Ω. For eg. increasingly
sorted but just first  ele-
ments swapped.
WORST Ο    −   Decreasingly Exactly  instance requires
( )= ( )
Case    Sorted Maximum time. But >  in-
 −  stances have time function ∈
=
 Ο. For eg. same as in best
case explanation.
In- Θ +  ,   −  +  ,   ,  --- ---
stance


11. Speeding up Search time for finding correct position…possibilities

Given sorted
Algo1 fixes correct position of  ℎ element, in an already sorted array  …  −  using linear search backwards. array B of size n,
Since, linear search requires Ο −  comparisons, we can speed up search in the worst case using binary search on and a search key
say x, binary search
 …  −  which takes Οlog  −  comparisons.
requires O(log n)
comparisons.
Insertion Sort (A, n)
1 For  =  to n
2  = , , , ; //Find correct position of A[k] using binary search on sorted A[1…k]
3 if(  ≠  ) // Need to work only if  is not ’s correct position already
4 currentval = Ak; // Insert A[k] into it’s proper position in A[1…k-1]
5 i = k − ;
6 While i ≥ currentpos // Go backwards starting from current position
7  +  = ; // Shift elements higher than currentval to 1 position right, to make place
8  =  − ;
9  +  = ; // Correct position found…copy currentval to this position
Algo2 Insertion Sort with Binary Search

• How would binary search affect number of comparisons? Will there be any change in asymptotic time complexity of
insertion sort using binary search?
• Consider fixing . ALGO1 requires worst case Ο comparisons due to linear search. Above algorithm would O(log n!)=O(n log n)
require worst case Οlog  comparisons to find correct position of .
• Summing search time in the worst case of line 2 in above algorithm, over  ≤  ≤ , we get worst case search
comparisons  Οlog  = Οlog ! = Ο log .
• Hence, it is sure that ALGO3 is better than ALGO1 in terms of number of comparisons in worst case.
• But what about copy 1 place to right assignment operations? That remains same in both the algorithms no matter
whether it is the best case input instance or worst case input instance. So, in worst case run time is still Ο  due
1
to number of inversions. So asymptotic time complexity of ALGO3 is same as  = Ο .

 P a g e | 11

• The worst case for ALGO1 remains worst case for ALGO2 and worst case asymptotic run time is hence
 =  .
• What about best case of ALGO1 given as input to ALGO2? This time increasingly sorted array input is not best
case input for ALGO2. Lets see why.
To find correct position of  in  …  − , ALGO1 requires  comparison in best case because  −  <
 so linear search terminates with 1 comparison. But ALGO2 requires log −  comparisons before finding out
that  −  < . This is so because binary search starts from middle index always. So in case input is already
increasingly sorted then for ALGO2 Θ log −  = log! = Θ log   comparisons and hence  =
Θ log . Hence, increasingly sorted input DOES NOT result into same asymptotic time complexity for
binary insertion sort.

• So what is the best case time complexity of binary insertion sort? There are 2 components. Search time and num-
ber of inversions. Best case instance should be the one which minimizes both together. But there is a slight prob-
lem here. To search correct position of  ,when will comparisons be reduced to minimum i,e 1 or 2 or some con-
11
stant? This happens if  [  ] <  i.e correct position is almost the middle index. So search terminates with
Θ comparisons. But this means nearly half the elements need to move 1 place to right because they are greater

than  i.e there are nearly  copy to 1 place right assignments. Summing over all  ≤  ≤  we get

Θ (   + Θ () ) = Θ −  + Θ  = Θ . That’s as bad as worst case asymptotically. This happens be-
cause of large number of inversion present in input.

• So lets see when can we have minimized inversions. This happens only when input is already sorted. But we have
already seen that run time in this case is  = Θ log .

• So clearly we can not have best case input which minimizes both, number of comparisons and number of inver-
sions. Why? If an input has minimum i.e Θ comparisons (not necessarily 1 or 2) resulting out of binary search. That

means correct position of  is nearly middle position of  …  − . This means, as seen above, nearly  shifts
are required, resulting in total run time of Θ . And if an input has minimum i.e 0 or Θ inversions then it means
correct position of  is near to the end of  …  − . As a result, binary search would require Οlog  −  i.e
high comparisons to reach to such an index. Clearly, we can not have any input array where we have Θ compar-
isons and Θ inversions simultaneously.

• Considering above discussion, we can figure out that best case running time of ALGO2 belongs to Θ log .
We can summarize ALGO1 and ALGO2 behaviour in below table,

Best Case Worst Case Minimum Maximum Min number Max number Best case Worst
Asymptotic Asymptotic Number of Number of of shift right of shift right input in- case in-
Run time Run time Compari- Compari- operations operations stance put in-
sons in Best sons in Best stance
case case
Linear Ω Ο  −  −    −  Increasingly Decreas-
search  Happens in  sorted array ingly
Inser- Happens in Happens in best case i/p Happens in sorted ar-
tion best case i/p worst case i/p worst case i/p ray
Sort

Binary Ωn log  Ο  −  log   Such case is Increasingly Decreas-
Insertio Such an i/p is Such an i/p is Such case is worst case i/p sorted array ingly
sort worst case i/p best case i/p best case i/p sorted ar-
ray

In a nutshell, we can say that as input instances are nearly as worst then ALGO2 is definitely better (not asymptotically
better) than ALGO1 since number of comparisons are bound to Ο log  in ALGO2. Also, if in certain situations cost of a
comparison is greater than cost of a copy right assignment then ALGO2 has an upper hand. This may happen when array
elements are not Θ size, as assumed throughout this whole document. If elements are of variable size then a single
comparison itself take time proportional to element size. In such a case, comparisons should be avoided as much as pos-
sible. But it all depends on how variable size element collection is handled either as an array or using some other data
structure.

12. Block Copy to Right Assignments

As seen in either ALGO1 or ALGO2, given , elements in  …  −  greater than  are copied  place to right
individually. We can rewrite these copy right assignments as a block write operations, as given below.

Insertion Sort (A, n)

1 For  =  to n
2 currentval= Ak;
3 i = k − ;
4 While i >  and Ai > currentval //counting number of elements greater than 
6  =  − ;
7  =  + ;
8 While  <  // copy to right greater elements
9  +  = ;
 P a g e | 12

10  +  = ;
Algo3 Insertion Sort with Block copy right

• Is there any difference in asymptotic time complexity or even in exact number of steps required compared to
ALGO1? NO. As an algorithm, ALGO3 is exactly as efficient as ALGO1.
• ALGO3 turns out to be slightly more efficient as a computer program than ALGO1 implemented as a program. This
happens only if programming language has efficient low level functions to do block memory copy. If such a facility
is available then while loop of line 8 and 9 can be converted to a single efficient programming language statement.
• Example:- memmove() function in C/C++

13. Solved Exercises

1 Consider sorting an  size array of strings where length of each string is Θ, for some fixed, unknown and very
large  ∈ ℕ i.e ∀1 || ∈ , where || means size of element .
Derive time and space complexity behaviour of Insertion Sort algorithm (Algo1) for this array.

Insertion Sort (A, n) COST

1 For  =  to n   +  +  
2 currentval = Ak;   − 
3 i = k − ;   − 
4 While i >  and  >      
5  +  = ;   − 
6  =  − ;   − 
7  +  = ;   − 

ANSWER:-

• As in Equation(1.1), analysis goes same way. Difference lies in costing of line 4, as shown above.
• Since array elements don’t have constant size, a comparison of array elements is not Θ, as assumed pre-
viously in Algo1. Even if  may be fixed, if it is too large then comparing two  sized elements can not be con-
sidered to be Θ time operation.
• Hence, a comparison of two Θ size elements takes Θ time. This has been reflected in cost of line 4
above.

 =   +  +   +   −  +     +   − 


=   +   ∑  +  +  



• We can use Best and Worst case analysis done to Equation 1.1 as it is with cost of line 4 modified as given.


 =   +   ∑  +  +   = 4 +   −  +   Best
Case


=  −  , for constant  =  +  and after dropping constant 4 and
recognizing that   ∈ 

 ∈ Ω


 =   +   ∑  +  +   Worst

Case

=  +   +  +   +  −  +  
=  +  −  , for constant  =  +  and after recognizing that
 +  Θ ∈ Θ and  +  Θ +  ∈ Θ

 ∈ Ο 

• above algorithm.

Thtrhgrrt
P a g e | 13