Asymptotic Notation
Asymptotic Notation
Analysis of Algorithms
Week #1-2
About the course
Course material
Textbook:
Introduction to Algorithms
by Cormen et al.
Lecture slides
05/19/25 18:14 2
Why take this course?
Very basic (especially for CS and SE/CE) and intellectually enlightening
course
Get familiar with typical problems and learn the bounds of power of
algorithms (undecidability and NP-completeness)
05/19/25 18:14 3
Tentative Outline
Introduction
Asymptotic Notation Dynamic programming
Divide and Conquer Greedy algorithms
Paradigm Graph algorithms
Recurrences Undecidability
Solving Recurrences NP-Completeness
Quicksort Practical Evaluation of
Sorting in linear time Algorithms
Binary search trees
Red-Black trees
05/19/25 18:14 4
INTRODUCTION
05/19/25 18:14 5
What is an algorithm?
Sequence of trivial
steps
input output
Algorithm
05/19/25 18:14 6
The statement of the problem defines what is
the relationship between the input and the
output.
05/19/25 18:14 7
An example computational
problem…
Given a function
f : {1,2,..., n}
find a bijection
g : {1,2,..., n} {1,2,..., n}
such that
i {1,2,..., n 1} : f ( g (i )) f ( g (i 1))
This is nothing but a very formal definition of
the sorting problem
(the problem as described above asks for an algorithm that sorts the input numbers
in nondecreasing order)
05/19/25 18:14 8
An example computational
problem…
The problem definition is not always given as formal
as in the previous example:
05/19/25 18:14 9
Sorting example
Given a sequence of numbers as input such as
[ 15, 42, 17, 34, 3, 17 ]
The output should be
[ 3, 15, 17, 17, 34, 42 ]
05/19/25 18:14 11
Not the sorting problem
again !!!
Sorting is a fundamental operation in many
disciplines and it is used as a part of other
algorithms.
A lot of research has been made on the
sorting problem.
A lot of algorithms have been developed.
It is a very simple and interesting problem to
explain basic ideas of algorithm design and
analysis techniques.
05/19/25 18:14 12
Correctness of algorithms
An algorithm is correct if for every instance
of the problem, it halts (terminates) and it
produces the correct answer.
Otherwise (i.e. if there are some instances for which
the algorithm does not halt, or it produces an
incorrect answer), it is called an incorrect algorithm.
05/19/25 18:14 13
Insertion sort
Basic idea: Given a nondecreasing sequence
[a1, a2, …, an] and a number k
the sequence
[a1, a2, … ai, k, ai+1,…, an]
is a nondecreasing sequence if ai ≤ k ≤ ai+1
For example:
[a1, a2, a3, a4, a5] k
05/19/25 18:14 15
An Implementation for
Insertion sort Considers each element
Searches for the
Insertion-Sort(A) { one-by-one. Note that, the
correctisplace
first element to insert
assumed to
for (j=2; j≤n; j=j+1) { form thetheinitial
next sorted
element.
num = A[j]; sequence.
i = j-1;
// find the correct place for num
while (i>0 and A[i]>num) { If it sees that num is
A[i+1] = A[i]; smaller than A[i], it
shifts A[i] one position
i=i-1; to the right
}
A[i+1] = num;
When the correct
} place is found, it will
} be already empty
05/19/25 18:14 16
Is Insertion sort the solution for
the sorting problem?
Insertion sort is only a solution for the sorting
problem.
“But we’ve just proved that it works correctly
for all the input sequences. Why do we need
other algorithms to solve the sorting
problem?”
There may be other algorithms better than
Insertion sort…
05/19/25 18:14 17
What does a “better
algorithm” mean?
A better algorithm uses less resources than the
other algorithms.
Then, just show us the best algorithm known.
We will only be using the best algorithm.
- Time (*)
- Space (i.e. memory)
Not that simple. Using
- Money less resource depends on
- Area
The number of input elements
- Bandwidth
The characteristics of the input
- Energy
- etc.
The resource we are interested in
So, the definition of “best” changes
05/19/25 18:14 18
Selecting the best algorithm
Selecting the best algorithm, first of all, requires to
have multiple algorithms for the solution of the same
problem. We will mainly use the RAM (random
access machine) model, where the
The resource on which our selection
algorithms will be
are implemented made
as single
should be known. threaded computer programs .
And, we must analyze the model,
In RAM available algorithms
statements to
are executed
understand how much oneofbythe
one,type of resource we
sequentially.
05/19/25 18:14 19
Analysis of Insertion sort
05/19/25 18:14 21
Definition of the running
time
We can use a 1990 PC AT computer or a
05/19/25 18:14 22
Definition of the running
time
Our notion of “running time” should be an
abstract notion that is as independent as
possible from such concerns.
05/19/25 18:14 23
Running time of insertion
cost times executed
sort
Insertion-Sort(A) {
for (j=2; j≤n; j=j+1) { c n 1
num = A[j]; c2 n-1
i = j-1; c3 n-1
// find the correct place for num
n
c4 kj
while (i>0 and A[i]>num) { j 2
n
A[i+1] = A[i]; c5 j 2
(k j 1)
i=i-1;
n
c6 (k j 1)
j 2
}
A[i+1] = num; c7 n-1
}
}
05/19/25 18:14 24
Running time of insertion
sort
The total running time can be calculated as:
T (n) c1n c2 (n 1) c3 (n 1) c4 j 2 k j c5 j 2 (k j 1) c6 j 2 (k j 1) c7 (n 1)
n n n
05/19/25 18:14 25
Running time of insertion sort
(best case)
Recall that kj is the number of times that the
“while loop” condition is checked to find the
correct place of a number
Under the best scenario, it will never iterate
for all j, hence kj =1 for all j
This corresponds to the case where the input
is already sorted
In this case n
T (n) a1n a2 j 2 k j a3 a1n a2 j 2 1 a3 (a1 a2 )n a3 a2
n
05/19/25 18:14 26
Running time of insertion sort
(worst case)
Under the worst scenario, the while loop will
iterate the maximum amount of time possible
Therefore, kj = j for all j
T (n) a1n a2 j 2 k j a3
n
a1n a2 j 2 j a3
n
n(n 1)
a1n a2 1 a3
2
b1n 2 b2 n b3
05/19/25 18:14 27
Running time of insertion sort
(average case)
On the average, the while loop will iterate half
of the maximum possible number of times
Therefore, kj = j/2 for all j
T (n) a1n a2 j 2 k j a3
n
j
a1n a2 j 2 a3
n
2
a n(n 1)
a1n 2 1 a3
2 2
d1n 2 d 2 n d 3
05/19/25 18:14 28
Running time of insertion
sort
Best case: T (n) (a1 a2 )n a3 a2
Linear function of n
2
Average case: T (n) b1n b2 n b3
Quadratic function of n
2
Worst case: T ( n ) d1 n d 2n d3
Quadratic function of n
05/19/25 18:14 29
Which running time we
should use?
In order to compare the running time of
algorithms, usually the “worst case running
time” is used, because
It gives an upper bound (it cannot go worse)
Murphy’s law (the worst case happens)
Average case is usually the same as the worst
case.
05/19/25 18:14 30
Asymptotic Analysis
05/19/25 18:14 31
Asymptotic Analysis
Suppose we have two algorithms for sorting
A1 and A2
Let the exact running time of them be
T1 (n) 2n 3 and T2 (n) 50n 2
Assume A1 is executed on a fast machine
(109 instructions per second)
Assume A2 is executed on a slow machine
(106 instructions per second)
Assume we will be sorting 105 numbers
05/19/25 18:14 32
Asymptotic Analysis
T1 (n) 2n 3 and T2 (n) 50n 2
A1 on the fast computer will need
210 instructions
5 3
6
9
2.10 2 million seconds
10 instructions/sec
A2 will run four times faster
A2 on the slow computer will need
5 2
50 10 instructions 4
50.10 0.5 million seconds
6
10 instructions/sec
05/19/25 18:14 33
Asymptotic Analysis
Look at growth of T(n) as n → ∞
Θ-notation:
Ignore lower order terms
Ignore leading constants
Leading
For example: constant
T (n) d1n 2 d 2 n d 3
Lower order
T (n) n 2 terms
05/19/25 18:14 35
Asymptotic Analysis
time
n0 input size
05/19/25 18:14 36
Algorithm Design Techniques
In general, there is no recipe for coming up with an
algorithm for a given problem.
05/19/25 18:14 37
Divide and Conquer
Another such design approach is
Divide and Conquer
05/19/25 18:14 38
Three steps of divide and
conquer
1. Divide: The problem is divided into several
smaller subproblems
2. Conquer: The subproblems are attacked
recursively by the
For example, dividesame
the algorithm. When
the size ofsequence
given the subproblem
of gets
Combine sorted small
subsequences
numbers into smaller together to form the sorted form
enough, the problemof is
sequences. the solved in a
original sequence.
straightforward manner.
3. Combine:
As long asThe solutions
we have to element
more than one the subproblems
to
combined to
sort, keep form
dividing.the
Whensolution of the original
we have a single
element to sort, it is already a sorted sequence.
problem.
05/19/25 18:14 39
Merge Sort
Basic idea: Given two sorted sequences
[a1,a2,…,an] and [b1,b2,…bm]
these two sequences can be merged into a
single sorted sequence efficiently.
For example:
[1,5,7] and [2,3,6]
05/19/25 18:14 41
Execution of the merge sort
3 8 4 1
divide
3 8 4 1
divide
3 8 1 4
merge
1 3 4 8
05/19/25 18:14 42
Execution of the merge sort
3 8 4
3 8 4
Works even when the number
of items is not an exact power
of two 3 8 4
3 8 4
3 4 8
05/19/25 18:14 43
A Merge Sort
Implementation
Lowest index of the subsequence Highest index of the subsequence
Merge-Sort(A,p,r) {
If there are at least two numbers, we still need to
1. if (p < r) { divide. Otherwise do nothing…
05/19/25 18:14 45
Analysis of Divide and Conquer
Algorithms
A recurrence for running time T(n) of a divide and
conquer algorithm is based on the three steps of the
design approach:
Suppose at each step the problem of size n is divided into
a subproblems each of size n/b. Furthermore suppose
dividing takes D(n) time.
05/19/25 18:14 46
Analysis of Divide and Conquer
Algorithms
The recurrence for a divide and conquer
algorithm will then be:
(1) if n c
T (n)
aT (n / b) D(n) C (n) otherwise
if the problem is small
enough,
time we
number spend
required
size a solve
ofof each
to
work required
workforrequired for
constant amount
subproblems
each of dividing
subproblem
subproblem combining
into the solution
recursive definition of the function
time to solve it subproblemsof the subproblems
05/19/25 18:14 47
Analysis of Merge Sort
Note that the pseudo code for Merge-Sort works
correctly when the number of elements is not even.
05/19/25 18:14 48
Merge-Sort(A,p,r) {
Analysis of Merge Sort 1. if (p<r) {
Merge-Sort(A,p,r) { 2. q =
floor((p+r)/2);
1. if (p<r) {
Merge-
2. Divide: q The divide step just computes the
=
3.
Sort(A,p,q);
floor((p+r)/2);
middle of the array, hence 4. it takes a constant
Merge-
3. Merge-
amount of
Sort(A,p,q); time → D(n) = Θ(1)
Sort(A,q+1,r);
5. Merge(A,p,q,r);
Conquer:
4.
Need
Merge- to solve recursively two
Sort(A,q+1,r); }
Merge-Sort(A,p,r) {
5.
subproblems, each of 1.which
Merge(A,p,q,r); } has size n/2.
if (p<r) {
Hence}this step takes 2.2T(n/2) time. q =
} floor((p+r)/2);
Combine: As we have 3.seen earlier, Merge-
merging
two sorted sequences with n elements takes
Sort(A,p,q);
linear time → C(n) = Θ(n) 4. Merge-
Sort(A,q+1,r);
5. Merge(A,p,q,r);
05/19/25 18:14 49
}
Analysis of Merge Sort
05/19/25 18:14 50
Comparison of Insertion and
Merge Sort
Insertion Sort : Θ(n2)
Merge Sort: Θ(n log2 n)
Merge Sort is asymptotically more efficient
than Insertion Sort.
Does this mean we should never use
Insertion Sort?
No. Note that we omitted the constant
coefficients…
05/19/25 18:14 51
Comparison of Insertion and
Merge Sort
time Insertion Sort
Merge Sort
n0 input size
Due to constant coefficients, Insertion Sort may still be
preferable for small input sizes.
Also, Insertion Sort has a good performance on almost
sorted inputs
05/19/25 18:14 52