Mathematical Background

• Set concepts and notations

• Proof by induction (induction proofs)

• Proof by contradiction

• Summations
– Formulas for sums and products of series

• Recursion analysis

• logarithms
1
 Definition (Algorithm)
• What exactly is an algorithm?

– An algorithm is a clearly specified set of simple instructions to be

followed to solve a problem

– Any well-defined computational procedure that takes some value (or

set of values called input) as an input and produces some value (or set
of values called output) as an output

– A sequence of computational steps that transforms the input into the

output

– A set of rules (well-defined, finite) used for calculation or problem

solving (for example with computer)

2
 Definition (Algorithm)
• What exactly is an algorithm?

– A finite set of instructions that, if followed, accomplishes a

particular task

– Is a precise, systematic method for producing a specified

result

– Is a tool for solving a well-defined computational problem

• The statement of the problem specifies (in general terms)
the desired input/output relationships
• The algorithm describes a specific computational
procedure for achieving the input/output relationships
3
 Example Algorithm (1)
• The sorting problem

– Statement of the sorting problem

• Input: a sequence of n number <a1, a2, …,an>
• Output: a permutation (reordering) <a1', a2',
…,an'> such that a1'≤ a2' ≤ … ≤ an '.

– That is, given an input sequence such as <31, 41, 59,

26, 41, 58> a sorting algorithm returns as an output
the sequence <26, 31, 41, 41, 58, 59>
4
 Example Algorithm (1)
• A sorting algorithm (insertion sort)

INSERTION-SORT(A)
1. for j = 2 to length[A]
2. do key ← A[j]
3. //insert A[j] to sorted sequence A[1..j-1]
4. i ← j-1
5. while i >0 and A[i]>key
6. do A[i+1] ← A[i] //move A[i] one position
right
7. i ← i-1
8. A[i+1] ← key
5
 Example Algorithm (2)
• Finding the maximum element problem

– Input: an array A storing n integers

– Output: the maximum element in A

– That is, given array A=[31, 41, 26, 41, 58], the maximum
algorithm returns 58 which is the maximum element in the
array

6
 Example Algorithm (2)
• An algorithm for finding the maximum element

7
 Example Algorithm (3)
• Finding the greatest common devisor problem of
two natural numbers
– Problem statement
• Determine the greatest common devisor of two natural
numbers x and y (where x < y )
– Input
• Two natural numbers a and b
– Output
• The largest natural number d which is the common
devisor of a and b

8
 Example Algorithm (3)
• Algorithm for finding the greatest common
devisor of two natural numbers problem

1. Divide y by x with remainder r

2. Replace y by x, and x by r
3. Repeat step 1 until r is zero

– When this algorithm terminates, y is the highest

common factor of

9
Example Algorithm (3)

• GCD(33,21)=3
• 33 = 1*21 + 12
• 21 = 1*12 + 9
• 12 = 1*9 + 3
• 9 = 3*3

10
 Why learn about algorithms?
• Three main reasons
1. We must create algorithms so that other people or
computers can help us achieve our goals

2. We will find ourselves following algorithms created by

others

3. Include the third reason

11
 Purpose Algorithm in programs
• To accept input
– There are zero or more quantities which are externally
supplied

• To change values hold by data structures

– It is through an algorithm that you change values

• To produce an output
– At least one quantity is produced

• To reorganize or reshuffle the data structure

– Sorting
– Deleting an item from the data structure
– Adding an item into the data structure
12
 Properties of Algorithms
• Every algorithm possesses the following properties
– I/O
– Finiteness
– Definiteness
– Effectiveness
– Correctness
– Efficiency
– Sequence (sequential)
– simplicity
– Language Independence
– Completeness

13
 Essential Properties of Algorithms

• Input specified

– The inputs are the data that will be transformed during the
computation to produce the output

– Every algorithm should have a specified number (zero or

more) input values (or quantities) which are externally
supplied

– We must specify
• the type of data
• The amount of data
14
 Essential Properties of Algorithms
• Output specified

– The outputs are the data resulting from the computation (the
intended result)

– Every algorithm should have one or more output procedures

– A possible output for some computations is a statement that

there can be no output, i.e., no solution is possible, i.e., at
least one quantity is produced

15
 Essential Properties of Algorithms
• Finiteness (It Must Terminate)
– An algorithm must have the property of finiteness
– That is, every valid algorithm should have finite number of
steps
– It must complete after a finite number of steps
– It must eventually stop either with the right output or with a
statement that no solution is possible
– If you trace out the instructions of an algorithm, then for all
cases the algorithm must terminate after a finite number of
steps
– Finiteness is an issue for computer algorithms because if the
algorithm doesn’t specify when to stop (computer algorithms
often repeat instructions), the computer will continue to
repeat the instructions for ever
16
 Essential Properties of Algorithms
• Definiteness (Absence of Ambiguity)

– Definiteness means specifying the sequence of operations

for transforming the inputs into the outputs

– Algorithms must specify every step

– Every step in the algorithm has to be clearly defined

– Each step must be clearly defined, having one and only one
interpretation

– Every detail of each step must be spelled out, including how

to handle errors 17
 Essential Properties of Algorithms
• Definiteness (Absence of Ambiguity)

– At each point in computation, one should be able to tell

exactly what happens next

– It must be composed of concrete steps

– There can be no ambiguity as to which step will be

performed

– Definiteness ensures that if the algorithm is performed at

different times or by different agents (people or
computers) using the same data, the output will be the
same
18
 Essential Properties of Algorithms
• Effectiveness (Feasibility)

– It must be possible to perform each step exactly and in a

finite amount of time.

– It is not enough that each operation be definite as in

definiteness, but it must also be feasible

– It must be possible to perform each instruction

– Whereas definiteness specifies which operations to do,

effectiveness means that they are doable

19
 Essential Properties of Algorithms
• Correctness

– An algorithm must compute correct answer for

all possible legal inputs

– Is the most important criteria for evaluating

algorithms

– If the algorithm is not doing what it is supposed

to do, it is worthless
20
 Essential Properties of Algorithms
• Efficiency (focus of this chapter)

– An algorithms should make efficient use of the computer

resources

– In particular, we would like an algorithm to run as fast as

possible using as little memory as possible

– An algorithm must solve a problem with the least amount

of computational resources such as time and space.

– To keep things simple, we will concentrate on the running

time of algorithms and will not look at the space (the
amount of memory needed)
21
 Essential Properties of Algorithms
• Sequential

– Every algorithm should have a beginning (start step) and a

halt (end) step

– Between the two every step should halve preceding and

succeeding steps

– That is, Each step must have a unique defined preceding

and succeeding step

– The first step (start step) and last step (halt step) must be
clearly noted
22
 Essential Properties of Algorithms
• Simplicity

– We would like our algorithms to be easy to understand

and easy to implement

– There often (but not always) a trade of between efficiency

and simplicity

– More efficient algorithms tend to be more complicated

– It depends on particular task which of the two criteria is

more important

23
 Essential Properties of Algorithms
• Language Independence

– It must not depend on any one programming language

– Completeness

– must solve the problem completely

• Any process having these properties will be

called an algorithm
24
 Languages in algorithms
• Because algorithms are developed by people, but executed by
some other agent, they must be written in some language

• The most important requirement of the language is that the

person who creates the instructions and the agent that performs
them interpret the instructions the same way

• An algorithm can be described in many ways

– Natural language
– Programming language
– Pseducode

25
 Models of Computation
A model of computation specifies the primitive
operations a computer is assumed to support.

•The Random Access Machine (RAM)

•Arithmetic operations: +, –, *, /, √, x , x
•Logical operations: and, or, not
•Array indexing: A[x], where x is a variable
•if-then-else
•while-loops
•for-loops
•procedure calls
26
 Analysis of algorithms
• What does it mean by analyzing algorithms?

– It means predicting the resources that the algorithm requires

• Occasionally, resources such as memory, communication,

bandwidth etc are primary concern in analyzing resources that
an algorithm requires

• Most often, computational time is the primary concern in

analyzing resources

27
 Analysis of algorithms
• The objective of algorithm analysis is
– to determine how quickly an algorithm executes in
practice
– To measure either time or space requirements of an
algorithm

• What to measure
– Space utilization- amount of memory required
– Time efficiency- amount of time required to process the
data

28
 Analysis of algorithms
• What to analyze
– The most important recourse to analyze is generally the
running time
– Several factors affect the running time of an a program
• Compiler used
• Computer used
• The algorithm used
• The input to the algorithm
– The first two are beyond the scope of theoretical model
– Although thy are important, we will not deal with them in the
course
– The last two are the main factors that we deal
– Typically, the size of the input is an important consideration

29
 Analysis of algorithms
• Space utilization and Time efficiency depends on
many factors
– Size of input
– Speed of machine (computer used)
– Quality of source code (used algorithm)
– Quality of compiler

• Focuses on measuring the running time of

an algorithm

30
 Running Time of Algorithm
• The running time of an algorithm depends on a number of
factors

• The running time of an algorithm is affected by the factors

mentioned in the previous slide

• But, for most algorithm, the running time depends on the input
– An already sorted sequence is easier to sort

• for most algorithm, the running time depends on size of the

input
– Short sequences are easier to sort than long ones
– That is the running time of most algorithms varies with the input and
typically grows with the input size

31
 Running Time of Algorithm
• Generally, we seek an upper bound on the running time,
because everybody likes a guarantee

• We can measure the running time of an algorithm in two ways

– Empirically

– Theoretically

32
 Empirical Analysis
• What you should do

– Write a program that implements the algorithm

– Run the program with the data sets (the inputs) of varying
size and composition

– Use the method like clock() ( or

System.CurrentTime.Millis() ) to get an accurate measure
of the running time

– Plot the results

33
 Empirical Analysis

• The resulting data set 9000

should look something 8000

7000
like this 6000

Time (ms)
5000

4000

3000

2000

1000

0
0 50 100
Input Size

34
 Empirical Analysis
• Is an experimental study

• The result comes from the running time of the program

• It uses system time

35
 Limitations of Empirical Analysis
• It can’t be used in estimating the efficiency of algorithms
because the result varies due to variations in

– Processor speed

– Input size

– Current processor load

– SW environment

• So, in order to compare two algorithms, the same

hardware and software environment must be used
36
 Limitations of Empirical Analysis
• Involves implementing the algorithm and testing it on
various instances

– It is necessary to implement and test the algorithm in order

to determine its running time

• But, implementing the algorithm may be difficult

• This is not the case for theoretical analysis

37
 Limitations of Empirical Analysis
• The difficulty to know which instances to test it on

• Experiments can be done only on a limited set of inputs, and

may not be indicative of the running time on other inputs not
included in the experiment

• That is, results may not be indicative of the running time on

other inputs not included in the experiment

38
 Exercise
• To be included

39
 Theoretical Analysis
• Is in contrast to the “experimental approach”

• Allows us to evaluate the speed (efficiency) of an algorithm in a

way that is independent from the HW and SW environment

• Is a general methodology for analyzing the running time of an

algorithm

• Uses a high-level description of the algorithm instead of testing

one of its implementation

• Takes into account all possible inputs

40
 Theoretical Analysis
• Takes an algorithm and produces a function T(n) which
depends on the number of operations

• That is, running time is expressed as T(n) for some function T

on input size n

• The output comes from the quantity of resources using

mathematical concepts

• Does not use system time

41
 Theoretical Analysis
• Rather it uses the number of operations (which are expressed in
time units) because the number of operations do not vary with

– Processor speed
– Input size
– Current processor load
– SW environment

• Theoretical analysis allows us to separate the effect of these

factors

42
 Theoretical Analysis
• What will be the number of operations for the algorithm
below using a computer having
– 100 MHZ and 750 MHZ
– Using DOS OS
– Using UNIX OS
– While printing, browsing

for(i=1; i <= 100; i++)

cout << i;

43
 Theoretical Analysis
• Is an approach that can be used to measure or estimate the
complexity of an algorithm as the approach do not vary due to
variations in the computer systems

• The only factor that affects measure of complexity is the input

size of the algorithm

• To avoid this, measure complexity of algorithm for arbitrary

number n as n Æ ∞

44
 How do measure complexity of algorithms
• Two steps or phases for this

– Analysis of algorithm (i.e., theoretical analysis)

– Order of magnitude

45
 How do measure complexity of algorithms
• Step one - analysis of algorithm

– At this stage we should have an algorithm or select an

algorithm

– By analyzing this algorithm we end up with a function T(n),

which is the computing time of an algorithm for input size n

– T(n)= the number of operations of time units

– This first step is used to measure the complexity of the

algorithm
46
 Calculating complexity of an algorithm
• Step one - analysis of algorithm

– In analyzing an algorithm written in psedocode to determine

its complexity, determine the number of operations on the
underlying model of computation

– To obtain an estimate of this number, we count primitive

operations, chosen to reflect the complexity of the total cost
of all operations

47
 Calculating complexity of an algorithm
• Step one - analysis of algorithm

– Primitive operations

• Basic computations performed by an algorithm

• Identifiable in pseudocode

• Largely independent from the programming language

• Exact definition not important

48
 Primitive or Basic Operations
• Step one - analysis of algorithm
– Examples of basic operations could be
• An assignment (Assigning a value to a variable)
• Calling a method
• Returning from a method
• Evaluating an expression
– An arithmetic operation between two variable (e.g.,
Performing an arithmetic operations such as addition)
• A Comparison between to variables (two numbers)
• Indexing into an array

49
Primitive or Basic Operations

50
 Running time- T(n)
• We measure execution (running) time in terms of any of the
following
– Arithmetic operations
– Assignment operations
– Loop iterations
– Comparisons
– Procedure calls (calling a method)
– Returning from a method

51
 Examples
• Include examples on computing running
time- T(n)

52
 Order of Magnitude
• Refers to the rate at which the storage or time grows as a function
of problem size

• Input: T(n), the timing function

• Output: order of T(n) written as O(T(n))

• The output of helps to determine the category to which the

algorithm belongs to some known functions (functions whose
complexity is known through research and experiment)

– See next slide for a list of the known functions

• Thus, the output is expressed in terms of its relationship to some

known functions 53
 Order of Magnitude
• Include list of functions whose complexity is known through
research and experiment

• Include also their graph

• Arrange such known functions by their order of complexity

54
 Functions whose complexity is known through
research and experiment
• Constant function
– T(n) ε O(1) : Great. This means your algorithm takes only
constant time. You can’t beat this
• loglogn
– T(n) ε O(loglogn): super fast! For all intents this is as fast as
a constant time
• Logarithmic time
– T(n) ε O(logn): Very good. This is called logarithmic time.
This is what we look for most data structures. Nite that
log1000 ≈ 10 and log 1, 000, 000 ≈ 20 (log’s base 2)
• Polylogarithmic time
– T(n) ε O((logn)k):(where k is a constant ). This is called
polylogarithmic time. Not bad, when simple logarithmic is55not
achievable
 Functions whose complexity is known through
research and experiment
• Linear time
– T(n) ε O(n) :this is called linear time. It is about the best that
one can hope for if your algorithm has to look at all the data.
– In data structure the game is usually to avoid this through
• nlogn
– T(n) ε O(nlogn) : This one is famous, because this is the time
needed to sort a list of numbers. It arises in a number of other
problems as well
• Quadratic time
– T(n) ε O(n2) : Okay if n is in the thousands, but rough when n
gets into the millions

56
 Functions whose complexity is known through
research and experiment

• Polynomial time
– T(n) ε O(n2) : (where k is a constant). This is called
polynomial time. Practical if k is not too large
• Exponential time
– T(n) ε O(2n) , T(n) ε O(nn) , T(n) ε O(n!) : algorithms taking
this much time are only practical for smallest values of n
(e.g., n ≤ 10 or maybe n ≤ 20)

57
 Order of Magnitude
• This type of analysis is called asymptotic analysis

• For large enough inputs, the multiplicative constants and lower-

order terms of an exact running time are dominated by the
effects of the input size itself

• When we look at input sizes large enough to make only the order
of growth of the running time relevant, we are studying
asymptotic efficiency of algorithms

• That is, we are concerned with how the running time of an

algorithm increases with the size of the input in the limit, as the
size of the input increases without bound

• Usually, an algorithm that is asymptotically more efficient will

be the best choice for all but very small inputs
58
More examples on computing T(n) and O(T(n))
• Include such examples

59
 More examples on computing T(n) and O(T(n))

• If we had to perform all these work every time we need to

analyze a program, the task would quickly be infeasible

• Since, we are giving the answer in terms of Big-Oh, there are lots
of short cuts that can be taken without affecting the final answer

• For example- consider example…

– Line 6 is obviously O(1) statement (per execution), so it is silly to count
precisely whether it is two, three or four units- it doesn’t matter
– Line 3 is obviously insignificant compared to the four loop, so it is silly to
waste time there

• This leads to several obvious general rules also called analysis

rules 60
 Analysis Rules
• These are guidelines for finding out the time
complexity of a piece of code

1. We assume an arbitrary time unit

2. Unlike real computers, it takes exactly one time unit to do

any thing (simple). Thus, execution of one of the following
operations takes time one (1)
– assignment statements
– Single input/output statements
– Single Boolean expression evaluation
– Single arithmetic
– function returns
61
 Analysis Rules
• Weakness of the above assumption

– In real life, not all operations take exactly the same time

– In our model (i.e., according to the assumption), one disk

read counts the same as an addition, even though the
addition is typically several orders of magnitude faster

62
 Analysis Rules
3. If-then-else statement

– The running time of an if-then-else statement is never more than the

running time of the test plus the larger of the running times of S1 and S2

• The total time for condition evaluation + the greater complexity of

then/else clause
• The total time for condition evaluation + the maximum of the
running times for the individual clauses in the selection
• The total time for condition evaluation + max (time taken by
alternatives)

– Clearly, this can be an over-estimate in some cases, but it is never an

under estimate

63
 Analysis Rules
• Example - Running time of if-then- else statement
If (x>5)
executed for (i=1; i<=n; i++)
n times
cout << i constant time
else
cout << “hello”; constant time

– Total time = 1 + max (n,1) = 1+ n = O(n)

64
 Analysis Rules
• Example - Running time of if-then- else statement

Worst-case running time: the test, plus either the

then part or the else part (whichever is the larger).

test: if (depth( ) != otherStack.depth( ) ) {

constant return false; then part:
} constant
else {
for (int n = 0; n < depth( ); n++) {
else part:
another if : if (!list[n].equals(otherStack.list[n]))
(constant +
constant + constant return false;
} constant) * n
(no else part)
}

Total time = c0 + c1 + (c2 + c3) * n = O(n)

65
 Analysis Rules
4. Consecutive statements
Add the time complexities of each statement.
•These just add (which means that the maximum is the one that
counts)
constant time x = x +1;
for (i=1; i<=n; i++) {
m = m + 2; executed
constant time
} n times
for (i=1; i<=n; i++) {
outer loop for (j=1; j<=n; j++) { inner loop
executed k = k+1; executed
n times } n times
constant time
}

Total time = c0 + c1n + c2n2 = O(n2)

66
 Analysis Rules
• Example - Consecutive statements
for (int i=0; i<n; i++)
A[i]=0;
for (i=0; i<n; i++)
{
for (int j=0; j<n; j++)
{
A[i] += i + j;
}
}

• This program fragment, which has O(n) work followed by

O(n2) work, is also O(n2)

67
 Analysis Rules
5. Running time of a Loop statement
The running time of a loop is, at most, the running
time of the statements inside the loop (including
tests) multiplied by the number of iterations.

for (i=1; i<=n; i++)

executed {
n times m = m + 2; constant time
}
Total time = a constant c * n = cn = O(n)

•Always assume the loop executes the maximum number of times

68
 Analysis Rules
6. Running time of Nested Loop
• Analyze inside out
• The running time of a statement inside a group of nested
loops is the running time of the statement multiplied by the
product of the sizes of all the loops

for (i=1; i<=n; i++) {

outer loop for (j=1; j<=n; j++) { inner loop
executed k = k+1; executed
n times } n times
} constant time

Total time = c * n * n * = cn2 = O(n2)

Total running time is the product of the
sizes of all the loops.
69
 Analysis Rules
7. Function call

– Running time of a function call is

• 1 + time taken for parameter calculation + time required

for the execution of function body (body time)

8. while loop – analyze like for loop

9. Switch statement- take the complexity of the most expensive

case

70
 Analysis Rules
• Example

– Find the complexity of the algorithm below using the

analysis rules discussed earlier

– Find the actual complexity for this algorithm and compare

it with the result obtained in (a)

71
 Algorithm Analysis Categories
• An algorithm may run faster on certain data sets than
others

• Algorithms must be examined under different situations to

correctly determine their efficiency for accurate
comparison

• Three categories to analyze algorithms

– Best case
– Worst case
– Average case

72
 Best Case Analysis
• Best case is defined as which input of size n is the cheapest
among all inputs of size n

• Best case is obtained from the in the input data set that results in
best possible performance

73
 Best Case Analysis
• Input
– Assumes the input data are found in the most advantageous
order for the algorithm
– For sorting, the most advantageous order is that data are
arranged in the required order
– For searching, the best case is that the required item is found
at the first position
– Question
• Which order is the most advantageous for an algorithm
that sorts 2 5 8 1 4 (ascending order)
• Input size
– Assumes that the input size is the minimum possible
74
 Best Case Analysis
• This case executes or causes the fewest number of executions

• So, it computes the lower bound of T(n) and You can not do better

• Is the amount of time the algorithm takes on the smallest possible

set of inputs

• Best case behavior is usually not informative and hence usually

uninteresting (why?)

• The function is denoted by Tbest(n) – best case running time of an

algorithm for n inputs

75
 Worst Case Analysis
• Is the complexity of an algorithm based on worst input of each
size

• Is the amount of time the algorithm takes on the worst possible

set of inputs

• Is the longest running time for any input of size n

76
 Worst Case Analysis
• Input (i.e., arrangement of data)
– Assumes that the data are arranged in the most
disadvantageous order
– For sorting, the worst case assumes that data are arranged in
opposite order
– For searching, the worst case assumes that the required item
is found at last position or missing
• Input size
– Assumes the input size to be infinite
– That is, it assumes that the input size is a very large number
– That is, it considers n Æinfinity

77
 Worst Case Analysis
• This case, causes highest number of executions to be performed
by the algorithm

• So, it computes the upper bound of T(n), the longest running

time for any input size n

• Commonly used for computing T(n)

• Is a function and is denoted by Tworst(n) – the worst case running

time of the algorithm for n inputs

78
 Worst Case Analysis
• Worst case is easier to analyze and can yield useful information

• In certain application domains (e.g., air traffic control, surgery)

knowing of the worst case time complexity is of crucial
importance

79
 Average Case Analysis
• Is the complexity of an algorithm averaged over all inputs of
each size

• Varies between the best case and worst case

• Gives the average performance of an algorithm

• Computes the optimal (the one which is found most of the time)
bound of T(n)

• The average case can be as bad as the worst case

80
 Average Case Analysis
• Input (i.e., arrangement of data)

– Assumes that data are arranged in random order

– For searching, it assumes that the required item is found at

any position or can be missing

– For sorting, it assumes that data are arranged in a random

order

• Input size

– The input size can be any random number as small as the best
case or as large as the worst case
81
 Average Case Analysis
• Problems with performing an average case analysis

– It may not be apparent what constitutes an “average” input for

a particular problem

• There is no clue as to what can be an average input

• It is difficult to envision the data that would cause an

algorithm (e.g., alg. for insertion sort) to exhibit its
average behavior

– Average case is difficult to determine

82
 Average Case Analysis
• Problems with performing an average case analysis

– The average operation count is often

• Quite difficult to determine or define
• Quite difficult to analyze

– Often we assume that all inputs of a given size are equally

likely
• In practice, this assumption may be violated, but
randomized algorithms can sometimes force this to hold

83
 Average Case Analysis
• As a result, in several of the algorithms, we limit our
analysis to determining the best and worst counts

• We focus on the worst case running time

– Easier to analyze

– Crucial to applications such as games, finance and

robotics

84
 Graphical view of best, average and worst cases
best case
• The running average case
time of an worst case
algorithm 120

varies with the 100

input and

Running Time
80
typically 60
grows with the
40
input size
20

0
1000 2000 3000 4000
Input Size
85
 Which Analysis to Use?
• Question
– How can we determine the complexity of an algorithm on any
data set?
– When is the worst case time important?

• Any of these categories have their own pros and cons

depending on the data set (arrangement of the data and
input size)

– While average time appears to be the fairest measure, it may be

difficult to determine

– Typically, the worst case analysis is used to estimate the

complexity of an algorithm (That is, we compare algorithms at
their worst case)
86
 Why use a worst case analysis
• The following are reasons for using worst case analysis as an
estimate for the complexity of an algorithm

1. The worst case running time of an algorithm provides

an upper bound on the running time for all inputs
(i.e., all cases)

• Thus, there is no bad operation than worst case

• So, knowing it gives us guarantee that the algorithm will

never take any longer

87
 Why use a worst case analysis
2. For some algorithms, the worst case occurs fairly often
– That is , Worst case "may" be a typical case

3. The average case is often roughly as bad as the worst case

– Average case may be approximately the worst case (( tj = j / 2 ) is
roughly quadratic)

4. Often it is hard to pick average cases and compute expected

times.

– The average case bounds are usually more difficult to

compute

Conclusion

– Unless otherwise specified, running time is the worst case

88
 Example
• Compute the best, average and worst case time complexities
of the Insertion sort

89
 Insertion Sort Algorithm
Input: a sequence of n number <a1, a2, …,an>
Output: a permutation (reordering) <a1', a2', …,an'>
such that a1'≤ a2' ≤ … ≤ an '.
• Idea: (similar to sort a hand of playing cards)
– Every time, take one card, and insert the card to
correct position in already sorted cards.

90
 Asymptotic Notations
• Goal: to simplify analysis by getting rid of unneeded
information (like “rounding” 1,000,001≈1,000,000)

• We want to say in a formal way 3n2 ≈ n2

• These are notations used to describe the running time of an

algorithm

• Are defined in terms of functions whose domain is the set of

natural numbers including 0 (range is the set of real numbers)

• But domain for asymptotic notations can be extended to the

domain of real numbers or restricted to the subset of natural
numbers 91
 Asymptotic Notations
• The following are the type of asymptotic notations
– ο
Big-Oh ( ) notation
– Ω
Big-Omega ( ) notation
– θ
Big-Theta ( ) notation
– Little-oh (o ) notation
– ω
Little-omega ( ) notation

92
 Big-Oh Notation
• Big-Oh notation is a way of comparing algorithms

• It is used for computing the complexity of algorithms

– That is, the amount of time that an algorithm takes to run

• Used to express the number of primitive operations executed as

a function of the input size.

• It is only concerned with what happens for a very large value

of n

• Thus only the largest term in the expression is needed

93
 Big-Oh
• Formal definition of Big-Oh
Let T, f : INÆ IR be functions. We say T(n) is O(f(n))
(pronounced T(n) is big-oh of f(n)) which we write T(n) ε
O(f(n)) or f(n)=O(g(n)), if there is

n0 ε IN and a constant c > 0 in IR susch that for all int egers n ≥ n0 , T (n) ≤ c. f (n)

• Mathematically precise, we can define O(f(n)) to be the

following class of functions

O( f (n)) = {T : IN → IR : there is n0 ε IN and a c > 0 in IR susch that for all n ≥ n0 we have T (n) ≤ c. f (n)}

• That is, for a given function f(n) we denote by O(g(n)) the set of
functions
O( f (n)) = {T (n) : IN → IR : there is n0 ε IN and a c > 0 in IR susch that for all n ≥ n0 we have 0 ≤ T (n) ≤ c. f (n)
94
 Big-Oh Notation
• O(f(n)) is best thought as a set of functions

• For a given function T(n), O(f(n)) denotes a set of functions

O(f(n)) = {T(n): c > 0 and n0 > 0 such that
0 ≤ T(n) ≤ cf(n) for all n ≥ n0 }

• Any function T(n) ε O(f(n)) cannot exceed the value f(n) when
n is sufficiently large, for n ≥ n0

• Definition indicates that O(f(n)) represent a set of functions

• For convenience we continuously write the notation T(n) =

O(f(n)), which means that T(n) belongs to O(f(n)), to mean
T(n) ε O(f(n))
95
 Big-Oh
• Remarks
– The functions T(n) and f(n) are none-negative for for all
integer n ≥ 0.
– c and n0 should not depend on n
– T(n) is O(f(n)) simply means T(n) ε O(f(n))
– The statement T(n) is O(f(n)) if T(n) grows at most as fast as
f(n)
– We think of T(n) ε O(f(n)) as corresponding to T(n) ≤ f(n)
– O( . ) is used to to asymptotically upper bound a function

96
 Big-Oh
•This definition says that eventually there is some point n0 past
which c.f(n) is always at least as large as T(n), so that if the
constant factors are ignored, f(n) is at leas as big as T(n)

T(n) = O(f(n))
97
 Big-Oh Notation
• For example
– If T(n) = 1000n and f(n) = n2, then
• n0 = 1,000 and c = 1
• We could also use n0 = 10 and c = 100
– Thus we say that 100n = O(n2)

• The notation is known as Big-Oh notation

• Frequently, instead of saying “order…” we say “Big-Oh …”

• If we use traditional inequality, the Big-Oh definiton says that the

rate of T(n) is greater than or equal to ( ≥ ) that of f(n)

98
 Big-Oh Example 1
• Consider the function f(n)=8n+128 shown in Figure

• Clearly, f(n) is non-negative for all integers n ≥ 0

• We wish to show that f(n) = O(n2)

• According to the definition of Big-Oh, in order to show this we

need to find an integer n0 and a constant c>0 such that for all
integers n ≥ n0, f(n) ≤ c. n2

• It does not matter what the particular constants are--as long as

they exist!

99
 Big-Oh Example 1
• E.g., suppose we choose c=1. Then

• Since (n+8)>0 for all values of n ≥ 0, we conclude that n0

– 16 ≥ 0. That is n0 = 16
• So, we have that for c=1 and n0 = 16, f(n) ≤ c. n2 for all
integers n ≥ n0
• Hence, f(n) = O(n2)
• The figure in the next slide clearly shows that the function
f(n) = n2 is greater than the function f(n)=8n+128 to the
right of n=16.

100
Big-Oh Example 1

101
 Big-Oh Example 1
• Of course, there are many other values of c and n0 that
will do.

• For example,

c = 2 and n0 = 2 + 2 17 ≈ 10.2 will do

• Another example

c = 4 and n0 = 1 + 33 ≈ 6.7 will do as well

102
 Big-Oh Example 2
For functions f(n) f(n) = 2n + 6
and g(n) (to the
right) there are
positive constants c
and n0 such that: c g(n) = 4n
f(n)≤c g(n) for n ≥ n0

conclusion:
g(n) = n
2n+6 is O(n).
n
103
 Big-Oh Example 3

On the other hand…

n2 is not O(n) because there is
no c and n0 such that:
n2 ≤ cn for n ≥ n0
(As the graph to the right
illustrates, no matter how large
a c is chosen there is an n big
enough that n2>cn ) .

104
 Some Facts to Use for Big-Oh Problems
• 1 ≤ n for all n ≥ 1

• n ≤ n2 for all n ≥ 1

• 2n ≤ n! for all n ≥ 4

• log n ≤ n for all n ≥ 2

• n ≤ n .log n for all n ≥ 2

• Exercise
– If f(n) = 10n + 5 and g(n) = n, then show that f(n) is
O(g(n))
– If f(n) = 3n2 + 4n + 1 then show that f(n) = O(n2)
105
 Big-Oh Notation
• Alternate definition

T ( n)
T (n) ε Ο( f (n)) if lim
n →∞ f ( n)

Is either zero or constant but not infinity

• Some people prefer the alternative definition because it is a little
easier to work with
• Exercise
– show that
T (n) = 13n 3 + 42n 2 + 2n log n + 3 n = Ο(n 3 )

106
 Remarks on the definition of Big-Oh notation
• Certain conventions have evolved which concern how big oh
expressions are normally written:

– First, it is common practice when writing big oh expressions

to drop all but the most significant terms.
• Thus, instead of we simply
write O(n2)

– Second, it is common practice to drop constant coefficients

• Thus, instead of O(3n2 ), we simply write O(n2 )
– As a special case of this last rule, if the function is a
constant, instead of, say O(1024), we simply write O(1).
107
 Remarks on the definition of Big-Oh notation
• The statement T(n) = O(f(n)) states only that c.f(n) is un upper
bound on the value of T(n) for all n, n ≥ n0

• It does not say any thing about how good or tight this bound is

• Example
n = O(n2), n = O(n2.5), n = O(n3), n = O(2n)

• For T(n) = O(f(n)) to be informative, f(n) should be as small as a

function of n as possible for which T(n) = O(f(n))

• So, although we often say 3n + 3 = O(n), we almost never say 3n

+ 3 = O(n2) even though the later statement is correct
108
 Remarks on the definition of Big-Oh notation
• 3n + 3 = O(n) is a better statement than 3n + 3 = O(n2)

• That is, one should make the approximation as tight as

possible

• Of course, in order for a particular big oh expression to be the

most useful, we prefer to find a tight asymptotic bound (see
Definition )

• Therefore
– Big-Oh describes the tight upper bound of T(n), that is the best
that we can do

• The definition states that the function f is at most c times the

function g except possibly when n is smaller than n0
109
Problems with the definition of Big-Oh notation
• It states only that there must exist certain c and n0, but does not
give any hint on how to calculate theses values

• It does not put any restriction on these values and gives little
advice in situations when there are many candidates

• If fact, there are infinite many pars of cs and n0s that can be
given for the same pair of functions T(n) and f(n)

• Exercise
– For T(n) = 2n2 + 3n + 1 = O(n2) where f(n) = n2, find
candidate values for c and n0 such that T(n) ≤ c.f(n) for all n
≥ n0
110
 Properties of the O notation
• If k is a constant, then k is O(1)
• Constant factors may be ignored
ƒ ∀k > 0, kf is O( f)
• Higher powers grow faster
– nr is O( ns) if 0 ≤ r ≤ s
• Fastest growing term dominates a sum
– If f is O(g), then f + g is O(g)
– eg an4 + bn3 is O(n4 )
• Polynomial’s growth rate is determined by leading term
– If f is a polynomial of degree d,
then f is O(nd)
– That is, a polynomial is O (the term containing the highest
power)
111
 Properties of the O notation
• f is O(g) is transitive
– If f is O(g) and g is O(h) then f is O(h)
• Product of upper bounds is upper bound for the product
– If f is O(g) and h is O(r) then fh is O(gr)
• Exponential functions grow faster than powers
– nk is O( bn ) ∀ b > 1 and k ≥ 0
eg n20 is O( 1.05n)
• Logarithms grow more slowly than powers
– logbn is O( nk) ∀ b > 1 and k > 0
eg log2n is O( n0.5)
112
 Properties of the O notation
• All logarithms grow at the same rate
– logbn is O(logdn) ∀ b, d > 1

• Sum of first n rth powers grows as the (r+1)th power

•. n

∑k
k =1
r
is Θ(n r +1
)
eg
n
n(n + 1)
∑
k =1
i=
2
is θ (n ) 2

113
 Properties of the O notation
• If f1(n) = O(g1(n)) and f2(n) = O(g2(n)), then
f1(n) + f2(n) = O(max(g1(n), g2(n)) )

114
 Big-Omega
• Definition (Big-Omega)
Consider a function f(n) which is non-negative for all integer n
≥ 0. We say that “f(n) is Omega of g(n),'' which we write
f(n)=Ω(g(n)), if there exists an integer

n0 and a constant c > 0 in IR susch that for all integers n ≥ n0 , f (n) ≥ c.g (n)

115
 Big-Omega- Example 1
• Consider the function f(n) = 5n2 – 64n + 256

• Clearly, f(n) is non negative for all integers n

• We wish to show that f(n) = Ω(n2)

• According to the definition of big Omega, in order to show this

we need to find intger n0 and a constant c > 0 such that for all
integers n ≥ n0, f(n) ≥ c.n2

• As with big oh, it does not matter what the particular constants
are--as long as they exist!

116
 Big-Omega- Example 1
• E.g., suppose we choose c=1. Then

• Since (n – 8)2 > 0 for all values of n ≥ 0, we conclude that n0 =

0

• So, we have that for c=1 and n0 = 0, f(n) ≥ cn2 for all integers n
≥ n0

• Hence, f(n) = Ω(n2)

117
 Big-Omega- Example 1
• The figure in the next slide clearly shows that the function
f(n) = n2 is less than the function f(n) = 5n2 – 64n + 256 for
all values of n ≥ 0

• Of course, there are many other values of c and n0 that will

do

• Example- c=2 and n0 = 16

118
Big-Omega- Example 1

119
 More examples on Big-Omega
• Include examples

120
 Big-Omega
• Alternative definition
f ( n)
f (n) ε Ω( g (n)) if lim is either a constant or ∞ (but not zero)
n →∞ g ( n)

121
 Big-Omega
• The definition of omega is almost identical to that of big-oh

• The only difference is in the comparision

– For big-oh f(n) ≤ c.g(n)
– For omega f(n) ≥ c.g(n)

• All the same conventions and caveats apply to omega as they

do to big-oh
– Ω notations suffer from the same profusion problem as
does Big-Oh notation
– There are unlimited number of choices for the constants c
and n
– See example on this
122
 Big-Omega
• The intuition behind Ω notation is shown in the figure below

f(n) = Ω(g(n)) 123

 Big-Omega
• As n increases f(n) grows no slower than g(n)

• This describes the tight lower bound of f(n)

• When we write f(n) = Ω(g(n)), we are saying that f is at least c

times the function g except possibly when n is smaller than n0

• Here, c is some positive constant

• Thus, g is the lower bound (except for a constant factor c) on the

value of f for all suitably large n

• We normally use only simple functional forms for g

124
 Asymptotic Lower Bound - Omega
• Big omega is transitive

– Assuming f(n)=Ω(g(n)) and g(n)=Ω(h(n)), we have

constants c1 > 0, c2 > 0 and integers n0, n1 such that
0 ≤ c1.g(n)≤f(n) for all n > n0, and 0≤ c2.h(n)≤g(n) for all n>n1

– multiplying the second set of inequalities throughout by c1,

0≤c1.c2h(n)≤c1g(n) provided n>max{n0,n1}

– Therefore, f(n)= Ω(h(n)) is true

125
 Asymptotic Lower Bound - Omega
• Transpose symmetry

– Big oh and big omega are related by transpose symmetry:

f(n) = O(g(n)) iff g(n) = Ω(f(n))

– 0≤f(n)≤c1g(n), where c1 > 0 for all n > n0

– Divide throughout by c1 and let c2=(1/c1)

– 0≤c2f(n)≤g(n), where c2 > 0 for all n > n0

– Which imply f(n) = Ω(g(n))

126
 Big-Omega
• Remarks
– As in the case of big-oh, there are several functions g(n) for
which f(n)=Ω(g(n))

– g(n) is only a lower bound on f(n)

– For the statement f(n) = Ω(g(n)) to be informative, g(n)

should be as large as the a function of n as possible for which
statement f(n) = Ω(g(n)) is true

– So, although, we say that 3n + 3 = Ω(n), 62n + n2 = Ω(2n), we

never say that 3n + 3 = Ω(1) or 62n + n2 = Ω(1) even though
both statements are true
127
 Big-Omega
• That is big-omega represent tight bound for running time

• If T(n)=Ω(f(n)) then T(n) and f(n) are no more than a

constant factor apart from each other

• A tight bound is always preferable than a non tight bound

128
 Theta Notation (Θ)
• The Theta notation is used when the function f can be bounded
both from above and below by the same function

• Definition (Theta)
Consider a function f(n) which is non-negative for all integer n
≥ 0. We say that “f(n) is Theta g(n),'' which we write
f(n)=θ(g(n)), if there exists positive constants c1 and c2 and
n0 such that

c1 g (n) ≤ f (n) ≤ c2 g (n) for all integers n ≥ n0

129
 Theta Notation (Θ) – Examples
• Include examples

130
 Theta Notation (Θ)
• Alternative definition
f ( n)
f (n) ε θ ( g (n)) if lim is a non−zero constant (not zero or ∞)
n →∞ g ( n)

• Example
n 3 + 3n 2 + 2 n
show that f ( n ) = ε θ (n 3 )
6

• Note that it follows that T(n) ε Ο(n3), and T(n) ε Ω(n3)

131
 Theta Notation (Θ)
• For a given function g(n), θ(g(n)) denotes a set of functions

• As n increases f(n) grows as fast as g(n)

• It describes the tight optimal bound of f(n)
• It shows the average case analysis

132
 Theta Notation (Θ)
• The figure below gives an intuitive picture of functions f(n) and
g(n), where f(n) = θ(g(n))
The Theta Notation asymptotically bounds a function form
above and below

f(n) = θ(g(n)) 133

 Theta Notation (Θ)
• For all values of n to the right of n0, the value of f(n) lies at or
above c1g(n) and at or below c2g(n)

• In other words, for all n ≥ n0, the function f(n) is equal to g(n) to
within a constant factor

• We say g(n) is asymptotically tight bound for f(n)

• The definition of θ(g(n)) requires that every member of θ(g(n))

be asymptotically non-negative

134
 Theta Notation (Θ)
• Therefore, we assume that every function used with θ- notation
is asymptotically nonnegative

• A function f(n) belonges to the set θ(g(n)) if there exist positive

constants c1 and c2 such that it can be “sandwiched” between
c1g(n) and c2g(n), for sufficiently large n

• When we write f(n)= θ(g(n)), we are saying that f lies between

c1 times the function g and c2 times the function g except
possibly when n is smaller than n0 (c1 and c2 are positive
constants)

135
 Theta Notation (Θ)
• Thus g is both the lower bound and upper bound (except for a constant factor
c) on the values of f for all suitably large n

• That is, another way to view the θ- notation is that it is both Ω(g(n)) and
Ο(g(n))

• That is, f (n) = θ ( g (n))

⇒ f (n) ∈ Ο( g (n)) and
g (n) ∈ Ω( g (n))

• Prove the following (exercise)

136
 Properties of Theta
• Reflexive
f(n) = θ(f(n))

• Transitive
T(n)=θ(f(n)), f(n)=θ(g(n))ÆT(n)=θ(g(n))

• Symmetry property
T(n)=θ(f(n)) iff f(n)=θ(T(n))

• T(n)=Ω(f(n)), T(n)=O(f(n)) iff T(n)=θ(f(n))

• T(n)=O(f(n)), T(n)≠θ(f(n))ÆT(n)≠Ω(f(n))

137
 Little-oh
• Is also called small-oh notation

• Is an extension of the big- Οh notation

• The asymptotic upper bound provided by Ο- notation may or

may not be asymptotically tight

• Example
– The bound 2n2 = Ο(n2) is asymptotically tight
– The bound 2n = Ο(n2) is not asymptotically tight
– Ο(n3) is not a tight upper bound for the expression (1/2)n2 +
(1/6)n where as Ο(n2) is
– There for (1/2)n2 + (1/6)n = ο(n3) but not ο(n2)
138
 Little-oh
• We use the ο-notation to denote an upper bound that is not
asymptotically tight

• That is, it represents upper bound like big- Οh but the upper
bound is not tight

139
 Little-oh
• Definition (Little-oh )
T(n) is said to be in little-oh of f(n), written as T(n) =o(f(n)), if

∃c, n0 > 0 ∋ T (n) < c. f (n) ∀n ≥ n0

• As n increases, T(n) grows strictly slower than f(n)

• Little-oh
– describes the non-tight upper bound of T(n)
– shows the worst case of an algorithm

140
 Little-oh
• Definition (Little-oh )

– T(n)=o(f(n)) (read as “T of n is in little oh of f of n) iff

T(n)=O(f(n)) and T(n)≠Ω(f(n))

• Example
– Consider the function T(n) = n + 1
– Clearly, T(n) = O(n2)
– Clearly, T(n) ≠ Ω(n2) since no mater what c we choose,
for large enough n, c.n2 ≥ n + 1
– Thus, we may write T(n) = o(n2)

141
 Little-oh
• Definition (Little-oh )

– Consider a function T(n) which is non-negative for all

integers n ≥ 0. We say that “T(n) is little-oh f(n)”, which we
write T(n)=o(f(n)), if and only if T(n) is O(f(n)) but T(n) is
not θ(f(n))

• Little oh notation represents a kind of loose asymptotic

bound in the sense that if we are given that T(n)=o(f(n)), then
we know that f(n) is an asymptotic upper bound since
T(n)=O(f(n)), but f(n) is not an asymptotic lower bound since
T(n)=O(f(n)) and T(n) ≠ θ(f(n)) implies that T(n) ≠ Ω(f(n))

142
 Little-oh
• T(n) = o(f(n)) then applying idea of limits as n increased
indefinitely T(n) becomes insignificant compared to f(n), i.e.,
T ( n)
lim =0
n →∞ f ( n)

– Note: if f(n) is o(g(n)), then f(n) = O(g(n))

• Example
– Show 4n3 + 3n2 + 5 = o(n4 - 3n3 - 5n – 4)
– This is equivalent to
n4 - 3n3 - 5n – 4 = ω (4n3 + 3n2 + 5)

143
 Little-oh
• Little-oh is not reflexive

– T(n) ≠ o(T(n))

– It is easy to check that

T ( n)
lim ≠0
n →∞ T ( n)

– Therefore, relation little-oh can not be reflexive

144
 Little-oh (Small-οh)
• Little-oh is not symmetric:
– T(n) = o(f(n))
⇒ f(n)=o(T(n))

– It holds because the following two conditions cannot hold

simultaneously

1. The magnitude of T(n) is insignificant with relative to f(n)

as nÆinfinity

2. The magnitude of f(n) is significant with relative to T(n) as

nÆinfinity

145
 Little-oh
• Little-oh is transitive

– If T(n)=o(f(n)) and f(n)=o(h(n)) Æ T(n)=o(h(n))

• It Means that something which grows no bigger than o(f(n))

must grow no more than o(h(n)) as f(n)=o(h(n))

146
 Little omega
• we use little ω to denote lower bound that is not
asymptotically tight

• For a given function f(n), ω(f(n)) denotes a set of functions

• Ω(f(n))={T(n): for any c > 0, there exist a constant n0 > 0

such that 0≤c.f(n)<T(n) for all n≥n0}

• Little omega defines the relation which is inverse of little-oh

• If T(n)=ω(f(n)) then the rate of growth of T(n) is faster

compared to f(n)

147
 Little omega
• Definition of little omega

– f(n) = ω(g(n)) if there exists a constant n0 > 0,

such that 0 ≤ c.g(n) < f(n) for all n ≥ n0

148
 Little omega
• So as nÆinfinity, the magnitude of g(n) is insignificant
comared to that of f(n)

• This implies that

f ( n)
lim =∞
n →∞ g ( n)

• Therefore,

g ( n)
lim = 0 ⇒ g (n) = o( f (n))
n →∞ f ( n)

149
 Comparison of Ω, Θ, and Ο
• Ο(g): functions that grow no faster than g

• Θ(g): functions grow at the same rate as g

• Ω(g): functions grow at least as fast as g

– Together Big O and Ω set establish the lower and upper

boundaries of asymptotic order.

– Θ(g) = Ο(g) ∩ Ω(g)

150
 Clarification
• When talking about Big O, Ω, θ, and little o, ϖ, we do not talk
about specific functions

– That is, there is no function that is exclusively Big O of

something else

• What we mean when we say that function f is big O of function g

is that the function f is a member of the set of functions which
satisfy the given criteria for big O.

• The proper notation is to say that f is an element in the set of

functions which is identified as O(g), or more explicitly
– f ∈ O(g),
– Remember, f ≠ O(g)
151