Analysis of Algorithms - Issues

• Issues:
CSC 344 – Algorithms and – Correctness – Does it work as advertised?
– Time Efficiency – Are time requirements
Complexity minimized?
– Space Efficiency – Are space requirements
Lecture #2 – Analyzing Algorithms minimized?
and Big O Notation – Optimality – Do we have the best balance between
minimizing time and space?

Theoretical Analysis Of Time

Analysis of Algorithms
Efficiency
• Issues: • Time efficiency is analyzed by determining the
number of repetitions of the basic operation as a
– Correctness function of input size
– Time Efficiency • Basic operation: the operation that contributes most
– Space Efficiency towards the running time of the algorithm
– Optimality
T(n) ≈ copC(n)
• Approaches:
– Theoretical Analysis Running Time
Execution Time
Number Of Times
Basic Operation Is
– Empirical Analysis For Basic
Executed
Operation
Input Size And Basic Operation Examples Best-Case, Average-Case, Worst-Case
Problem Input size measure Basic operation
• For some algorithms efficiency depends on
Searching for key in a
Number of list’s items, i.e. n Key comparison
form of input:
list of n items
– Worst case: Cworst(n) – maximum over inputs of
Multiplication of two Matrix dimensions or total Multiplication of two size n
matrices number of elements numbers
– Best case: Cbest(n) – minimum over inputs of
Checking primality of n’size = number of digits (in size n
Division
a given integer n binary representation) – Average case: Cavg(n) – “average” over inputs of
Typical graph Visiting a vertex or size n
#vertices and/or edges
problem traversing an edge

Empirical Analysis Of Time Efficiency Average-Case

• Select a specific (typical) sample of inputs • Average case: Cavg(n) – “average” over inputs
• Use physical unit of time (e.g., milliseconds) of size n
– Number of times the basic operation will be executed on
or typical input
• Count actual number of basic operation’s – NOT the average of worst and best case
executions – Expected number of basic operations considered as a
random variable under some assumption about the
• Analyze the empirical data probability distribution of all possible inputs
 Example: Sequential Search Order of Growth
• Most important: Order of growth within a
constant multiple as n→∞
• Example:
– How much faster will algorithm run on computer
that is twice as fast?
– How much longer does it take to solve problem of
• Best case?
double input size?
• Worst case?
• Average case?

Types Of Formulas For Basic Operation’s

Values of Some Important Functions as n →∞
Count
• Exact formula n log2n n nlog2n n2 n3 2n n!
10 3.3 101 3.3×101 102 103 103 3.6×106
e.g., C(n) = n(n-1)/2
102 6.6 102 6.6×102 104 106 1.3×1030 9.3×10157

• Formula indicating order of growth with specific 103 10 103 1.0×104 106 109
104 13 104 1.3×105 108 1012
multiplicative constant
105 17 105 1.7×106 101 1015
e.g., C(n) ≈ 0.5 n2 0

106 20 106 2.0×107 101 1018

2
• Formula indicating order of growth with unknown
multiplicative constant
e.g., C(n) ≈ cn2
 Asymptotic Order Of Growth Big Omega
• A way of comparing functions that ignores
constant factors and small input sizes
– O(g(n)) - class of functions f(n) that grow no faster
than g(n)
– Θ(g(n)) - class of functions f(n) that grow at same
rate as g(n)
– Ω(g(n)) - class of functions f(n) that grow at least
as fast as g(n)

Big O Big Theta

 Establishing Order Of Growth Using Establishing Order Of Growth Using
The Definition Limits
0 order of growth of T(n) < order of growth of g(n)
• Definition: f(n) is in O(g(n)) if order of growth of
f(n) ≤ order of growth of g(n) (within constant lim T(n)/g(n) = c > 0 order of growth of T(n) = order of growth of g(n)
n→∞
multiple), i.e., there exist positive constant c and non- ∞ order of growth of T(n) > order of growth of g(n)
negative integer n0 such that
f(n) ≤ c g(n) for every n ≥ n0 Examples:
• 10n vs. n2
• Examples:
• n(n+1)/2 vs. n2
– 10n is O(n2)
– 5n+20 is O(n)

Some Properties Of Asymptotic Order L’Hôpital’s Rule And Stirling’s

Of Growth Formula
• f(n) ∈ O(f(n)) • L’Hôpital’s rule:
• f(n) ∈ O(g(n)) iff g(n) ∈ Ω(f(n)) If limn→∞ f(n) = limn→∞ g(n) = ∞ and
the derivatives f´, g´ exist,
• If f (n) ∈ O(g (n)) and g(n) ∈ O(h(n)) ,
then f(n) ∈ O(h(n)) then lim f(n) = lim f ´(n)
n→∞
→∞ g(n) n→∞
→∞ g ´(n)
Note similarity with a ≤ b – Example: log n vs. n
• If f1(n) ∈ O(g1(n)) and f2(n) ∈ O(g2(n)) , • Stirling’s formula: n! ≈ (2πn)1/2 (n/e)n
then f1(n) + f2(n) ∈ O(max{g1(n), g2(n)})
– Example: 2n vs. n!
Orders Of Growth Of Some Important Time Efficiency Of Nonrecursive
Functions Algorithms
• All logarithmic functions loga n belong to the same class General Plan for Analysis
Θ(log n) no matter what the logarithm’s base a > 1 is
• Decide on parameter n indicating input size
• All polynomials of the same degree k belong to the same class: • Identify algorithm’s basic operation
aknk + ak-1nk-1 + … + a0 ∈ Θ(nk) • Determine worst, average, and best cases for
nput of size n
• Exponential functions an have different orders of growth for • Set up a sum for the number of times the basic
different a’s operation is executed
• order log n < order nα (α>0) < order an < order n! < order nn
• Simplify the sum using standard formulas and
rules

Useful Summation Formulas And

Basic Asymptotic Efficiency Classes
Rules
1 constant Σl≤i≤u1 = 1+1+ ⋯ +1 = u - l + 1
log n logarithmic In particular, Σl≤i≤u1 = n - 1 + 1 = n ∈ Θ(n)
n linear
n log n n-log-n or linearithmic Σ1≤i≤n i = 1+2+ ⋯ +n = n(n+1)/2 ≈ n2/2 ∈ Θ(n2)
n2 quadratic
Σ1≤i≤n i2 = 12+22+ ⋯ +n2 = n(n+1)(2n+1)/6 ≈ n3/3 ∈ Θ(n3)
n3 cubic
2n exponential Σ0≤i≤n ai = 1 + a + ⋯ + an = (an+1 - 1)/(a - 1) for any a ≠ 1
n! factorial In particular, Σ0≤i≤n 2i = 20 + 21 + ⋯ + 2n = 2n+1 - 1 ∈ Θ(2n )

Σ(ai ± bi ) = Σai ± Σbi Σcai = cΣai Σl≤i≤uai = Σl≤i≤mai +

Σm+1≤i≤uai
Example 1 - Maximum Element Example 3 - Matrix Multiplication

Example 2 - Element Uniqueness

Example 4: Counting Binary Digits
problem