Asymptotic Notations

• Iterative Algorithms and their analysis

• Asymptotic Notations
– Big O, Notations

• Review of Discrete Math

– Summations
– Logarithms

1
 Example I:
Finding the sum of an array of numbers
int Sum(int A[], int N) { • How many steps does
int sum = 0; this algorithm take to
for (i=0; i < N; i++){
finish?
sum += A[i]; – We define a step to be
} //end-for a unit of work that can
be executed in
return sum; constant amount of
} //end-Sum time in a machine.

2
 Example I:
Finding the sum of an array of numbers
Times
Executed
int Sum(int A[], int N) {
int sum = 0; 1

for (i=0; i < N; i++){

sum += A[i]; N
} //end-for N
return sum;
} //end-Sum 1
--------
Total: 1 + N + N + 1 = 2N + 2
• Running Time: T(N) = 2N+2
– N is the input size (number of ints) to add
3
 Searching
• One of the most important problems in
computer science
• Problem: Given an array of “N” numbers,
search to find if a given “key” exists or not
– Linear Search
– Binary Search
– Recall Search Trees

4
 Linear Search

int LinearSearch(int A[N], int N, Times

int key) { Executed
int i= 0; 1

while (i < N){ N

if (A[i] == key) return i; N
i++; N
} //end-while

return -1; 1
} //end-LinearSearch ---------
Total: 1+3*N+1 = 3N+2
5
 Example V: Binary Search
• Problem: You are given a sorted array of
integers, and you are searching for a key
− Linear Search – T(n) = 3n+2 (Worst case)
− Can we do better?
− E.g. Search for 55 in the sorted array below

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

6
 Example V: Binary Search
• Since the array is sorted, we can reduce our search space
in half by comparing the target key with the key
contained in the middle of the array and continue this way
• Example: Let’s search for 55

0 1 2 3 4 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

(left  right )
left middle  right
2

0 1 2 3 4 6 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

left middle right Eliminated

7
 Binary Search (continued)

0 1 2 3 4 5 6 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

Eliminated left middle right

0 1 2 3 4 5 6 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

lef
t ig ht
Eliminated r
middle
• Now we found 55 Successful search
• Had we searched for 57, we would have terminated at the
next step unsuccessfully
8
 Binary Search - Algorithm
// Return the index of the array containing the key or –1 if key not found
int BinarySearch(int A[], int N, int key){
left = 0;
right = N-1;

while (left <= right){

int middle = (left+right)/2; // Index of the key to test against
if (A[middle] == key) return middle; // Key found. Return the index
else if (key < A[middle]) right = middle – 1; // Eliminate the right side
else left = middle+1; // Eliminate the left side
} //end-while

return –1; // Key not found

} //end-BinarySearch

• Worst case running time: T(n) = 3 + 5*log2N. Why?

9
 Asymptotic Notations
• Summation – T(N) = 2N + 2
• Linear Search – T(N) = 3N + 2
• Binary Search – T(N) = 3 + 5*log2N

• In general, what really matters is the “asymptotic”

performance as N → ∞, regardless of what happens for
small input sizes N.

• 3 asymptotic notations
– Big-O --- Asymptotic upper bound
– Omega --- Asymptotic lower bound
– Theta --- Asymptotic tight bound

10
Big-Oh Notation: Asymptotic Upper Bound
• T(n) = O(f(n)) [T(n) is big-Oh of f(n) or order of f(n)]
– If there are positive constants c & n0 such that
T(n) <= c*f(n) for all n >= n0

c*f(n)
Running Time

T(n)

n0 Input size N

– Example: T(n) = 50n is O(n). Why?

– Choose c=50, n0=1. Then 50n <= 50n for all n>=1
– many other choices work too!
11
 Notation: Asymptotic Lower Bound
• T(n) = (f(n))
– If there are positive constants c & n0 such that
T(n) >= c*f(n) for all n >= n0

T(n)
Running Time

c*f(n)

n0 Input size N

– Example: T(n) = 2n + 5 is (n). Why?

– 2n+5 >= 2n, for all n >= 1
– T(n) = 5*n2 - 3*n is (n2). Why?
– 5*n2 - 3*n >= 4*n2, for all n >= 4 12
  Notation: Asymptotic Tight Bound
• T(n) = (f(n))
– If there are positive constants c1, c2 & n0 such that
c1*f(n) <= T(n) <= c2*f(n) for all n >= n0
c2*f(n)
Running Time

T(n)
c1*f(n)

n0 Input size N

– Example: T(n) = 2n + 5 is (n). Why?

2n <= 2n+5 <= 3n, for all n >= 5
– T(n) = 5*n2 - 3*n is (n2). Why?
– 4*n2 <= 5*n2 - 3*n <= 5*n2, for all n >= 4 13
 Big-Oh, Theta, Omega
Tips to guide your intuition:
• Think of O(f(N)) as “less than or equal to” f(N)
– Upper bound: “grows slower than or same rate as” f(N)

• Think of Ω(f(N)) as “greater than or equal to” f(N)

– Lower bound: “grows faster than or same rate as” f(N)

• Think of Θ(f(N)) as “equal to” f(N)

– “Tight” bound: same growth rate

• (True for large N and ignoring constant factors)

14
 Some Math
N
N ( N  1)
S(N) = 1 + 2 + 3 + 4 + … N = 
i 1
i
2
N
N * ( N  1) * (2n  1) N 3
Sum of Squares:  i  
2

i 1 6 3
N 1
N
A 1
Geometric Series: 
i 0
A 
i

A 1
A>1

N 1
N
1  A A<1

i 0
A i

1 A
 (1)

15
 Some More Math
Linear Geometric
( n 1)
n
( n  1) x  nx n
x
Series:  ix i  x  2 x 2  3x 3  ...  nx n 
i 0 ( x  1)2

n
1 1 1 1
Harmonic Series: H n    1    ...   (ln n) O(1)
i 1 i 2 3 n

Logs: log A B  B * log A

log( A * B)  log A  log B
A
log( )  log A  log B
B
16
 More on Summations
b b a 1
• Summations with general
bounds:  f (i)  f (i)   f (i)
i a i 0 i 0

n n n
• Linearity of Summations:  (4i
i 1
2
 6i )  4 i  6 i
i 1
2

i 1

17