Asymptotic Notations

• Iterative Algorithms and their analysis

• Asymptotic Notations
– Big O, Notations

• Review of Discrete Math

– Summations
– Logarithms

1
 Example I:
Finding the sum of an array of
numbers
int Sum(int A[], int N) { • How many steps
int sum = 0; does this algorithm
for (i=0; i < N; i++){
take to finish?
sum += A[i]; – We define a step to
} //end-for be a unit of work that
can be executed in
return sum; constant amount of
} //end-Sum time in a machine.

2
 Example I:
Finding the sum of an array of
numbers
Times
Execute
int Sum(int A[], int N) {
int sum = 0;
d
1
for (i=0; i < N; i++){
sum += A[i];
} //end-for N
N
return sum;
} //end-Sum
1
--------
Total: 1 + N + N + 1 = 2N
+2
• Running Time: T(N) = 2N+2
– N is the input size (number of ints) to3
 Searching
• One of the most important problems in
computer science
• Problem: Given an array of “N” numbers,
search to find if a given “key” exists or not
– Linear Search
– Binary Search
– Recall Search Trees

4
 Linear Search

int LinearSearch(int A[N], int Times

N, Executed
int key) { 1
int i= 0;
N
while (i < N){ N
if (A[i] == key) return i; N
i++;
} //end-while
1
return -1; ---------
} //end-LinearSearch Total: 1+3*N+1 =
3N+2 5
 Example V: Binary Search
• Problem: You are given a sorted array of
integers, and you are searching for a key
− Linear Search – T(n) = 3n+2 (Worst case)
− Can we do better?
− E.g. Search for 55 in the sorted array below

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

6
 Example V: Binary Search
• Since the array is sorted, we can reduce our search space in
half by comparing the target key with the key contained in
the middle of the array and continue this way
• Example: Let’s search for 55

0 1 2 3 4 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

(left  right )
left middle  right
2

0 1 2 3 4 6 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

left middle right Eliminated

7
 Binary Search (continued)

0 1 2 3 4 5 6 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

Eliminated left middle right

0 1 2 3 4 5 6 7 8 11 15
3 8 10 11 20 50 55 60 65 70 72 90 91 94 96 99

l ef t
t h
Eliminated r ig
middle
• Now we found 55 Successful search
• Had we searched for 57, we would have terminated
at the next step unsuccessfully
8
 Binary Search - Algorithm
// Return the index of the array containing the key or –1 if key not found
int BinarySearch(int A[], int N, int key){
left = 0;
right = N-1;

while (left <= right){

int middle = (left+right)/2; // Index of the key to test against
if (A[middle] == key) return middle; // Key found. Return the index
else if (key < A[middle]) right = middle – 1; // Eliminate the right side
else left = middle+1; // Eliminate the left side
} //end-while

return –1; // Key not found

} //end-BinarySearch

• Worst case running time: T(n) = 3 + 5*log2N. Why?

9
 Asymptotic Notations
• Summation – T(N) = 2N + 2
• Linear Search – T(N) = 3N + 2
• Binary Search – T(N) = 3 + 5*log2N

• In general, what really matters is the “asymptotic”

performance as N → ∞, regardless of what happens
for small input sizes N.

• 3 asymptotic notations
– Big-O --- Asymptotic upper bound
– Omega --- Asymptotic lower bound
– Theta --- Asymptotic tight bound

10
 Big-Oh Notation: Asymptotic Upper
Bound
• T(n) = O(f(n)) [T(n) is big-Oh of f(n) or order of f(n)]
– If there are positive constants c & n0 such that
T(n) <= c*f(n) for all n >= n0
Running Time
c*f(n)

T(n)

n0 Input size N

– Example: T(n) = 50n is O(n). Why?

– Choose c=50, n0=1. Then 50n <= 50n for all
n>=1
– many other choices work too! 11
  Notation: Asymptotic Lower
•
Bound
T(n) = (f(n))
– If there are positive constants c & n0 such that
T(n) >= c*f(n) for all n >= n0

T(n)
Running Time

c*f(n)

n0 Input size N

– Example: T(n) = 2n + 5 is (n). Why?

– 2n+5 >= 2n, for all n >= 1
– T(n) = 5*n2 - 3*n is (n2). Why?
– 5*n2 - 3*n >= 4*n2, for all n >= 4 12
  Notation: Asymptotic Tight
Bound
• T(n) = (f(n))
– If there are positive constants c1, c2 & n0 such that
c1*f(n) <= T(n) <= c2*f(n) for all n >= n0

c2*f(n)
Running Time

T(n)
c1*f(n)

n0 Input size N

– Example: T(n) = 2n + 5 is (n). Why?

2n <= 2n+5 <= 3n, for all n >= 5
– T(n) = 5*n2 - 3*n is (n2). Why?
13
– 4*n2 <= 5*n2 - 3*n <= 5*n2, for all n >= 4
 Big-Oh, Theta, Omega
Tips to guide your intuition:
• Think of O(f(N)) as “less than or equal to” f(N)
– Upper bound: “grows slower than or same rate as” f(N)

• Think of Ω(f(N)) as “greater than or equal to” f(N)

– Lower bound: “grows faster than or same rate as” f(N)

• Think of Θ(f(N)) as “equal to” f(N)

– “Tight” bound: same growth rate

• (True for large N and ignoring constant factors)

14
 Some Math
N ( N  1)
N

S(N) = 1 + 2 + 3 + 4 + … N = i
2
i 1

N
N * ( N  1) * (2n  1) N 3
Sum of Squares:  i 
2

i 1 6 3
N 1
N
A 1
Geometric Series:  A  A  1
i
A>1
i 0

N 1
N
1  A A<1

i 0
A i

1 A
(1)

15
 Some More Math
Linear Geometric
( n 1) n
n
( n  1) x  nx x
Series:  ix i x  2 x 2  3x 3  ...  nx n  2
i 0 ( x  1)

n
1 1 1 1
Harmonic Series: H n  1    ...  (ln n) O(1)
i 1 i 2 3 n

Logs: log A B B * log A

log( A * B) log A  log B
A
log( ) log A  log B
B
16
 More on Summations
b b a 1
• Summations with general
bounds:  f (i)  f (i)   f (i)
i a i 0 i 0

n n n
• Linearity of Summations:
 (4i
i 1
2
 6i ) 4 i  6 i
i 1
2

i 1

17