Analysis of Algorithms

CS 477/677

Midterm Exam Review

Instructor: George Bebis
 General Advice for Study

• Understand how the algorithms are working

– Work through the examples we did in class

– “Narrate” for yourselves the main steps of the

algorithms in a few sentences

• Know when or for what problems the

algorithms are applicable
• Do not memorize algorithms

2
 Asymptotic notations
• A way to describe behavior of functions in the limit
– Abstracts away low-order terms and constant factors

– How we indicate running times of algorithms

– Describe the running time of an algorithm as n goes to 

• O notation: asymptotic “less than”: f(n) “≤”

g(n)
  notation: asymptotic “greater than”: f(n) “≥”
g(n)
  notation: asymptotic “equality”: f(n) “=” g(n)
3
 Exercise
• Order the following 6 functions in increasing
order of their growth rates:
– nlog n, log2n, n2, 2n, n, n.

log2n
n
n
nlogn
n2
2n

4
 Running Time Analysis
• Algorithm Loop1(n)
p=1
O(n)
for i = 1 to 2n
p = p*i

• Algorithm Loop2(n)
O(n2)
p=1
for i = 1 to n2
p = p*i

5
 Running Time Analysis
Algorithm Loop3(n)
s=0 O(n2)

for i = 1 to n
for j = 1 to i
s=s+I

6
 Recurrences
Def.: Recurrence = an equation or
inequality that describes a function in
terms of its value on smaller inputs,
and one or more base cases
• Recurrences arise when an algorithm contains
recursive calls to itself
• Methods for solving recurrences
– Substitution method
– Iteration method
– Recursion tree method
– Master method
• Unless explicitly stated choose the simplest method for solving
recurrences 7
Analyzing Divide and Conquer Algorithms
• The recurrence is based on the three steps of the
paradigm:
– T(n) – running time on a problem of size n
– Divide the problem into a subproblems, each of size
n/b: takes D(n)
– Conquer (solve) the subproblems aT(n/b)
– Combine the solutions C(n)
(1) if n ≤ c
aT(n/b) + D(n) + C(n)
T(n) = otherwise

8
 Master’s method
• Used for solving recurrences of the form:

 n
T (n) aT    f (n)
b
where, a ≥ 1, b > 1, and f(n) > 0
Compare f(n) with nlogba:

Case 1: if f(n) = O(nlogba - ) for some  > 0, then: T(n) = (nlogba)

Case 2: if f(n) = (nlogba), then: T(n) = (nlogba lgn)

Case 3: if f(n) = (nlogba +) for some  > 0, and if

af(n/b) ≤ cf(n) for some c < 1 and all sufficiently large n, then:
regularity condition T(n) = (f(n)) 9
 Sorting
• Insertion sort
– Design approach: incremental
– Sorts in place: Yes
– Best case: (n)
– Worst case: (n2)

• Bubble Sort
– Design approach: incremental
– Sorts in place: Yes
– Running time: (n2)

10
 Analysis of Insertion Sort

INSERTION-SORT(A) cost times

c1 n
for j ← 2 to n
c2 n-1
do key ← A[ j ]
0 n-1
Insert A[ j ] into the sorted sequence A[1 . . j
-1] c4 n-1
n 2
/2 comparisons
c5
 t
n
i←j-1 j 2 j
c6
 (t
n
while i > 0 and A[i] > key j 2 j  1)
c7
do A[i + 1] ← A[i]  (t
n
 1)
n
2
/2 exchanges j 2 j
c8 n-1
i←i–1
A[i + 1] ← key 11
 Analysis of Bubble-Sort
Alg.: BUBBLESORT(A)
for i  1 to length[A]
do for j  length[A] downto i + 1
Comparisons:  do if A[j] < A[j -1] Exchanges:  n2/2
n2/2 then exchange A[j]  A[j-1]
n n n

T(n) =c1(n+1) +c2  (n  i  1)  c3  (n  i )  c4  (n  i )

i 1
i 1 i 1
n

= (n) +(c2 + c2 +  (n  i )
i 1
n cn 4) n n ( n  1) n 2
n
 ( n  i )  n   i n 
2
 
i 1 i 1 i 1 2 2 2
T(n) = (n2) 12
 Sorting
• Selection sort
– Design approach: incremental
– Sorts in place: Yes
– Running time: (n2)

• Merge Sort
– Design approach: divide and conquer
– Sorts in place: No
– Running time: (nlgn)

13
 Analysis of Selection Sort
Alg.: SELECTION-SORT(A) cost times

n ← length[A] c1 1

for j ← 1 to n - 1 c2 n

do smallest ← j c3 n-1
n2/2 c4 n 1

comparisons for i ← j + 1 to n  j 1
(n  j  1)
c5 n 1
do if A[i] < A[smallest]  j 1
(n  j )
n c6 n 1
exchanges then smallest ← i  j 1
(n  j )
c7 n-1
exchange A[j] ↔ A[smallest]
14
 MERGE-SORT Running Time
• Divide:
– compute q as the average of p and r: D(n) = (1)
• Conquer:
– recursively solve 2 subproblems, each of size n/2 
2T (n/2)
• Combine:
– MERGE on an n-element subarray takes (n) time 
C(n) = (n)
(1) if n =1
T(n) = 2T(n/2) + (n) if n > 1

15
 Quicksort
• Quicksort
Partition the array A into 2 subarrays A[p..q] and A[q+1..r],
– Idea: such that each element of A[p..q] is smaller than or equal
to each element in A[q+1..r]. Then sort the subarrays
recursively.

– Design approach: Divide and conquer

– Sorts in place: Yes

– Best case: (nlgn)

– Worst case: (n2)

• Partition
– Running time (n)
(nlgn) – on average
• Randomized Quicksort (n2) – in the worst case
16
Analysis of Quicksort

17
 The Heap Data Structure
• Def: A heap is a nearly complete binary tree
with the following two properties:
– Structural property: all levels are full, except
possibly the last one, which is filled from left to right
– Order (heap) property: for any node x
Parent(x) ≥ x

7 4
5 2

Heap

18
 Array Representation of Heaps
• A heap can be stored as an
array A.
– Root of tree is A[1]
– Parent of A[i] = A[ i/2 ]
– Left child of A[i] = A[2i]
– Right child of A[i] = A[2i + 1]
– Heapsize[A] ≤ length[A]
• The elements in the subarray
A[(n/2+1) .. n] are leaves
• The root is the max/min
element of the heap
A heap is a binary tree that is filled in order
19
 Operations on Heaps
(useful for sorting and priority queues)
– MAX-HEAPIFY O(lgn)
– BUILD-MAX-HEAP O(n)
– HEAP-SORT O(nlgn)
– MAX-HEAP-INSERT O(lgn)
– HEAP-EXTRACT-MAX O(lgn)
– HEAP-INCREASE-KEY O(lgn)
– HEAP-MAXIMUM O(1)
– You should be able to show how these algorithms
perform on a given heap, and tell their running time 20
Lower Bound for Comparison Sorts
Theorem: Any comparison sort algorithm requires
(nlgn) comparisons in the worst case.
Proof: How many leaves does the tree have?
– At least n! (each of the n! permutations if the input appears as
some leaf)  n!
– At most 2h leaves

h
 n! ≤ 2h
 h ≥ lg(n!) = (nlgn)
leaves

21
 Linear Time Sorting
• We can achieve a better running time for sorting
if we can make certain assumptions on the input
data:
– Counting sort:
• Each of the n input elements is an integer in the

range [0 ... r]
• r=O(n)

22
 Analysis of Counting Sort
Alg.: COUNTING-SORT(A, B, n, k)
1. for i ← 0 to r (r)
2. do C[ i ] ← 0
3. for j ← 1 to n
(n)
4. do C[A[ j ]] ← C[A[ j ]] + 1
5. C[i] contains the number of elements equal to i
6. for i ← 1 to r
(r)
7. do C[ i ] ← C[ i ] + C[i -1]
8. C[i] contains the number of elements ≤ i
9. for j ← n downto 1
10. do B[C[A[ j ]]] ← A[ j ] (n)
11. C[A[ j ]] ← C[A[ j ]] - 1
Overall time: (n + r) 23
 RADIX-SORT
Alg.: RADIX-SORT(A, d)
for i ← 1 to d
do use a stable sort to sort array A on digit i
• 1 is the lowest order digit, d is the highest-order digit

(d(n+k))

24
 Analysis of Bucket Sort
Alg.: BUCKET-SORT(A, n)

for i ← 1 to n O(n)

do insert A[i] into list B[nA[i]]

for i ← 0 to n - 1
(n)
do sort list B[i] with quicksort sort

concatenate lists B[0], B[1], . . . , B[n -1]

O(n)
together in order

return the concatenated lists

(n)
25