Algorithms Analysis & Design

Analysis I
(Normal Code: statements, conditions, loops)
 PRE-TEST
• Two solutions to calculate the variance… Which is Faster?
ALGORITHM1 ALGORITHM2
Variance(A[], N) Variance(A[], N)
var = 0 var = 0
for i= 1 to N M = Mean(A,N)
var += (A[i] – Mean(A,N)) * for i= 1 to N
(A[i] – Mean(A,N)) var += Pow(A[i] – M, 2)
return var return var

• What’s the Worst- & Best-Case Complexity of Each?

 Agenda
• Space & Time Trade-offs
• Running Time vs. Order PART I
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes PART II
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 Learning Outcomes

• Give examples that illustrate time-space trade-offs of algorithms.

• State the formal definition of order (big O).

• Explain what is meant by “best”, “expected”, and “worst” case

behavior of an algorithm.
• Determine informally the time complexity of simple algorithms.

• List and contrast standard complexity classes.

 References
• Chapter 2
– Section 2.1
– Section 2.2
 Algorithmic Performance
• Time Analysis
• Instructions take time.
• How fast does the algorithm perform?
• What affects its runtime?
• Space Analysis
• Data structures take space.
• What kind of data structures can be used?
• How does choice of data structure affect the runtime?
 We will focus on time:
1. How to estimate the required time of an algorithm  [ANALYSIS]
2. How to reduce the required time  [DESIGN]
6
 Space & Time Trade-offs
• Sometimes we increase space to reduce time
• Ex.: Fibonacci 0, 1, 1, 2, 3, 5, 8, 13, 21, …
F(0)=0, F(1)=1 F(n)=F(n-1)+F(n-2) for n>1
ALGORITHM1 ALGORITHM2
F(n) F(n)
if n≤1 return n F[0] = 0; F[1] = 1
else for i = 2 to n do
return F(n-1)+F(n-2) F[i] = F[i-1] + F[i-2]
return F[n]
Time = O(2N) Time = O(N)
Space = O(1) Space = O(N)
 Agenda
• Space & Time Trade-offs
• Running Time vs. Order PART I
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes PART II
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 Running Time vs. Order
FIRST: Running Time T(N)
– Counting # steps that the algorithm takes as a function in the input size
– Each operation in an algorithm has a cost.

Ex.1: Sequence of Operations

count = count + 1  take a certain amount of time, but it is constant
Cost: c1
sum = sum + count Cost: c2

 Total Cost = c1 + c2
 Running Time vs. Order
FIRST: Running Time T(N)
Ex.2: Simple Loop
Cost Times
i = 1; c1 1
sum = 0; c2 1
while (i ≤ n) c3 n+1
{
i = i + 1; c4 n
sum = sum + i; c5 n
}
T(N) = c1 + c2 + (n+1)×c3 + n×c4 + n×c5
 The time required for this algorithm is proportional to n
 Running Time vs. Order
SECOND: Order/Complexity O(…)
– Max dominant factor in the running time without any constants
– Examples:
1. T(N) = c1 + c2 + (n+1)×c3 + n×c4 + n×c5  O(N)
2. T(N) = 2xN2 + 106xN + 1000  O(N2)
3. T(N) = (20xN)7  O(N7)
4. T(N) = 108  O(1)
5. T(N) = log(N100) + 1010  O(log(N))
Note: log(NA) = A x log(N)

Refer to this doc for a review on a set of important math. rules

 Running Time vs. Order
SECOND: Order/Complexity O(…)

Is it rationale to use order O() instead of T()?

In time analysis… What we care about?

(small or large input size)
 Running Time vs. Order
SECOND: Order/Complexity O(…) use this online graph representation to compare different functions
103 log(N) vs N
At large input size:
– Constants are not important …
N T(N) = 106 Log2 N T(N) = 20 N Comparison
8 3 x 106 160 O(N) is better
106 20 x 106 20 x 106 Equals
109 30 x 106 20 x 109 O(Log2 N) is better

– Largest factor is dominating…

N T(N) = N2 + N Log N + 10 N O(N2) T(N) / O(N)
10 210 100 2.1
1000 106 + 1000x3 + 10x1000 = 106 + 13,000 106 1.013
 Agenda
• Space & Time Trade-offs
• Running Time vs. Order
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 How to Calculate the Order O()?
• Statements:
– O(1)
– Exception for the function calls!
• Conditions:
– Chose the max body order
– Take care of the condition(s)!
• Loops:
– Number of iterations × body order
 How to Calculate the Order O()?
Ex.2: Simple Loop [Revisited]
Type Order
i = 1; statement 1
sum = 0; statement 1
while (i ≤ n) loop #iterations x body
{
i = i + 1 statement 1
sum= sum + i statement 1
}

#iterations = N, Body O(1)  Final Order is O(N)

 How to Calculate the Order O()?
Ex.3: Does any of array A or array B contains element T?
for I = 1 to N
If A[I] == T return True

for I = 1 to N
If B[I] == T return True

return False
 How to Calculate the Order O()?
Ex.4: Do array A & array B have a number in common?
for I = 1 to N
for J = 1 to N
If A[I] == B[J]
return True

return False
 How to Calculate the Order O()?
Ex.5: Does array A have duplicate entries?
for I = 1 to N
for J = I + 1 to N
If A[I] == A[J]
return True

return False
Ex.6: Test yourself? (#1)
For the following code, answer the give questions
1: Fact = 1
2: For I = 1 to Max(1000, N)
3: Fact = Fact x I
4: End for
Ex.6: Test yourself? (#2) 2) What's the upper 3) What's the final order
For the following code, answer the give questions bound (i.e. max) of the entire code?
i = 1; number of iterations
sum = 0;
for the inner loop?
while (i <= n)
{
j=1;
while (j <= i)
{
sum = sum + i;
j = j + 1;
}
i = i + 1;
}
1) What's the number of iterations for outer loop?
Ex.6: Test yourself? (#3) 2) What's the lower 3) What's the final order
bound (i.e. min) numberof the entire code?
For the following code, answer the give questions
i=1;
of comparisons for the
sum = 0; inner loop?
j=1;
while (i <= n) {
while (j <= n) {
sum = sum + i;
j = j + 1;
}
i = i + 1;
}
1) What's the number of iterations for outer loop?
 Agenda
• Space & Time Trade-offs
• Running Time vs. Order PART I
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes PART II
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 Worst, Best, Expected Cases
• Worst case:
– A situation (i.e. input) that leads the algorithm to behave at its worst time

• Best case:
– A situation (i.e. input) that leads the algorithm to behave at its best time

• Expected (Average) case:

– The expected time of the algorithm when it run over random
(unexpected) input
 Worst, Best, Expected Cases
Ex.8: Linear search for an item in the array
– Best case: item is found at first place  O(1)
– Worst case: item is not found  O(N)
– Expected (Average) case: item is found at any place
• To find it at any location (i):
– Equally like probability = 1/N
– Number of comparisons = i
– Order:

Refer to this doc for a review on a set of important math. rules

 Agenda
• Space & Time Trade-offs
• Running Time vs. Order
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 Common Complexity Classes
Function Growth Rate Name
c Constant
log N Logarithmic
log2N Log-squared
N Linear
N log N
N2 Quadratic
N3 Cubic
2N Exponential
 Common Complexity Classes
Running times for small inputs

29
 Common Complexity Classes
Running times for moderate inputs

30
 More Examples
Ex.9:
sum = 0;
for(i = 1 ; i < N; i*=2)
{
sum += i;
}
 Agenda
• Space & Time Trade-offs
• Running Time vs. Order
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
INSERTION-SORT

Sorting a hand of cards using insertion sort

33
 Insertion Sort
• Iteration i: repeatedly swap element i with the one to its left if
smaller.

• Property: after ith iteration, a[0] through a[i] contain first i+1
elements in ascending order.

1 i j n

A:

key
sorted
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 7.42 0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 1: step 0.

35
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 0.56
7.42 7.42
0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 2: step 0.

36
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 2.78
0.56 0.56 7.42 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 2: step 1.

37
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 2.78 7.42 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 2: step 2.

38
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 2.78 1.12
7.42 7.42
1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 3: step 0.

39
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12
2.78 2.78
1.12 7.42 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 3: step 1.

40
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 2.78 7.42 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 3: step 2.

41
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 2.78 1.17
7.42 7.42
1.17 0.32 6.21 4.42 3.14 7.71

Iteration 4: step 0.

42
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17
2.78 2.78
1.17 7.42 0.32 6.21 4.42 3.14 7.71

Iteration 4: step 1.

43
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 2.78 7.42 0.32 6.21 4.42 3.14 7.71

Iteration 4: step 2.

44
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 2.78 0.32
7.42 7.42
0.32 6.21 4.42 3.14 7.71

Iteration 5: step 0.

45
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 0.32
2.78 2.78
0.32 7.42 6.21 4.42 3.14 7.71

Iteration 5: step 1.

46
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 0.32
1.17 1.17
0.32 2.78 7.42 6.21 4.42 3.14 7.71

Iteration 5: step 2.

47
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.32
1.12 1.12
0.32 1.17 2.78 7.42 6.21 4.42 3.14 7.71

Iteration 5: step 3.

48
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.56
0.32 0.32 1.12 1.17 2.78 7.42 6.21 4.42 3.14 7.71

Iteration 5: step 4.

49
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 7.42 6.21 4.42 3.14 7.71

Iteration 5: step 5.

50
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21
7.42 7.42
6.21 4.42 3.14 7.71

Iteration 6: step 0.

51
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21 7.42 4.42 3.14 7.71

Iteration 6: step 1.

52
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21 4.42
7.42 7.42
4.42 3.14 7.71

Iteration 7: step 0.

53
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42
6.21 6.21
4.42 7.42 3.14 7.71

Iteration 7: step 1.

54
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 6.21 7.42 3.14 7.71

Iteration 7: step 2.

55
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 6.21 3.14
7.42 7.42
3.14 7.71

Iteration 8: step 0.

56
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 3.14
6.21 6.21
3.14 7.42 7.71

Iteration 8: step 1.

57
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14
4.42 4.42
3.14 6.21 7.42 7.71

Iteration 8: step 2.

58
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71

Iteration 8: step 3.

59
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71

Iteration 9: step 0.

60
 Insertion Sort

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71

Iteration 10: DONE.

61
 Insertion Sort
INSERTION-SORT (A, n) ⊳ A[1 . . n]
for j = 2 to n do
key = A[j]
i = j –1
while i > 0 and A[i] > key do
A[i+1] = A[i]
i = i –1
A[i+1] = key

• #iterations of outer for loop = N-1

• #iterations of inner while loop = ??
• All others are statements of O(1)
 Insertion Sort
Best case
• already sorted numbers
• inner loop body never executed (i.e. O(1))
• while loop always break at conditon A[i] > key
• Final order: O(N)

Worst case
• reversely sorted numbers
• inner loop body executed for all previous elements (i.e. O(N) as upper
bound)
• while loop always break at 1st condition i > 0
• Final order: O(N2)
 Agenda
• Space & Time Trade-offs
• Running Time vs. Order
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 WARM-UP COMPETITION – BONUS
Document Distance:
• Given: two documents D1 & D2
• Req.: Calculate distance between them d(D1, D2)?
CASE INSENSITIVE
• Apps:
– Search for similar documents
– Detecting duplicates (e.g. Wikipedia mirrors)
– Plagiarism detection
• Definitions:
– Word = sequence of alpha-numeric characters ONLY
– Document = sequence of words (ignore space, punctuation, …etc.)
• Treat all upper-case letters as if they are lower-case, so “Cat" & “cat" are same
• Word end at a non-alphanumeric char, so "can't" contains 2 words: "can" & "t"
 WARM-UP COMPETITION – BONUS
Document Distance:
• Idea: define distance in terms of shared words.
– Think of document D as a vector
– D[w] = # occurrences of word w
• Example: D1 = “the cat” D2 = “the dog”
– D1[“the”] = 1, D1[“cat”] = 1
– D2[“the”] = 1, D2[“dog”] = 1
• d(D1,D2) = angle between 2 vectors.
– 0◦: identical, 90◦: no common words
 WARM-UP COMPETITION – BONUS
Document Distance:
• Algorithm:
1. Split each document into words
2. Count word frequencies (document vectors)
• NOTE: Max Freq. Value BOUNDED BY 100,000

3. Compute distance
 WARM-UP COMPETITION – BONUS
Document Distance:
• Competition Rules:
– Startup Code: C# Template – Check LMS
– Criteria:
1. Correct O/P
2. Execution time
– Delivery:
• DUE TO: SUN 19 FEB 23:59
• Submit “DocDistance.cs” to the LMS
– Bonuses:
1. TOP 50
2. TOP 5
 Agenda
• Space & Time Trade-offs
• Running Time vs. Order
• How to Calculate the Order?
• Worst, Best, Expected Cases
• Complexity Classes
• Insertion Sort
• Warm-up Competition [bonus]
• Summary & Questions
IMP: Refer to this doc for a review on a set of important math. rules
 POST-TEST
• Compare Expected Time?
ALGORITHM1 ALGORITHM2
Variance(A[], N) Variance(A[], N)
var = 0 var = 0
for i= 1 to N M = Mean(A,N)
var += (A[i] – Mean(A,N)) * for i= 1 to N
(A[i] – Mean(A,N)) var += Pow(A[i] – M, 2)
return var return var

• What’s the Worst- & Best-Case Complexity of Each?

 Summary
• Sometimes: increase space to reduce time
• This course about time!
– how to calculate? [Analysis]
– How to reduce? [Design]
• Guiding Principles:
1. At large input: use Order O()! Don’t pay attention to constants & lower orders
2. Worst case analysis  useful for “general-purpose” routines
• To calculate order:
– Statements  O(1) Conditions  O(Max Body) Loops  O(Body) x #iterations
• Insertion Sort:
– Fast at small input
– Worst & Average O(N2), Best O(N)
 Questions
• Test yourself @this online quiz.
– Questions: 1, 2, 3
– Answers are available