Elementary analysis

Huynh Tuong Nguyen

Chapter 1
Elementary analysis
Contents
Advanced algorithms on March 3, 2021
Definition and
notations
Algorithm
Complexity
Formulas

Basic methods for

asymptotic behaviour
analysis
Counting number of
elementary operations
Solving recurrences

Huynh Tuong Nguyen

Faculty of Computer Science and Engineering
University of Technology - VNUHCM
Ch.1, p.1/32
 Elementary analysis
Contents
Huynh Tuong Nguyen

1 Definition and notations Contents

Algorithm Definition and
notations
Complexity Algorithm

Formulas Complexity
Formulas

Basic methods for

asymptotic behaviour
analysis
Counting number of
2 Basic methods for asymptotic behaviour analysis elementary operations
Solving recurrences
Counting number of elementary operations
Solving recurrences

Ch.1, p.2/32
 Elementary analysis
Algorithm
Huynh Tuong Nguyen

What is an algorithm?
An algorithm is a finite set of precise instructions for performing a
computation or for solving a problem. Contents

Definition and
notations
Properties of algorithms Algorithm
Complexity

• Input from a specified set, Formulas

Basic methods for

• Output from a specified set (solution), asymptotic behaviour
analysis
• Definiteness of every step in the computation, Counting number of
elementary operations

• Correctness of output for every possible input, Solving recurrences

• Finiteness of the number of calculation steps,

• Effectiveness of each calculation step and
• Generality for a class of problems.

Ch.1, p.3/32
 Elementary analysis
Complexity
Huynh Tuong Nguyen

• Generally, not much interested in time and space complexity for

small inputs.
• Given two algorithms A and B for solving problem P .
Contents

Definition and
notations
Input size Algorithm A Algorithm B
Algorithm
n 5000 n 1.2n Complexity

10 50,000 6 Formulas

Basic methods for

100 500,000 2,817,975 asymptotic behaviour
1, 000 5,000,000 1.5 × 1079 analysis
Counting number of
100, 000 5 × 108 1.3 × 107918 elementary operations
Solving recurrences

• B cannot be used for large inputs, while A is still feasible.

• So what is important is the growth of the complexity functions.
• Growth of time and space complexity with increasing input size n
is a suitable measure for the comparison of algorithms.

Ch.1, p.4/32
 Elementary analysis
Types of formulas for basic operation count
Huynh Tuong Nguyen

• Exact formulas, e.g., C(n) = n(n − 1)/2.

• Formula indicating order of growth with specific multiplicative
constant e.g., C(n) ≈ 0.5n2 .
Contents
• Formula indicating order of growth with unknown multiplicative Definition and
constant e.g., C(n) ≈ c.n2 notations
Algorithm

• Most important: Order of growth within a constant multiple as Complexity

Formulas
n→∞ Basic methods for
asymptotic behaviour
analysis
Asymptotic growth rate Counting number of
elementary operations
Solving recurrences
A way of comparing functions that ignores constant factors and
small input sizes
• O(g(n)): class of functions f (n) that grow no faster than g(n)
• Θ(g(n)): class of functions f (n) that grow at the same rate as
g(n)
• Ω(g(n)): class of functions f (n) that grow at least as fast as g(n)

Ch.1, p.5/32
 Elementary analysis
Complexity classes - a small vocabulary
Huynh Tuong Nguyen

• Constant: O(1) (independing on the input size)

Contents
• Sub-linear or logarithmic: O(log n)
Definition and
• Linear: O(n) notations
Algorithm

• Quasi-linear: O(n log n) Complexity

Formulas

• Quadratic: O(n )
2
Basic methods for
asymptotic behaviour
• Cubic: O(n3 ) analysis
Counting number of
• Polynomial: O(np ) (O(n2 ), O(n3 ), etc) elementary operations
Solving recurrences

• Quasi-polynomial: O(nlog(n) )
• Exponential: O(2n )
• Factorial: O(n!)

Ch.1, p.6/32
 Elementary analysis
Asymptotic upper bound - worst case
Huynh Tuong Nguyen

Asymptotic upper bound “big O”

T (n) = O(f (n)) iif ∃c ∈ R+ , c > 0 and ∃n0 ∈ N , n0 > 0 Contents

such that ∀n > n0 : T (n) ≤ c × f (n) Definition and

notations
Algorithm
Complexity
Example Formulas

Let T (n) = 2n + 3n3 + 5. T (n) is in O(n3 ) with: Basic methods for

asymptotic behaviour
analysis
• (c = 10 and n0 = 1) or (c = 5 and n0 = 2) Counting number of
elementary operations
Solving recurrences

Principle: the lower-order terms are negligible.

Ch.1, p.7/32
 Elementary analysis
Asymtotic lower bound - best case
Huynh Tuong Nguyen

“big Omega” Contents

+ Definition and
T (n) = Ω(f (n)) iif ∃c ∈ R , c > 0 and ∃n0 ∈ N , n0 > 0 notations

such that ∀n > n0 : T (n) ≥ c × f (n) Algorithm

Complexity
Formulas

Basic methods for

Example asymptotic behaviour
analysis
Let T (n) = 2n + 3n3 + 5. T (n) is in Ω(n3 ) with: Counting number of
elementary operations
• (c = 1 and n0 = 1) Solving recurrences

Ch.1, p.8/32
 Elementary analysis
Asymtotic approximating bound - average case
Huynh Tuong Nguyen

“big Theta”
T (n) = Θ(f (n)) iif ∃c1 , c2 ∈ R+ , c1 > 0, c2 > 0 and ∃n0 ∈ N ,
Contents
n0 > 0 Definition and
such that ∀n > n0 : c1 × f (n) ≤ T (n) ≤ c2 × f (n) notations
Algorithm
Complexity
Formulas
Property
Basic methods for
asymptotic behaviour
T (n) = O(f (n)) and T (n) = Ω(f (n)) =⇒ T (n) = Θ(f (n)) analysis
Counting number of
elementary operations

Example Solving recurrences

Let T (n) = 2n + 3n3 + 5.

So, T (n) is in O(n3 ) and in Ω(n3 ).
Consequently, T (n) is in Θ(n3 ).

Ch.1, p.9/32
 Elementary analysis
Other properties
Huynh Tuong Nguyen

Not transitive
• f (n) = n2 ; g(n) = n
• ⇒ f (n) = O(n2 ) = g(n) but f (n) 6= g(n)
Contents

Definition and
Transitivity notations
Algorithm

• f (n) = O(g(n)) & g(n) = O(h(n)) ⇒ f (n) = O(h(n)) Complexity

Formulas

• f (n) = Ω(g(n)) & g(n) = Ω(h(n)) ⇒ f (n) = Ω(h(n)) Basic methods for
asymptotic behaviour
analysis
• f (n) = Θ(g(n)) & g(n) = Θ(h(n)) ⇒ f (n) = Θ(h(n)) Counting number of
elementary operations
Solving recurrences

Additivity

• f (n) = O(h(n)) & g(n) = O(h(n)) ⇒ f (n) + g(n) = O(h(n))

• f (n) = Ω(h(n)) & g(n) = Ω(h(n)) ⇒ f (n) + g(n) = Ω(h(n))
• f (n) = Θ(h(n)) & g(n) = Θ(h(n)) ⇒ f (n) + g(n) = Θ(h(n))

Ch.1, p.10/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Contents

Definition and
notations
Algorithm
a T (n) = 3 + 5n2 ⇒ T (n) = Θ(n2 ) ? Complexity
Formulas

2n + 5 when n is even
b if T (n) = , then T (n) = O(?)
n2 − n + 1 when n is odd Basic methods for
asymptotic behaviour
and T (n) = Ω(?). analysis
Counting number of
elementary operations
Solving recurrences

Ch.1, p.11/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Compare the asymptotic behaviours of

Contents
1 2n and 10n Definition and
notations
2 log2 n and log10 n Algorithm
Complexity
Formulas

Basic methods for

asymptotic behaviour
analysis
1 Prove that for any positive functions f and g, f (n) + g(n) Counting number of

and max(f (n); g(n)) are asymptotically equivalent. elementary operations

Solving recurrences

2 Give a (necessary and sufficient) condition on positive

functions f and g to ensure that f (n) + g(n) and f (n) are
asymptotically equivalent.

Ch.1, p.12/32
 Elementary analysis
Common asymptotic behaviours
Huynh Tuong Nguyen

Contents

Definition and
Size Approximate computational time notations
n Θ(log n) Θ(n) Θ(n log n) Θ(n2 ) Θ(2n ) Θ(n!) Algorithm

10 3.10−9 s 10−8 s 3.10−8 s 10−7 s 10−6 s 3.10−3 s Complexity

102 7.10−9 s 10−7 s 7.10−7 s 10−5 s 4.1013 y

Formulas
*
103 10−8 s 10−6 s 10−5 s 10−3 s * * Basic methods for
asymptotic behaviour
104 1, 3.10−8 s 10−5 s 10−4 s 10−1 s * * analysis

105 1, 7.10−8 s 10−4 s 2.10−3 s 10s * * Counting number of

elementary operations
106 2.10−8 s 10−3 s 2.10−2 s 17m * * Solving recurrences

Ch.1, p.13/32
 Elementary analysis
Example
Huynh Tuong Nguyen

Contents
1. (1)Var int: d = 0 Definition and
notations
2. (n)For i from 1 to n do Algorithm

1. (1)d = d + 1 Complexity
Formulas
2. (1)a[i] = a[i] × a[i] + d × d
Basic methods for
asymptotic behaviour
3. Endfor analysis
Counting number of
elementary operations
Solving recurrences

Number of elementary operations: 1 + n × (1 + 1) = 2n + 1.

Ch.1, p.14/32
 Elementary analysis
Linear loop example
Huynh Tuong Nguyen

Contents

1. Var int: i = 1 1. Var int: i = n Definition and

notations
2. While i ≤ n do 2. While i ≥ 1 do Algorithm
Complexity

1. Write “Bonjour” 1. Write “Bonjour” Formulas

2. i=i+1 2. i=i−1 Basic methods for

asymptotic behaviour
analysis
3. EndWhile 3. EndWhile Counting number of
elementary operations
Solving recurrences

Number of elementary operations: 2n + 1.

Ch.1, p.15/32
 Elementary analysis
Logarithmic loop example
Huynh Tuong Nguyen

Contents

1. Var int: i = 1 1. Var int: i = n Definition and

notations
2. While i ≤ n do 2. While i ≥ 1 do Algorithm
Complexity

1. Write “Bonjour” 1. Write “Bonjour” Formulas

2. i=i×2 2. i = i/2 Basic methods for

asymptotic behaviour
analysis
3. EndWhile 3. EndWhile Counting number of
elementary operations
Solving recurrences

Number of elementary operations: 1 + 2 log2 (n).

Ch.1, p.16/32
 Elementary analysis
Nested loop example
Huynh Tuong Nguyen

Nb of iterations = nb of iterations of external loop × nb of iterations of

internal loop
Contents

Definition and
notations
Algorithm

1. Var int: i = 1 Complexity

Formulas

2. While i ≤ n do Basic methods for

asymptotic behaviour
1. Var int: j = 1 analysis

2. While j ≤ n do Number of elementary Counting number of

elementary operations

1. Write “Bonjour” operations: Solving recurrences

2. j = j × 3 1 + 2n + 2n × log3 (n).
3. EndWhile
4. i=i+1
3. EndWhile

Ch.1, p.17/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Function XYZ(array: a[])

Contents
1. (1)Var int: i
Definition and
notations
2. (n)For i from 1 to n do Algorithm
Complexity
1. (1)Var int: t = a[i] Formulas

2. (1)Var int: j Basic methods for

3. (?)For j from i − 1 to 0 do asymptotic behaviour
analysis
1. (1)a[j + 1] = a[j] Counting number of
elementary operations

4. EndFor Solving recurrences

5. (1)a[j + 1] = t
3. EndFor

Ch.1, p.18/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Give algorithms having number of elementary operations as below.

• T1 (n) = 3 + 5n
Contents
• T2 (n) = n log2 n
Definition and
notations
• T3 (n) = n3
Algorithm

• T4 (n) = (3n)! Complexity

Formulas
• T5 (n) = log2 (3n) Basic methods for
asymptotic behaviour
• T6 (n) = 2 log3 (2n) analysis
Counting number of
• T7 (n) = n2 log4 n elementary operations

√ Solving recurrences
• T8 (n) = n
√
• T9 (n) = 3 n2
• T10 (n) = 2n
• T11 (n) = n!

Ch.1, p.19/32
 Elementary analysis
Solving recurrences
Huynh Tuong Nguyen

Logarithmic recurrence
T (n) = 1 + T (n/2) = 1 + (1 + T (n/4)) Contents

= k + T (1) where 2k ≤ n < 2k+1 , i.e. k = dlog2 ne Definition and

notations
is O(log2 n) Algorithm
Complexity
Formulas

Linear recurrence Basic methods for

asymptotic behaviour
T (n) = n + T (n/2) = n + (n/2 + T (n/4)) analysis

= n + n/2 + . . . + n/2k + T (n/2k ) Counting number of

elementary operations

= n(2 − 1/2k ) + T (1) Solving recurrences

(where 2k ≤ n < 2k+1 , i.e. k = dlog2 ne)

is O(n)

Ch.1, p.23/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Determine the recurrence relation of the following algorithm.

Function Factorial(n: integer): integer

1. If n ≤ 1 then
Contents
1. Return 1
Definition and
notations
2. Else Algorithm

1. Return n × Factorial(n-1) Complexity

Formulas

3. EndIf Basic methods for

asymptotic behaviour
analysis
Counting number of
Function SumUp(n, a[]: integer): integer elementary operations
Solving recurrences

1. If a[n] ≤ 1 then
1. Return 1

2. Else
1. Return a[n] + SumUp(n-1,a[])
3. EndIf

Ch.1, p.24/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Determine complexity of algorithms having following recurrence Contents

relations Definition and

notations
Algorithm
1. T1 (n) = 2T1 (n/2) + 1 Complexity
Formulas
2. T2 (n) = 2T2 (n/2) + n
Basic methods for
3. T3 (n) = T3 (n − 1) + 3n + 1 asymptotic behaviour
analysis
4. T4 (n) = T4 (n − 2) + n2 + 2 Counting number of
elementary operations

5. T5 (n) = T5 (n − 1) + T5 (n − 2) + 2 Solving recurrences

Ch.1, p.25/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Determine corresponding recurrences

• Constant: O(1) (independing on the input size)
Contents
• Sub-linear or logarithmic: O(log n) Definition and
notations
• Linear: O(n) Algorithm
Complexity
• Quasi-linear: O(n log n) Formulas

• Quadratic: O(n2 ) Basic methods for

asymptotic behaviour
• Cubic: O(n3 ) analysis
Counting number of

• Polynomial: O(np ) (O(n2 ), O(n3 ), etc) elementary operations

Solving recurrences

• Quasi-polynomial: O(nlog(n) )
• Exponential: O(2n )
• Factorial: O(n!)

Ch.1, p.26/32
 Elementary analysis
Exercise
Huynh Tuong Nguyen

Contents

Definition and
notations
√ Algorithm
Solve recurrence f (n) = 5f ( n) + log3 n.
3
Complexity

(Hint: change variables, with g(m) = f (3m ).) Formulas

Basic methods for

asymptotic behaviour
analysis
Counting number of
elementary operations
Solving recurrences

Ch.1, p.32/32