Discrete Mathematics and Its Application 2

(CS147)
Lecture 2&3: Algorithm analysis: worst-case asymptotic runing time, big-O notation

Fanghui Liu

Department of Computer Science, University of Warwick, UK

Centre for Discrete Mathematics and its Applications (DIMAP), Warwick
Target
Analyzing the runtime of an algorithm for a computational problem using Big-O notation.

CS147 | Fanghui Liu, [email protected] Slide 2/ 34

Target
Analyzing the runtime of an algorithm for a computational problem using Big-O notation.

▶ Q1: What is a computational problem?

CS147 | Fanghui Liu, [email protected] Slide 2/ 34

Target
Analyzing the runtime of an algorithm for a computational problem using Big-O notation.

▶ Q1: What is a computational problem?

▶ Q2: What is an algorithm?

CS147 | Fanghui Liu, [email protected] Slide 2/ 34

Target
Analyzing the runtime of an algorithm for a computational problem using Big-O notation.

▶ Q1: What is a computational problem?

▶ Q2: What is an algorithm?
▶ Q3: How to define an algorithm’s runtime?

CS147 | Fanghui Liu, [email protected] Slide 2/ 34

Target
Analyzing the runtime of an algorithm for a computational problem using Big-O notation.

▶ Q1: What is a computational problem?

▶ Q2: What is an algorithm?
▶ Q3: How to define an algorithm’s runtime?
▶ Q4: What is big-O notation? Why do we need big-O notation?

CS147 | Fanghui Liu, [email protected] Slide 2/ 34

Computational problem (decision problem)

Example
Input - An array A[1, 2, . . . , n] of n numbers.
⇔ A[1], A[2], A[3], · · · , A[n]
For each i ∈ {1, 2, · · · , n}, A[i] is a real number.

CS147 | Fanghui Liu, [email protected] Slide 3/ 34

Computational problem (decision problem)

Example
Input - An array A[1, 2, . . . , n] of n numbers.
Output
▶ Yes if there exist indices i, j ∈ {1, 2, · · · , n} with i , j such that A[i] + A[j] = 0.
▶ No otherwise.

CS147 | Fanghui Liu, [email protected] Slide 3/ 34

Computational problem (decision problem)

Example
Input - An array A[1, 2, . . . , n] of n numbers.
Output
▶ Yes if there exist indices i, j ∈ {1, 2, · · · , n} with i , j such that A[i] + A[j] = 0.
▶ No otherwise.

Computational problem P (say)

◦ the class of decision problems that are solvable in polynomial time

CS147 | Fanghui Liu, [email protected] Slide 3/ 34

Computational problem (decision problem)
In the previous example, for all inputs I, either solutions(I)=Yes or Solutions(I) = No. A
computational problem P consists of:

CS147 | Fanghui Liu, [email protected] Slide 4/ 34

Computational problem (decision problem)
In the previous example, for all inputs I, either solutions(I)=Yes or Solutions(I) = No. A
computational problem P consists of:

▶ decision problem: the answer for every input is either yes or no

CS147 | Fanghui Liu, [email protected] Slide 4/ 34

Computational problem (decision problem)
In the previous example, for all inputs I, either solutions(I)=Yes or Solutions(I) = No. A
computational problem P consists of:

▶ decision problem: the answer for every input is either yes or no

▶ search problem: can be cast as decision problem

CS147 | Fanghui Liu, [email protected] Slide 4/ 34

Computational problem (decision problem)
In the previous example, for all inputs I, either solutions(I)=Yes or Solutions(I) = No. A
computational problem P consists of:

▶ decision problem: the answer for every input is either yes or no

▶ search problem: can be cast as decision problem
▶ counting problem, optimization problem... (not included in CS147)
CS147 | Fanghui Liu, [email protected] Slide 4/ 34
Algorithm

Definition
An algorithm for a computational problem P is a step by step procedure such that given any
input I, outputs a valid solution for I.

CS147 | Fanghui Liu, [email protected] Slide 5/ 34

Algorithm

Definition
An algorithm for a computational problem P is a step by step procedure such that given any
input I, outputs a valid solution for I.

Input Output
Algorithm

▶ Input: I
▶ Output: A valid solution for I

CS147 | Fanghui Liu, [email protected] Slide 5/ 34

An algorithm for our example problem

An example algorithm (Pseudocode)

For i = 1, 2, . . . , n
For j = i + 1, i + 2, . . . , n
If A[i] + A[j] = 0
Return YES.
Return No.
You can implement this algorithm by writing a computer program in C, C++, Java etc.

CS147 | Fanghui Liu, [email protected] Slide 6/ 34

An algorithm for our example problem

An example algorithm (Pseudocode)

For i = 1, 2, . . . , n
For j = i + 1, i + 2, . . . , n
If A[i] + A[j] = 0
Return YES.
Return No.
You can implement this algorithm by writing a computer program in C, C++, Java etc.

CS147 | Fanghui Liu, [email protected] Slide 6/ 34

Algorithms vs. Programs

▶ The same algorithm can be implemented in different programming languages.

CS147 | Fanghui Liu, [email protected] Slide 7/ 34

Algorithms vs. Programs

▶ The same algorithm can be implemented in different programming languages.

▶ An algorithm is an abstract mathematical object, independent of
◦ the programming language it has been implemented in
◦ the machine (computer) it is running on.

CS147 | Fanghui Liu, [email protected] Slide 7/ 34

Runtime of an program and algorithm1

Statement
Runtime of a computer program = the actual time (say, in microseconds) if it takes to finish
execution.

▶ It depends on
◦ the input
◦ the programming language
◦ the machine (computer)

1 good reference: https://web.stanford.edu/class/archive/cs/cs106b/cs106b.1206/lectures/big-o/

CS147 | Fanghui Liu, [email protected] Slide 8/ 34
*Algorithm Analysis: Primitive Operations

Figure: Source from CS106B in Stanford.

CS147 | Fanghui Liu, [email protected] Slide 9/ 34

*Algorithm Analysis: Primitive Operations

Figure: Source from CS106B in Stanford.

▶ currentMax = v[0] has two

operations (variable assignment and
indexing into a Vector) and is executed
once (total operations = 2)

CS147 | Fanghui Liu, [email protected] Slide 9/ 34

*Algorithm Analysis: Primitive Operations

Figure: Source from CS106B in Stanford.

▶ currentMax = v[0] has two

operations (variable assignment and
indexing into a Vector) and is executed
once (total operations = 2)
▶ n = v.size() has two operations
(variable assignment and calling a
function) and is executed once (total
operations so far: = 4)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
*Algorithm Analysis: Primitive Operations
▶ The int i = 1 in the for loop declaration is executed
once, with 1 operation (total = 5)

Figure: Source from CS106B in Stanford.

▶ currentMax = v[0] has two

operations (variable assignment and
indexing into a Vector) and is executed
once (total operations = 2)
▶ n = v.size() has two operations
(variable assignment and calling a
function) and is executed once (total
operations so far: = 4)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
*Algorithm Analysis: Primitive Operations
▶ The int i = 1 in the for loop declaration is executed
once, with 1 operation (total = 5)
▶ The i < n in the for declaration is executed for
checking n times (total: n + 5)

Figure: Source from CS106B in Stanford.

▶ currentMax = v[0] has two

operations (variable assignment and
indexing into a Vector) and is executed
once (total operations = 2)
▶ n = v.size() has two operations
(variable assignment and calling a
function) and is executed once (total
operations so far: = 4)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
*Algorithm Analysis: Primitive Operations
▶ The int i = 1 in the for loop declaration is executed
once, with 1 operation (total = 5)
▶ The i < n in the for declaration is executed for
checking n times (total: n + 5)
▶ The i++ in the for declaration is executed n − 1
times, and is two operations (arithmetic and
assignment) (total: 3n + 3 operations so far)
Figure: Source from CS106B in Stanford.

▶ currentMax = v[0] has two

operations (variable assignment and
indexing into a Vector) and is executed
once (total operations = 2)
▶ n = v.size() has two operations
(variable assignment and calling a
function) and is executed once (total
operations so far: = 4)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
*Algorithm Analysis: Primitive Operations
▶ The int i = 1 in the for loop declaration is executed
once, with 1 operation (total = 5)
▶ The i < n in the for declaration is executed for
checking n times (total: n + 5)
▶ The i++ in the for declaration is executed n − 1
times, and is two operations (arithmetic and
assignment) (total: 3n + 3 operations so far)
Figure: Source from CS106B in Stanford. ▶ if (currentMax < v[i]) is executed n − 1 times
and is two operations (comparison and vector
▶ currentMax = v[0] has two indexing) (total: 5n + 1)
operations (variable assignment and
indexing into a Vector) and is executed
once (total operations = 2)
▶ n = v.size() has two operations
(variable assignment and calling a
function) and is executed once (total
operations so far: = 4)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
*Algorithm Analysis: Primitive Operations
▶ The int i = 1 in the for loop declaration is executed
once, with 1 operation (total = 5)
▶ The i < n in the for declaration is executed for
checking n times (total: n + 5)
▶ The i++ in the for declaration is executed n − 1
times, and is two operations (arithmetic and
assignment) (total: 3n + 3 operations so far)
Figure: Source from CS106B in Stanford. ▶ if (currentMax < v[i]) is executed n − 1 times
and is two operations (comparison and vector
▶ currentMax = v[0] has two indexing) (total: 5n + 1)
operations (variable assignment and ▶ currentMax = v[i] is executed at most n − 1
indexing into a Vector) and is executed times, and is two operations (assignment and vector
once (total operations = 2) indexing) but as few as zero times (if we never have
▶ n = v.size() has two operations to update the max!) (total: either 7n − 1 or 5n + 1)
(variable assignment and calling a
function) and is executed once (total
operations so far: = 4)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
*Algorithm Analysis: Primitive Operations
▶ The int i = 1 in the for loop declaration is executed
once, with 1 operation (total = 5)
▶ The i < n in the for declaration is executed for
checking n times (total: n + 5)
▶ The i++ in the for declaration is executed n − 1
times, and is two operations (arithmetic and
assignment) (total: 3n + 3 operations so far)
Figure: Source from CS106B in Stanford. ▶ if (currentMax < v[i]) is executed n − 1 times
and is two operations (comparison and vector
▶ currentMax = v[0] has two indexing) (total: 5n + 1)
operations (variable assignment and ▶ currentMax = v[i] is executed at most n − 1
indexing into a Vector) and is executed times, and is two operations (assignment and vector
once (total operations = 2) indexing) but as few as zero times (if we never have
▶ n = v.size() has two operations to update the max!) (total: either 7n − 1 or 5n + 1)
(variable assignment and calling a ▶ return currentMax is executed once, and is one
function) and is executed once (total operation (returning from a function) (total
operations so far: = 4) operations: either 7n or 5n + 2)
CS147 | Fanghui Liu, [email protected] Slide 9/ 34
Algorithm Analysis: Simplify!

▶ Do we really need this much detail? Nope!

CS147 | Fanghui Liu, [email protected] Slide 10/ 34

Algorithm Analysis: Simplify!

▶ Do we really need this much detail? Nope!

▶ Let’s simplify: we want a big picture approach.

CS147 | Fanghui Liu, [email protected] Slide 10/ 34

Algorithm Analysis: Simplify!

▶ Do we really need this much detail? Nope!

▶ Let’s simplify: we want a big picture approach.
▶ It is enough to know that vectorMax() grows linearly proportional to n

Claim
For very large values of n, there is no significant difference between 5n and 7n!

CS147 | Fanghui Liu, [email protected] Slide 10/ 34

Algorithm Analysis: Simplify!

▶ Do we really need this much detail? Nope!

▶ Let’s simplify: we want a big picture approach.
▶ It is enough to know that vectorMax() grows linearly proportional to n

Claim
For very large values of n, there is no significant difference between 5n and 7n!

Remove constants and less significant factors!

CS147 | Fanghui Liu, [email protected] Slide 10/ 34

 Big-O notation...

CS147 | Fanghui Liu, [email protected] Slide 11/ 34

 Big-O notation...

Target
Talk about the rate at which some function changes as its argument grows (or shrinks),
without worrying to much about the detailed form.

CS147 | Fanghui Liu, [email protected] Slide 11/ 34

Basic definitions (I)

Consider any two functions f (n) and g(n) satisfying

f (n) > 0, g(n) > 0 for all positive integer n

CS147 | Fanghui Liu, [email protected] Slide 12/ 34

Basic definitions (I)

Consider any two functions f (n) and g(n) satisfying

f (n) > 0, g(n) > 0 for all positive integer n

▶ g(n) ∈ O(f (n)) [Big O of f (n)] if there exist constants c > 0 and N such that
g(n) ≤ cf (n) for all n > N .

CS147 | Fanghui Liu, [email protected] Slide 12/ 34

Basic definitions (I)

Consider any two functions f (n) and g(n) satisfying

f (n) > 0, g(n) > 0 for all positive integer n

▶ g(n) ∈ O(f (n)) [Big O of f (n)] if there exist constants c > 0 and N such that
g(n) ≤ cf (n) for all n > N .
▶ g(n) ∈ Ω(f (n)) [Big Omega of f (n)] if there exist constants c > 0 and N such that
g(n) ≥ cf (n) for all n > N .

CS147 | Fanghui Liu, [email protected] Slide 12/ 34

Basic definitions (I)

Consider any two functions f (n) and g(n) satisfying

f (n) > 0, g(n) > 0 for all positive integer n

▶ g(n) ∈ O(f (n)) [Big O of f (n)] if there exist constants c > 0 and N such that
g(n) ≤ cf (n) for all n > N .
▶ g(n) ∈ Ω(f (n)) [Big Omega of f (n)] if there exist constants c > 0 and N such that
g(n) ≥ cf (n) for all n > N .
▶ g(n) ∈ Θ(f (n)) [Big Theta of f (n)] if g(n) ∈ O(f (n)) and g(n) ∈ Ω(f (n)).
⇔ there exist constants c1 , c2 > 0 and N such that c1 f (n) ≤ g(n) ≤ c2 f (n) for all
n > N.

CS147 | Fanghui Liu, [email protected] Slide 12/ 34

Examples

Example
The function n is in O(n3 ).

CS147 | Fanghui Liu, [email protected] Slide 13/ 34

Examples

Example
The function n is in O(n3 ).

Proof.
Set c = 1, N = 1, then n ≤ cn3 for all n ≥ N . □

CS147 | Fanghui Liu, [email protected] Slide 13/ 34

Examples

Example
The function n is in O(n3 ).

Proof.
Set c = 1, N = 1, then n ≤ cn3 for all n ≥ N . □

Example
The function n3 is not in O(n).

CS147 | Fanghui Liu, [email protected] Slide 13/ 34

Examples

Example
The function n is in O(n3 ).

Proof.
Set c = 1, N = 1, then n ≤ cn3 for all n ≥ N . □

Example
The function n3 is not in O(n).

Proof by contradiction.
3
√ constants c > 0 and N such that n ≤ cn for all n > N . But we have
Suppose that there exist
3
n > cn for all n > c. This leads to a contradiction, and concludes the proof. □

CS147 | Fanghui Liu, [email protected] Slide 13/ 34

Basic definitions (II)

Consider any two functions f (n) and g(n) satisfying

f (n), g(n) > 0 for all positive integer n

CS147 | Fanghui Liu, [email protected] Slide 14/ 34

Basic definitions (II)

Consider any two functions f (n) and g(n) satisfying

f (n), g(n) > 0 for all positive integer n

▶ g(n) ∈ o(f (n)) [little o of f (n)] if for every c > 0, there exists an N such that
g(n) ≤ cf (n) for all n > N .

g(n)
⇔ lim = 0.
n→∞ f (n)
▶ g(n) ∈ ω(f (n)) [little omega of f (n)] if for every c > 0, there exists an N such that
g(n) ≥ cf (n) for all n > N .

g(n)
⇔ lim = ∞.
n→∞ f (n)

CS147 | Fanghui Liu, [email protected] Slide 14/ 34

Intuitions

▶ g(n) ∈ O(f (n)) means g ≤ f “asymptotically".

▶ g(n) ∈ Ω(f (n)) means g ≥ f “asymptotically".
▶ g(n) ∈ Θ(f (n)) means g = f “asymptotically".
▶ g(n) ∈ o(f (n)) means g < f “asymptotically".
▶ g(n) ∈ ω(f (n)) means g > f “asymptotically".

CS147 | Fanghui Liu, [email protected] Slide 15/ 34

Abusing the equals sign

▶ We write g(n) = O(f (n)) to denote g(n) ∈ O(f (n)).

CS147 | Fanghui Liu, [email protected] Slide 16/ 34

Abusing the equals sign

▶ We write g(n) = O(f (n)) to denote g(n) ∈ O(f (n)).

Statement
for all f (n) ∈ O(n2 ) and g(n) ∈ O(n3 ), we have f (n) + g(n) + 1 ∈ O(n3 ).

CS147 | Fanghui Liu, [email protected] Slide 16/ 34

Abusing the equals sign

▶ We write g(n) = O(f (n)) to denote g(n) ∈ O(f (n)).

Statement
for all f (n) ∈ O(n2 ) and g(n) ∈ O(n3 ), we have f (n) + g(n) + 1 ∈ O(n3 ).

Proof.
Prove by definition. □

CS147 | Fanghui Liu, [email protected] Slide 16/ 34

Abusing the equals sign

▶ We write g(n) = O(f (n)) to denote g(n) ∈ O(f (n)).

Statement
for all f (n) ∈ O(n2 ) and g(n) ∈ O(n3 ), we have f (n) + g(n) + 1 ∈ O(n3 ).

Proof.
Prove by definition. □
n2 n3
◦ Example: 2 + 4 = O(n2 ) + O(n3 ) = O(n3 )

CS147 | Fanghui Liu, [email protected] Slide 16/ 34

Abusing the equals sign

▶ We write g(n) = O(f (n)) to denote g(n) ∈ O(f (n)).

Statement
for all f (n) ∈ O(n2 ) and g(n) ∈ O(n3 ), we have f (n) + g(n) + 1 ∈ O(n3 ).

Proof.
Prove by definition. □
2 3
◦ Example: n2 + n4 = O(n2 ) + O(n3 ) = O(n3 )
◦ We can generalize it!

CS147 | Fanghui Liu, [email protected] Slide 16/ 34

Principles: useful simplifications

▶ If g(n) = O(f (n)), we have

O(f (n)) + O(g(n)) = O(f (n)) .

CS147 | Fanghui Liu, [email protected] Slide 17/ 34

Principles: useful simplifications

▶ If g(n) = O(f (n)), we have

O(f (n)) + O(g(n)) = O(f (n)) .

◦ Example: 2n4 + n7 /3 = O(n7 ).

CS147 | Fanghui Liu, [email protected] Slide 17/ 34

Principles: useful simplifications

▶ If g(n) = O(f (n)), we have

O(f (n)) + O(g(n)) = O(f (n)) .

◦ Example: 2n4 + n7 /3 = O(n7 ).

▶ If c is a constant, we have
O(cf (n)) = O(f (n)) .

CS147 | Fanghui Liu, [email protected] Slide 17/ 34

Principles: useful simplifications

▶ If g(n) = O(f (n)), we have

O(f (n)) + O(g(n)) = O(f (n)) .

◦ Example: 2n4 + n7 /3 = O(n7 ).

▶ If c is a constant, we have
O(cf (n)) = O(f (n)) .
◦ Example 1: O(5n2 ) = O(n2 ).

CS147 | Fanghui Liu, [email protected] Slide 17/ 34

Principles: useful simplifications

▶ If g(n) = O(f (n)), we have

O(f (n)) + O(g(n)) = O(f (n)) .

◦ Example: 2n4 + n7 /3 = O(n7 ).

▶ If c is a constant, we have
O(cf (n)) = O(f (n)) .
◦ Example 1: O(5n2 ) = O(n2 ).
◦ Example 2: O(loga n) = O(logb n) if a, b > 0 are constants.

CS147 | Fanghui Liu, [email protected] Slide 17/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition,

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n).

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality.

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality. why?

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality. why?
√
ln n
2 ≤ cn0.0001

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality. why?
√
ln n
√
2 ≤ cn0.0001 ⇔ ln n ln 2 ≤ ln c + 0.0001 ln n

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality. why?
√
ln n
√
2 ≤ cn0.0001 ⇔ ln n ln 2 ≤ ln c + 0.0001 ln n
√
⇐ ln n ln 2 ≤ 0.0001 ln n

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality. why?
√
ln n
√
2 ≤ cn0.0001 ⇔ ln n ln 2 ≤ ln c + 0.0001 ln n
√
⇐ ln n ln 2 ≤ 0.0001 ln n
√ 8
(ln 2)2
⇔ ln n ≥ 104 ln 2 ⇔ n ≥ e10 := N .

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Example

Example
√
ln n
Given f (n) = 2 and g(n) = n0.0001 , check f (n) = O(g(n)) and f (n) , Ω(g(n)).

Proof.
We can prove it by the definition.
We prove f (n) = O(g(n)) as an example, by definition, we need to find c > 0 and n > N such
that f (n) ≤ cg(n). Here we assume c ≥ 1 without loss of generality. why?
√
ln n
√
2 ≤ cn0.0001 ⇔ ln n ln 2 ≤ ln c + 0.0001 ln n
√
⇐ ln n ln 2 ≤ 0.0001 ln n
√ 8
(ln 2)2
⇔ ln n ≥ 104 ln 2 ⇔ n ≥ e10 := N .
8
(ln 2)2
We conclude the proof by taking c ≥ 1 and N := e10 . □

CS147 | Fanghui Liu, [email protected] Slide 18/ 34

Comparison

Figure: source from https://en.wikipedia.org/wiki/Big_O_notation.

CS147 | Fanghui Liu, [email protected] Slide 19/ 34

Principles: using limits

f (n)
▶ If limn→∞ g(n) exists (and is finite), then f (n) = O(g(n)).

CS147 | Fanghui Liu, [email protected] Slide 20/ 34

Principles: using limits

▶ If limn→∞ fg(n)
(n)
exists (and is finite), then f (n) = O(g(n)).
▶ We can often apply L’Höpital’s rule to calculate this limit

f (n) f ′ (n)
lim = lim ′ .
n→∞ g(n) n→∞ g (n)

Remark: Remember the condition when using the L’Höpital’s rule.

CS147 | Fanghui Liu, [email protected] Slide 20/ 34

Principles: using limits

▶ If limn→∞ fg(n)
(n)
exists (and is finite), then f (n) = O(g(n)).
▶ We can often apply L’Höpital’s rule to calculate this limit

f (n) f ′ (n)
lim = lim ′ .
n→∞ g(n) n→∞ g (n)

Remark: Remember the condition when using the L’Höpital’s rule.

Example
Consider two functions f (n) = 2n and g(n) = 3n , we have f (n) = O(g(n)) and
f (n) , Ω(g(n)).

CS147 | Fanghui Liu, [email protected] Slide 20/ 34

Asymptotic notations and summations (I)
1−xn+1
Pn
▶ Consider an geometric series f (n) = i=1 xi = 1−x .

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
1−xn+1
Pn
▶ Consider an geometric series f (n) = i=1 xi = 1−x .
◦ If x > 1, then f (n) = Θ(xn ).

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
1−xn+1
Pn
▶ Consider an geometric series f (n) = i=1 xi = 1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
1−xn+1
Pn
▶ Consider an geometric series f (n) = i=1 xi = 1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n
X n
X n
X
(ai + b) = ai + b
i=1 i=1 i=1

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n n n
X X X n(n + 1)
(ai + b) = ai + b=a + bn
i=1 i=1 i=1
2

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n n n
X X X n(n + 1)
(ai + b) = ai + b=a + bn = Θ(n2 ) + Θ(n) = Θ(n2 ) .
i=1 i=1 i=1
2

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n n n
X X X n(n + 1)
(ai + b) = ai + b=a + bn = Θ(n2 ) + Θ(n) = Θ(n2 ) .
i=1 i=1 i=1
2

▶ Dealing with an harmonic series:

n
X 2n
=
i=1
i

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n n n
X X X n(n + 1)
(ai + b) = ai + b=a + bn = Θ(n2 ) + Θ(n) = Θ(n2 ) .
i=1 i=1 i=1
2

▶ Dealing with an harmonic series:

n n
! n
!
X 2n X n X 1
=Θ =Θ n
i=1
i i=1
i i=1
i

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n n n
X X X n(n + 1)
(ai + b) = ai + b=a + bn = Θ(n2 ) + Θ(n) = Θ(n2 ) .
i=1 i=1 i=1
2

▶ Dealing with an harmonic series:

n n
! n
!
X 2n X n X 1
=Θ =Θ n = Θ(n log n) .
i=1
i i=1
i i=1
i
Pn 1
◦ Harmonic series: i=1 i = ln n + γ + ϵn with the Euler–Mascheroni constant
1
γ ≈ 0.577 and ϵn ≈ 2n .

CS147 | Fanghui Liu, [email protected] Slide 21/ 34

Asymptotic notations and summations (I)
n+1
▶ Consider an geometric series f (n) = ni=1 xi = 1−x
P
1−x .
◦ If x > 1, then f (n) = Θ(xn ).
◦ If x = 1, then f (n) = Θ(n).
◦ If 0 < x < 1, then f (n) = Θ(1).
▶ Dealing with an arithmetic series with two constants a > 0, b > 0
n n n
X X X n(n + 1)
(ai + b) = ai + b=a + bn = Θ(n2 ) + Θ(n) = Θ(n2 ) .
i=1 i=1 i=1
2

▶ Dealing with an harmonic series:

n n
! n
!
X 2n X n X 1
=Θ =Θ n = Θ(n log n) .
i=1
i i=1
i i=1
i
Pn 1
◦ Harmonic series: i=1 i = ln n + γ + ϵn with the Euler–Mascheroni constant
1
γ ≈ 0.577 and ϵn ≈ 2n .
◦ Check more information about this series
https://en.wikipedia.org/wiki/Harmonic_series_(mathematics)
CS147 | Fanghui Liu, [email protected] Slide 21/ 34
Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 .

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 . Prove that f (n) = Θ(n4 ).

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 . Prove that f (n) = Θ(n4 ).

Proof.
◦ upper bound
n
X
f (n) = i3 ≤
i=1

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 . Prove that f (n) = Θ(n4 ).

Proof.
◦ upper bound
n
X n
X
3
f (n) = i ≤ n3 = O(n4 ) .
i=1 i=1

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 . Prove that f (n) = Θ(n4 ).

Proof.
◦ upper bound
n
X n
X
3
f (n) = i ≤ n3 = O(n4 ) .
i=1 i=1

◦ lower bound
n
X
f (n) = i3 ≥
i=1

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 . Prove that f (n) = Θ(n4 ).

Proof.
◦ upper bound
n
X n
X
3
f (n) = i ≤ n3 = O(n4 ) .
i=1 i=1

◦ lower bound
n n
X X
3n
f (n) = i ≥ ( )3
i=1 n 2
i= 2

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

Asymptotic notations and summations (II)

Example
Pn
Bounding parts of the sum: consider f (n) = i=1 i3 . Prove that f (n) = Θ(n4 ).

Proof.
◦ upper bound
n
X n
X
3
f (n) = i ≤ n3 = O(n4 ) .
i=1 i=1

◦ lower bound
n n
X X
3n
f (n) = i ≥ ( )3 = Ω(n4 ) .
i=1 n 2
i= 2

4
Accordingly, we have f (n) = Θ(n ). □

CS147 | Fanghui Liu, [email protected] Slide 22/ 34

 Using Big-O notation for algorithm analysis...

CS147 | Fanghui Liu, [email protected] Slide 23/ 34

Back to vectorMax()

▶ For vectorMax(): ignore the original two variable initializations, the return statement,
the comparison, and the setting of currentMax in the loop.

CS147 | Fanghui Liu, [email protected] Slide 24/ 34

Back to vectorMax()

▶ For vectorMax(): ignore the original two variable initializations, the return statement,
the comparison, and the setting of currentMax in the loop.
▶ Notice that the important part of the function is the fact that the loop conditions will
change with the size of the array: for each extra element, there will be one more
iteration. This is a linear relationship, and therefore O(n).

CS147 | Fanghui Liu, [email protected] Slide 24/ 34

Nested Loops

CS147 | Fanghui Liu, [email protected] Slide 25/ 34

Nested Loops

▶ The inner loop (variable j) has a complexity of O(n), as our analysis from the previous
slides would show. However, the entire inner loop happens n times, as well! This squares
the number of times result is incremented.

CS147 | Fanghui Liu, [email protected] Slide 25/ 34

Nested Loops

▶ The inner loop (variable j) has a complexity of O(n), as our analysis from the previous
slides would show. However, the entire inner loop happens n times, as well! This squares
the number of times result is incremented.
▶ Now we have a quadratic relationship between n and the time to complete the function:
O(n2 ).

CS147 | Fanghui Liu, [email protected] Slide 25/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.
▶ Line 1. total time ≤ c1 n

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.
▶ Line 1. total time ≤ c1 n
▶ Line 2. total time ≤ ni=1 c2 (n − i)
P

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.
▶ Line 1. total time ≤ c1 n
▶ Line 2. total time ≤ ni=1 c2 (n − i)
P
Pn
▶ Line 3. total time ≤ i=1 c3 (n − i)

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.
▶ Line 1. total time ≤ cP
1n
n
▶ Line 2. total time ≤ Pi=1 c2 (n − i)
n
▶ Line 3. total time ≤ Pi=1 c3 (n − i)
n
▶ Line 4. total time ≤ i=1 c4 (n − i)

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.
▶ Line 1. total time ≤ cP
1n
n
▶ Line 2. total time ≤ Pi=1 c2 (n − i)
n
▶ Line 3. total time ≤ Pi=1 c3 (n − i)
n
▶ Line 4. total time ≤ i=1 c4 (n − i)
▶ Line 5. total time ≤ c5

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Example: Runtime of an algorithm (Nested loops)

An example algorithm (Pseudocode)

(take c1 time) 1. For i = 1, 2, . . . , n
(take c2 time) 2. For j = i + 1, i + 2, . . . , n
(take c3 time) 3. If A[i] + A[j] = 0
(take c4 time) 4. Return YES.
(take c5 time) 5. Return No.
▶ Line 1. total time ≤ c1 n
▶ Line 2. total time ≤ ni=1 c2 (n − i)
P
Pn
▶ Line 3. total time ≤ i=1 c3 (n − i)
▶ Line 4. total time ≤ ni=1 c4 (n − i)
P
▶ Line 5. total time ≤ c5
Total time spent on an input of size n
n
X n
X n
X
≤ c1 n + c2 (n − i) + c3 (n − i) + c4 (n − i) + c5 = O(n2 )
i=1 i=1 i=1

CS147 | Fanghui Liu, [email protected] Slide 26/ 34

Input sensitivity

▶ On every input of size n, the algorithm spends at most O(n2 ) time.

CS147 | Fanghui Liu, [email protected] Slide 27/ 34

Input sensitivity

▶ On every input of size n, the algorithm spends at most O(n2 ) time.

▶ Input sensitivity: There are inputs of size n on which the algorithm spends only Θ(1)
time. [e.g., A[1] + A[2] = 0]

CS147 | Fanghui Liu, [email protected] Slide 27/ 34

Input sensitivity

▶ On every input of size n, the algorithm spends at most O(n2 ) time.

▶ Input sensitivity: There are inputs of size n on which the algorithm spends only Θ(1)
time. [e.g., A[1] + A[2] = 0]
▶ Input sensitivity: There are inputs of size n on which the algorithm spends Θ(n2 ) time.
[e.g., A[1] = 1, A[2] = 2, . . . , A[n − 1] = n − 1 and A[n] = −(n − 1)]

CS147 | Fanghui Liu, [email protected] Slide 27/ 34

Input sensitivity

▶ On every input of size n, the algorithm spends at most O(n2 ) time.

▶ Input sensitivity: There are inputs of size n on which the algorithm spends only Θ(1)
time. [e.g., A[1] + A[2] = 0]
▶ Input sensitivity: There are inputs of size n on which the algorithm spends Θ(n2 ) time.
[e.g., A[1] = 1, A[2] = 2, . . . , A[n − 1] = n − 1 and A[n] = −(n − 1)]
We will say that the algorithm has a runtime of O(n2 ) in the worst case.

CS147 | Fanghui Liu, [email protected] Slide 27/ 34

Runtime of an algorithm (Worst case)

Formally, let

f (n) = max (runtime of the algorithm on input I)

input I of size n

We will focus on how f (n) grows with input size n, asymptotically.

⇔ Worst-case asymptotic running time of an algorithm.

CS147 | Fanghui Liu, [email protected] Slide 28/ 34

Worst-case asymptotic running time

▶ Is a feature of an algorithm for a given computational problem

CS147 | Fanghui Liu, [email protected] Slide 29/ 34

Worst-case asymptotic running time

▶ Is a feature of an algorithm for a given computational problem

▶ Independent of
◦ programming language
◦ specific input
◦ machine (computer)

CS147 | Fanghui Liu, [email protected] Slide 29/ 34

Worst-case asymptotic running time

▶ Is a feature of an algorithm for a given computational problem

▶ Independent of
◦ programming language
◦ specific input
◦ machine (computer)
▶ Allows us to compare two different algorithms for the same problem.

CS147 | Fanghui Liu, [email protected] Slide 29/ 34

Worst-case asymptotic running time

▶ An O(n2 ) time algorithm is better than an O(n3 ) time algorithm

▶ An O(n log n) time algorithm is better than an O(n2 ) time algorithm

Goal
Given a computational problem, our goal will be to find an algorithm for it with smallest
possible worst-case asymptotic runtime.

CS147 | Fanghui Liu, [email protected] Slide 30/ 34

Runtime of an algorithm (Worst case)

Statement
▶ constant time O(1): arithmetic/logic operation, access one element in an array
▶ linear time O(n): merge two sorted arrays (Lecture 5)
▶ quadratic time O(n2 ): bubble sort (Lecture 4)
▶ logarithmic time O(log n): binary search (Lecture 6)
▶ linearithmic time O(n log n): merge sort (Lecture 5)

CS147 | Fanghui Liu, [email protected] Slide 31/ 34

Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best
▶ the average case: averaged over all possible inputs for randomized algorithm

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best
▶ the average case: averaged over all possible inputs for randomized algorithm
▶ runtime analysis vs. memory analysis

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best
▶ the average case: averaged over all possible inputs for randomized algorithm
▶ runtime analysis vs. memory analysis
▶ be careful of hidden loop: e.g., for an array A[n], insert a value
◦at index 1
◦ at index n+1

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best
▶ the average case: averaged over all possible inputs for randomized algorithm
▶ runtime analysis vs. memory analysis
▶ be careful of hidden loop: e.g., for an array A[n], insert a value
◦at index 1
◦ at index n+1

Be careful!
What is the difference between “at index 1” and “index n+1”?

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best
▶ the average case: averaged over all possible inputs for randomized algorithm
▶ runtime analysis vs. memory analysis
▶ be careful of hidden loop: e.g., for an array A[n], insert a value
◦at index 1
◦ at index n+1

Be careful!
What is the difference between “at index 1” and “index n+1”?

▶ O(1) time for inserting at “index n+1”2

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
Beyond the worst case runtime analysis

▶ the best case: find one input that the algorithm can perform the best
▶ the average case: averaged over all possible inputs for randomized algorithm
▶ runtime analysis vs. memory analysis
▶ be careful of hidden loop: e.g., for an array A[n], insert a value
◦at index 1
◦ at index n+1

Be careful!
What is the difference between “at index 1” and “index n+1”?

▶ O(1) time for inserting at “index n+1”2

▶ O(n) time for inserting at “index 1”

2 If we don’t consider the memory allocation.

CS147 | Fanghui Liu, [email protected] Slide 32/ 34
*Examples in TCS, ML theory

Figure: time complexity and parameter complexity [AZLS19].

Figure: Recall the example in Lecture 1: sample complexity and time complexity [CKM22].

CS147 | Fanghui Liu, [email protected] Slide 33/ 34

Wrap up!

◦ Useful materials for reading

▶ Notes on Discrete Mathematics [Aspnes, Chapter 7]
▶ https:
//web.stanford.edu/class/archive/cs/cs106b/cs106b.1206/lectures/big-o/
▶ Algorithms [DPV, Chapter 7]

CS147 | Fanghui Liu, [email protected] Slide 34/ 34

Wrap up!

◦ Useful materials for reading

▶ Notes on Discrete Mathematics [Aspnes, Chapter 7]
▶ https:
//web.stanford.edu/class/archive/cs/cs106b/cs106b.1206/lectures/big-o/
▶ Algorithms [DPV, Chapter 7]
◦ Next week: Sorting algorithms

CS147 | Fanghui Liu, [email protected] Slide 34/ 34

References I

[0] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song, A convergence theory for deep learning via
over-parameterization, International Conference on Machine Learning, PMLR, 2019,
pp. 242–252.
(Cited on page 119.)

[0] Sitan Chen, Adam R Klivans, and Raghu Meka, Learning deep relu networks is
fixed-parameter tractable, 2021 IEEE 62nd Annual Symposium on Foundations of
Computer Science (FOCS), IEEE, 2022, pp. 696–707.
(Cited on page 119.)

CS147 | Fanghui Liu, [email protected] Slide 1/ 1