Complexity of Programs/Functions

Cosmin E. Oancea
[email protected]

Department of Computer Science (DIKU)

University of Copenhagen

November 2022
Today’s Lecture
First hour:
examine two solutions to the problem of finding the index at
which a target element appears in an input array:
I when the array is not sorted,
I when the array is sorted.
learn how to reason empirically on whether two algorithms
have the same complexity (only from their runtimes).

Second hour:
introduce the notion of (asymptotic) complexity of an
algorithm, which models how the running time depends on
the “input size” (or problem size), where the input size can be
I the magnitude of the integer parameter, or
I the length of the input array, etc.
give examples from several complexity classes:
O(1), O(log(n)), O(n), O(n · log(n)), O(n2 ), O(2n ) . . ..
examine the problem of computing all the prime numbers up
to n — Eratosthenes’ sieve algorithm, which has complexity
O(n · log(log(n)))
Finding the Index at which an Element Appears in an Array
When the input array is NOT SORTED
When the input array IS SORTED

Introducing the Notion of Complexity of Functions/Programs

Computing all Prime Numbers up to N

Searching for a Target Element in an Unsorted Array
Problem: For a given array and a given target element, find the
index in the array that contains an element that is equal to the
target element. If the target element does not appear in the array
then the result should be −1.

Rephrased: I have a row of boxes in front of me, each containing a

piece of paper with a number on it. I am tasked to find the index
of a box that contains a certain number, e.g., 33. What do I do?

function linearSearch( array, target_elem ) {

???

}
Searching for a Target Element in an Unsorted Array
Problem: For a given array and a given target element, find the
index in the array that contains an element that is equal to the
target element. If the target element does not appear in the array
then the result should be −1.
function linearSearch ( array , target elem ) {
f o r ( l e t i = 0 ; i < a r r a y . l e n g t h ; i = i + 1 ) { / / t r a v e r s e b o xe s
l e t c u r r e n t e l e m = a r r a y [ i ] ; / / l o o k i n s i d e t h e c u r r e n t box
i f ( c u r r e n t e l e m === t a r g e t e l e m ) { / / j u s t got l u cky
r e t u r n i ; / / y e l l t h e box ’ s i n d e x and l e a v e i m m e d i a t e l y
}
}
r e t u r n −1; / / u n l u c k y : t a r g e t e l e m e n t i s n o t i n a r r a y
}
Searching for a Target Element in an Unsorted Array
Problem: For a given array and a given target element, find the
index in the array that contains an element that is equal to the
target element. If the target element does not appear in the array
then the result should be −1.
function linearSearch ( array , target elem ) {
f o r ( l e t i = 0 ; i < a r r a y . l e n g t h ; i = i + 1 ) { / / t r a v e r s e b o xe s
l e t c u r r e n t e l e m = a r r a y [ i ] ; / / l o o k i n s i d e t h e c u r r e n t box
i f ( c u r r e n t e l e m === t a r g e t e l e m ) { / / j u s t got l u cky
r e t u r n i ; / / y e l l t h e box ’ s i n d e x and l e a v e i m m e d i a t e l y
}
}
r e t u r n −1; / / u n l u c k y : t a r g e t e l e m e n t i s n o t i n a r r a y
}

How many operations do I need to execute in the worst case, e.g.,

in how many boxes do I have to look?
Searching for a Target Element in an Unsorted Array
Problem: For a given array and a given target element, find the
index in the array that contains an element that is equal to the
target element. If the target element does not appear in the array
then the result should be −1.
function linearSearch ( array , target elem ) {
f o r ( l e t i = 0 ; i < a r r a y . l e n g t h ; i = i + 1 ) { / / t r a v e r s e b o xe s
l e t c u r r e n t e l e m = a r r a y [ i ] ; / / l o o k i n s i d e t h e c u r r e n t box
i f ( c u r r e n t e l e m === t a r g e t e l e m ) { / / j u s t got l u cky
r e t u r n i ; / / y e l l t h e box ’ s i n d e x and l e a v e i m m e d i a t e l y
}
}
r e t u r n −1; / / u n l u c k y : t a r g e t e l e m e n t i s n o t i n a r r a y
}

How many operations do I need to execute in the worst case, e.g.,

in how many boxes do I have to look?
Answer: in the worst case, for an array of length n, I will perform
order-n operations, i.e., k1 × n + k2 where k1 and k2 are constants.
(For example because I need to look inside each of the n boxes).
Finding the Index at which an Element Appears in an Array
When the input array is NOT SORTED
When the input array IS SORTED

Introducing the Notion of Complexity of Functions/Programs

Computing all Prime Numbers up to N

Binary Search on a Sorted Array
Binary search problem: finds the position of a target value in a
sorted array, and -1 if not there.
Similar game: I am thinking of a number between 1 and 128.
What number am I thinking of? Use at most 8 questions to win!
Binary Search on a Sorted Array
Binary search problem: finds the position of a target value in a
sorted array, and -1 if not there.
Similar game: I am thinking of a number between 1 and 128.
What number am I thinking of? Use at most 8 questions to win!
Binary search begins by comparing the element at the middle of
the array with the target value:
if target matches the element, its position is returned;
if target < element, the search continues in the lower half;
if target > element, the search continues in the upper half;
essentially the algorithm eliminates the half in which the
target value cannot lie in each iteration, and thus requires a
(sublinear) number of operations at worst proportional with
log2 (n), where n denotes the size of the array.
(We will explain later what log2 (n) is).

Binary search requires a loop implementation, i.e., cannot be

implemented by map, reduce, filter.
Binary Search on a Sorted Array
Binary search problem: finds the position of a target value in a
sorted array, and -1 if not there. The example below searches 23.

Implementation on the whiteboard.

Each steps halves the size of the to-be-searched subarray, hence the algorithm takes
sublinear time (log (n) steps), where n is the size of the original array.
Binary Search on a Sorted Array
Variables L, H denote the low (start) and high (end) indices of the
current (to-be-searched) sub-array, and m the middle position.
function binarySearch( array, target ) {
let L = 0, H = array.length - 1;
while (L <= H) {
let m = Math.floor( (L + H) / 2 );
let elem = array[m];
if (elem === target) {
return m; // target found at index m
} else {
// compute the sub-array for next step
if (elem < target) L = m + 1;
else H = m - 1;
}
}
return -1;
}
Binary Search: Number of Operations in Worst Case
Assume the length of the original array is n. Each iteration halves
the length of the to-be-searched subarray. The worst case is when
the target element is not part of the array.
After iteration 1, the length of the to-be-searched subarray is n
2
= n
21
After iteration 2, the length of the to-be-searched subarray is n
4
= n
22
After iteration 3, the length of the to-be-searched subarray is n
8
= n
23
...
After iteration i, the length of the to-be-searched subarray is n
2i

The loop ends, say after x iterations, when the to-be-searched subarray has length 1.

It follows that x satisfies: n

2x = 1 which is equivalent with 2x = n

Definition of Logarithm in base 2

x = log2 (n) is the solution of the equation 2x = n

log2 (n) grows much, much slower than n (see next slide)!
Runtime of Searching in an Unsorted vs Sorted Array
Assume the length of the original array is n.
searching in an unsorted array takes time n
binary search on a sorted array takes time log2 (n)

n 64 1024 16384 262144 4194304 67108864 1073741824

log2 (n) 6 10 14 18 22 26 30
Theoretical
Speedup 11× 102× 1170× 14564× 190650× 2581110× 35791394×
= log n(n)
2

Runtime of Original Program

Speedup =
Runtime of Optimized Program

As we increase the size of the problem (n),

binary search keeps getting faster and faster in
comparison with linear search (i.e., in an unsorted array).
That means that they have different complexities!
More Reading on Binary Search on a Sorted Array
You may google your favorite tutorial.
Here are two links (of many) that you may find useful:

Tutorial from tutorialspoint (click me).

From wikipedia (click me).

Finding the Index at which an Element Appears in an Array
When the input array is NOT SORTED
When the input array IS SORTED

Introducing the Notion of Complexity of Functions/Programs

Computing all Prime Numbers up to N

What is the complexity of an algorithm?
The complexity of a function/program models the manner in which the running time of
the function varies (grows) when the function is applied to solve larger problems
(i.e., how does the runtime scale with the magnitude of the parameters).

Informally, it expresses the worst-case number of operations performed by that algorithm

as a function of the input size, from which we keep only the leading (maximal) term and
ignore the other terms (that grow slower), and
ignore multiplicative constants

Example: assume a program that receives n and m as integral arguments, and performs, in
the worst case, a number of operations equal to
3 · m · n + 1000000 · m + 100 · n + 1000000000 + 235 · m · n2 .

Its complexity is:

What is the complexity of an algorithm?
The complexity of a function/program models the manner in which the running time of
the function varies (grows) when the function is applied to solve larger problems
(i.e., how does the runtime scale with the magnitude of the parameters).

Informally, it expresses the worst-case number of operations performed by that algorithm

as a function of the input size, from which we keep only the leading (maximal) term and
ignore the other terms (that grow slower), and
ignore multiplicative constants

Example: assume a program that receives n and m as integral arguments, and performs, in
the worst case, a number of operations equal to
3 · m · n + 1000000 · m + 100 · n + 1000000000 + 235 · m · n2 .

Its complexity is: O(m · n2)

we kept only the “fastest-growing” term 235 · m · n2 and
removed its constant factor 235.
The big O notation denotes this simplification when the
number of operations corresponds to the worst-case scenario.
The running time of programs in the same complexity class
will differ by a constant factor for large-enough inputs.
 Classes of Algorithmic Complexity

O(1) denotes a constant number of operations (i.e., does not depends on the input).
O(log(n)) denotes logarithmic time, i.e., the solution x of equation n = 2x . Can efficiently
compute for very large values of parameter n, think: binary search.
O(n) denotes linear time (proportional with the input n).
Can compute large sizes, think: linear search, map, reduce, filter, etc.
O(n · log(n)) think: computing n binary searches on an array of size n. Can compute largish sizes.
O(n2 ) denotes quadratic time; impractical but for small n.
O(2n ) denotes exponential time; impractical but for tiny n.
Constant Time Complexity O(1)
The number of operations is constant (does not depend on input)
function f ( a , b ) {
let x = a * b;
let y = a + b;
return x * y ;
}

f u n c t i o n min ( a , b ) {
i f ( a < b ) return a ;
else return b ;
}

function g ( array ) {
let s = 0;
f o r ( i = 0 ; i < 1000000; i = i + 1 ) {
s = s + arr [ i ] ;
}
return s ;
}

1000000 is big, but it is still a constant. The complexity of

function g is still O(1)!
Linear Complexity O(n)
The number of operations is proportional with the input n
function f ( array ) {
/ / f i l t e r empty s t r i n g s and t r a n s f o r m s t o numbers
r e t u r n a r r a y . f i l t e r ( s => s ! = = ” ” ) . map ( a => Number ( a ) ) ;
}

f u n ct i o n g ( a r r a y ) { / / f i n d s the minimal element

r e t u r n a r r a y . r e d u ce ( min , I n f i n i t y ) ;
}

function h ( array ) {
let s = 0;
for ( i = 0; i < array . length ; i = i +1) {
s = s + arr [ i ] ;
}
return s ;
}

The number of operations of the functions above is proportional

with the length of the input array, denoted n, hence O(n)
Quadratic Complexity O(n2 )
The number of operations is proportional with the square of the
input:
f u n c t i o n m u l t Ta b l e ( n ) {
l e t t a b l e = new A r r a y ( n ) ;
f o r ( l e t i = 0 ; i <n ; i = i + 1 ) {
t a b l e [ i ] = new A r r a y ( n ) ;

f o r ( l e t j = 0 ; j <n ; j = j + 1 ) {
table [ i ] [ j ] = ( i +1) * ( j + 1 ) ;
}
}
return table ;
}

Building a multiplication table with n rows and n column requires

a number of operations proportional with n2 , hence its complexity
is O(n2 )
Why Do We Care about Complexity?
(1) So that you can reason what sizes are infeasible to compute:
O(n2 ) you should probably not try to run a quadratic algorithm with
n = 106
O(n3 ) you should probably not try to run n × n matrix multiplication
on matrices whose dimensions are in the tens (or hundred) of
thousands;
O(2n ) you should probably not try to run an exponential algorithm
when the input n in over a hundred.
Why Do We Care about Complexity?
(1) So that you can reason what sizes are infeasible to compute:
O(n2 ) you should probably not try to run a quadratic algorithm with
n = 106
O(n3 ) you should probably not try to run n × n matrix multiplication
on matrices whose dimensions are in the tens (or hundred) of
thousands;
O(2n ) you should probably not try to run an exponential algorithm
when the input n in over a hundred.

(2) Because it allows us to reason and classify the efficiency of a

program. Programs whose runtime differ by a constant factor
are in the same (complexity) class.
(3) Because solving a problem with an algorithm that has
sub-optimal complexity is a catastrophic/unforgivable
mistake.
Why Do We Care about Complexity?
(1) So that you can reason what sizes are infeasible to compute:
O(n2 ) you should probably not try to run a quadratic algorithm with
n = 106
O(n3 ) you should probably not try to run n × n matrix multiplication
on matrices whose dimensions are in the tens (or hundred) of
thousands;
O(2n ) you should probably not try to run an exponential algorithm
when the input n in over a hundred.

(2) Because it allows us to reason and classify the efficiency of a

program. Programs whose runtime differ by a constant factor
are in the same (complexity) class.
(3) Because solving a problem with an algorithm that has
sub-optimal complexity is a catastrophic/unforgivable
mistake.
(4) For the purpose of this course, it is fine if your program is a
constant-factor away from the optimal running time, as long
as it has the optimal complexity.
Finding the Index at which an Element Appears in an Array
When the input array is NOT SORTED
When the input array IS SORTED

Introducing the Notion of Complexity of Functions/Programs

Computing all Prime Numbers up to N

What is a Prime Number?
A prime number is a positive integer who does not have divisors other than 1 and itself.
examples of prime numbers: 2, 3, 5, 7, 13, 17, 23, . . .
examples of numbers that are not primes:
21 = 3 ∗ 7, 33 = 3 ∗ 11, 57 = 3 ∗ 19
What is a Prime Number?
A prime number is a positive integer who does not have divisors other than 1 and itself.
examples of prime numbers: 2, 3, 5, 7, 13, 17, 23, . . .
examples of numbers that are not primes:
21 = 3 ∗ 7, 33 = 3 ∗ 11, 57 = 3 ∗ 19

One can test whether a√ number n is prime by checking whether it is divisible with any
integer between 2 and n.
we will use the modulo operator: x % y is the reminder resulted from dividing x
by y as integers; the reminder is always smaller than y, for example:
14 % 3 results in 2 because 14 = 3*4 + 2
15 % 2 results in 1 because 15 = 2*7 + 1

f u n c t i o n i s Pr i m e ( n ) {
i f ( n < 2 | | Math . f l o o r ( n ) ! = = n )
r e t u r n false ;

f o r ( l e t i = 2 ; i <= Math . s q r t ( n ) ; i = i + 1 ) {
i f ( ( n % i ) === 0 )
r e t u r n false ;
}
r e t u r n true ;
}

What is the complexity of this program?

What is a Prime Number?
A prime number is a positive integer who does not have divisors other than 1 and itself.
examples of prime numbers: 2, 3, 5, 7, 13, 17, 23, . . .
examples of numbers that are not primes:
21 = 3 ∗ 7, 33 = 3 ∗ 11, 57 = 3 ∗ 19

One can test whether a√ number n is prime by checking whether it is divisible with any
integer between 2 and n.
we will use the modulo operator: x % y is the reminder resulted from dividing x
by y as integers; the reminder is always smaller than y, for example:
14 % 3 results in 2 because 14 = 3*4 + 2
15 % 2 results in 1 because 15 = 2*7 + 1

f u n c t i o n i s Pr i m e ( n ) {
i f ( n < 2 | | Math . f l o o r ( n ) ! = = n )
r e t u r n false ;

f o r ( l e t i = 2 ; i <= Math . s q r t ( n ) ; i = i + 1 ) {
i f ( ( n % i ) === 0 )
r e t u r n false ;
}
r e t u r n true ;
}

What is the complexity of this program? √

In the worst case, it takes time proportional with n.
Computing all Prime Numbers up to N – Dummy
A prime number is a positive integer who does not have divisors
other than 1 and itself.
Problem: compute all the prime numbers less than or equal to n.
The result is an array of length n + 1 of booleans in which the
element at index i is true if i is prime and false otherwise.
What would be a direct (dummy) implementation?
Computing all Prime Numbers up to N – Dummy
A prime number is a positive integer who does not have divisors
other than 1 and itself.
Problem: compute all the prime numbers less than or equal to n.
The result is an array of length n + 1 of booleans in which the
element at index i is true if i is prime and false otherwise.
What would be a direct (dummy) implementation?
f u n c t i o n primesUpTo ( n ) {
l e t primes = [ 0 ,0 ] ;
f o r ( l e t i = 2 ; i <=n ; i + + ) {
l e t i s p r i m e = i s Pr i m e ( i ) ;
p r i m e s . pushBack ( i s p r i m e ) ;
}
}
What is the complexity of this dummy?
Computing all Prime Numbers up to N – Dummy
A prime number is a positive integer who does not have divisors
other than 1 and itself.
Problem: compute all the prime numbers less than or equal to n.
The result is an array of length n + 1 of booleans in which the
element at index i is true if i is prime and false otherwise.
What would be a direct (dummy) implementation?
f u n c t i o n primesUpTo ( n ) {
l e t primes = [ 0 ,0 ] ;
f o r ( l e t i = 2 ; i <=n ; i + + ) {
l e t i s p r i m e = i s Pr i m e ( i ) ;
p r i m e s . pushBack ( i s p r i m e ) ;
}
}
What is the complexity of this dummy?
The implementation takes time proportional with
√ √ √ √ 3
2 + 3 + . . . + n ∼ n · n = n2 .
Computing all Prime Numbers up to N – Dummy
A prime number is a positive integer who does not have divisors
other than 1 and itself.
Problem: compute all the prime numbers less than or equal to n.
The result is an array of length n + 1 of booleans in which the
element at index i is true if i is prime and false otherwise.
What would be a direct (dummy) implementation?
f u n c t i o n primesUpTo ( n ) {
l e t primes = [ 0 ,0 ] ;
f o r ( l e t i = 2 ; i <=n ; i + + ) {
l e t i s p r i m e = i s Pr i m e ( i ) ;
p r i m e s . pushBack ( i s p r i m e ) ;
}
}
What is the complexity of this dummy?
The implementation takes time proportional with
√ √ √ √ 3
2 + 3 + . . . + n ∼ n · n = n 2 . Unforgivable mistake!
Computing Prime Numbers up to N in Near-Linear Time
A prime number is a positive integer who does not have divisors
other than 1 and itself.
Algorithm known as the sieve of Eratosthenes (Eratosthenes of
Cyrene, 3rd century BC) link to wikipedia (click me):
Algorithm sieve:
Input: an integer n ≥ 2
Output: an array of n + 1 booleans, such that the element at index i is true if i is a prime
number and false otherwise.
Implementation:
(1) create an array A of n + 1 booleans and initialize all its elements with true. In
JavaScript you can use let A = new Array(n+1).fill(true);
(2) set the first two elements to false, i.e., 0 and 1 are not considered prime numbers
√
(3) for i = 2, 3, 4, ..., not exceeding n do
I if A[i] is false then do nothing
I else if A[i] is true do (i.e., i is known to be a prime number)
I for j = i*i, i*i + i, i*i + 2*i, i*i + 3*i,. . .
not exceeding n do
set A[j] to false;
(4) return A
The implementation takes near linear time: n · log(log(n)).
Computing Prime Numbers up to N in Near-Linear Time
Initially the array A contains only true values. After step (2):
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T T T T T T T T T T T T T T
Computing Prime Numbers up to N in Near-Linear Time
Initially the array A contains only true values. After step (2):
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T T T T T T T T T T T T T T
Iteration i=2 finds A[i] to be true and
sets locations j = 4, 6, 8, 10, 12, 14, 16 to False (F)
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F T F T F T F T F
Computing Prime Numbers up to N in Near-Linear Time
Initially the array A contains only true values. After step (2):
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T T T T T T T T T T T T T T
Iteration i=2 finds A[i] to be true and
sets locations j = 4, 6, 8, 10, 12, 14, 16 to False (F)
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F T F T F T F T F
Iteration i=3 finds A[i] to be true and sets locations j = 9, 12, 15 to F
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F F F T F T F F F
Computing Prime Numbers up to N in Near-Linear Time
Initially the array A contains only true values. After step (2):
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T T T T T T T T T T T T T T
Iteration i=2 finds A[i] to be true and
sets locations j = 4, 6, 8, 10, 12, 14, 16 to False (F)
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F T F T F T F T F
Iteration i=3 finds A[i] to be true and sets locations j = 9, 12, 15 to F
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F F F T F T F F F

Iteration i=4 finds A[i] to be false, hence does not execute the
inner loop.
Computing Prime Numbers up to N in Near-Linear Time
Initially the array A contains only true values. After step (2):
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T T T T T T T T T T T T T T
Iteration i=2 finds A[i] to be true and
sets locations j = 4, 6, 8, 10, 12, 14, 16 to False (F)
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F T F T F T F T F
Iteration i=3 finds A[i] to be true and sets locations j = 9, 12, 15 to F
The indices (top) and values (bottom) of array A just before the loop:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
F F T T F T F T F F F T F T F F F

Iteration i=4 finds A[i] to be false, hence

√ does
√ not execute the
inner loop.The loop then exits because n = 16 = 4.
Summary
We have introduced the notion of complexity that models how
fast (or slow) the running time scales with the magnitude of the
input. It allows to:
reason at a high-level about the efficiency/quality of the
programs we are writing: writing an implementation that has
sub-optimal complexity is an unpardonable mistake!
group/classify programs into a smallish number of classes
that have similar running times (that can differ up to a
constant factor);
predict problem sizes that are infeasible to compute.

We have studied two smart algorithms for computing:

the index at which a target element appears in a sorted array
of length n, which has complexity O(log(n)),
all the prime numbers up to some (input number) n, which
has complexity O(n · log(log(n))).