Algorithms – Rosen Section 3.1-3.

Anna-Simone Frank1

MNF130

Spring 2024

1
Slides created by Tom Michoel and modified by Erik Maartensson and
Anna-Simone Frank
Algorithms

An algorithm is a finite sequence of precise instructions for performing a

computation or solving a problem.

Importance in discrete math course:

I Prove that a proposed algorithm solves the given problem.
I Compute the complexity of an algorithm.
I Many applications of discrete math cannot be discussed without
computational considerations (e.g. cryptography).

We describe algorithms using pseudocode instead of in a specific

programming language.
Asymptotic complexity

Given an algorithm that solves some problem in f (n) operations when the
problem size is n.
I How do we compare this algorithm against other algorithms solving
the same problem?
I How do we compare the algorithm against itself for other problem
sizes.
Asymptotic complexity

Given an algorithm that solves some problem in f (n) operations when the
problem size is n.
I How do we compare this algorithm against other algorithms solving
the same problem?
I How do we compare the algorithm against itself for other problem
sizes.

I The result of implementing the algorithm and testing it will depend

highly on our available hardware and the efficiency of our
implementation.
I Calculating the exact number of operations needed is tedious and
might not give a good picture of how this changes with the problem
size.
Asymptotic complexity

Given an algorithm that solves some problem in f (n) operations when the
problem size is n.
I How do we compare this algorithm against other algorithms solving
the same problem?
I How do we compare the algorithm against itself for other problem
sizes.

I The result of implementing the algorithm and testing it will depend

highly on our available hardware and the efficiency of our
implementation.
I Calculating the exact number of operations needed is tedious and
might not give a good picture of how this changes with the problem
size.
We use perhaps the most beautiful concept in computer science -
asymptotic time complexity!
Definitions of asymptotic complexity
Let f (n) denote the number of operations needed for an algorithm to
solve a problem of size n. We use the following asymptotic notation. The
first definition is the most frequently used one.
Definitions of asymptotic complexity
Let f (n) denote the number of operations needed for an algorithm to
solve a problem of size n. We use the following asymptotic notation. The
first definition is the most frequently used one.
Definition (Asymptotic upper bound)
We say that f (n) is O(g (n)) if there is a constant C such that
f (n) ≤ C · g (n) for all sufficiently large n.
Definitions of asymptotic complexity
Let f (n) denote the number of operations needed for an algorithm to
solve a problem of size n. We use the following asymptotic notation. The
first definition is the most frequently used one.
Definition (Asymptotic upper bound)
We say that f (n) is O(g (n)) if there is a constant C such that
f (n) ≤ C · g (n) for all sufficiently large n.
Definition (Asymptotic lower bound)
We say that f (n) is Ω(g (n)) if there is a constant C such that
f (n) ≥ C · g (n) for all sufficiently large n.
Definitions of asymptotic complexity
Let f (n) denote the number of operations needed for an algorithm to
solve a problem of size n. We use the following asymptotic notation. The
first definition is the most frequently used one.
Definition (Asymptotic upper bound)
We say that f (n) is O(g (n)) if there is a constant C such that
f (n) ≤ C · g (n) for all sufficiently large n.
Definition (Asymptotic lower bound)
We say that f (n) is Ω(g (n)) if there is a constant C such that
f (n) ≥ C · g (n) for all sufficiently large n.
Definition (Combined upper and lower bound)
We say that f (n) is Θ(g (n)) if f (n) is both O(g (n)) and Ω(g (n)).
Definitions of asymptotic complexity
Let f (n) denote the number of operations needed for an algorithm to
solve a problem of size n. We use the following asymptotic notation. The
first definition is the most frequently used one.
Definition (Asymptotic upper bound)
We say that f (n) is O(g (n)) if there is a constant C such that
f (n) ≤ C · g (n) for all sufficiently large n.
Definition (Asymptotic lower bound)
We say that f (n) is Ω(g (n)) if there is a constant C such that
f (n) ≥ C · g (n) for all sufficiently large n.
Definition (Combined upper and lower bound)
We say that f (n) is Θ(g (n)) if f (n) is both O(g (n)) and Ω(g (n)).
Example
Consider the function f (n) = n3 . Then f (n) is
I O(n4 ) and O(n3 ),
I Ω(n3 ) and Ω(n2 ),
I Θ(n3 ).
Some basic rules for computing asymptotic complexity

1. The logarithmic function log(n) grows slower than a polynomial

function like n2 , which grows slower than a exponential function like
2n .
2. If f1 (n) is O(g1 (n)) and f2 (n) is O(g2 (n)), then f1 (n) · f2 (n) is
O(g1 (n) · g2 (n)).
3. Asymptotically, the sum of some functions is equal to the dominant
term.
Some basic rules for computing asymptotic complexity

1. The logarithmic function log(n) grows slower than a polynomial

function like n2 , which grows slower than a exponential function like
2n .
2. If f1 (n) is O(g1 (n)) and f2 (n) is O(g2 (n)), then f1 (n) · f2 (n) is
O(g1 (n) · g2 (n)).
3. Asymptotically, the sum of some functions is equal to the dominant
term.

Example (2.)
If f1 (n) is O(n2 ) and f2 (n) is O(log(n)), then f1 (n) · f2 (n) is O(n2 log(n)).
Some basic rules for computing asymptotic complexity

1. The logarithmic function log(n) grows slower than a polynomial

function like n2 , which grows slower than a exponential function like
2n .
2. If f1 (n) is O(g1 (n)) and f2 (n) is O(g2 (n)), then f1 (n) · f2 (n) is
O(g1 (n) · g2 (n)).
3. Asymptotically, the sum of some functions is equal to the dominant
term.

Example (2.)
If f1 (n) is O(n2 ) and f2 (n) is O(log(n)), then f1 (n) · f2 (n) is O(n2 log(n)).

Example (3.)
A polynomial p(n) = ak nk + ak−1 nk−1 + · · · a1 n + a0 is O(nk ).
Some basic rules for computing asymptotic complexity

1. The logarithmic function log(n) grows slower than a polynomial

function like n2 , which grows slower than a exponential function like
2n .
2. If f1 (n) is O(g1 (n)) and f2 (n) is O(g2 (n)), then f1 (n) · f2 (n) is
O(g1 (n) · g2 (n)).
3. Asymptotically, the sum of some functions is equal to the dominant
term.

Example (2.)
If f1 (n) is O(n2 ) and f2 (n) is O(log(n)), then f1 (n) · f2 (n) is O(n2 log(n)).

Example (3.)
A polynomial p(n) = ak nk + ak−1 nk−1 + · · · a1 n + a0 is O(nk ).

Example (1. and 3.)

The function log2 (n) + n3 + 3n is O(3n ).
Asymptotic complexity of a simple example algorithm

The worst-case complexity of an algorithm is the largest number of

operations performed by an algorithm on input of a given size.
Algorithm complexity is usually expressed with O-notation: we care about
the overall growth rate, not the precise constants of proportionality.

Example (Finding the maximum element in a list)

procedure max(a1 , . . . , an : integers)
max := a1
for i := 2, . . . , n do
if ai > max then
max := ai
return max
The algorithm performs n − 1 comparisons (ai > max). Hence its
(worst-case) complexity is O(n).
Linear or sequential search

The searching problem is the problem of locating an element x in a list

(finite sequence) of distinct elements a1 , a2 , . . . , an or determining that it
is not in the list.

Algorithm (Linear search)

Loop over all elements, test if they match x, quit once x is found.

procedure linear search(x integer, a1 , . . . , an : integers)

i := 1
while i ≤ n and x 6= ai do
i := i + 1
if i ≤ n then
location := i
else
location := 0
return location
Complexity:
I At every step of while loop, 2 comparisons are made.
I After while loop one more comparison is made.
I If x = ai , then 2i + 1 comparisons are made.
I In the worst case, x is not in the list. Then an extra comparison is
made for i = n + 1, and the total number of comparisons is 2n + 2
Hence the worst-case complexity is O(n).
Insertion sort

The sorting problem is the problem of ordering the elements of a list.

Algorithm (Insertion sort)

Insertion sort is how you sort a deck of cards, picking a card from the
deck and inserting it in the right location among the already sorted cards.
procedure insertion sort(a1 , . . . , an : real numbers with n ≥ 2)
for j := 2, . . . , n do
i := 1
while aj > ai do
i := i + 1
m := aj
for k := 0, . . . , j − i − 1 do
aj−k := aj−k−1
ai := m
return {a1 , . . . , an } in increasing order
I At the 1st step, the 2nd element is put in the right position relative
to the 1st element.
I At the 2nd step, the 3rd element is put in the right position relative
to the first two (sorted) elements.
I ...
I At the jth step, the (j + 1)st element is put in the right position
relative to the first j (sorted) elements.
I ...
I At the (n − 1)st step, the nth element is put in the right position
relative to the first n − 1 (sorted) elements.
 I In the worst case, at the jth step, j + 1 comparisons (to the j
already sorted elements and to itself) are needed to put the (j + 1)st
element in the right position.
I Hence in the worst case (when we start with an already sorted list),
the total number of comparisons is
n−1
X n(n − 1) (n + 2)(n − 1)
(j + 1) = + (n − 1) =
2 2
j=1

Hence the (worst-case) complexity of insertion sort is O(n2 ).

Bubble sort

Algorithm (Bubble sort)

Successively compare adjacent elements:
procedure bubble sort(a1 , . . . , an : real numbers with n ≥ 2)
for i := 1, . . . , n − 1 do
for j := 1, . . . , n − i do
if aj > aj+1 then
m := aj
aj := aj+1
aj+1 := m
return {a1 , . . . , an } in increasing order
After one pass (i = 1), the last element is the largest element in the list.
Proof.
Let a = maxi ai . If in the original list an = a, it will still be there after
one pass, because any element (old or new) at position n − 1 will be
smaller than an (that is, the j = n − 1 comparison will have no effect.
If ak = a for some k < n, then by the same reasoning the j = k − 1
comparison has no effect and ak remains in place for the first k − 1 steps.
From the j = kth step onwards, aj > aj+1 will be true always, until at
last the old ak has moved down to the last position.
I The 1st pass guarantees that the largest element is in correct position.
I The 2nd pass guarantees that the 2 largest elements are in correct
position.
I ...
I The (n − 1)st pass guarantees that all elements are in correct position.

Figure 1: Bubble sort

 I During the ith pass, n − i comparisons are made.
I Hence the total number of comparisons is
n−1
X n−1
X n(n − 1) n(n − 1)
(n − i) = n(n − 1) − i = n(n − 1) − =
2 2
i=1 i=1

Hence the (worst-case) complexity of bubble sort is O(n2 ).

We can write bubble sort as a recursive algorithm:

Algorithm (Bubble sort recursive)

procedure bubble sort(a1 , . . . , an : real numbers with n ≥ 2)
for j := 1, . . . , n − 1 do
if aj > aj+1 then
m := aj
aj := aj+1
aj+1 := m
if n = 2 then
return {a1 , a2 }
else
return {bubble sort(a1 , . . . , an−1 ), an }
Bubble sort takes input of size n and reduces the problem to sorting
input of size n − 1, needing n − 1 comparisons to do so.
Writing fn for the number of comparisons on input of size n, we get the
recurrence relation

fn = fn−1 + (n − 1)

with initial condition f2 = 1.

We propose a solution of the form fn = αn2 + βn. Plugging this into the
recurrence relation gives:

αn2 + βn = α(n − 1)2 + β(n − 1) + n − 1

(1 − 2α)n + (α − β − 1) = 0

This is true for all n if and only if 1 − 2α = 0 and α − β − 1 = 0, or

α = 12 and β = − 21 . Hence we find again:

n(n − 1)
fn =
2
Binary search
Searching for an element in a list can be done faster if the list is sorted.

Algorithm (Binary search)

procedure binary search(x integer, a1 , . . . , an : increasing integers)
i := 1
j := n
while i < j do
m := b(i + j)/2c
if x > am then
i := m + 1
else
j := m
if x = ai then
location := i
else
location := 0
return location
The complexity of binary search is best understood by taking n = 2k − 1,
and representing all possible comparisons x > am as a binary tree with
the midpoint values of m at the nodes.
The number of steps (= maximum number of comparisons) from the
root to one of the leaves is exactly k = log2 n.
Hence the worst-case complexity is O(log n)
Example (n = 15, x = a11 )
We test m = 8, 12, 10, 11
8

4 12

2 6 10 14
1 3 5 7 9 11 13 15
We can write binary search as a recursive algorithm:
Algorithm (Binary search recursive)
procedure binary search(x integer, a1 , . . . , an : increasing integers)
if n = 0 (empty list) then
location := 0
return location
else if x = am then
location := m
return location
else if x > am then
binary search(x, am+1 , . . . , an )
else
binary search(x, a1 , . . . , am−1 )
if x = ai then
location := i
else
location := 0
return location
Binary search takes input of size n, and reduces the problem to a search
on input of size n/2, needing two comparisons to do so: one to determine
whether the midpoint is the searched for element, and one to determine
whether to proceed with the left or right half of the list.
Let f (n) be the number of comparison in binary search for input of size
n. Then
n

f (n) = f +2
2
Taking again n = 2k and writing ak = f (2k ), we get a recurrence relation

ak = ak−1 + 2

with initial condition a1 = 1.

We propose that the solution is an arithmetic progression:

ak = a + dk

Plugging this into the recurrence relation and initial condition gives

a + dk = a + d(k − 1) + 2 a1 = a + d = 1
d =2 a = −1

Hence

f (2k ) = ak = 2k − 1

or

f (n) = 2 log2 (n) − 1

We find again that the worst-case complexity of binary search is

O(log n).
Matrices

An m × n matrix is a rectangular array of numbers with m rows and n

columns. If m = n, the matrix is called square. Two matrices are equal if
they have the same number of rows and columns and the same entries in
every position.
We write A = (aij ) as a shorthand for
 
a11 a12 ... a1n
 a21 a22 ... a2n 
 
A= . .. .. 
 .. . . 
am1 am2 ... amn

The sum of two m × n matrices A = (aij ) and B = (bij ) is the matrix

A + B = (aij + bij ).
The product of an m × r P
matrix A =
(aij ) and r × n matrix B = (bij ) is
r
the m × n matrix AB = k=1 aik bkj .
Complexity of Matrix Operations

Let A and B be n × n matrices. The complexity of computing A + B is

clearly n2 = θ(n2 ).
Complexity of Matrix Operations

Let A and B be n × n matrices. The complexity of computing A + B is

clearly n2 = θ(n2 ).

In the product AB, every element is the result of a scalar product

between an n-dimensional row matrix and an n-dimensional column
matrix. This computation requires n multiplications and n − 1 additions.
Thus the total complexity of calculating all n2 elements is

n2 (n + (n − 1)) = 2n3 − n2 = θ(n3 ).