Radboud University Nijmegen

Course: Algorithms and Datastructures

Professor: Frits Vaandrager

Algorithms and Datastructures

Summary

Author:
Martan van der Straaten

June 13, 2021

 Table of Contents

Contents
Page

Week 1 2
What makes an algorithm good? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Big O, Big Omega and Big Thèta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Week 2 3
Elementary Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Big O cheatsheet for datastructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Data Structures - explained . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Breadth First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Week 3 10
Depth First Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Week 4 11
Dijkstra’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Dijkstra’s idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Heaps and Priority Queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Week 5 13
Bellman Ford Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
NetworkFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Week 6 15
Capacity scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Shortest Augmenting Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Week 7 17
Divide and Conquer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Week 8 19
Dynamic Programming - Bottom up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Week 9 21

Week 10 23
Binary SearchTrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Week 11 25
Red-Black Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Week 12 27
Hashmaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Week 13 29
Greedy Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

PAGE 1
 Week 1
Big O, Big Omega and Big Thèta

Week 1
What makes an algorithm good?
1. It should be correct, meaning it should output the intended output for any input.
2. It should be efficient. Meaning it should be as fast as possible while retaining (1). This can
also refer to efficiency regarding other resources like the covid medicine, computing power, etc.

Big O, Big Omega and Big Thèta

Big O
Suppose f,g measure the number of ”steps” of two algorithms in the worst case. Then the 𝒪 notation
gives a good way of comparing their respective growth; tells us a lot about relative execution time of
the two.

Definition of 𝒪:

𝒪(𝑔) = 𝑓 ∶ ℕ → ℝ+ ∣ ∃𝑐 ∈ ℝ+ [∃𝑛0 ∈ ℕ[∀𝑛 ≥ 𝑛0 [𝑓(𝑛) ≤ 𝑐 ∗ 𝑔(𝑛)]]]

Big Omega
Just as 𝒪 provides an asymptotic upper bound on a function, Ω provides an asymptotic lower bound:
Definition of Ω:

Ω(𝑔) = 𝑓 ∶ ℕ → ℝ+ ∣ ∃𝑐 ∈ ℝ+ [∃𝑛0 ∈ ℕ[∀𝑛 ≥ 𝑛0 [𝑐 ∗ 𝑔(𝑛) ≤ 𝑓(𝑛)]]]

Big Theta
We use the Θ-notation to denoteasymptotic tight bounds

Definition of Θ:

Θ(𝑔) = 𝒪(𝑔) ∩ Ω(𝑔)

Limits
An alternative to the methods described above is to compute the limit, and use the rules below:

1.
𝑓(𝑛)
lim = 0 , then𝑓 ∈ 𝒪(𝑔), 𝑓 ∉ Ω(𝑔)
𝑛→∞ 𝑔(𝑛)

2.
𝑓(𝑛)
lim = ∞ , then𝑓 ∉ 𝒪(𝑔), 𝑓 ∈ Ω(𝑔)
𝑛→∞ 𝑔(𝑛)

3.
𝑓(𝑛)
lim = 𝑐 , then for 𝑐 < 0 < ∞, 𝑓 ∈ Θ(𝑔)
𝑛→∞ 𝑔(𝑛)

PAGE 2
 Week 2
Data Structures - explained

Week 2
Elementary Data Structures
A data structure is a particular way of organizing data in a computer so that it can be used efficiently.

Dynamic Sets
Sets are as fundamental to computer science as they are to mathematics. Sets manipulated by
algorithms can grow, shrink, and change over time; we call such sets dynamic. Algorithms require
different operations to be performed on sets (insert, delete, membership,..).Dynamic sets that supports
these operations are called dictionaries.

Figure 1: Visualization of dictionary

Big O cheatsheet for datastructures

Figure 2: Cheatsheet for datastructures

PAGE 3
 Week 2
Data Structures - explained

Data Structures - explained

Array
Works as we know it from any programming language. Insertion and deletion are in 𝒪(1) if keys are
indices in the array (they normally are). This is however NOT a dynamic data-structure, so it is very
memory inefficient.

Stack
Operations

• Insert operation is called Push

• Delete operation is called Pop (Element that is deleted is pre-specified to be most recently
inserted element)

Figure 3: Visualization of a stack

Operations - defined

Stack operations

1 Empty(S)
2 if (s .top==0):
3 return True
4 return False
5

6 Push(S,x):
7 S.top = S.top + 1
8 S[S.top] = x
9

10 Pop(S):
11 if (Empty(S)):
12 error ”underflow”
13 else:
14 S.top = S.top−1
15 RETURN S[S.top+1]

Each of these are in 𝒪(1)

PAGE 4
 Week 2
Data Structures - explained

Queue
Operations

• Insert operation is called Enqueue

• Delete operation is called Dequeue (Element that is deleted is pre-specified to be most recently
inserted element)

Figure 4: Visualization of a queue

Operations - defined

Queue operations

1 Empty(Q)
2 if (Q.head == Q.tail):
3 return True
4 return False
5

6 Enque(Q,x):
7 Q[Q.tail] = x
8 if Q.tail == Q.length:
9 Q.tail = 1
10 else:
11 Q.tail = Q.tail +!
12

13 Deque(Q):
14 x = Q[Q.head]
15 if Q.head == Q.length:
16 Q.head = 1
17 else:
18 Q.head = Q.head+1
19 RETURN x

Each of these are in 𝒪(1)

PAGE 5
 Week 2
Data Structures - explained

Linked List
Operations

• Search, insert and delete

Figure 5: Visualization of several linked lists

Operations - defined

Queue operations

1 Search(L,k)
2 x = L.head
3 while x!=NIL and x.key!=k:
4 x=x.next
5 return x
6

7 Insert (L,x)
8 x.next = L.head
9 if L.head != NIL:
10 L.head.prev = x
11 L.head = x
12 x.prev = NIL
13

14 Delete(L,x)
15 if x.prev != NIL:
16 x.prev.next = x.next
17 else L.head = x.next
18 if x.next != NIL:
19 x.next.prev = x.prev

Insertion and Deletion are both in 𝒪(1), while Search is in 𝒪(𝑛)

PAGE 6
 Week 2
Graphs

Graphs
Graph basics
1. A graph is a pair of two sets, 𝐺 = (𝑉 , 𝐸). Here 𝑉 is the set of vertices and 𝐸 ⊆ 𝑉 × 𝑉 is the
set of edges that connect them.
2. (𝑢, 𝑣) ∈ 𝐸 is a vertex from u to v, denoted as 𝑢 → 𝑣. Self loops are possible.
3. Graphs can be directed or undirected. Directed works as expected, and in undirected we have
the additional rule that (𝑢, 𝑣) ∈ 𝐸 → (𝑣, 𝑢) ∈ 𝐸.

4. If (𝑢, 𝑣) ∈ 𝐸 then vertex v is adjacent to vertex u

5. If G is connected then there is a path between each pair of vertexes.
6. If is undirected, connected and ∣ 𝐸 ∣=∣ 𝑉 ∣ −1, meaning each pair of nodes is connected via a
UNIQUE path.

Storage Methods
Adjacency List

1. Consists of an array (Adj) containing ∣ 𝑉 ∣ lists, so one list per vertex

2. for 𝑢 ∈ 𝑉 , Adj(u) consists of all vertexes adjacent to u

3. Total storage is Θ(𝐸 + 𝑉 ) for both directed and undirected graphs

Figure 6: Visualization from slides

Pros:
1. Space-efficient when graph is sparse

2. Can easily be modified to support many graph-variants

Cons:
1. Determining if an edge (𝑢, 𝑣) ∈ 𝐸 is not efficient, complexity could grow to be in 𝒪(𝑉 ) for
worst case.

PAGE 7
 Week 2
Graphs

Adjacency Matrix

1. ∣ 𝑉 ∣ × ∣ 𝑉 ∣ matrix A

2. We number the vertexes arbitrarily

3. A is then given by:

1 if (𝑖, 𝑗) ∈ 𝐸.
𝐴[𝑖, 𝑗] = {
0 otherwise.

4. Total storage is Θ(𝑉 2 ) for both directed and undirected graphs

Figure 7: Visualization from slides

Pros:
1. Determining if an edge (𝑢, 𝑣) ∈ 𝐸 is efficient, Θ(1)

2. Can store weights instead of bits for weighted graph

Cons:
1. Not space-efficient when graph is sparse
2. Determining if an edge (𝑢, 𝑣) ∈ 𝐸 is still in Θ(∣ 𝑉 ∣)

Lists vs Matrix
General rule of thumb:
Lists for spare and Matrix for dense graphs

PAGE 8
 Week 2
Breadth First Search

Breadth First Search

For a given graph we want to find 𝑑[𝑣] = 𝛿(𝑠, 𝑣) the distance from s to each vertex, and 𝑝[𝑣] = 𝑢
such that (u,v) is the last edge on the sortest path from s to v. Aka v’s predecessor.

How it works
Expands frontier between discovered and undiscovered vertices uniformly across the breadth of the
frontier. A vertex is “discovered”the first time it is encountered during the search and a vertex is
“finished”if all vertices adjacent to it have been discovered.

This means we will explore from each vertex all those vertexes that have not yet been explored.
We do this (in practice) by appending them to a queue, and when we have discovered all adjacent
vertexes for the current vertex we pop the first element of this que and repeat the process. We do
this until the queue is empty.

Sometimes colours are used to desribe this process

• White - undescovered vertex
• Gray - discovered but not finished
• Black - finished vertex

Pseudocode
BFS algorithm

1 BFS(G,s)
2 foreach vertex u in V[G] – {s}
3 do color[u] ← white
4 d[u] ← ∞
5 𝜋[u] ← nil
6 color [ s ] ← gray
7 d[s ] ← 0
8 𝜋[ s ] ← nil
9 Q←∅
10 enqueue(Q,s)
11 while Q ≠ ∅
12 do u ← dequeue(Q)
13 foreach v ← Adj[u]
14 do if color [v] = white
15 then color[v] ← gray
16 d[v] ← d[u] + 1
17 𝜋[v] ←u
18 enqueue(Q,v)
19 color [u] ←black

Complexity
Initialization is 𝒪(∣ 𝑉 ∣)
While searching:
- Each vertex is discovered at most once, so V vertexes are queued, which is in 𝒪(∣ 𝑉 ∣)
- All adjacency lists will be walked through at most once, length of all adjacency lists is in 𝒪(∣ 𝐸 ∣).
Total complexity is in 𝒪(∣ 𝐸 ∣ + ∣ 𝑉 ∣)

Proof
Look at slides and attempt this, summary is pointless here

PAGE 9
 Week 3
Depth First Search

Week 3
Depth First Search
The idea
We explore edges out of most recently discovered vertex v. When all edges of v have been explored,
we then backtrack to explore other edges leaving v’s predecessor search as deep as possible first.
We continue this until all vertices reachable from original source are discovered. If any undiscovered
vertices remain, then we choose one as a new source and repeat search from there.
We want to know for each vertex v:
• d[v] = discovery time (v turns from white to gray)
• f [v] = finishing time(v turns from gray to black)
• 𝜋v] : predecessor of v=u, such that v was discovered during the scan of u’s adjacency list

Pseudocode
DFS algorithm

1 DFS−Visit(G, u)
2 color [u] ← GRAY
3 time ← time+ 1
4 d[u] ← time
5 foreach v ← Adj[G][u]
6 do ifcolor [v] = WHITE
7 then 𝜋[v] ← u
8 DFS−Visit(G, v)
9 color [u] ← BLACK
10 time ← time + 1
11 f [u] ← time
12

13 DFS(G)
14 foreach vertex u ← V[G]
15 do color[u] ← WHITE
16 𝜋u] ←NIL
17 time ← 0
18 foreach vertex u ← V[G]
19 do if color [u] = WHITE
20 then DFS−Visit(G, u)

Complexity
Total complexity is in 𝒪(∣ 𝐸 ∣ + ∣ 𝑉 ∣)

Proof
Look at slides and attempt this, summary is pointless here

PAGE 10
 Week 4
Dijkstra’s idea

Week 4
Dijkstra’s Algorithm
Weighted Graphs
Consider a graph G= (V, E) with edge-weight function 𝑤 ∶ 𝐸 → 𝑅. The weight of path p =
𝑘−1
𝑣1 → 𝑣2 → ... → 𝑣𝑘 is defined to be ∑𝑖=1 𝑤(𝑣𝑖 , 𝑣𝑖 + 1).

A shortest path from u to v is a path of minimum weight from u to v. The shortest-path weight from
u to v is defined as 𝛿(𝑢, 𝑣) = 𝑚𝑖𝑛{𝑤(𝑝) ∶ 𝑝is a path from u to v}. 𝛿(𝑢, 𝑣) = ∞ if no path from u to
v exists.

Dijkstra’s idea
From a given source vertex 𝑠 ∈ 𝑉 , find the shortest-path weights 𝛿(𝑠, 𝑣)for all 𝑣 ∈ 𝑉 . If all edge
weights w(u, v)are non-negative, all shortest-path weights must exist.

1. Maintain a set S of vertices whose shortest-path distances from s are known.

2. At each step add to S the vertex 𝑣 ∈ 𝑉 –𝑆 whose distance estimate from s is minimal.
3. Update the distance estimates of vertices adjacent to v

Dijkstra’s algorithm

Dijkstra’s algorithm

1 d[s ] ← 0
2 foreach v∈ V – {s}
3 do d[v] ← ∞
4 S← ∅
5 Q←V
6

7 while Q ��
8 do u ← EXTRACT−MIN(Q)
9 S←�S{u}
10 foreach v∈Adj[u]
11 do if d[v] > d[u] + w(u, v)
12 then d[v] ← d[u] + w(u, v)

Proofs
Again in Slides

No weights
Dijkstra on a weightless graph is the same as doing BFS. You could run the algorithm in paper to
see if this makes sense.

PAGE 11
 Week 5
Bellman Ford Algorithm

Complexity

Figure 8: Dijkstra’s algorithm complexity

Heaps and Priority Queues

Heapsort
A nice sorting algorithm. Its running time is 𝒪(𝑛 log 𝑛) and it sorts in place. It is rather unique
because it introduces an algorithm design technique, namely creating a data structure(heap) to
manage information during the execution of the algorithm. Such a heap has other applications beside
sorting such as Priority Queues.

Binary Heap
In a binary heap we store the data as an array, which is really easily stored and accessed. However
logically we view it as a binary tree, with parents, children, etc.

Max-Heap
In a Max-Heap structure every node excluding the root, value is at most that of its parent: A[parent[i]]
≥ A[i].

Min-Heap
In a Min-Heap structure every node excluding the root, value is at least that of its parent: A[parent[i]]
≤ A[i].

Priority Que
Operations on a max-priority queue:

• Insert(S, x) - inserts the element x into the set S (𝒪(log 𝑛), restore heap)

• Maximum(S) - returns the element of S with the largest key (𝒪(1), top-element)
• Extract-Max(S) - removes and returns the element of S with the largest key (𝒪(log 𝑛), restore
heap)
• Increase-Key(S, x, k) –increases the value of element x’s key to the new value k (𝒪(log 𝑛),
restore heap)
Similar for Min-heap, but now with minimum and decrease.

PAGE 12
 Week 5
NetworkFlow

Week 5
Bellman Ford Algorithm
Issue with Dijkstra’s algorithm: If a graph G = (V,E) contains a negative-weight cycle, then some
shortest paths may not exist. The solution? Bellman-Ford algorithm: Finds all shortest-path lengths
from a source 𝑠 ∈ 𝑉 to all 𝑣 ∈ 𝑉 or determines that a negative-weight cycle exists.
Bellman Ford

1 d[s ] ← 0
2 foreach v∈–V {s}
3 do d[v] ← ∞
4 for i ← 1 to |V–| 1
5 do for each edge (u, v) ∈E
6 do if d[v] > d[u] + w(u, v)
7 then d[v] ← d[u] + w(u, v)
8 for each edge (u, v) ∈E
9 do if d[v] > d[u] + w(u, v)
10 then report that a negative−weight cycle exists

Proofs
Makes a lot of sense, if we have not found it after |V|-1 passes, then clearly we are stuck in a loop
somewhere. For more indepth proof :) read slides.

NetworkFlow
Flow networks
Abstraction for material flowing through the edges. Consists of a graph G = (V, E) with source
𝑠 ∈ 𝑉 and sink 𝑡 ∈ 𝑉 . And a non-negative integer capacity c(e) for each 𝑒 ∈ 𝐸.

Figure 9: Flow network visualised

PAGE 13
 Week 5
NetworkFlow

Min-cut problem
A st-cut (cut) is a partition (A, B) of the vertices with 𝑠 ∈ 𝐴 and 𝑡 ∈ 𝐵. Its capacity is the sum of
the capacities of the edges from A to B.
Min-cut problem. Find a cut of minimum capacity.

Max-flow problem
An st-flow (flow) f is a function that satisfies:
• ∀𝑒 ∈ 𝐸, 0 ≤ 𝑓(𝑒) ≤ 𝑐(𝑒)

• ∀𝑣 ∈ 𝑉 − {𝑠, 𝑡}, ∑e into v 𝑓(𝑒) = ∑e out of v 𝑓(𝑒)

The value of a flow f is: val(f) = ∑e out of v 𝑓(𝑒)
Max-flow problem. Find a flow of maximum value

Ford Fulkerson Algorithm

Finding the max-flow of a graph

Figure 10: Ford Fulkerson algorithm

Lemmas and Theorems

• Flow value lemma. Let f be any flow and let (A, B) be any cut. Then, the net flow across
(A, B) equals the value of f.
• Weak duality. Let f be any flow and (A, B) be any cut. Then, v(f) ≤cap(A, B).
• Augmenting path theorem. A flow f is a max-flow iff no augmenting paths.

• Max-flow min-cut theorem Value of the max-flow = capacity of min-cut.

These lemmas are known and proven. Proofs can be found in slides.

PAGE 14
 Week 6
Capacity scaling

Week 6
Capacity scaling
Why
Generic Ford Fulkerson does not always select its paths efficiently. In the following diagram it is
possible that each iteration the flow is only increased by 1 for each run, meaning we will have to
make 2C augmenting paths. This is inefficient.

Figure 11: Bad paths

To improve the algorithm we must use care when selecting augmenting pathsSome choices lead to
exponential algorithms, however clever choices lead to polynomial algorithms. But there is an issue!
If capacities are irrational then the algorithm is not guaranteed to terminate!

The Intuition
We want to choose the augmenting path with highest bottleneck capacity: it increases our flow by
max possible amount in given iteration. However we don’t want to worry about finding exact highest
bottleneck path instead we maintain a scaling parameter 𝛿. We then let 𝐺𝑓 (𝛿) be the subgraph of
the residual graph consisting only of arcs with capacity ≥ 𝛿.

The algorithm

Capacity Scaling

1 CAPACITY−SCALING(G, s, t, c)
2 foreach edge e∈E :
3 f(e) ← 0
4 𝛿 ← largest power of 2 ≤ C
5 while (𝛿 ≥ 1):
6 𝐺𝑓 (𝛿) ← Δ−residual graph
7 while (there exists an augmenting path P in 𝐺𝑓 (𝛿))
8 f ← AUGMENT (f, c, P)
9 Update 𝐺𝑓 (𝛿)
10 𝛿 ← 𝛿 /2
11 return f.

PAGE 15
 Week 6
Matching

Shortest Augmenting Path

We can also choose to pick the shortest path. Finding the path can be done using BFS. This is also
called the Edmonds–Karp algorithm.

The algorithm

Shortest Augmenting Path

1 SHORTEST−AUGMENTING−PATH(G, s, t, c)
2 foreach e ∈ E : f(e) ← 0
3 𝐺𝑓 ← residual graph
4 while (there exists an augmenting path in 𝐺𝑓 )
5 P← BFS(𝐺𝑓 , s, t)
6 f ← AUGMENT (f, c, P)
7 Update 𝐺𝑓
8 return f

The shortest augmenting path algorithm runs in O(m2 n) time.

Matching
Given an undirected graph G = (V, E) a subset of edges 𝑀 ⊆ 𝐸 is a matching if each node appears
in at most one edge in M.
A graph G is bipartite if the nodes can be partitioned into two subsets L and R such that every edge
connects a node in L to one in R.

Bipartite Matching
Given a bipartite graph G = (L∪R, E), find a max cardinality matching.
1. Create digraph G’ = (L∪R∪ {s, t}, E’ ).
2. Direct all edges from L to R, and assign infinite (or unit) capacity.
3. Add source s, and unit capacity edges from s to each node in L.
4. Add sink t, and unit capacity edges from each node in R to t.

Visualisation

Figure 12: Bipartite Matching

Small flow-algorithms
See other tiny algortithms in 07NetworkFlowII.pdf

PAGE 16
 Week 7
Divide and Conquer

Week 7
Divide and Conquer
The Idea
1. Divide a problem into sub-problems (smaller instances of the original problem)
2. Conquer the sub-problems(trivial for small sizes,use the same strategy otherwise)
3. Combine solutions of sub-problems into a solution of the original problem

Merge Sort
Algorithm invented by John von Neumann in 1945 that uses a divide and conquer strategy:
1. Divide: Divide vector in two vectors of similar size
2. Conquer: recursively sort the two vectors (trivial for size 1)
3. Combine: Combine two ordered vectors into a new ordered vector
To do this we need an auxiliary merge function.

The algorithm

Merge-sort

1 void merge_sort(int A [], int p, int r) {

2 if (p < r) { /∗ base case ? ∗/
3 q = (p+r)/2; /∗ Divide ∗/
4 merge_sort(A,p,q); /∗ Conquer ∗/
5 merge_sort(A,q+1,r); /∗ Conquer ∗/
6 merge(A,p,q,r); /∗ Combine ∗/
7 }
8 }

Initial call on an array with n elements would be mergesort(A,1,n).

Merging
This still leaves us with the task of defining merge(). The input is:
• Array A and indices p,q,r such that p≤q≤r
• Subarrays A[p..q] and A[q+ 1..r] are sorted
• By restrictions on p,q,r, neither subarray is empty
Output: The two subarrays are merged into a single sorted subarray in A[p..r].

Idea behind merging

Think of two piles of cards.Each pile is sorted and placed face-up on a table with the smallest cards
on top. We will merge these into a single sorted pile,face-down on the table.
1. Choose the smaller of the two top cards
2. Remove it from its pile, thereby exposing a new top card
3. Place the chosen card face-down onto the output pile.
4. Repeatedly perform basic steps until one input pile is empty.Once one input pile empties, just
take the remaining input pile and place it face-down onto the output pile.
5. We do at most n operations of 𝒪(1) this should take 𝒪(𝑛) time.

PAGE 17
 Week 7
Complexity

Complexity
The substitution method
Two steps:
1. Guess the form of the solution.
2. Use mathematical induction to find the constants and show that the solution works

Example:
𝑐, if n = 1
𝑇 (𝑛) = {
2𝑇 (𝑛/2) + 𝑐𝑛, if n > 2𝑘 with 𝑘 > 0
Claim 𝑇 (𝑛) = 𝑐𝑛(log 𝑛 + 1)

Base case:
𝑇 (1) = 𝑐 = 𝑐 ∗ 1(log 1 + 1)

𝑇 (𝑛) = 2𝑇 (𝑛/2) + 𝑐𝑛
𝐼𝐻 𝑛 𝑛
= 2𝑐 (log + 1) + 𝑐𝑛
2 2
𝑛
= 𝑐𝑛(log + 1) + 𝑐𝑛
2
= 𝑐𝑛(log 𝑛 − log 2 + 1) + 𝑐𝑛
= 𝑐𝑛(log 𝑛 − 1 + 1) + 𝑐𝑛
= 𝑐𝑛(log 𝑛) + 𝑐𝑛
= 𝑐𝑛(log 𝑛 + 1)

How to find a good guess

Mostly based on intuition and pattern recognition. As example below:

Base case:
𝑇 (𝑛) = 2𝑇 (𝑛/2) + 𝑛
= 2(2𝑇 (𝑛/4) + 𝑛/2) + 𝑛
= 2(2(2𝑇 (𝑛/8) + 𝑛/4) + 𝑛/2) + 𝑛
𝑘
= 2𝑘 𝑇 ( 𝑘 + 𝑛 ∗ 𝑘
2
This is still hard to solve, so we still need some guess-work. Namely what value k should have. Taking
𝑘 = log 𝑛 gives us:

log 𝑛
2log 𝑛 ⋅ 𝑇 ( ) + 𝑛 ∗ log 𝑛 = 𝑛𝑇 (1) + 𝑛 log 𝑛
2log 𝑛
= 𝑛 ∗ 0 + 𝑛 log 𝑛
= 𝑛 log 𝑛

PAGE 18
 Week 8
Dynamic Programming - Bottom up

Week 8
Dynamic Programming - Bottom up
Memoization
When we use recurrence relations it is easily possible that we recompute something quite a lot of
times. This is very inefficient, and can slow down our algorithm enormously. A solution to this is
dynamic programming. Dynamic programming trades space-complexity for time-complexity by stor-
ing results of computations such that the next time we don’t have to recompute them. This is called
Memoization.

Fibonacci with memoization

A good example of something that will become a lot quicker when using memoization is a function
computing the Fibonacci sequence. Without it we would have to do a lot of recomputations, giving
us an exponential(!) time complexity. With memoizations each number has to be calculated only
once, so we get linear (!) time complexity. The cost for this is that we also have to store all these
solutions, giving us linear space complexity.
We can improve this space complexity by working our way down in the tree breadth first rather than
depth first. This would allow us to only store the 2 solutions below it, since we don’t yet need any
others.

Floyd Warshal Algorithm

Given a weighted graph G = (V,E) without negative cost cycles it will find the shortest paths between
all pairs of vertices.

Naive approach
1.Enumerate all pairs of vertices (s,t)
2.Use Bellman-Ford to compute shortest path from s to t
Compexity: 𝒪(𝑛2 ) pairs, 𝒪(𝑛2 ) to get shortest path gives 𝒪(𝑛4 )

A Sequence of Distance Matrixes

Length of the shortest path from i to j that only goes through vertices {1,2,...,k}. The matrix with
k = n contains our solutions.

Figure 13: Distance Matrix

This gives us the following recurrence reletation:

𝑘 𝑘−1 𝑘−1 𝑘−1
𝑑𝑖𝑗 = 𝑚𝑖𝑛(𝑑𝑖𝑗 , 𝑑𝑖𝑘 + 𝑑𝑘𝑗 )

Where the first argument of min is shortest path from i to j that does not go through k, and the
second argument is the path through k.

PAGE 19
 Week 8
Dynamic Programming - Bottom up

The algorithm

Shortest Augmenting Path

1 Floyd−Warshall(n x n matrix 𝐷0 )
2 for (k = 1 to n){
3 𝐷𝑘 = new n x n matrix;
4 for ( i = 1 to n)
5 for ( j = 1 to n)
6 𝐷𝑘 [ i ][ j ] = min(𝐷𝑘−1 [i][j] , 𝐷𝑘−1 [ i ][ k] + 𝐷𝑘−1 [k][ j ]);
7 }
8 return 𝐷𝑛 ;

This has a time- and space-complexity of 𝒪(𝑛3 ).

Matrix chain Product

Given a list of matrixes find a parenthesization that minimizes the number of multiplicative operations.

Naive approach
List all possible parenthesizations and pick the best one. This has an 𝒪(𝑛4 ) time complexity, not ideal.

Recurrence relation

0, if i = j
𝑚[𝑖, 𝑗] = {
𝑚𝑖𝑛{𝑚[𝑖, 𝑘] + 𝑚[𝑘 + 1, 𝑗] + 𝑝𝑖−1 𝑝𝑘 𝑝𝑗 }, if 𝑖 < 𝑗, for 𝑖 ≤ 𝑘 < 𝑗
In the first case we only have 𝐴𝑖 .
In the second case we take the min, over all possible breaking positions k. By adding the breaks+their
product.

Reconstructing the solution

Keep another matrix s, where s[i,j] contains the optimal breaking position for the sequence
Ai, Ai+1,...,Aj

Example
A1: 5×4
A2: 4×6
A3: 6×2
A4: 2×7

Figure 14: Matrix parenthesis example worked out

PAGE 20
 Week 9
Dynamic Programming - Bottom up

Week 9
Weighted Interval Scheduling
Given jobs with starting- and ending times and a value, give the maximal subset of compatible jobs.

Recurrence relation

0, if 𝑗 = 0
𝑚[𝑖, 𝑗] = {
𝑚𝑎𝑥{𝑣𝑗 + 𝑉 [𝑝(𝑗)], 𝑣[𝑗 − 1]}, if 𝑗 > 0
First case is if we have no jobs.
Second case is if we have jobs, then we take the best choice amongst Including j and the compatible
jobs with j and not including j.

Exercise

Figure 15: Exercise for reader, find maximal solution

Text Justification
Given a sequence of words, how do we split them nicely over lines.

Badness
Suppose words 𝑤𝑖 , ...𝑤𝑗 form one line. Where lines have length L.

∞, if length(𝑖, 𝑗) > 𝐿
badness(i,j) = {
(𝐿 − 𝑙𝑒𝑛𝑔𝑡ℎ(𝑖, 𝑗))3 , otherwise

where 𝑙𝑒𝑛𝑔𝑡ℎ(𝑖, 𝑗) = 𝑝𝑖 + ... + 𝑝𝑗+ spaces between words

Best arrangement
𝑘
Split words into lines 𝑙1, 𝑙2, ..., 𝑙𝑘 that minimize ∑𝑖=1 𝑏𝑎𝑑𝑛𝑒𝑠𝑠(𝑙𝑖 )

T[j] = minimal badness for words 𝑤1 𝑤2 ...𝑤𝑗 .

0, if𝑗 = 0
𝑇 [𝑗] = {
min {𝑇 [𝑖 − 1] + 𝑏𝑎𝑑𝑛𝑒𝑠𝑠(𝑖, 𝑗)}, 𝑗 > 0
1≤𝑖≤𝑗

PAGE 21
 Week 9
Dynamic Programming - Bottom up

Exercise

Figure 16: Exercise for reader, find maximal solution

Others
• Longest common substructure

• Optimal Binary SearchTree

• Cutting Cloth

Attempt these, finding own recurrence relation relying on bottom-up dynamic programming.

PAGE 22
 Week 10
Binary SearchTrees

Week 10
Binary SearchTrees
A binary search tree is a datastructure. In this datastructure, ordered elements can be stored in such
a way that insertion and lookups can be done really quick. We can do this quicker than in say an
array because we can use the tree structure to find our element in 𝒪(log 𝑛) rather than 𝒪(𝑛) time.

Figure 17: Binary search tree

A binary searchtree consists of nodes. Each nodes has at most 2 children, with the left being smaller
than or equal to its parent and the right child larger than or equal to its parent.

Search
We move through our tree structure based on the fact that 𝑥.𝑙𝑒𝑓𝑡 ≤ 𝑥 ≤ 𝑥.𝑟𝑖𝑔ℎ𝑡 until we have found
our key.
Iterative find

1 search(x,k)
2 #Original x is top−node of tree
3 while x != NULL and k!= x.key:
4 if (k<x.key):
5 x=x.left
6 else:
7 x=x.right
8 return x

Insert
We move through our tree structure based on the fact that 𝑥.𝑙𝑒𝑓𝑡 ≤ 𝑥 ≤ 𝑥.𝑟𝑖𝑔ℎ𝑡 until we have found
the correct position to insert the new value such that the tree structure remains intact.

PAGE 23
 Week 10
Binary SearchTrees

Iterative find

1 insert(T, z):
2 y = NULL
3 x = T.root
4 while x!= NULL:
5 y=x
6 if z.key < x.key:
7 x=x.left
8 else:
9 x=x.right
10 z.p = y
11 if (y==NIL):
12 T.root = z
13 elif z.key<y.key:
14 y. left = z
15 else:
16 y. right = z

Delete
Deletion is harder than insertion. We must namely remove an element and still keep the tree-structure
intact.

Lazy deletion
We could use lazy deletion, where we don’t actually remove it, we just flip a bit stating it is no longer
active. This makes deletion a lot quicker, but lookup times may grow quickly and we might have to
modify certain functions.

Deletion
If we use regular deletion we have 3 different cases:
1. Leaf case: Just remove the node

2. One child case: Replace pointer with pointer to child, and remove node.
3. Two child case: Replace with successor
The successor named above is a descendant whose value is guaranteed to be between the left and
right subtrees.

AVL Trees
AVL trees are balanced Binary Search Trees. This means that insertion and deletion become more
complex but the lookup-time is decreased since we no longer have a worst case of 𝒪(𝑛) but rather of
about 𝒪(log 𝑛).

PAGE 24
 Week 11
Red-Black Trees

Week 11
Red-Black Trees
Red Black Trees are a form of binary search trees with 1 extra bit per node: the attribute color, which
is either red or black. All empty trees (leaves) are colored black. We use a single sentinel, NIL, for
all the leaves of red-black tree T, with color[nil]= black.The root’s parent is also nil[T].

Properties
1. Every node is either red or black
2. The root is black
3. Every leaf (nil) is black
4. If a node is red, then both its children are black
5. For each node, all paths from node to descendant leaves contain same number of black nodes

Height of a node:
h(x) = number of edges in a longest path to a leaf

Black-height of a node x, bh(x):

bh(x)= number of black nodes (including nil[T ]) on the path from x to leaf, not counting x (This is
well defined because of 5.).

Black-height of red-black tree is black-height of its root.

relation:

𝑏ℎ(𝑥) ≤ ℎ(𝑥) ≤ 2𝑏ℎ(𝑥)

Red-black trees are balanced, meaning all operations can be performed in 𝒪(log 𝑛) time. Opera-
tions are more difficult though, searching for an element can be done exactly as in BSTs, but insertion
and deletion are not straightforward.

Rotations
Rotations are the basic tree-restructuring operation for almost all balanced search trees. Rotation
takes a search tree and a node and changes pointers to change the local structure making sure they
won’t violate the binary-search-tree property.
Left rotation and right rotation are inverses.

Figure 18: Rotations in a tree

Time complexity is 𝒪(1) for both Left-Rotate and Right-Rotate, since a constant number of pointers
are modified.

PAGE 25
 Week 11
Red-Black Trees

Insertion
Insertion must preserve all red-black properties. The basic idea is to use a slightly modified version
of Tree-Insert from BST to insert a node x into T. We color the node x red, and fix the modified tree
by re-coloring nodes and performing rotation to restore RB tree property.
The idea
We want to move the problem up the tree. We do this with a while loop with condition:p[z] is red.
z initially points to inserted node.
Loop invariant: We know that z is red, we can then say:
If p[z] is the root, then p[z] is black
There is at most one red-black violation:
Property 2: z is a red root, or
Property 4: z and p[z] are both red

From this we get 3 cases we can distinguish between that will solve all situations and make sure
we end with a valid red-black structure and a black parent node. Meaning we will always have a valid
tree after them.

Figure 19: Case 1, we have a red uncle y

Figure 20: Case 2, we have a black uncle y and z is a right child

Figure 21: Case 3, we have a black uncle y and z is a left child

The algorithm for insertion is now quite straightforward. We insert the node into the tree as we
normally would, and after that we run a while loop with the condition named above. In it we test for
these three cases, fixing the tree structure. At the end we must make sure to set the root-node to
black.

Deletion
The same idea but different cases. Look at slides for exact details, but try this for yourself first.
Important difference is that we now might have too many or too few black-nodes in this subtree,
meaning we have to fix that.

PAGE 26
 Week 12
Hashmaps

Week 12
Hashmaps
We want to use a hash-function to map our record we want to insert into our table. The issue is that
since hash-functions contain less possibilities that the entire universe of possibilities, we will come
across some collisions. The way we deal with them depends on which algorithm we use.

Chaining
Link records in the same slot into a list.
The worst case is that everything hashes into the same list, this will result in an 𝒪(𝑛) time complexity
if S=|n|.

Actual Complexity
We make the assumption of simple uniform hashing: Each key 𝑘 ∈ 𝑆 is equally likely to be hashed
to any slot of table T, independent of where other keys are hashed. Let n be the number of keys in
the table, and let m be the number of slots. Define the load factor of T to be 𝛼 = 𝑛/𝑚 =average
number of keys per slot.

The expected time for an unsuccessful search for a record with a given key is = Θ(1 + 𝛼).The
expected search time = Θ(1)if 𝛼 = 𝒪(1), or equivalently, if n= 𝒪(𝑚).
For a succesful search there is a very good and concise slide set online. A good hash-function
We have to have a hash-function that guarantees the assumption of simple uniform hashing.

The division method:

Assuming all keys are integers, we define ℎ(𝑘) = 𝑘 mod 𝑚. Pick m to be a prime not too close to
a power of 2or 10 and not otherwise used prominently in the computing environment.

Multiplication method:
Assuming that all keys are integers, m= 2r, and our computer has w-bit words. Define ℎ(𝑘) = (𝐴 ⋅ 𝑘
mod 2𝑤 )rsh(𝑤–𝑟).
Where rsh is the “bitwise right-shift” operator and A is an odd integer in the range 2𝑤–1 < 𝐴 < 2𝑤.
We don’t want to pick A too close to 2𝑤−1 or 2𝑤 .

Pros are that multiplication modulo 2𝑤 is fast compared to division and the rsh operator is fast.

Open addressing
When inserting, we systematically probe the table until an empty slot is found. The hash function
depends on both the key and probe number, h: 𝑈 × {0, 1, ..., 𝑚–1} → {0, 1, ..., 𝑚–1}. The probe
sequence ℎ(𝑘, 0), ℎ(𝑘, 1), ..., ℎ(𝑘, 𝑚–1) should be a permutation of {0, 1, ..., 𝑚–1}.

The downside is that the table may fill up, and deletion is difficult (but not impossible).

Linear Probing
Given an ordinary hash function h’(k), linear probing uses the hash function h(k,i) = (h’(k) +i) mod
m. This method, though simple, suffers from primary clustering, where long runs of occupied slots
build up, increasing the average search time. Moreover, the long runs of occupied slots tend to get
longer.

Double hashing
Given two ordinary hash functions h1(k) and h2(k), double hashing uses the hash function h(k,i)
= (h1(k) +i �h2(k)) mod m. This method generally produces excellent results, but h2(k) must be
relatively prime to m. One way is to make m a power of 2 and design h2(k) to produce only odd
numbers.

PAGE 27
 Week 12
Hashmaps

Actual Complexity
We make use of the assumption of uniform hashing: Each key is equally likely to have any one of the
m! permutations as its probe sequence.

Theorem.
Given an open-addressed hash table with load factor 𝛼 = 𝑛/𝑚 < 1, the expected number of probes
in an unsuccessful search is at most 1/(1–𝛼).
Proof of the theorem:

• At least one probe is always necessary.

• With probability n/m, the first probe hits an occupied slot, and a second probe is necessary.
• With probability (n–1)/(m–1), the second probe hits an occupied slot, and a third probe is
necessary.

• etc
𝑛−𝑖 𝑛
Observe that 𝑚−𝑖 <𝑚 = 𝛼 for i= 1, 2, ..., n.
Therefore, the expected number of probes is
𝑛 𝑛−1 𝑛−2
1+ (1 + (1 + (...)))
𝑚 𝑚−1 𝑚−2
Which simplifies to

1 + 𝛼(1 + 𝛼(1 + 𝛼(...)))

Giving us
∞
1
∑ 𝛼𝑖 =
𝑖=0
1−𝛼

Meaning that if 𝛼 is constant, then accessing an open-addressed hash table takes constant time.

PAGE 28
 Week 13
Greedy Algorithms

Week 13
Greedy Algorithms
Greedy Algorithms are easy and fun. They are basically the definition of, what if we just try the thing
that makes the most sense, and then it immediately works. There is some more nuance to it, but
that is the gist of them. Below some examples will be provided.

Stable Marriage problem

Matching
Consider a set M = 𝑚1 , ..., 𝑚𝑛 of men and a set W = 𝑤1 , ..., 𝑤𝑛 of women. A matching is a set S
of ordered pairs in 𝑀 × 𝑊 such that each member of M and each member of W appears in at most
one pair. A perfect matching is a matching S with the property that each member of M and each
member of W appears in exactly one pair.

Ranking
Each man ranks all the woman; ties are not allowed, each woman, analogously, ranks all the man.
Let S be a perfect matching and let (B,A) and (L,M) be two pairs in S. We say that (B,M) is an
instability of S if B prefers M over A, and M prefers B over L. Matching S is stable if it contains no
instability.
Gale-Shapley Algorithm

Figure 22: The Gale-Shapley Algorithm

This guarantees that each man will end up with the highest possible person on their list, taking into
account that the matching must be stable. For woman this is not guaranteed.

Cashiers algorithm
Given currency denominations: 1, 5, 10, 25, 100, devise a method to pay amount to customer using
fewest number of coins.

The Algorithm
The cashiers algorithm tells us to, at each iteration, add coin of the largest value that does not take
us past the amount to be paid. Correctness
This is correct for the named integers, but other examples might not hold. Given stamps worth 100,

PAGE 29
 Week 13
Greedy Algorithms

70 and 1 we should pay 140. The cashiers algorithm tells us to take 100 at first, but then we’re
already doomed, since the solution is 70+70.

Interval Scheduling
Job j starts at 𝑠𝑗 and finishes at 𝑓𝑗 . Two jobs are compatible if they don’t overlap. We want to find
the maximum subset of mutually compatible jobs.

The Algorithm
Sort on finishing time and add job if compatible with already picked jobs.

Interval Partitioning
The same can be done when partitioning resources. You sort them on finishing time as well, and
assign a resource if available otherwise allocate a new resource.

Minimizing Lateness
A single resource processes one job at a time. Job j requires 𝑡𝑗 units of processing time and is due at
time 𝑑𝑗 . If j starts at time 𝑠𝑗 , it finishes at time 𝑓𝑗 = 𝑠𝑗 + 𝑡𝑗 . Lateness: 𝑙𝑗 = 𝑚𝑎𝑥{0, 𝑓𝑗 –𝑑𝑗 }.

We want to schedule all jobs minimizing maximum lateness L = 𝑚𝑎𝑥𝑗 𝑙𝑗

The Algorithm
Sort on earliest deadline first. Start with t=0. Assign the current job to the timeframe [𝑡, 𝑡 + 𝑡𝑗 ] and
set 𝑠𝑗 = 𝑡 and 𝑓𝑗 = 𝑡 + 𝑡𝑗 . Then increase t with 𝑡𝑗 . This means we will never be idle.

Minimal Spanning Trees

Properties of Spanning Tree

• T is a spanning tree of G.
• T is acyclic and connected.
• T is connected and has n – 1 edges.
• T is acyclic and has n – 1 edges.
• T is minimally connected: removal of any edge disconnects it.
• T is maximally acyclic: addition of any edge creates a cycle.
• T has a unique simple path between every pair of nodes.
A minimum spanning tree is a spanning tree whose sum of edge costs is minimized.

Cycles and Cutsets

A cycle is a path with no repeated nodes or edges other than the starting and ending nodes.

A cut is a partition of the nodes into two nonempty subsets S and V–S. The cutset of a cut S
is the set of edges with exactly one endpoint in S.

The Greedy Algorithm

• Red rule: Let C be a cycle with no red edges. Select an uncolored edge of C of max weight
and color it red.
• Blue rule: Let D be a cutset with no blue edges. Select an uncolored edge in D of min weight
and color it blue.
• Greedy algorithm: Apply the red and blue rules (non-deterministically!) until all edges are
colored. The blue edges form an MST. Note: you can stop once n – 1 edges colored blue.

PAGE 30
 Week 13
Greedy Algorithms

Prims Algorithm
Initialize S = any node.
Repeat n – 1 times:

• Add to tree the min weight edge with one endpoint in S.

• Add new node to S.
This is just a special case of the greedy algorithm (namely just the blue rule repeatedly applied to S)

Kruskal’s Algorithm
Consider edges in ascending order of weight: Add to tree unless it would create a cycle.

This is just a special case of greedy algorithm with 2 cases.

• Case 1: both endpoints of e in same blue tree then color red by applying red rule to unique
cycle.

• Case 2. If both endpoints of e are in different blue trees then color blue by applying blue rule
to cutset defined by either tree.

PAGE 31