Minimum Spanning Trees

Graphs - Definitions

Graph

a set of vertices (nodes) and edges connecting them

we say G = ( V, E ) where

V is a set of vertices: V = { vi }

An edge connects two vertices: e = ( vi , vj )

E is a set of edges: E = { (vi , vj ) }

Vertices
Edges

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 2

Graphs - Definitions

Path – p of length, k, is a sequence of connected
vertices

p = <v0,v1,...,vk> where (vi,vi+1) ∈ E

< i, c, f, g, h >
Path of length 5

< a, b >
Path of length 2

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 3

Graphs - Definitions

Cycle

A graph contains no cycles if there is no path

p = <v0,v1,...,vk> such that v0 = vk
< i, c, f, g, i >
is a cycle

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 4

Graphs - Definitions

Spanning Tree

A spanning tree is a set of |V|-1 edges that connect all
the vertices of a graph

The red path connects

all vertices,
so it’s a spanning tree

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 5

Graphs - Minimum Spanning Tree

Generally there is more than one spanning tree

If a cost cij is associated with edge eij=(vi,vj)
then the minimum spanning tree is the set of edges Espan such that
C=Σ
C=Σ(cij|for all eij belonging to Espan)
Espan) is a minimum

Other ST’s can be

formed ..
• Replace 2 with 7
• Replace 4 with 11

The red tree is the

Min ST

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 6

Generic MST

Loop invariant

Prior to each iteration, A is a subset of some MST.

Add the safe edge

At each step, investigate (u,v) that can be added to A without
violating the constraint.

GENERIC-MST(G, w)
1. A ← Φ
2. while A does not form a spanning tree
3. do find an edge (u,v) that is safe for A
4. A ← A U {(u,v)}
5. return A

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 7

Graphs - Kruskal’s Algorithm

How to find the safe edge ?!!!

Calculate the minimum spanning tree

Put all the vertices into single node trees by themselves

Put all the edges in a priority queue

Repeat until we’ve constructed a spanning tree

Extract the cheapest edge

If it forms a cycle, ignore it,
else add it to the forest of trees
(it will join two trees into a larger tree)

Return the spanning tree

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 8

Graphs - Kruskal’s Algorithm

Calculate the minimum spanning tree

Put all the vertices into single node trees by themselves

Put all the edges in a priority queue

Repeat until we’ve constructed a spanning tree

Extract cheapest edge

If it forms a cycle, ignore it
else add it to the forest of trees
(it will join two trees into a larger tree)

Return the spanning tree

Note that this algorithm makes no attempt

• to be clever
• to make any sophisticated choice of the next edge
• it just tries the cheapest one!

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 9

Kruskal’s Algorithm

Cycle detection

Uses a Union-find structure

For which we need to understand a partition of a set

Partition

A set of sets of elements of a set

Every element belongs to one of the sub-sets

No element belongs to more than one sub-set

Formally:

Set, S = { si }

Partition(S) = { Pi }, where Pi = { si }
• ∀ si∈ S, si ∈ Pj

∀ j, k Pj ∩ Pk = ∅

S = ∪ Pj
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 10
Kruskal’s Algorithm

Partition

The elements of each set of a partition

are related by an equivalence relation

equivalence relations are
x~x

reflexive

transitive if x ~ y and y ~ z, then x ~ z

symmetric if x ~ y, then y ~ x

The sets of a partition are equivalence classes

Each element of the set is related to every other element

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 11

Kruskal’s Algorithm

Partitions

In the MST algorithm,
the connected vertices form equivalence classes

“Being connected” is the equivalence relation

Initially, each vertex is in a class by itself

As edges are added,
more vertices become related
and the equivalence classes grow

Until finally all the vertices are in a single equivalence
class

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 12

Kruskal’s Algorithm

Representatives

One vertex in each class may be chosen as the
representative of that class

We arrange the vertices in lists that lead to the
representative

This is the union-find structure

Cycle determination

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 13

Kruskal’s Algorithm

Cycle determination

If two vertices have the same representative,
they’re already connected and adding a further
connection between them is pointless

Procedure:

For each end-point of the edge that you’re going to add

follow the lists and find its representative

if the two representatives are equal,
then the edge will form a cycle

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 14

Kruskal’s Algorithm in operation
All the vertices are in
single element trees

Each vertex is its

own representative
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 15
Kruskal’s Algorithm in operation
All the vertices are in
single element trees

Add it to the forest,

joining h and g into a
2-element tree

The cheapest edge

is h-g

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 16

Kruskal’s Algorithm in operation
The cheapest edge
is h-g

Add it to the forest, Choose g as its

joining h and g into a representative
2-element tree
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 17
Kruskal’s Algorithm in operation
Add it to the forest,
The next cheapest edge
joining c and i into a
is c-i
2-element tree

Choose c as its
Our forest now has 2 two-element trees representative
and 5 single vertex ones
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 18
Kruskal’s Algorithm in operation
Add it to the forest,
The next cheapest edge joining a and b into a
is a-b 2-element tree

Choose b as its
representative

Our forest now has 3 two-element trees

and 4 single vertex ones
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 19
Kruskal’s Algorithm in operation
Add it to the forest,
The next cheapest edge
merging two
is c-f 2-element trees

Choose the rep of one

as its representative

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 20

Kruskal’s Algorithm in operation
The next cheapest edge The rep of g is c
is g-i
The rep of i is also c

∴ g-i forms a cycle

It’s clearly not needed!

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 21

Kruskal’s Algorithm in operation
The next cheapest edge
The rep of c is c
is c-d

The rep of d is d

∴ c-d joins two

trees, so we add it

.. and keep c as the representative

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 22

Kruskal’s Algorithm in operation
The next cheapest edge
is h-i

The rep of h is c

The rep of i is c

∴ h-i forms a cycle,

so we skip it

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 23

Kruskal’s Algorithm in operation
The next cheapest edge The rep of a is b
is a-h

The rep of h is c

∴ a-h joins two trees,

and we add it

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 24

Kruskal’s Algorithm in operation
The next cheapest edge
is b-c But b-c forms a cycle

So add d-e instead

... and we now have a spanning tree

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 25
Kruskal’s Algorithm
Algorithm Kruskal(Graph G(V,E),
Weight_function w)
←Φ
A←
for each vertex v ∈ V[G]
do MAKE-SET(v)
Sort the edges of E into nondecreasing order
by weight w
for each edge (u,v) ∈ E,
≠ FIND-SET(u)
do if FIND-SET(u)≠
then A ← A U {(u,v)}
UNION(u,v)
return A

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 26

Greedy Algorithms

At no stage did we attempt to “look ahead”

We simply made the naïve choice

Choose the cheapest edge!

Therefore, MST is an example of a greedy
algorithm

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 27

Proving MST

MST Proof

“Proof by contradiction” is usually the best approach!

Note that

any edge creating a cycle is not needed
∴ Each edge must join two sub-trees

Suppose that the next cheapest edge, ex, would join

trees Ta and Tb

Suppose that instead of ex we choose ez - a more
expensive edge, which joins Ta and Tc

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 28

Proving MST

But we still need to join Tb to Ta or some other
tree to which Ta is connected

The cheapest way to do this is to add ex

So we should have added ex instead of ez

Proving that the greedy approach is correct for
MST

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 29

MST - Time complexity

Steps

Initialise forest O( |V| )

Sort edges O( |E|log|E| )

Check edge for cycles O( |V| ) x

Number of edges O( |V| ) O( |V|2 )

Total O( |V|+|E|log|E|+|V|2 )

Since |E| = O( |V|2 ) O( |V|2 log|V| )

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 30

MST - Time complexity

Thus we would class MST as O( n2 log n )
for a graph with n vertices

This is an upper bound,
some improvements on this are known ...

Prim’s Algorithm can be O( |E|+|V|log|V| )
using Fibonacci heaps

even better variants are known for restricted cases,
such as sparse graphs ( |E| ≈ |V| )

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 31

Prim’s Algorithm

Follows the natural greedy approach

starting with the source vertex to create the spanning
tree,

add an edge to the tree that is attached at exactly one
end to the tree & has minimum weight among all such
edges.

Prim's algorithm starts from one vertex and grows
the rest of the tree an edge at a time.

As a greedy algorithm, which edge should we pick?

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 32

Why is Prim’s algorithm correct?

Theorem: Let G be a connected, weighted graph and let
E’⊂E be a subset of the edges in a MST T=(V, Er). Let V'
be the vertices incident with edges in E'. If (x,y) is an edge
of minimum weight such that x Є V’ and y is not in V', then
E U {x,y} is a subset of a minimum spanning tree.

Proof:

If the edge is in T, this is trivial.

Suppose (x,y) is not in T Then there must be a path in T from x to y
since T is connected.

If (v,w) is the first edge on this path with one edge in V', if we
delete it and replace it with (x, y) we get a spanning tree.

This tree must have smaller weight than T, since W(v,w)>W(x,y).

Thus T could not have been the MST.

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 33

Approach

In Kruskal,

the selection function

uses a greedy approach….

chooses an edge that in minimum weighted edge …. then in

increasing order…….

does not worry too much about their connection to the
previously chosen edges…except cycles.

the result is a sort of forest of trees that grows haphazardly and

later merge into a tree.

While in Prim,

the MST grows in a natural manner…

staring from an arbitrary root.

at each step, a new edge is added to a tree already constructed.

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 34

Algorithm Prim(G=(V, E), weight)
{initialization}
T ← ∅
B ← { an arbitrary memebr of N}
while B ≠ N do
find E = {u, v} of minimum
weight such that
u Є B and v Є N\B
T ← T U {u}
B ← B U {v}
return T

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 35

Kruskal and Prim – Operational differences

Consider the graph shown below:

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 36

Running Kruskal on above
Step Edge consi Connected components

Init - {1} {2} {3} {4} {5} {6} {7}

1 {1 2} {1 2} {3} {4} {5} {6} {7}
2 {2 3} {1 2 3} {4} {5} {6} {7}
3 {4 5} {1 2 3} {4 5} {6} {7}
4 {6 7} {1 2 3} {4 5} {6 7}
5 {1 4} {1 2 3 4 5} {6 7}
6 {2 5} Rejected
7 {4 7} {1 2 3 4 5 6 7}

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 37

Kruskal and Prim – Operational differences

Consider the graph shown below:

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 38

Running Prim on above
Step {u v} B
Init - {1}
1 {1 2} {1 2}
2 {2 3} {1 2 3}
3 {1 4} {1 2 3 4}
4 {4 5} {1 2 3 4 5}
5 {4 7} {1 2 3 4 5 7}
6 {5} rejected
7 {4 7} {1 2 3 4 5 6 7}

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 39

Algorithm Prim(G(V,E), weight[1..n,1..n, r)
1. for each u Є V[G]
2. do key[u] ← ∞
3. π[u]←NIL
4. key[root]←0
5. Q ← V[G] //Q is a priority queue as min-heap on key
6. while Q ≠ ∅
7. do u ← EXTRACT-MIN(Q)
8. for each v Є Adj[u]
9. do if v Є Q and(w(u,v) < key[v])
10. then π[v]←u
11. key[v]←w(v,u)

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 40

Kruskal and Prim – Operational differences

Consider the graph shown below:

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 41

Running this algorithm on above
Step Vertex Edge Key[v] Minimum in Tree V–Q
inspected inspected removed from Q
- - ∞ - Nil All vertices
Init
1 - 0 1 {1} V – {1}
1
1 1-2 key[2]=2 - -do- -do-
2
1 1-4 key[4]=4 - -do- -do-
3
2 - - 2 {1,2} V – {1 2}
3
2 2-3 key[3]=2 - -do- -do
4
2 2-5 key[5]=4 - -do- -do-
5
2 2-4 Rejected - -do- -do-
6
3 - - 3 {1 2 3} V- {1 2 3}
7
3 3-5 Rejected - -do- -do-
8
3 3-6 key[6]=6 - -do- -do-
9
4 - - 4 {1 2 3 4} V – {1 2 3 4}
10
4 4-5 key[5]=3 - -do- -do-
11
4 4-7 key[7]=4 - -do- -do-
12

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 42

continued from above
4 4-2 & 4-1 Rejected - -do- -do-
13
5 - - 5 {1 2 3 4 5} V-…
14
5 5-4, 5-7, 5-6 Rejected - -do- -do-
15
7 - - 7 {1 2 3 4 5 7} V-…
16
7 7-3 Key[6]=3 - -do- -do-
17
7 7-5, 7-4 Rejected - -do- -do-
18
6 - - 6 {1 2 3 4 5 7 6} NIL
19
Since Q is null now no further iterations
20

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 43

Clustering Problem:
Application of MST
The Clustering Problem

arises when a collection of objects are to be
classified or organized into coherent groups.

it is natural to look for how similar or dissimilar
each pair of objects is

define a distance function

objects at a larger distance from each other are less
similar to each other.

the distance may not be in terms of physical distance

distance may take more abstract meaning

e.g.

distance between two species - to be the no of years in the
course of the evolution

distance between two images – number of corresponding
pixels at which their intensity values differ by at least some
threshold
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 45
The Clustering Problem (contd)

Given a distance function on the objects

divide the objects into groups so that

objects within the same group are close

objects within the different groups are far apart.

Formally

given a set U of n objects, labeled p1,p2,p3,p4…pn

for each pair pi and pj, we have a numerical distance
d(pi,pj) with d(pi,pj)=0 and d(pi,pj)>0.

also d(pi,pj)=d(pj,pi)

given a parameter k, a k-clustering of U is a partition of
U into k nonempty sets C1, C2, C3, C4….Ck.

spacing of a k-clustering is defined as the minimum
distance between any two points lying in different clusters.
11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 46
The Clustering Problem (contd)

Given that points in different clusters should be far
apart from each other,

a natural goal is to seek a k-clustering with the
maximum possible spacing.

How many k-clustering of a set u can be there ?

exponentially many

finding out then one with maximum spacing.

How ?

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 47

Algorithm Approach

consider growing a graph on the vertex set U.

connected components become the clusters.

bring the nearby points into the same cluster.

draw an edge between the closest pair of points

add edges between pairs of points in order of increasing
distance d(pi,pj).

if pi and pj belong to the same cluster, do not add that
edge.

single link clustering

a special case of agglomerative clustering

How is this related to Kruskal’s MST.

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 48

Algorithm Approach (contd)

Follow the same method as Kruskal’s algorithm

but stop when obtaining k-clustering.

How to stop at having k connected components ?

stop just before it adds the last k – 1 edges.

Run Kruskal’s MST on the given graph fully and

then delete the k-1 most expensive edges

define the resulting structure to be k connected
components.

an example of 3-clustering

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 49

11/29/2006 Mr D C Jinwala, Algorithms & Computational Complexity, M Tech(CO), SVNIT, 200607 50