BACKGROUND: A Brief Introduction Graphs – definitions & representations

to Graph Theory
ä Graph theory is a fundamental tool in sparse matrix techniques.
• General definitions; Representations;
DEFINITION. A graph G is defined as a pair of sets G = (V, E)
• Graph Traversals; with E ⊂ V × V . So G represents a binary relation. The graph
• Topological sort; is undirected if the binary relation is symmetric. It is directed
otherwise. V is the vertex set and E is the edge set.

If R is a binary relation between elements in V then, we can represent

it by a graph G = (V, E) as follows:
(u, v) ∈ E ↔ u R v

Undirected graph ↔ symmetric relation

4-2 – graphBG

1 2 1 2 Graphs – Examples and applications

ä Applications of graphs are numerous.

1. Airport connection system: (a) R (b) if there is a non-stop flight
4 3 4 3
from (a) to (b).
(1 R 2); (4 R 1); (2 R 3); (3 (1 R 2); (2 R 3); (3 R 4); (4 2. Highway system;
R 2); (3 R 4) R 1)
3. Computer Networks;
- 1 Given the numbers 5, 3, 9, 15, 16, show the two graphs
representing the relations 4. Electrical circuits;
R1: Either x < y or y divides x. 5. Traffic Flow;
R2: x and y are congruent modulo 3. [ mod(x,3) = mod(y,3)] 6. Social Networks;
ä |E| ≤ |V |2. For undirected graphs: |E| ≤ |V |(|V | + 1)/2. 7. Sparse matrices;
ä A sparse graph is one for which |E|  |V |2. ...
4-3 – graphBG 4-4 – graphBG
 Basic Terminology & notation: Representations of Graphs

ä If (u, v) ∈ E, then v is adjacent to u. The edge (u, v) is ä A graph is nothing but a collection of vertices (indices from 1 to
incident to u and v. n), each with a set of its adjacent vertices [in effect a ’sparse matrix
ä If the graph is directed, then (u, v) is an outgoing edge from u without values’]
and incoming edge to v ä Therefore, can use any of the sparse matrix storage formats -
ä Adj(i) = {j|j adjacent to i} omit the real values arrays.
Adjacency matrix Assume V =
ä The degree of a vertex v is the number of edges incident to v. 
{1, 2, · · · , n}. Then the adjacency 1 if (i, j) ∈ E
Can also define the indegree and outdegree. (Sometimes self-edge ai,j =
matrix of G = (V, E) is the n × n 0 Otherwise
i → i omitted)
matrix, with entries:
ä |S| is the cardinality of set S [so |Adj(i)| == deg( i) ]
ä A subgraph G0 = (V 0, E 0) of G is a graph with V 0 ⊂ V and
E 0 ⊂ E.

4-5 – graphBG 4-6 – graphBG

Representations of Graphs (cont.) Dynamic representation: Linked lists

1 2 1
  Null

1 2 Null
 1 
  3
 1 1
Null

1 4 3 4 Null

Example: 5 Null
  2
1 1 1 6 Null

1 1 
 
 1 1 ä An array of linked lists. A linked list associated with vertex i,
1 1 contains all the vertices adjacent to vertex i.
4 3
ä General and concise for ’sparse graphs’ (the most practical situ-
ations).
ä Not too economical for use in sparse matrix methods
4-7 – graphBG 4-8 – graphBG
 More terminology & notation - 2 Find cycles in this graph: A path in an indirected graph
1 5 1 5

ä For a given Y ⊂ X, the section graph of Y is the subgraph 2 2

3 3
GY = (Y, E(Y )) where 4 4

E(Y ) = {(x, y) ∈ E|x ∈ Y, y in Y }

7 6 7 6

ä A section graph is a clique if all the nodes in the subgraph are ä A path w0, . . . , wk is simple if the vertices w0, . . . , wk are
pairwise adjacent (→ dense block in matrix) distinct (except that we may have w0 = wk for cycles).
ä A path is a sequence of vertices w0, w1, . . . , wk such that ä An undirected graph is connected if there is path from every
(wi, wi+1) ∈ E for i = 0, . . . , k − 1. vertex to every other vertex.
ä The length of the path w0, w1, . . . , wk is k (# of edges in the ä A digraph with the same property is said to be strongly connected
path)
ä A cycle is a closed path, i.e., a path with wk = w0.
ä A graph is acyclic if it has no cycles.
4-9 – graphBG 4-10 – graphBG

ä The undirected form of a directed graph the undirected graph Root 1 5 Root 1 5

obtained by removing the directions of all the edges.

2 2
ä Another term used “symmetrized” form -
3 3 4
4
ä A directed graph whose undirected form is connected is said to
6 6
be weakly connected or connected. 9 9
8 8
ä Tree = a graph whose undirected form, i.e., symmetrized form,
10
is acyclic & connected 10
7 7
ä Forest = a collection of trees
ä Parent-Child relation: immediate neighbors of root are children.
ä In a rooted tree one specific vertex is designated as a root.
Root is their parent. Recursively define children-parents
ä Root determines orientation of the tree edges in parent-child
ä In example: v3 is parent of v6, v8 and v6, v8 are chidren of v3.
relation
ä Nodes that have no children are leaves. In example: v10, v7, v8, v4
ä Descendent, ancestors, ...
4-11 – graphBG 4-12 – graphBG
 Tree traversals Graphs Traversals – Depth First Search

ä Tree traversal is a process of visiting all vertices in a tree. Typically ä Issue: systematic way of visiting all nodes of a general graph
traversal starts at root.
ä Two basic methods: Breadth First Search (wll’s see later) & ...
ä Want: systematic traversals of all nodes of tree – moving from a
ä Depth-First Search.
node to a child or parent
ä Preorder traversal: Visit parent before children [recursively] Algorithm List = DF S(G, v) (DFS from v)
In example: v1, v2, v9, v10, v3, v8, v6, v7, v5, v4 • Visit and Mark v;
• for all edges (v, w) do
ä Postorder traversal: Visit children before parent [recursively]
– if w is not marked then List = DF S(G, w)
In example : v10, v9, v2, v8, v7, v6, v3, v4, v5, v1
– Add v to top of list produced above

ä If G is undirected and connected, all nodes will be visited

ä If G is directed and strongly connected, all nodes will be visited
4-13 – graphBG 4-14 – graphBG

Depth First Search – undirected graph example Depth First Search – directed graph example
A A

B C
ä Assume adjacent nodes are B C
listed in alphabetical order.
- 3 DFS traversal from A ? D G

D G
E F

E F

ä Assume adjacent nodes are listed in alphabetical order.

- 4 DFS traversal from A?
4-15 – graphBG 4-16 – graphBG
 function [Lst, Mark] = dfs(u, A, Lst, Mark) Depth-First-Search Tree: Consider the parent-child relation: v is
%% function [Lst, Mark] = dfs(u, A, Lst, Mark)
%% dfs from node u -- Recursive a parent of u if u was visited from v in the depth first search
%%----------------------------------- algorithm. The (directed) graph resulting from this binary relation
[ii, jj, rr] = find(A(:,u)); is a tree called the Depth-First-Search Tree. To describe tree: only
Mark(u) = 1;
for k=1:length(ii) need the parents list.
v = ii(k);
if (~Mark(v)) ä To traverse all the graph we need a DFS(v,G) from each node v
[Lst, Mark] = dfs(v, A, Lst, Mark); that has not been visited yet – so add another loop. Refer to this as
end
end DFS(G)
Lst = [u,Lst]
ä When a new vertex is visited in DFS, some work is done. Example:
we can build a stack of nodes visited to show order (reverse order:
easier) in which the node is visited.

4-17 – graphBG 4-18 – graphBG

EXAMPLE 3 5 We assume adjacency list

o o is in increasing order. Back edges, forward edges, and cross edges
1111
000000
11 A
| | [e.g: Adj(4)=(1,5,6,7)] 0000
1111
0000
1111
0 111
1
00
11
0
1
00
11
0
1
000
000
111
| | 0000
1111
0000
11110 111
1
0
1
000
000
111
0000
1111 000
111
0 111
1 000
0000
11110
1 000
111
| |4 6 0000
1111
0000
11110
1
0
1 000
111
000
111
1 o-----o-----o 0000
11110
1
0000
1111
B11
000
1 0 111
000
111111
000000
0000
1111
00
110
1 1
0
1
0
1
00
11
0C
1
00
11
/ \ | / 0000
1111
0
1
0000
11110
1 00
11
0 1
1
0000
11110 00
11
0
1
0000
11110
1 00
11
/ \ | / 0
1
0 1
0000
1111
1
0000
1111
0
1
0
0
1
00
11
00
11
00
11
/ \ | / 0000
1111
0
1 0
1 00
11
0000
11110
1
0 1
1 0 00
11
0000
1111
0 1
1 0 00
11
/ \ | / 0000
1111
0
1
0000
1111
0
1
0000
1111
0
1
0
1
0
1
00
11
00
11
00
11
/ \|/ 0000
1111
0
1 0
1
0
1 00 G
11
0000
1111
D00
11
00
0000
1111
0000
1111 0 1
1 0 000
111
11
00
11
0000
1111
0000
1111 000
111
11
00
2 o 7 o 0000
1111
0000
1111
0000
1111
0000
1111
0000
1111
0
1
0000
1111 000
111
0 111
1
0000
1111
0
1 000
0000
1111
0000
1111
0000
11110 111
0000
1111
1
0000
1111
0
1
000
000
111
DFS traversal: 1 --> 2 --> 3 --> 4 --> 5 --> 6 --> 7 0000
1111
0000
1111
0000
1111
0
1 000
111
0000
1111
0000
1111
0000
11110000
1111
0
1 000
111
Parents list: 1 1 1 4 4 6 00
11 0000
1111
0000
11110000
1111
11111111
00000000 0
1 000
111
00
11
E11 0000
1111
0000
1111
00
11000
111
00 00 F
11
3 o o 5
| | ä Thick red lines: DFS traversal tree from A
| |4 6
1 o------o------o <----- Depth First ä A → F is a Forward edge
/ / Search Tree
/ / ä F → B is a Back edge
2 o 7 o
ä C → B and G → F are Cross-edges.
4-19 – graphBG 4-20 – graphBG
 A
Properties of Depth First Search
ä Consider the ‘List’ produced by
B C ä If G is a connected undirected (or strongly connected) graph,
DFS. then each vertex will be visited once and each edge will be inspected
Lst=[A, C, G, B, D, F, E] at least once.
ä Order in list is important for D G
some algorithms ä Therefore, for a connected undirected graph, The cost of DFS is
O(|V | + |E|)
E F
ä If the graph is undirected, then there are no cross-edges. (all
ä Notice: Label nodes in List from 1 to n . Then: non-tree edges are called ‘back-edges’)
• Tree-edges / Forward edges : labels increase in → Theorem: A directed graph is acyclic iff a DFS search of G yields
• Cross edges : labels in/de-crease in → [depends on labeling] no back-edges.

• Back-edges : labels decrease in → ä Terminology: Directed Acyclic Graph or DAG

4-21 – graphBG 4-22 – graphBG

Topological Sort Topological Sorting: A first algorithm

The Problem: Given a Directed Acyclic Graph (DAG), order the Property exploited: An acyclic Digraph must have at least one
vertices from 1 to n such that, if (u, v) is an edge, then u appears vertex with indegree = 0.
before v in the ordering.
- 5 Prove this
ä Equivalently, label vertices from 1 to n so that in any (directed)
Algorithm:
path from a node labelled k, all vertices in the path have labels >k.
ä Many Applications ä First label these vertices as 1, 2, . . . , k;

ä Prerequisite requirements in a program ä Remove these vertices and all edges incident from them

ä Scheduling of tasks for any project ä Resulting graph is again acyclic ... ∃ nodes with indegree = 0.
label these nodes as k + 1, k + 2, . . . ,
ä Parallel algorithms;
ä Repeat..
ä ...
- 6 Explore implementation aspects.
4-23 – graphBG 4-24 – graphBG
 Alternative methods: Topological sort from DFS Lst = DFS(G,v)

ä Depth first search traversal of graph. • Visit and Mark v;

ä Do a ‘post-order traversal’ of the DFS tree. • for all edges (v, w) do
– if w is not marked then Lst = DF S(G, w)
Algorithm Lst = T sort(G) (post-order DFS from v)
Mark = zeros(n,1); Lst = ∅ • Lst = [v, Lst]
for v=1:n do: ä Topological order given by the final Lst array of Tsort
if (Mark(v)== 0)
[Lst, Mark] = dfs(v, G, Lst, Mark); - 7 Explore implementation issue
end - 8 Implement in matlab
end
- 9 Show correctness [i.e.: is this indeed a topol. order? hint: no
back-edges in a DAG]
ä dfs(v, G, Lst, Mark) is the DFS(G,v) which adds v to the top of
Lst after finishing the traversal from v
4-25 – graphBG 4-26 – graphBG

GRAPH MODELS FOR SPARSE MATRICES Graph Representations of Sparse Matrices. Recall:
• See Chap. 3 of text
Adjacency Graph G = (V, E) of an n × n matrix A :
• Sparse matrices and graphs.
V = {1, 2, ...., N } E = {(i, j)|aij 6= 0}
• Bipartite model, hypergraphs

• Application: Paths in graphs, Markov chains ä G == undirected if A has a symmetric pattern

Example:
  2
  2
1 1
? ? ?
   
 ?  ? ? 
   
 ? ?  ? ?
? 4 3 ? ? 4 3

4-28 – graph
 - 10 Show the matrix pattern for the 1 5
Example: Adjacency graph of:
graph on the right and give an interpre- 2
3
 
tation of the path v4, v2, v3, v5, v1 on 4

? ?
the matrix 7 6 ? ? 
 
ä A separator is a set Y of vertices such that the graph GX−Y is  
 ? ? ? ?
disconnected. A= .
? ? 
 
Example: Y = {v3, v4, v5} is a separator in the above figure  ? ?
? ?

Example: For any adjacency matrix A, what is the graph of

A2? [interpret in terms of paths in the graph of A]

4-29 – graph 4-30 – graph

ä Two graphs are isomorphic is there is a mapping between the Bipartite graph representation
vertices of the two graphs that preserves adjacency.
- 11 Are the following 3 graphs isomorphic? If yes find the mappings ä Each row is represented by a vertex; Each column is represented
between them. by a vertex.
1
ä Relations only between rows and columns: Row i is connected
to column j if aij 6= 0
4 1 1 2
 
2 3
?
 
 ? 
3 2
Example:  
4 5
? ? 
4  
5
 ? ?
? ?
5 6 6 3 6

ä Bipartite models used only for specific cases [e.g. rectangular

ä Graphs are identical – labels are different matrices, ...] - By default we use the standard definition of graphs.

4-31 – graph 4-32 – graph

 Interpretation of graphs of matrices Paths in graphs

- 12 In which of the following cases is the underlying physical mesh - 16 What is the graph of Ak ?
the same as the graph of A (in the sense that edges are the same):
Theorem Let A be the adjacency matrix of a graph G = (V, E).
• Finite difference mesh [consider the simple case of 5-pt and 7-pt Then for k ≥ 0 and vertices u and v of G, the number of paths
FD problems - then 9-point meshes. ] of length k starting at u and ending at v is equal to (Ak )u,v .
• Finite element mesh with linear elements (e.g. triangles)?
• Finite element mesh with other types of elements? [to answer Proof: Proof is by induction.
this question you would have to know more about higher order ä Recall (definition): A matrix is reducible if it can be permuted
elements] into a block upper triangular matrix.
- 13 What is the graph of A + B (for two n × n matrices)? ä Note: A matrix is reducible iff its adjacency graph is not (strongly)
connected, i.e., iff it has more than one connected component.
- 14 What is the graph of AT ?
- 15 What is the graph of A.B?
4-33 – graph 4-34 – graph

C
●
B
A B C ä Definition : a graph is d regular if each vertex has the same
● ●

●
●
● ● ●
degree d.
ä No edges from
●
●
●
● Proposition: The spectral radius of a d regular graph is equal to d.
● ●
C to A or B. No
●
●
edges from B to A. Proof: The vector e of all ones is an eigenvector of A associated
● A

● ●
with the eigenvalue λ = d. In addition this eigenvalue is the largest
●

●
possible (consider the infinity norm of A). Therefore e is the Perron-
Frobenius vector u1.
Theorem: Perron-Frobenius An irreducible, nonnegative n × n
matrix A has a real, positive eigenvalue λ1 such that:
(i) λ1 is a simple eigenvalue of A;
(ii) λ1 admits a positive eigenvector u1 ; and
(iii)|λi| ≤ λ1 for all other eigenvalues λi where i > 1.
ä The spectral radius is equal to the eigenvalue λ1

4-35 – graph 4-36 – graph

 Application: Markov Chains ä Spectral radius is ≤ 1 [Why?]
ä Assume P is irreducible. Then:
ä Read about Markov Chains in Sect. 10.9 of:
https://www-users.cs.umn.edu/∼saad/eig book 2ndEd.pdf ä Perron Frobenius → ρ(P ) = 1 is an eigenvalue and associated
eigenvector has positive entries.
ä The stationary probability satisfies the equation:
ä Probabilities are obtained by scaling π by its sum.
πP = π
ä Example: One of the 2 models used for page rank.
Where π is a row vector. Example: A college Fraternity has 50 students at various stages of college
ä P is the probabilty transition matrix and it is ‘stochastic’: (Freshman, Sophomore, Junior, Senior). There are 6 potential stages for the
following year: Freshman, Sophomore, Junior, Senior, graduated, or left-without
A matrix P is said to be stochastic if :
degree. Following table gives probability of transitions from one stage
(i) pP ij ≥ 0 for all i, j
n to next
(ii) j=1 pij = 1 for i = 1, · · · , n
(iii) No column of P is a zero column.

4-37 – graph 4-38 – graph

To From Fr So. Ju. Sr. Grad lwd A few words about hypergraphs
Fr. .2 0 0 0 0 0
So. .6 .1 0 0 0 0 ä Hypergraphs are very general.. Ideas borrowed from VLSI work
Ju. 0 .7 .1 0 0 0
ä Main motivation: to better represent communication volumes
Sr. 0 0 .8 .1 0 0
when partitioning a graph. Standard models face many limitations
Grad 0 0 0 .75 1 0
lwd .2 .2 .1 .15 0 1 ä Hypergraphs can better express complex graph partitioning prob-
lems and provide better solutions.
- 17 What is P ? Assume initial population is x0 = [10, 16, 12, 12, 0, 0]
and do a follow the population for a few years. What is the probability ä Example: completely nonsymmetric patterns ...
that a student will graduate? What is the probability that s/he leaves ä .. Even rectangular matrices. Best illustration: Hypergraphs are
without a degree? ideal for text data

4-39 – graph 4-40 – graph

Example: V = {1, . . . , 9} and E = {a, . . . , e} with A few words on computational graphs
a = {1, 2, 3, 4}, b = {3, 5, 6, 7}, c = {4, 7, 8, 9}, f(x,y,z) = g(a(x,y,z), b(x,y,z))
d = {6, 7, 8}, and e = {2, 9}
f(x,y,z)
ä Computational graphs: graphs where
3 5
nodes represent computations whose
1✖
a
✖ ✖ Boolean matrix: evaluation depend on other (incoming) a(x,y,z) b(x,y,z)
b
●
● 1 2 3 4 5 6 7 8 9 nodes.
4
2✖ ✖ 7 6
1 1 1 1 a
✖ ✖
1 1 1 1 b
e● A= 1 1 1 1 c Consider the follow- g(x, y) = (x + y − 2) ∗ (y + 1)
c
d ä
● ●
1 1 1 d ing expression:
net d
✖9 ✖ 1 1 e 
net e 8 z = x+y
We can decompose this as v = y + 1

g = (z − 2) ∗ v

4-41 – graph 4-42 – graph

g = (z−2)*v
Back-Propagation
ä Corresponding compu- z = x+y
v = y+1 ä Often we want to compute the gradient of the function at the
tational graph:
root, once the nodes have been evaluated
y
x
ä The derivatives can be calculated by going backward (or down
ä Given values of x, y we want to (a) Evaluate the nodes and also the tree)
(b) derivatives of g w.r.t x, y at the nodes ä Here is a very simple example from Neural Networks
(a) is trivial - just follow the graph up - starting from the leaves (that  1
contain x and y)  L = 2 (y − t)2 x t

y = σ(z) w L
(b): Use the chain rule – here shown for x only using previous setting  z y

z = wx + b b
∂g ∂g da ∂g db
∂x
= ∂a dx
+ ∂b dx
ä Note that t (desired output) and x (input) are constant.
- 18 For the above example compute values at nodes and derivatives
when x = −1, y = 2.
4-43 – graph 4-44 – graph
 ∂f
Back-Propagation: General computational graphs ä Let δk = ∂vk
(called ‘errors’). Then
∂vj ∂vl ∂vm
δk = δj + δl + δm
∂vk ∂vk ∂vk

Representation: a DAG ä To compute the δk ’s once the vj ’s

have been computed (in a ‘forward’ prop-
ä Last node (vn) is the target function. Let us rename it f . agation) – proceed backward.
ä Nodes vi, i = 1, · · · , e with indegree 0 are the variables ä δj , δl, δm available and ∂vi/∂vk
computable. Nore δn ≡ 1.
ä Want to compute ∂f /∂v1, ∂f /∂v2, · · · , ∂f /∂ve
ä Simply use the chain rule. Look for example at node vk in figure ä However: cannot just do this in any order. Must follow a
∂f ∂f ∂vj ∂f ∂vl ∂f ∂vm topological order in order to obey dependencies.
= + +
∂vk ∂vj ∂vk ∂vl ∂vk ∂vm ∂vk

4-45 – graph 4-46 – graph

Example:

4-47 – graph