CSE408

Optimal binary search

tree and
Knapsack problem
Lecture # 23
Optimal binary search trees

• e.g. binary search trees for 3, 7, 9, 12;

3 7 7 12

3 12 3 9 9
7

9 9 12 3

7
12
(a) (b) (c) (d)

8 -2
Optimal binary search trees

• n identifiers : a1 <a2 <a3 <…< an

Pi, 1in : the probability that ai is searched.
Qi, 0in : the probability that x is searched
where ai < x < ai+1 (a0=-, an+1=).
n n

 P  Q
i 1
i
i 1
i 1

8 -3
 10
• Identifiers : 4, 5, 8, 10, 11,
12, 14
• Internal node : successful
5 14

search, Pi
4 8 11 E7
• External node : unsuccessful
E0 E1 E2 E3 E4 12 search, Qi

E5 E6

 The expected cost of a binary tree:

n n

 P  level(a
1
i i )   Qi  (level(E i )  1)
 Thenlevel of the root : 1 n 0

8 -4
The dynamic programming approach

• Let C(i, j) denote the cost of an optimal binary search

tree containing ai,…,aj .
• The cost of the optimal binary search tree with ak as
its root :
  k 1
  n

C(1, n)  min Pk  Q 0   Pi  Q i   C1, k  1  Q k   Pi  Q i   Ck  1, n  
1 k  n
  i 1   i  k 1 

ak
P1...Pk-1 Pk+1...Pn
Q0...Qk-1 Qk...Qn

a1 ...a
k-1
ak+1 ...a
n

C(1,k-1) C(k+1,n) 8 -5
General formula

  k 1

C(i, j)  min  Pk  Qi-1   Pm  Q m   Ci, k  1
i k  j
  m i 
 j

 Q k   Pm  Q m   Ck  1, j 
 m  k 1 
 j

 min Ci, k  1  Ck  1, j  Qi-1   Pm  Q m 
i k  j
 m i 
ak
P1...Pk-1 Pk+1...Pn
Q0...Qk-1 Qk...Qn

a1 ...a
k-1
ak+1 ...a
n

C(1,k-1) C(k+1,n) 8 -6
Computation relationships of sub trees

• e.g. n=4
C(1,4)

C(1,3) C(2,4)

C(1,2) C(2,3) C(3,4)

• Time complexity : O(n3)
when j-i=m, there are (n-m) C(i, j)’s to compute.
Each C(i, j) with j-i=m can be computed in O(m) time.

O(  m(n  m) )  O(n )
1 m  n
3

8 -7
Knapsack problem

There are two versions of the problem:

1. “0-1 knapsack problem”
• Items are indivisible; you either take an item or not. Some
special instances can be solved with dynamic programming

2. “Fractional knapsack problem”

• Items are divisible: you can take any fraction of an item
0-1 Knapsack problem

• Given a knapsack with maximum capacity W,

and a set S consisting of n items
• Each item i has some weight wi and benefit
value bi (all wi and W are integer values)
• Problem: How to pack the knapsack to achieve
maximum total value of packed items?
0-1 Knapsack problem

• Problem, in other words, is to find

max  bi subject to w W i
iT iT

 The problem is called a “0-1” problem,

because each item must be entirely
accepted or rejected.
0-1 Knapsack problem: brute-force approach

Let’s first solve this problem with a

straightforward algorithm
• Since there are n items, there are 2n
possible combinations of items.
• We go through all combinations and find
the one with maximum value and with total
weight less or equal to W
• Running time will be O(2n)
0-1 Knapsack problem: dynamic programming pproach

• We can do better with an algorithm based on

dynamic programming

• We need to carefully identify the subproblems

Defining a Subproblem

• Given a knapsack with maximum capacity W,

and a set S consisting of n items
• Each item i has some weight wi and benefit
value bi (all wi and W are integer values)
• Problem: How to pack the knapsack to achieve
maximum total value of packed items?
Defining a Sub problem

• We can do better with an algorithm based on

dynamic programming

• We need to carefully identify the subproblems

Let’s try this:

If items are labeled 1..n, then a subproblem
would be to find an optimal solution for
Sk = {items labeled 1, 2, .. k}
 CSE408
Matrix Chain
Multiplication
Lecture # 24
Matrix-chain Multiplication

• Suppose we have a sequence or chain A1, A2,

…, An of n matrices to be multiplied
– That is, we want to compute the product
A1A2…An

• There are many possible ways

(parenthesizations) to compute the product

11-2
Matrix-chain Multiplication …contd

• Example: consider the chain A1, A2, A3, A4 of

4 matrices
– Let us compute the product A1A2A3A4
• There are 5 possible ways:
1. (A1(A2(A3A4)))
2. (A1((A2A3)A4))
3. ((A1A2)(A3A4))
4. ((A1(A2A3))A4)
5. (((A1A2)A3)A4)
11-3
Matrix-chain Multiplication …contd

• To compute the number of scalar

multiplications necessary, we must know:
– Algorithm to multiply two matrices
– Matrix dimensions

• Can you write the algorithm to multiply two

matrices?

11-4
Algorithm to Multiply 2 Matrices
Input: Matrices Ap×q and Bq×r (with dimensions p×q and q×r)
Result: Matrix Cp×r resulting from the product A·B

MATRIX-MULTIPLY(Ap×q , Bq×r)
1. for i ← 1 to p
2. for j ← 1 to r
3. C[i, j] ← 0
4. for k ← 1 to q
5. C[i, j] ← C[i, j] + A[i, k] · B[k, j]
6. return C
Scalar multiplication in line 5 dominates time to compute
CNumber of scalar multiplications = pqr
11-5
Matrix-chain Multiplication …contd

• Example: Consider three matrices A10100,

B1005, and C550
• There are 2 ways to parenthesize
– ((AB)C) = D105 · C550
• AB  10·100·5=5,000 scalar multiplications Total:
• DC  10·5·50 =2,500 scalar multiplications 7,500
– (A(BC)) = A10100 · E10050
• BC  100·5·50=25,000 scalar multiplications
• AE  10·100·50 =50,000 scalar multiplications
Total:
11-6
75,000
Matrix-chain Multiplication …contd

• Matrix-chain multiplication problem

– Given a chain A1, A2, …, An of n matrices, where
for i=1, 2, …, n, matrix Ai has dimension pi-1pi
– Parenthesize the product A1A2…An such that the
total number of scalar multiplications is
minimized
• Brute force method of exhaustive search
takes time exponential in n

11-7
Dynamic Programming Approach

• The structure of an optimal solution

– Let us use the notation Ai..j for the matrix that
results from the product Ai Ai+1 … Aj
– An optimal parenthesization of the product
A1A2…An splits the product between Ak and Ak+1
for some integer k where1 ≤ k < n
– First compute matrices A1..k and Ak+1..n ; then
multiply them to get the final matrix A1..n

11-8
Dynamic Programming Approach …contd

– Key observation: parenthesizations of the

subchains A1A2…Ak and Ak+1Ak+2…An must also be
optimal if the parenthesization of the chain
A1A2…An is optimal (why?)

– That is, the optimal solution to the problem

contains within it the optimal solution to
subproblems

11-9
Dynamic Programming Approach …contd

• Recursive definition of the value of an

optimal solution
– Let m[i, j] be the minimum number of scalar
multiplications necessary to compute Ai..j
– Minimum cost to compute A1..n is m[1, n]
– Suppose the optimal parenthesization of Ai..j
splits the product between Ak and Ak+1 for some
integer k where i ≤ k < j

11-10
Dynamic Programming Approach …contd

– Ai..j = (Ai Ai+1…Ak)·(Ak+1Ak+2…Aj)= Ai..k · Ak+1..j

– Cost of computing Ai..j = cost of computing Ai..k +
cost of computing Ak+1..j + cost of multiplying Ai..k
and Ak+1..j
– Cost of multiplying Ai..k and Ak+1..j is pi-1pk pj

– m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj for i ≤

k<j
– m[i, i ] = 0 for i=1,2,…,n

11-11
Dynamic Programming Approach …contd

– But… optimal parenthesization occurs at one

value of k among all possible i ≤ k < j
– Check all these and select the best one

0 if i=j
m[i, j ] =
min {m[i, k] + m[k+1, j ] + pi-1pk pj } if i<j
i ≤ k< j

11-12
Dynamic Programming Approach …contd

• To keep track of how to construct an optimal

solution, we use a table s
• s[i, j ] = value of k at which Ai Ai+1 … Aj is split
for optimal parenthesization
• Algorithm: next slide
– First computes costs for chains of length l=1
– Then for chains of length l=2,3, … and so on
– Computes the optimal cost bottom-up

11-13
Algorithm to Compute Optimal Cost
Input: Array p[0…n] containing matrix dimensions and n
Result: Minimum-cost table m and split table s

MATRIX-CHAIN-ORDER(p[ ], n)
for i ← 1 to n Takes O(n3) time
m[i, i] ← 0
Requires O(n 2) space
for l ← 2 to n
for i ← 1 to n-l+1
j ← i+l-1
m[i, j] ← 
for k ← i to j-1
q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]
if q < m[i, j]
m[i, j] ← q
s[i, j] ← k
return m and s
11-14
Constructing Optimal Solution

• Our algorithm computes the minimum-cost

table m and the split table s
• The optimal solution can be constructed
from the split table s
– Each entry s[i, j ]=k shows where to split the
product Ai Ai+1 … Aj for the minimum cost

11-15
Example

• Show how to multiply this Matrix Dimension

matrix chain optimally
A1 30×35

• Solution on the board A2 35×15

– Minimum cost 15,125 A3 15×5
– Optimal parenthesization
((A1(A2A3))((A4 A5)A6)) A4 5×10
A5 10×20
A6 20×25

11-16
 CSE408
Longest Common Sub
Sequence
Lecture # 25
 Dynamic programming

• It is used, when the solution can be

recursively described in terms of solutions
to subproblems (optimal substructure)
• Algorithm finds solutions to subproblems
and stores them in memory for later use
• More efficient than “brute-force methods”,
which solve the same subproblems over
and over again
2
Longest Common Subsequence (LCS)

Application: comparison of two DNA strings

Ex: X= {A B C B D A B }, Y= {B D C A B A}
Longest Common Subsequence:
X= AB C BDAB
Y= B D CAB A
Brute force algorithm would compare each
subsequence of X with the symbols in Y
3/2/2022 3
 LCS Algorithm
• if |X| = m, |Y| = n, then there are 2m
subsequences of x; we must compare each
with Y (n comparisons)
• So the running time of the brute-force
algorithm is O(n 2m)
• Notice that the LCS problem has optimal
substructure: solutions of subproblems are
parts of the final solution.
• Subproblems: “find LCS of pairs of prefixes
of X and Y”
3/2/2022 4
 LCS Algorithm
• First we’ll find the length of LCS. Later we’ll
modify the algorithm to find LCS itself.
• Define Xi, Yj to be the prefixes of X and Y of
length i and j respectively
• Define c[i,j] to be the length of LCS of Xi and
Yj
• Then the length of LCS of X and Y will be
c[m,n]
c[i  1, j  1]  1 if x[i ]  y[ j ],
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise
3/2/2022 5
 LCS recursive solution
c[i  1, j  1]  1 if x[i]  y[ j ],
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise
• We start with i = j = 0 (empty substrings of x
and y)
• Since X0 and Y0 are empty strings, their LCS
is always empty (i.e. c[0,0] = 0)
• LCS of empty string and any other string is
empty, so for every i and j: c[0, j] = c[i,0] = 0
3/2/2022 6
 LCS recursive solution
c[i  1, j  1]  1 if x[i]  y[ j ],
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise
• When we calculate c[i,j], we consider two
cases:
• First case: x[i]=y[j]: one more symbol in
strings X and Y matches, so the length of LCS
Xi and Yj equals to the length of LCS of
smaller strings Xi-1 and Yi-1 , plus 1
3/2/2022 7
 LCS recursive solution
c[i  1, j  1]  1 if x[i]  y[ j ],
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise

• Second case: x[i] != y[j]

• As symbols don’t match, our solution is not
improved, and the length of LCS(Xi , Yj) is the
same as before (i.e. maximum of LCS(Xi, Yj-1)
and LCS(Xi-1,Yj)

Why not just take the length of LCS(Xi-1, Yj-1) ?

3/2/2022 8
 LCS Length Algorithm
LCS-Length(X, Y)
1. m = length(X) // get the # of symbols in X
2. n = length(Y) // get the # of symbols in Y
3. for i = 1 to m c[i,0] = 0 // special case: Y0
4. for j = 1 to n c[0,j] = 0 // special case: X0
5. for i = 1 to m // for all Xi
6. for j = 1 to n // for all Yj
7. if ( Xi == Yj )
8. c[i,j] = c[i-1,j-1] + 1
9. else c[i,j] = max( c[i-1,j], c[i,j-1] )
10. return c
3/2/2022 9
 LCS Example
We’ll see how LCS algorithm works on the
following example:
• X = ABCB
• Y = BDCAB

What is the Longest Common Subsequence

of X and Y?

LCS(X, Y) = BCB
X=AB C B
Y= BDCAB 10
 ABCB
LCS Example (0) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi

A
1

2 B

3 C

4 B

X = ABCB; m = |X| = 4
Y = BDCAB; n = |Y| = 5
Allocate array c[5,4]
11
 ABCB
LCS Example (1) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0

2 B
0

3 C 0

4 B 0

for i = 1 to m c[i,0] = 0
for j = 1 to n c[0,j] = 0
12
 ABCB
LCS Example (2) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 13
 ABCB
LCS Example (3) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 14
 ABCB
LCS Example (4) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 15
 ABCB
LCS Example (5) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 16
 ABCB
LCS Example (6) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 17
 ABCB
LCS Example (7) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 18
 ABCB
LCS Example (8) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 19
 ABCB
LCS Example (10) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 20
 ABCB
LCS Example (11) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 21
 ABCB
LCS Example (12) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 22
 ABCB
LCS Example (13) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 23
 ABCB
LCS Example (14) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2

if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 24
 ABCB
LCS Example (15) BDCAB
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2 3
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

3/2/2022 25
LCS Algorithm Running Time

• LCS algorithm calculates the values of each

entry of the array c[m,n]
• So what is the running time?

O(m*n)
since each c[i,j] is calculated in
constant time, and there are m*n
elements in the array
3/2/2022 26
 How to find actual LCS
• So far, we have just found the length of LCS,
but not LCS itself.
• We want to modify this algorithm to make it
output Longest Common Subsequence of X
and Y
Each c[i,j] depends on c[i-1,j] and c[i,j-1]
or c[i-1, j-1]
For each c[i,j] we can say how it was acquired:
2 2 For example, here
2 3 c[i,j] = c[i-1,j-1] +1 = 2+1=3
27
How to find actual LCS - continued
• Remember that
c[i  1, j  1]  1 if x[i]  y[ j ],
c[i, j ]  
 max( c[i, j  1], c[i  1, j ]) otherwise

 So we can start from c[m,n] and go backwards

 Whenever c[i,j] = c[i-1, j-1]+1, remember x[i]
(because x[i] is a part of LCS)
 When i=0 or j=0 (i.e. we reached the
beginning), output remembered letters in
reverse order
3/2/2022 28
 Finding LCS
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2 3

3/2/2022 29
 Finding LCS (2)
j 0 1 2 3 4 5
i Yj B D C A B

0 Xi
0 0 0 0 0 0
A
1 0 0 0 0 1 1

2 B
0 1 1 1 1 2

3 C 0 1 1 2 2 2

4 B 0 1 1 2 2 3
LCS (reversed order): B C B
LCS (straight order): B C B
(this string turned out to be a palindrome) 30
 CSE408
Minimum Spanning
Tree(Prims,Kruskshal)
Lecture #26
Minimum Spanning Tree

• A minimum spanning tree connects all nodes

in a given graph
• A MST must be a connected and undirected
graph
• A MST can have weighted edges
• Multiple MSTs can exist within a given
undirected graph
More about Multiple MSTs

• Multiple MSTs can be generated depending on

which algorithm is used
• If you wish to have an MST start at a specific
node
• However, if there are weighted edges and all
weighted edges are unique, only one MST will
exist
Real Life Application of a MST

A cable TV company is laying cable in a new

neighborhood. If it is constrained to bury the cable
only along certain paths, then there would be a
graph representing which points are connected by
those paths. Some of those paths might be more
expensive, because they are longer, or require the
cable to be buried deeper; these paths would be
represented by edges with larger weights. A
minimum spanning tree would be the network with
the lowest total cost.
Prim’s Algorithm

• Initially discovered in 1930 by Vojtěch Jarník,

then rediscovered in 1957 by Robert C. Prim
• Similar to Dijkstra’s Algorithm regarding a
connected graph
• Starts off by picking any node within the graph
and growing from there
Prim’s Algorithm Cont.

• Label the starting node, A, with a 0 and all

others with infinite
• Starting from A, update all the connected
nodes’ labels to A with their weighted edges if
it less than the labeled value
• Find the next smallest label and update the
corresponding connecting nodes
• Repeat until all the nodes have been visited
Prim’s Algorithm Example
Prim’s Algorithm Example
Kruskal’s Algorithm

• Created in 1957 by Joseph Kruskal

• Finds the MST by taking the smallest weight in the graph and
connecting the two nodes and repeating until all nodes are
connected to just one tree
• This is done by creating a priority queue using the weights as
keys
• Each node starts off as it’s own tree
• While the queue is not empty, if the edge retrieved connects
two trees, connect them, if not, discard it
• Once the queue is empty, you are left with the minimum
spanning tree
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
Kruskal’s Algorithm Example
 CSE408
Dijkstra,Huffmancoding

Lecture # 27
Dijkstra’s Algorithm
Edge Relaxation

• Consider an edge e = (u,z) such

that d(u) = 50
– u is the vertex most recently d(z) = 75
u e
added to the cloud s z
– z is not in the cloud

• The relaxation of edge e

updates distance d(z) as
follows:
d(u) = 50
d(z)  min{d(z),d(u) + weight(e)} d(z) = 60
u e
s z
Example
0 0
8 A 4 8 A 4
2 2
8 7 2 1 4 8 7 2 1 3
B C D B C D

 3 9  5 3 9 8
2 5 2 5
E F E F

0 0
8 A 4 8 A 4
2 2
8 7 2 1 3 7 7 2 1 3
B C D B C D

5 3 9 11 5 3 9 8
2 5 2 5
E F E F
Example (cont.)

0
8 A 4
2
7 7 2 1 3
B C D

5 3 9 8
2 5
E F
0
8 A 4
2
7 7 2 1 3
B C D

5 3 9 8
2 5
E F
Dijkstra’s Algorithm

• A priority queue stores Algorithm DijkstraDistances(G, s)

the vertices outside the Q  new heap-based priority queue
cloud for all v  G.vertices()
if v = s
– Key: distance setDistance(v, 0)
– Element: vertex else
• Locator-based methods setDistance(v, )
– insert(k,e) returns a l  Q.insert(getDistance(v), v)
locator setLocator(v,l)
– replaceKey(l,k) changes while Q.isEmpty()
the key of an item u  Q.removeMin()
for all e  G.incidentEdges(u)
• We store two labels with
{ relax edge e }
each vertex: z  G.opposite(u,e)
– Distance (d(v) label) r  getDistance(u) + weight(e)
– locator in priority queue if r < getDistance(z)
setDistance(z,r)
Q.replaceKey(getLocator(z),r)
Analysis

• Graph operations
– Method incidentEdges is called once for each vertex
• Label operations
– We set/get the distance and locator labels of vertex z O(deg(z)) times
– Setting/getting a label takes O(1) time
• Priority queue operations
– Each vertex is inserted once into and removed once from the priority
queue, where each insertion or removal takes O(log n) time
– The key of a vertex in the priority queue is modified at most deg(w) times,
where each key change takes O(log n) time
• Dijkstra’s algorithm runs in O((n + m) log n) time provided the graph is
represented by the adjacency list structure
– Recall that Sv deg(v) = 2m
• The running time can also be expressed as O(m log n) since the graph
is connected
Extension

• Using the template Algorithm DijkstraShortestPathsTree(G, s)

method pattern, we can
extend Dijkstra’s …
algorithm to return a
tree of shortest paths for all v  G.vertices()
from the start vertex to …
all other vertices setParent(v, )
…
• We store with each
vertex a third label: for all e  G.incidentEdges(u)
– parent edge in the { relax edge e }
shortest path tree z  G.opposite(u,e)
• In the edge relaxation r  getDistance(u) + weight(e)
step, we update the if r < getDistance(z)
parent label setDistance(z,r)
setParent(z,e)
Q.replaceKey(getLocator(z),r)
Why Dijkstra’s Algorithm Works

• Dijkstra’s algorithm is based on the greedy method.

It adds vertices by increasing distance.
 Suppose it didn’t find all shortest
distances. Let F be the first wrong 0
vertex the algorithm processed. 8 A 4
 When the previous node, D, on the 2
7 7 2 1 3
true shortest path was considered, its B C D
distance was correct.
5 3 9 8
 But the edge (D,F) was relaxed at that 2 5
time! E F

 Thus, so long as d(F)>d(D), F’s distance

cannot be wrong. That is, there is no
wrong vertex.
Purpose of Huffman Coding

• Proposed by Dr. David A. Huffman in 1952

– “A Method for the Construction of Minimum
Redundancy Codes”
• Applicable to many forms of data
transmission
– Our example: text files
The Basic Algorithm

• Huffman coding is a form of statistical coding

• Not all characters occur with the same
frequency!
• Yet all characters are allocated the same amount
of space
– 1 char = 1 byte, be it e or x
The Basic Algorithm

• Any savings in tailoring codes to frequency of

character?
• Code word lengths are no longer fixed like
ASCII.
• Code word lengths vary and will be shorter for
the more frequently used characters.
The (Real) Basic Algorithm

1. Scan text to be compressed and tally

occurrence of all characters.
2. Sort or prioritize characters based on number of
occurrences in text.
3. Build Huffman code tree based on
prioritized list.
4. Perform a traversal of tree to determine all code words.
5. Scan text again and create new file using the
Huffman codes.
Building a Tree Scan the original text

• Consider the following short text:

Eerie eyes seen near lake.

• Count up the occurrences of all characters in the

text
Building a Tree Scan the original text

Eerie eyes seen near lake.

• What characters are present?

E e r i space
ysnarlk.
Building a Tree Scan the original text

Eerie eyes seen near lake.

• What is the frequency of each character in the text?

Char Freq. Char Freq. Char Freq.

E 1 y 1 k 1
e 8 s 2 . 1
r 2 n 2
i 1 a 2
space 4 l 1
Building a Tree Prioritize characters

• Create binary tree nodes with character

and frequency of each character
• Place nodes in a priority queue
– The lower the occurrence, the higher the
priority in the queue
Building a Tree Prioritize characters

• Uses binary tree nodes

public class HuffNode
{
public char myChar;
public int myFrequency;
public HuffNode myLeft, myRight;
}
priorityQueue myQueue;
Building a Tree

• The queue after inserting all nodes

E i y l k . r s n a sp e
1 1 1 1 1 1 2 2 2 2 4 8

• Null Pointers are not shown

Building a Tree

• While priority queue contains two or more nodes

– Create new node
– Dequeue node and make it left subtree
– Dequeue next node and make it right subtree
– Frequency of new node equals sum of frequency of left
and right children
– Enqueue new node back into queue
Building a Tree

E i y l k . r s n a sp e
1 1 1 1 1 1 2 2 2 2 4 8
Building a Tree

y l k . r s n a sp e
1 1 1 1 2 2 2 2 4 8

E i
1 1
Building a Tree

y l k . r s n a sp e
2
1 1 1 1 2 2 2 2 4 8
E i
1 1
Building a Tree

k . r s n a sp e
2
1 1 2 2 2 2 4 8
E i
1 1

y l
1 1
Building a Tree

2
k . r s n a 2 sp e
1 1 2 2 2 2 4 8
y l
1 1
E i
1 1
Building a Tree

r s n a 2 2 sp e
2 2 2 2 4 8
y l
E i 1 1
1 1

k .
1 1
Building a Tree

r s n a 2 2 sp e
2
2 2 2 2 4 8
E i y l k .
1 1 1 1 1 1
Building a Tree

n a 2 sp e
2 2
2 2 4 8
E i y l k .
1 1 1 1 1 1

r s
2 2
Building a Tree

n a 2 sp e
2 2 4
2 2 4 8

E i y l k . r s
1 1 1 1 1 1 2 2
Building a Tree

2 4 e
2 2 sp
8
4
y l k . r s
E i 1 1 1 1 2 2
1 1

n a
2 2
Building a Tree

2 4 4 e
2 2 sp
8
4
y l k . r s n a
E i 1 1 1 1 2 2 2 2
1 1
Building a Tree

4 4 e
2 sp
8
4
k . r s n a
1 1 2 2 2 2

2 2

E i y l
1 1 1 1
Building a Tree

4 4 4
2 sp e
4 2 2 8
k . r s n a
1 1 2 2 2 2
E i y l
1 1 1 1
Building a Tree

4 4 4
e
2 2 8
r s n a
2 2 2 2
E i y l
1 1 1 1

2 sp
4
k .
1 1
Building a Tree

4 4 4 6 e
2 sp 8
r s n a 2 2
4
2 2 2 2 k .
E i y l 1 1
1 1 1 1

What is happening to the characters with a low number of occurrences?

Building a Tree

4
6 e
2 2 2 8
sp
4
E i y l k .
1 1 1 1 1 1
8

4 4

r s n a
2 2 2 2
Building a Tree

4
6 e 8
2 2 2 8
sp
4 4 4
E i y l k .
1 1 1 1 1 1
r s n a
2 2 2 2
Building a Tree

8
e
8
4 4
10
r s n a
2 2 2 2 4
6
2 2 2 sp
4
E i y l k .
1 1 1 1 1 1
Building a Tree

8 10
e
8 4
4 4
6
2 2
r s n a 2 sp
2 2 2 2 4
E i y l k .
1 1 1 1 1 1
Building a Tree

10
16
4
6
2 2 e 8
2 sp 8
4
E i y l k . 4 4
1 1 1 1 1 1

r s n a
2 2 2 2
Building a Tree

10 16
4
6
e 8
2 2 8
2 sp
4 4 4
E i y l k .
1 1 1 1 1 1
r s n a
2 2 2 2
Building a Tree

26

16
10

4 e 8
6 8
2 2 2 sp 4 4
4
E i y l k .
1 1 1 1 1 1 r s n a
2 2 2 2
Building a Tree

After enqueueing this node

there is only one node left in
priority queue.
26

16
10

4 e 8
6 8
2 2
2 sp 4 4
4
E i y l k .
1 1 1 1 1 1 r s n a
2 2 2 2
Building a Tree

Dequeue the single node left in the

queue.
26

This tree contains the new code 10

16
words for each character.
4 e 8
6 8
Frequency of root node should 2 2 2 sp 4 4
equal number of characters in text. 4
E i y l k .
1 1 1 1 1 1 r s n a
2 2 2 2

Eerie eyes seen near lake.  26 characters

Encoding the File Traverse Tree for Codes

• Perform a traversal of the

tree to obtain new code
words 26
• Going left is a 0 going right is 16
10
a1
• code word is only completed 4
6
e 8
8
when a leaf node is reached 2 2 2 sp 4 4
4
E i y l k .
1 1 1 1 1 1 r s n a
2 2 2 2
Encoding the File Traverse Tree for Codes

Char Code
E 0000
i 0001
y 0010
l 0011 26
k 0100 16
. 0101 10
space 011 4
e 10 e 8
6 8
r 1100 2 2 2
s 1101 sp 4 4
4
n 1110 E i y l k .
a 1111 1 1 1 1 1 1 r s n a
2 2 2 2
Encoding the File

• Rescan text and encode file

using new code words Char Code
E 0000
Eerie eyes seen near lake. i 0001
y 0010
l 0011
000010110000011001110001010110110100
k 0100
111110101111110001100111111010010010
. 0101
space 011
1 e 10
r 1100
s 1101
n 1110
a 1111
 Why is there no need for a
separator character?
.
Encoding the File

• Have we made things any

000010110000011001110001010110110100
better? 111110101111110001100111111010010010

• 73 bits to encode the text

1

• ASCII would take 8 * 26 =

208 bits

If modified code used 4 bits per

character are needed. Total bits
4 * 26 = 104. Savings not as great.
Decoding the File

• How does receiver know what the codes are?

• Tree constructed for each text file.
– Considers frequency for each file
– Big hit on compression, especially for smaller files
• Tree predetermined
– based on statistical analysis of text files or file types
• Data transmission is bit based versus byte based
Decoding the File

• Once receiver has tree it

scans incoming bit stream 26

• 0  go left 10
16

• 1  go right 4 e 8
6 8
2 2 2 sp 4 4
4
101000110111101111011 E i y l k .
1 1 1 1 1 1 r s n a
2 2 2 2
11110000110101
Summary

• Huffman coding is a technique used to compress files

for transmission
• Uses statistical coding
– more frequently used symbols have shorter code words
• Works well for text and fax transmissions
• An application that uses several data structures
 CSE408
Single Source Shortest
path

Lecture #28
Outline and Reading

• Weighted graphs
– Shortest path problem
– Shortest path properties
• Dijkstra’s algorithm
– Algorithm
– Edge relaxation
• The Bellman-Ford algorithm
• All-pairs shortest paths
Weighted Graphs

• In a weighted graph, each edge has an associated numerical value,

called the weight of the edge
• Edge weights may represent, distances, costs, etc.
• Example:
– In a flight route graph, the weight of an edge represents the distance in
miles between the endpoint airports

PVD
ORD
SFO
LGA

HNL
LAX
DFW
MIA
Shortest Path Problem

• Given a weighted graph and two vertices u and v, we want to find a

path of minimum total weight between u and v.
– Length of a path is the sum of the weights of its edges.
• Example:
– Shortest path between Providence and Honolulu
• Applications
– Internet packet routing
– Flight reservations
– Driving directions
PVD
ORD
SFO
LGA

HNL
LAX
DFW
MIA
Shortest Path Properties

Property 1:
A subpath of a shortest path is itself a shortest path
Property 2:
There is a tree of shortest paths from a start vertex to all the other vertices
Example:
Tree of shortest paths from Providence

PVD
ORD
SFO
LGA

HNL
LAX
DFW
MIA
Dijkstra’s Algorithm
Edge Relaxation

• Consider an edge e = (u,z) such

that d(u) = 50
– u is the vertex most recently d(z) = 75
u e
added to the cloud s z
– z is not in the cloud

• The relaxation of edge e

updates distance d(z) as
follows:
d(u) = 50
d(z)  min{d(z),d(u) + weight(e)} d(z) = 60
u e
s z
Example
0 0
8 A 4 8 A 4
2 2
8 7 2 1 4 8 7 2 1 3
B C D B C D

 3 9  5 3 9 8
2 5 2 5
E F E F

0 0
8 A 4 8 A 4
2 2
8 7 2 1 3 7 7 2 1 3
B C D B C D

5 3 9 11 5 3 9 8
2 5 2 5
E F E F
Example (cont.)

0
8 A 4
2
7 7 2 1 3
B C D

5 3 9 8
2 5
E F
0
8 A 4
2
7 7 2 1 3
B C D

5 3 9 8
2 5
E F
Dijkstra’s Algorithm

• A priority queue stores Algorithm DijkstraDistances(G, s)

the vertices outside the Q  new heap-based priority queue
cloud for all v  G.vertices()
if v = s
– Key: distance setDistance(v, 0)
– Element: vertex else
• Locator-based methods setDistance(v, )
– insert(k,e) returns a l  Q.insert(getDistance(v), v)
locator setLocator(v,l)
– replaceKey(l,k) changes while Q.isEmpty()
the key of an item u  Q.removeMin()
for all e  G.incidentEdges(u)
• We store two labels with
{ relax edge e }
each vertex: z  G.opposite(u,e)
– Distance (d(v) label) r  getDistance(u) + weight(e)
– locator in priority queue if r < getDistance(z)
setDistance(z,r)
Q.replaceKey(getLocator(z),r)
Analysis

• Graph operations
– Method incidentEdges is called once for each vertex
• Label operations
– We set/get the distance and locator labels of vertex z O(deg(z)) times
– Setting/getting a label takes O(1) time
• Priority queue operations
– Each vertex is inserted once into and removed once from the priority
queue, where each insertion or removal takes O(log n) time
– The key of a vertex in the priority queue is modified at most deg(w) times,
where each key change takes O(log n) time
• Dijkstra’s algorithm runs in O((n + m) log n) time provided the graph is
represented by the adjacency list structure
– Recall that Sv deg(v) = 2m
• The running time can also be expressed as O(m log n) since the graph
is connected
Extension

• Using the template Algorithm DijkstraShortestPathsTree(G, s)

method pattern, we can
extend Dijkstra’s …
algorithm to return a
tree of shortest paths for all v  G.vertices()
from the start vertex to …
all other vertices setParent(v, )
…
• We store with each
vertex a third label: for all e  G.incidentEdges(u)
– parent edge in the { relax edge e }
shortest path tree z  G.opposite(u,e)
• In the edge relaxation r  getDistance(u) + weight(e)
step, we update the if r < getDistance(z)
parent label setDistance(z,r)
setParent(z,e)
Q.replaceKey(getLocator(z),r)
Why Dijkstra’s Algorithm Works

• Dijkstra’s algorithm is based on the greedy method.

It adds vertices by increasing distance.
 Suppose it didn’t find all shortest
distances. Let F be the first wrong 0
vertex the algorithm processed. 8 A 4
 When the previous node, D, on the 2
7 7 2 1 3
true shortest path was considered, its B C D
distance was correct.
5 3 9 8
 But the edge (D,F) was relaxed at that 2 5
time! E F

 Thus, so long as d(F)>d(D), F’s distance

cannot be wrong. That is, there is no
wrong vertex.
Why It Doesn’t Work for Negative-Weight Edges

Dijkstra’s algorithm is based on the greedy method. It adds vertices by

increasing distance.

0
8 A 4
– If a node with a negative incident 6
edge were to be added late to the 7 7 5 1 4
cloud, it could mess up distances B C D
for vertices already in the cloud. 0 -8
5 9
2 5
E F

C’s true distance is 1, but it is already

in the cloud with d(C)=5!
Bellman-Ford Algorithm

• Works even with negative- Algorithm BellmanFord(G, s)

weight edges for all v  G.vertices()
• Must assume directed edges if v = s
(for otherwise we would have setDistance(v, 0)
negative-weight cycles) else
setDistance(v, )
• Iteration i finds all shortest
for i  1 to n-1 do
paths that use i edges. for each e  G.edges()
• Running time: O(nm). { relax edge e }
• Can be extended to detect a u  G.origin(e)
negative-weight cycle if it z  G.opposite(u,e)
exists r  getDistance(u) + weight(e)
– How? if r < getDistance(z)
setDistance(z,r)
Bellman-Ford Example
Nodes are labeled with their d(v) values

8 0 4 8 0 4
-2 -2
7 1 8 7 -2 1 4
     
3 9 3 9
-2 5 -2 5
   

8 0 4 8 0 4
-2 -2
5 8 7 1 -1 7 1
-2 4 5 -2 -1
1 3 9 3 9
-2 6 9 5 -2 4 5
  1 9
All-Pairs Shortest Paths

• Find the distance between Algorithm AllPair(G) {assumes vertices 1,…,n}

every pair of vertices in a for all vertex pairs (i,j)
weighted directed graph G. if i = j
• We can make n calls to D0[i,i]  0
else if (i,j) is an edge in G
Dijkstra’s algorithm (if no
D0[i,j]  weight of edge (i,j)
negative edges), which
else
takes O(nmlog n) time. D0[i,j]  + 
• Likewise, n calls to Bellman- for k  1 to n do
Ford would take O(n2m) for i  1 to n do
time. for j  1 to n do
• We can achieve O(n3) time Dk[i,j]  min{Dk-1[i,j], Dk-1[i,k]+Dk-1[k,j]}
using dynamic return Dn
programming (similar to the i Uses only vertices numbered 1,…,k
Floyd-Warshall algorithm). (compute weight of this edge)
j
Uses only vertices
numbered 1,…,k-1 k Uses only vertices
numbered 1,…,k-1
 CSE408
All pairs shortest path

Lecture #29
All pairs shortest path
• The problem: find the shortest path between every
pair of vertices of a graph
• The graph: may contain negative edges but no
negative cycles

• A representation: a weight matrix where

W(i,j)=0 if i=j.
W(i,j)= if there is no edge between i and j.
W(i,j)=“weight of edge”
• Note: we have shown principle of optimality
applies to shortest path problems
The weight matrix and the graph

1
1 2 3 4 5 3 v1 v2
9
1 0 1  1 5 5
1 2 3
2 9 0 3 2  v5

3   0 4  3
v4
2
v3
4   2 0 3 4

5 3    0
The subproblems

• How can we define the shortest distance di,j in

terms of “smaller” problems?

• One way is to restrict the paths to only include

vertices from a restricted subset.

• Initially, the subset is empty.

• Then, it is incrementally increased until it

includes all the vertices.
The subproblems

• Let D(k)[i,j]=weight of a shortest path from

vi to vj using only vertices from {v1,v2,…,vk}
as intermediate vertices in the path

– D(0)=W
– D(n)=D which is the goal matrix

• How do we compute D(k) from D(k-1) ?

The Recursive Definition:
Case 1: A shortest path from vi to vj restricted to using only
vertices from {v1,v2,…,vk} as intermediate vertices does not
use vk. Then D(k)[i,j]= D(k-1)[i,j].

Case 2: A shortest path from vi to vj restricted to using only

vertices from {v1,v2,…,vk} as intermediate vertices does use
vk. Then D(k)[i,j]= D(k-1)[i,k]+ D(k-1)[k,j].

Shortest path using intermediate vertices

{V1, . . . Vk } Vk

Vj
Vi

Shortest Path using intermediate vertices { V1, . . . Vk -1 }

The recursive definition
• Since
D(k)[i,j]= D(k-1)[i,j] or
D(k)[i,j]= D(k-1)[i,k]+ D(k-1)[k,j].
We conclude:
D(k)[i,j]= min{ D(k-1)[i,j], D(k-1)[i,k]+ D(k-1)[k,j] }.

Shortest path using intermediate vertices

{V1, . . . Vk } Vk

Vj
Vi

Shortest Path using intermediate vertices { V1, . . . Vk -1 }

The pointer array P

• Used to enable finding a shortest path

• Initially the array contains 0

• Each time that a shorter path from i to j is found

the k that provided the minimum is saved (highest
index node on the path from i to j)

• To print the intermediate nodes on the shortest

path a recursive procedure that print the shortest
paths from i and k, and from k to j can be used
Floyd's Algorithm Using n+1 D matrices

Floyd//Computes shortest distance between all pairs of

//nodes, and saves P to enable finding shortest paths
1. D0  W // initialize D array to W [ ]
2. P  0 // initialize P array to [0]
3. for k  1 to n
4. do for i  1 to n
5. do for j  1 to n
6. if (Dk-1[ i, j ] > Dk-1 [ i, k ] + Dk-1 [ k, j ] )
7. then Dk[ i, j ]  Dk-1 [ i, k ] + Dk-1 [ k, j ]
8. P[ i, j ]  k;
9. else Dk[ i, j ]  Dk-1 [ i, j ]
 Example
1 2 3
1 0 4 5
W= D0 =
5
2 2 0 
1
3  -3 0
4 2 3
1 2 3
-3 1 0 0 0
2
P= 2 0 0 0
3 0 0 0
 1 2 3
k = 1
1 5 D0 =
1 0 4 5 Vertex 1
4 3
2 
-3
2 2 0
can be
2 3  -3 0
1 2 3
intermediat
D1 =
1 0 4 5 1 0
e node
D [2,3] = min( D [2,3], 0 0
D [2,1]+D [1,3] )
2 2 0 7 = min (, 7)
=7
3  -3 0

1 2 3
D1[3,2] = min( D0[3,2], D0[3,1]+D0[1,2] )
1 0 0 0
= min (-3,)
P= 2 0 0 1 = -3
3 0 0 0
 1 2 3 k=2
1 5 D1 = 1
3 2
0 4 5
Vertices 1,
4 2 2 0 7

2
-3 3  -3 0
2 can be
1 2 3 intermediat
D2 =
1 0 4 5 2 1
e
D [1,3] = min( D [1,3],
= min (5, 4+7)
1 1
D [1,2]+D [2,3] )
2 2 0 7
=5
3 -1 -3 0

1 2 3
1 0 0 0
D2[3,1] = min( D1[3,1], D1[3,2]+D1[2,1] )
P= 2 0 0 1 = min (, -3+2)
3 2 0 0 = -1
 1 5 D2 =
1 2 3 k=3
1 0 4 5
4 2 3
2 2 0 7
Vertices 1,
2
-3
3 -1 -3 0 2, 3 can be
1 2 3 intermediat
D3 =
1 0 2 5 3 2e D [1,3]+D [3,2] )
D [1,2] = min(D [1,2], 2 2

2 2 0 7 = min (4, 5+(-3))

3 -1 -3 0 =2

1 2 3
1 0 3 0 D3[2,1] = min(D2[2,1], D2[2,3]+D2[3,1] )
= min (2, 7+ (-1))
P= 2 0 0 1
=2
3 2 0 0
Floyd's Algorithm: Using 2 D matrices

Floyd
1. D  W // initialize D array to W [ ]
2. P  0 // initialize P array to [0]
3. for k  1 to n
// Computing D’ from D
4. do for i  1 to n
5. do for j  1 to n
6. if (D[ i, j ] > D[ i, k ] + D[ k, j ] )
7. then D’[ i, j ]  D[ i, k ] + D[ k, j ]
8. P[ i, j ]  k;
9. else D’[ i, j ]  D[ i, j ]
10. Move D’ to D.
Can we use only one D matrix?

• D[i,j] depends only on elements in the kth

column and row of the distance matrix.
• We will show that the kth row and the kth
column of the distance matrix are unchanged
when Dk is computed
• This means D can be calculated in-place
The main diagonal values

• Before we show that kth row and column of D

remain unchanged we show that the main
diagonal remains 0

• D(k)[ j,j ] = min{ D(k-1)[ j,j ] , D(k-1)[ j,k ] + D(k-1)[ k,j

]}
= min{ 0, D(k-1)[ j,k ] + D(k-1)[ k,j ] }
=0
• Based on which assumption?
The kth column

• kth column of Dk is equal to the kth column of

Dk-1
• Intuitively true - a path from i to k will not
become shorter by adding k to the allowed
subset of intermediate vertices

• For all i, D(k)[i,k] =

= min{ D(k-1)[i,k], D(k-1)[i,k]+ D(k-1)[k,k] }
= min { D(k-1)[i,k], D(k-1)[i,k]+0 }
= D(k-1)[i,k]
The kth row

• kth row of Dk is equal to the kth row of Dk-1

For all j, D(k)[k,j] =

= min{ D(k-1)[k,j], D(k-1)[k,k]+ D(k-1)[k,j] }
= min{ D(k-1)[ k,j ], 0+D(k-1)[k,j ] }
= D(k-1)[ k,j ]
Floyd's Algorithm using a single D

Floyd
1. D  W // initialize D array to W [ ]
2. P  0 // initialize P array to [0]
3. for k  1 to n
4. do for i  1 to n
5. do for j  1 to n
6. if (D[ i, j ] > D[ i, k ] + D[ k, j ] )
7. then D[ i, j ]  D[ i, k ] + D[ k, j ]
8. P[ i, j ]  k;
Printing intermediate nodes on shortest path from q to r

path(index q, r) 1 2 3
if (P[ q, r ]!=0)
1 0 3 0
path(q, P[q, r])
println( “v”+ P[q, r]) P= 2 0 0 1
path(P[q, r], r)
3 2 0 0
return;
//no intermediate nodes
else return
1 5
Before calling path check D[q, r] < , and print
node q, after the call to
4 3
path print node r 2
2
-3
Example
 The final distance matrix and P

The values in parenthesis are the non zero P values.

 The call tree for Path(1, 4)

Path(1, 4)

Path(1, 6) Path(6, 4)
Print
v6
P(1, 6)=0

Path(6, 3) Print Path(3, 4)

v3

P(6, 3)=0 P(3, 4)=0

The intermediate nodes on the shortest path from 1 to 4 are v6, v3.
The shortest path is v1, v6, v3, v4.
 CSE408
Maximum Flow

Lecture #30
Maximum Flow

• Maximum Flow Problem

• The Ford-Fulkerson method
• Maximum bipartite matching
Flow networks:

• A flow network G=(V,E): a directed graph, where each edge

(u,v)E has a nonnegative capacity c(u,v)>=0.
• If (u,v)E, we assume that c(u,v)=0.
• two distinct vertices :a source s and a sink t.

12

16 20

s 10 9 7 t
4

13 4

14
Flow:

• G=(V,E): a flow network with capacity function c.

• s-- the source and t-- the sink.
• A flow in G: a real-valued function f:V*V  R satisfying
the following two properties:
• Capacity constraint: For all u,v V,
we require f(u,v)  c( u,v).
• Flow conservation: For all u V-{s,t}, we require

 f (e)   f (e)
e.in.v e.out.v
Net flow and value of a flow f:

• The quantity f (u,v) is called the net flow from vertex

u to vertex v.
• The value of a flow is defined as

f   f ( s, v )
– The total flow fromvsource
V
to any other vertices.
– The same as the total flow from any vertices to
the sink.
 12/12

15/20
11/16

s 10 1/4 4/9 7/7 t

8/13 4/4

11/14

A flow f in G with value . f  19

Maximum-flow problem:

• Given a flow network G with source s and sink t

• Find a flow of maximum value from s to t.

• How to solve it efficiently?

The Ford-Fulkerson method:

• This section presents the Ford-Fulkerson method for solving

the maximum-flow problem.We call it a “method” rather than
an “algorithm” because it encompasses several
implementations with different running times.The Ford-
Fulkerson method depends on three important ideas that
transcend the method and are relevant to many flow
algorithms and problems: residual networks,augmenting
paths,and cuts. These ideas are essential to the important
max-flow min-cut theorem,which characterizes the value of
maximum flow in terms of cuts of the flow network.
Continue:

• FORD-FULKERSON-METHOD(G,s,t)
• initialize flow f to 0
• while there exists an augmenting path p
• do augment flow f along p
• return f
Residual networks:

• Given a flow network and a flow, the residual

network consists of edges that can admit more net
flow.
• G=(V,E) --a flow network with source s and sink t
• f: a flow in G.
• The amount of additional net flow from u to v
before exceeding the capacity c(u,v) is the residual
capacity of (u,v), given by: cf(u,v)=c(u,v)-f(u,v)
in the other direction: cf(v, u)=c(v, u)+f(u, v).
 Example of residual network

12 4/12
16 v1 v3 20 4/16 v1 v3 20

s 10 4 7 t s 7
9 10 4 t
4/9
13 4 13 4/4
v2 v4 v2 v4
14 4/14

(a)
 Example of Residual network (continued)

8
12
v1 v3 20
4
4
s 10 4 4
7 t
5
13 10 4
v2 v4
4

(b)
Fact 1:

• Let G=(V,E) be a flow network with source s and sink t, and let
f be a flow in G.
• Let Gf be the residual network of G induced by f,and let f’ be a
flow in Gf.Then, the flow sum f+f’ is a flow in G with
value .
f  in
• f+f’: the flow  fsame
f ' the  f 'direction will be added.
the flow in different directions will be cnacelled.
Augmenting paths:

• Given a flow network G=(V,E) and a flow f, an augmenting

path is a simple path from s to t in the residual network Gf.

• Residual capacity of p : the maximum amount of net flow that

we can ship along the edges of an augmenting path p, i.e.,
cf(p)=min{cf(u,v):(u,v) is on p}.

2 3 1

The residual capacity is 1.

 Example of an augment path (bold edges)

8
12
v1 v3 20
4
4
s 10 4 4
7 t
5
13 10 4
v2 v4
4

(b)
The basic Ford-Fulkerson algorithm:

• FORD-FULKERSON(G,s,t)
• for each edge (u,v) E[G]
• do f[u,v] 0
• f[v,u] 0
• while there exists a path p from s to t in the residual network
Gf
• do cf(p) min{cf(u,v): (u,v) is in p}
• for each edge (u,v) in p
• do f[u,v] f[u,v]+cf(p)
•
Example:

• The execution of the basic Ford-Fulkerson algorithm.

• (a)-(d) Successive iterations of the while loop: The left side of
each part shows the residual network Gf from line 4 with a
shaded augmenting path p.The right side of each part shows
the new flow f that results from adding fp to f.The residual
network in (a) is the input network G.(e) The residual network
at the last while loop test.It has no augmenting paths,and the
flow f shown in (d) is therefore a maximum flow.
 12 4/12
16 v1 v3 20 4/16 v1 v3 20

s 10 4 7 t s 7
9 10 4 t
4/9
13 4 13 4/4
v2 v4 v2 v4
14 4/14

(a)
 8
12 4/12
v1 v3 20 v1 v3 7/20
4 11/16
4
s 10 4 4
7 t s 7/10 4 7/7 t
5 4/9
13 10 4 13 4/4
v2 v4 v2 v4
4 11/14

(b)
 8 12/12
5 v1 v3 13
4 11/16 v1 v3 15/20
11 4
s 3 11 7 7 t s 7/7
10 1/4 t
5 4/9
13 3
4 8/13 4/4
v2 v4 v2 v4
11 11/14

(c)
 12 12/12
5 v1 v3 5 11/16 v1 v3 19/20
11 4
s 11 3 7 15 t s 7/7
5 10 1/4 t
5 9
8 3
4 12/13 4/4
v2 v4 v2 v4
11 11/14

(d)
 12
5 v1 v3 1

s 3 19
1 11 9 7 t

12 3 4
v2 v4
11

(e)
Time complexity:

• If each c(e) is an integer, then time complexity is O(|E|f*),

where f* is the maximum flow.
• Reason: each time the flow is increased by at least one.
• This might not be a polynomial time algorithm since f* can be
represented by log (f*) bits. So, the input size might be log(f*).
The Edmonds-Karp algorithm

• Find the augmenting path using breadth-first search.

• Breadth-first search gives the shortest path for
graphs (Assuming the length of each edge is 1.)
• Time complexity of Edmonds-Karp algorithm is
O(VE2).
• The proof is very hard and is not required here.
Maximum bipartite matching:

• Bipartite graph: a graph (V, E), where V=LR, L∩R=empty, and

for every (u, v)E, u L and v R.
• Given an undirected graph G=(V,E), a matching is a subset of
edges ME such that for all vertices vV,at most one edge of
M is incident on v.We say that a vertex v V is matched by
matching M if some edge in M is incident on v;otherwise, v is
unmatched. A maximum matching is a matching of maximum
cardinality,that is, a matching M such that for any matching M’,
we have .

M  M'
 L R L R
(a) (b)

A bipartite graph G=(V,E) with vertex partition V=LR.(a)A matching with

cardinality 2.(b) A maximum matching with cardinality 3.
Finding a maximum bipartite matching:

• We define the corresponding flow network G’=(V’,E’) for the

bipartite graph G as follows. Let the source s and sink t be
new vertices not in V, and let V’=V{s,t}.If the vertex
partition of G is V=L R, the directed edges of G’ are given by
E’={(s,u):uL} {(u,v):u L,v R,and (u,v) E} {(v,t):v
R}.Finally, we assign unit capacity to each edge in E’.

• We will show that a matching in G corresponds directly to a

flow in G’s corresponding flow network G’. We say that a flow
f on a flow network G=(V,E) is integer-valued if f(u,v) is an
integer for all (u,v) V*V.
 s
t

L R L R
(a) (b)

(a)The bipartite graph G=(V,E) with vertex partition V=LR. A

maximum matching is shown by shaded edges.(b) The corresponding
flow network.Each edge has unit capacity.Shaded edges have
a flow of 1,and all other edges carry no flow.
Continue:

• Lemma .
• Let G=(V,E) be a bipartite graph with vertex partition
V=LR,and let G’=(V’,E’) be its corresponding flow network.If
M is a matching in G, then there is an integer-valued flow f in
G’ with value .Conversely, if f is an integer-valued
flow in G’,then there is a matchingfMinM G with
cardinality .
• Reason: The edges incidentM tof s and t ensures this.
– Each node in the first column has in-degree 1
– Each node in the second column has out-degree 1.
– So each node in the bipartite graph can be involved once in the flow.
Example:
Aug. path:
 1

1
1

1
1
1

Residual network. Red edges are new edges in the residual network.
The new aug. path is bold. Green edges are old aug. path. old flow=1.
 CSE408
Lower bound theory

Lecture # 31
Lower bound

• We always consider the algorithm with

smaller order of complexity as the better.
• How can we ensure that is there any faster
method available
• We need to find a function g(n) which is a
lower bound on the time that algorithm must
take, So if f(n) is the time for some algorithm
then we can write f(n)=Ω(g(n) where g(n) is
lower bound for f(n)
Lower bound theory

• Or we can say f(n) >= c * g(n) for all n>n0

• Where c & n0 are constants
• For many problems it is easy to calculate the
lower bound on n inputs
• e.g. consider the set of problems to find the
maximum of an ordered set of n integers. Clearly
every integer must be examined at least once. So
Ω(n) is a lower bound for that. For matrix
multiplication we have 2n2inputs so the lower
bound can be Ω(n2)
Sorting and Searching

• For all sorting & searching we use comparison

trees for finding the lower bound.
• For an unordered set the searching algorithm will
take Ω(n) as the lower bound. For an ordered set
it will take

• Ω(logn) as the lower bound. Similarly all the

sorting algorithms can not sort in less then
Ω(nlogn) time so Ω(nlogn) is the lower bound for
sorting algorithms.
Oracle and adversary arguments

• Given some computational model, the oracle tells

the outcome of each comparison. In order to
derive a good lower bound, the oracle tries its
best to cause the algorithm to work as hard as it
might.
• Take the example of a tournament where the
values are called players and the largest value is
interpreted as the winner and the second largest
is taken as the runner up. So the problem is the
similar to finding the largest and the second
largest number out of a set of n numbers.
Second largest

• Since we can’t determine the second largest

element without having determined the
largest element, at least n-1 comparisons are
necessary. We need to show that there is
always some sequence of comparisons which
forces the second largest to be fund in (logn)-1
additional comparisons
Tournament theory

• The winner of the tournament has played x matches then

there are x people who are candidates for the runner up
position. The runner up has lost only once to the winner
and other x-1 persons must have lost to one other person.
We can produce a oracle which decides the results of
matches in such a way that winner plays logn other people.
• In a match between a and b the oracle declares a as the
winner if a is previously undefeated and b has lost at least
once or if both a and b are undefeated but a has won more
match then b. In any other case the oracle can decide
arbitrarily as long as it remains consistent.
Winning match

• Corresponding to this tournament imagine

drawing a directed graph with n vertices. Each
vertex corresponds to one of the n players.
Draw a directed edge from vertex b to a iff
either player a has defeated b or a has
defeated another player who has defeated b.
Since for the overall winner there must be an
edge from each of the remaining n-1 vertices,
it follows that the winner must have played at
least logn matches.
Increasing matrix

• M is an nXn integer matrix in which the keys in

each row are in increasing order. Consider the
problem of finding the position of an integer x
in M. Determine a lower bound for the no. of
comparisons to be done in the problem.
 CSE408
Back tracking Algorithm

Lecture # 32
Backtracking
• Backtracking is a technique
used to solve problems with
a large search space, by
systematically trying and
eliminating possibilities.
Portion A
• A standard example of
backtracking would be going

Portion B
through a maze.
– At some point in a maze, you
might have two options of
which direction to go:
Backtracking
 One strategy would be
to try going through
Portion A of the maze.
Portion B
 If you get stuck before
you find your way out,
then you "backtrack"

Portion A
to the junction.

 At this point in time you

know that Portion A
will NOT lead you out
of the maze,
 so you then start
searching in Portion B
Backtracking
• Clearly, at a single junction
you could have even more
than 2 choices.

• The backtracking strategy

says to try each choice, one
after the other,
– if you ever get stuck,
"backtrack" to the junction C
and try the next choice. B
A

• If you try all choices and

never found a way out,
then there IS no solution to
the maze.
Backtracking – Eight Queens Problem
• Find an arrangement of 8
queens on a single chess board
such that no two queens are
attacking one another.

• In chess, queens can move all

the way down any row, column
or diagonal (so long as no
pieces are in the way).

– Due to the first two restrictions,

it's clear that each row and
column of the board will have
exactly one queen.
Backtracking – Eight Queens Problem
• The backtracking strategy is as Q
follows:
1) Place a queen on the first Q
available square in row 1. Q
Q
2) Move onto the next row,
placing a queen on the first Q Q
available square there (that
doesn't conflict with the
previously placed queens). Continue…
3) Continue in this fashion until
either:
a) you have solved the problem, or
Animated Example:
b) you get stuck. http://www.hbmeyer.de/backtrack
– When you get stuck, remove the
queens that got you there, until you /achtdamen/eight.htm#up
get to a row where there is another
valid square to try.
Backtracking – Eight Queens Problem

• When we carry out backtracking, an easy

way to visualize what is going on is a tree
that shows all the different possibilities
that have been tried.
• On the board we will show a visual
representation of solving the 4 Queens
problem (placing 4 queens on a 4x4 board
where no two attack one another).
Backtracking – Eight Queens Problem

• The neat thing about coding up backtracking,

is that it can be done recursively, without
having to do all the bookkeeping at once.
– Instead, the stack or recursive calls does most of
the bookkeeping
– (ie, keeping track of which queens we've placed,
and which combinations we've tried so far, etc.)
perm[] - stores a valid permutation of queens from index 0 to location-1.
location – the column we are placing the next queen
usedList[] – keeps track of the rows in which the queens have already been placed.

void solveItRec(int perm[], int location, struct onesquare usedList[]) {

if (location == SIZE) { Found a solution to the problem, so print it!

printSol(perm);
}
for (int i=0; i<SIZE; i++) { Loop through possible rows to place this queen.

if (usedList[i] == false) { Only try this row if it hasn’t been used

if (!conflict(perm, location, i)) { Check if this position conflicts with any previous
queens on the diagonal
perm[location] = i;
usedList[i] = true; 1) mark the queen in this row
solveItRec(perm, location+1, usedList); 2) mark the row as used
usedList[i] = false; 3) solve the next column location
} recursively
} 4) un-mark the row as used, so we
} can get ALL possible valid
} solutions.
Backtracking – 8 queens problem - Analysis
• Another possible brute-force algorithm is generate the permutations of
the numbers 1 through 8 (of which there are 8! = 40,320),
– and uses the elements of each permutation as indices to place a queen on
each row.
– Then it rejects those boards with diagonal attacking positions.

• The backtracking algorithm, is a slight improvement on the permutation

method,
– constructs the search tree by considering one row of the board at a time,
eliminating most non-solution board positions at a very early stage in their
construction.
– Because it rejects row and diagonal attacks even on incomplete boards, it
examines only 15,720 possible queen placements.

• A further improvement which examines only 5,508 possible queen

placements is to combine the permutation based method with the early
pruning method:
– The permutations are generated depth-first, and the search space is pruned
if the partial permutation produces a diagonal attack
 CSE408
Sub set sum , Assignment
problem
Lecture # 34
Subset problem
Sub set sum Algo
Branch and Bound
Branch and bound
Assignment problem
Assignment problem
Exmaple
 CSE408
Knapsack problem

Lecture # 35
Dynamic Programming
Dynamic Programming is a general algorithm design technique
for solving problems defined by or formulated as recurrences with
overlapping sub instances

• Invented by American mathematician Richard Bellman in the

1950s to solve optimization problems and later assimilated by CS

• “Programming” here means “planning”

• Main idea:
- set up a recurrence relating a solution to a larger instance to
solutions of some smaller instances
- solve smaller instances once
- record solutions in a table
- extract solution to the initial instance from that table
Knapsack Problem by DP
Given n items of
integer weights: w1 w2 … wn
values: v1 v2 … vn
a knapsack of integer capacity W
find most valuable subset of the items that fit into the
knapsack
Consider instance defined by first i items and capacity
j (j  W).
Let V[i,j] be optimal value of such an instance. Then
max {V[i-1,j], vi + V[i-1,j- wi]} if j- wi  0
{
V[i,j] =
V[i-1,j] if j- wi < 0
Initial conditions: V[0,j] = 0 and V[i,0] = 0
Knapsack Problem by DP (example)
Example: Knapsack of capacity W = 5
item weight value
1 2 $12
2 1 $10
3 3 $20
4 2 $15 capacity j
0 1 2 3 4 5
00 0 0
w1 = 2, v1= 12 1 0 0 12 Backtracing
w2 = 1, v2= 10 2 0 10 12 22 22 22 finds the actual
optimal subset,
w3 = 3, v3= 20 3 0 10 12 22 30 32 i.e. solution.
w4 = 2, v4= 15 4 0 10 15 25 30 37
Knapsack Problem by DP (pseudocode)

Algorithm DPKnapsack(w[1..n], v[1..n], W)

var V[0..n,0..W], P[1..n,1..W]: int
for j := 0 to W do
V[0,j] := 0
for i := 0 to n do Running time and space:
O(nW).
V[i,0] := 0
for i := 1 to n do
for j := 1 to W do
if w[i]  j and v[i] + V[i-1,j-w[i]] > V[i-1,j] then
V[i,j] := v[i] + V[i-1,j-w[i]]; P[i,j] := j-w[i]
else
V[i,j] := V[i-1,j]; P[i,j] := j
return V[n,W] and the optimal subset by backtracing
CSE408
TSP

Lecture # 36
Traveling Salesman Problem

• For each city i, 1≤ i ≤ n, find the sum si of the

distances from city i to the two nearest cities;
compute the sums of these n numbers, divide
the result by 2, and, if all the distances are
integers,round up the result to the nearest
integer:
Traveling Salesman Problem
Edge(a,d) and (d,a)
Example
 CSE408
Vertex Cover and Bin
Packing
Lecture # 37
The vertex-cover problem
Example
The set cover problem
Set cover algo
Bin Packing
• In the bin packing problem, objects of different volumes must
be packed into a finite number of bins or containers each of
volume V in a way that minimizes the number of bins used. In
computational complexity theory, it is a combinatorial NP-
hard problem.
• There are many variations of this problem, such as 2D
packing, linear packing, packing by weight, packing by cost,
and so on.
• They have many applications, such as filling up containers,
loading trucks with weight capacity constraints, creating file
backups in removable media and technology mapping in
Field-programmable gate array semiconductor chip design.
 The bin packing problem can also be seen as a special case of the cutting
stock problem.
 When the number of bins is restricted to 1 and each item is characterised
by both a volume and a value, the problem of maximising the value of
items that can fit in the bin is known as the knapsack problem.
 Despite the fact that the bin packing problem has an NP-hard
computational complexity, optimal solutions to very large instances of the
problem can be produced with sophisticated algorithms.
 In addition, many heuristics have been developed: for example, the first fit
algorithm provides a fast but often non-optimal solution, involving placing
each item into the first bin in which it will fit.
 It requires Θ(n log n) time, where n is the number of elements to be
packed.
Bin packing

• The algorithm can be made much more effective by first

sorting the list of elements into decreasing order
(sometimes known as the first-fit decreasing algorithm),
although this still does not guarantee an optimal solution,
and for longer lists may increase the running time of the
algorithm.
• It is known, however, that there always exists at least one
ordering of items that allows first-fit to produce an optimal
solution.[1]
• An interesting variant of bin packing that occurs in practice
is when items can share space when packed into a bin.
• Specifically, a set of items could occupy less space when
packed together than the sum of their individual sizes.
• This variant is known as VM packing[2] since when
virtual machines (VMs) are packed in a server,
their total memory requirement could decrease
due to pages shared by the VMs that need only
be stored once.
• If items can share space in arbitrary ways, the bin
packing problem is hard to even approximate.
• However, if the space sharing fits into a hierarchy,
as is the case with memory sharing in virtual
machines, the bin packing problem can be
efficiently approximated.
 CSE408
Modular Arithmetic &
Chinese Remainder
Theorem
Lecture # 38
Modular Arthmetic
• In mathematics, modular arithmetic (sometimes
called clock arithmetic) is a system of arithmetic for
integers, where numbers "wrap around" upon
reaching a certain value—the modulus.
Modular Arthmetic

• Modular arithmetic can be handled mathematically by

introducing a congruence relation on the integers that is
compatible with the operations of the ring of integers:
addition, subtraction, and multiplication. For a positive
integer n, two integers a and b are said to be congruent
modulo n, written:

• if their difference a − b is an integer multiple of n (or n

divides a − b). The number n is called the modulus of the
congruence.
• The properties that make this relation a congruence relation
(respecting addition, subtraction, and multiplication) are the
following.
• If
• And
• then:

• It should be noted that the above two properties would

still hold if the theory were expanded to include all real
numbers, that is if were not necessarily all integers. The
next property, however, would fail if these variables were
not all integers:
Chinese Theorem

• The Chinese remainder theorem is a result about congruences in

number theory and its generalizations in abstract algebra. It was
first published in the 3rd to 5th centuries by Chinese mathematician
Sun Tzu.
• In its basic form, the Chinese remainder theorem will determine a
number n that when divided by some given divisors leaves given
remainders.
• For example, what is the lowest number n that when divided by 3
leaves a remainder of 2, when divided by 5 leaves a remainder of 3,
and when divided by 7 leaves a remainder of 2?
• A common introductory example is a woman who tells a policeman
that she lost her basket of eggs, and that if she makes three
portions at a time out of it, she was left with 2, if she makes five
portions at a time out of it, she was left with 3, and if she makes
seven portions at a time out of it, she was left with 2.
• She then asks the policeman what is the minimum number of eggs
she must have had. The answer to both problems is 23.
• Suppose n1, n2, …, nk are positive integers that are pairwise
coprime. Then, for any given sequence of integers a1,a2, …, ak,
there exists an integer x solving the following system of
simultaneous congruences.
Example

Brute Force Technique:-

 CSE408
GCD and
Optimization Problem

Lecture # 39
 Euclidean Algorithm

m,n Euclidean
gcd(m,n)
Algorithm
integer euclid(pos. integer m, pos. integer n)
x = m, y = n
while(y > 0)
r = x mod y
x=y
y=r
return x

L11 2
Euclidean Algorithm. Example

gcd(33,77):
Step r = x mod y x y

0 - 33 77

L11 3
Euclidean Algorithm. Example

gcd(33,77):
Step r = x mod y x y

0 - 33 77
33 mod 77
1 77 33
= 33

L11 4
Euclidean Algorithm. Example

gcd(33,77):
Step r = x mod y x y

0 - 33 77
33 mod 77
1 77 33
= 33
77 mod 33
2 33 11
= 11

L11 5
Euclidean Algorithm. Example

gcd(33,77):
Step r = x mod y x y

0 - 33 77
33 mod 77
1 77 33
= 33
77 mod 33
2 33 11
= 11
33 mod 11
3 11 0
L11 =0 6
Euclidean Algorithm. Example

gcd(244,117):
Step r = x mod y x y
0 - 244 117

L11 7
Euclidean Algorithm. Example

gcd(244,117):
Step r = x mod y x y
0 - 244 117
1 244 mod 117 = 10 117 10

L11 8
Euclidean Algorithm. Example

gcd(244,117):
Step r = x mod y x y
0 - 244 117
1 244 mod 117 = 10 117 10
2 117 mod 10 = 7 10 7

L11 9
Euclidean Algorithm. Example

gcd(244,117):
Step r = x mod y x y
0 - 244 117
1 244 mod 117 = 10 117 10
2 117 mod 10 = 7 10 7
3 10 mod 7 = 3 7 3

L11 10
Euclidean Algorithm. Example

gcd(244,117):
Step r = x mod y x y
0 - 244 117
1 244 mod 117 = 10 117 10
2 117 mod 10 = 7 10 7
3 10 mod 7 = 3 7 3
4 7 mod 3 = 1 3 1

L11 11
Euclidean Algorithm. Example

gcd(244,117):

Step r = x mod y x y
0 - 244 117
1 244 mod 117 = 10 117 10
2 117 mod 10 = 7 10 7
3 10 mod 7 = 3 7 3
4 7 mod 3 = 1 3 1
5 3 mod 1=0 1 0
By
L11
definition  244 and 117 are rel. prime.
12
Euclidean Algorithm Correctness

The reason that Euclidean algorithm works is

gcd(x,y ) is not changed from line to line. If x’,
y’ denote the next values of x , y then:
gcd(x’,y’) = gcd(y, x mod y)
= gcd(y, x + qy) (the useful fact)
= gcd(y, x ) (subtract y -multiple)
= gcd(x,y)

L11 13
Optimization Problem

• In mathematics and computer science, an optimization

problem is the problem of finding the best solution
from all feasible solutions. Optimization problems can
be divided into two categories depending on whether
the variables are continuous or discrete.
• An optimization problem with discrete variables is
known as a combinatorial optimization problem. In a
combinatorial optimization problem, we are looking for
an object such as an integer, permutation or graph
from a finite (or possibly countable infinite) set.
 CSE408
Complexity Classes

Lecture # 40
NP-Completeness

• Poly time algorithm: input size n (in some

encoding), worst case running time – O(nc) for
some constant c.
• Three classes of problems
– P: problems solvable in poly time.
– NP: problems verifiable in poly time.
– NPC: problems in NP and as hard as any problem
in NP.
NP-Completeness (verifiable)

• Verifiable in poly time: given a certificate of a

solution, could verify the certificate is correct in
poly time.
• Examples (their definitions come later):
– Hamiltonian-cycle, given a certificate of a sequence
(v1,v2,…, vn), easily verified in poly time.
– 3-CNF, given a certificate of an assignment 0s, 1s,
easily verified in poly time.
– (so try each instance, and verify it, but 2n instances)
• Why not defined as “solvable in exponential
time?” or “Non Poly time”?
NP-Completeness (why NPC?)

• A problem p NP, and any other problem p

NP can be translated as p in poly time.
• So if p can be solved in poly time, then all
problems in NP can be solved in poly time.
• All current known NP hard problems have been
proved to be NPC.
Relation among P, NP, NPC

• P  NP (Sure)
• NPC  NP (sure)
• P = NP (or P  NP, or P  NP) ???
• NPC = NP (or NPC  NP, or NPC  NP) ???
• P  NP: one of the deepest, most perplexing
open research problems in (theoretical)
computer science since 1971.
Arguments about P, NP, NPC

• No poly algorithm found for any NPC problem

(even so many NPC problems)
• No proof that a poly algorithm cannot exist for
any of NPC problems, (even having tried so
long so hard).
• Most theoretical computer scientists believe
that NPC is intractable (i.e., hard, and P  NP).
View of Theoretical Computer Scientists on P, NP, NPC

NP NPC

P  NP, NPC  NP, P  NPC = 

Why discussion on NPC

• If a problem is proved to be NPC, a good evidence for

its intractability (hardness).
• Not waste time on trying to find efficient algorithm
for it
• Instead, focus on design approximate algorithm or a
solution for a special case of the problem
• Some problems looks very easy on the surface, but in
fact, is hard (NPC).
Decision VS. Optimization Problems

• Decision problem: solving the problem by giving an

answer “YES” or “NO”
• Optimization problem: solving the problem by
finding the optimal solution.
• Examples:
– SHORTEST-PATH (optimization)
• Given G, u,v, find a path from u to v with fewest edges.
– PATH (decision)
• Given G, u,v, and k, whether exist a path from u to v consisting of
at most k edges.
Decision VS. Optimization Problems (Cont.)

• Decision is easier (i.e., no harder) than optimization

• If there is an algorithm for an optimization problem,
the algorithm can be used to solve the
corresponding decision problem.
– Example: SHORTEST-PATH for PATH
• If optimization is easy, its corresponding decision is
also easy. Or in another way, if provide evidence that
decision problem is hard, then the corresponding
optimization problem is also hard.
• NPC is confined to decision problem. (also
applicable to optimization problem.)
– Another reason is that: easy to define reduction between
decision problems.
(Poly) reduction between decision problems

• Problem (class) and problem instance

• Instance  of decision problem A and instance
 of decision problem B
• A reduction from A to B is a transformation
with the following properties:
– The transformation takes poly time
– The answer is the same (the answer for  is YES if
and only if the answer for  is YES).
Implication of (poly) reduction

 
(Poly) Reduction Algorithm for B
Decision algorithm for A

1. If decision algorithm for B is poly, so does A.

A is no harder than B (or B is no easier than A)

2. If A is hard (e.g., NPC), so does B.

3. How to prove a problem B to be NPC ??

(at first, prove B is in NP, which is generally easy.)

3.1 find a already proved NPC problem A
3.2 establish an (poly) reduction from A to B
Question: What is and how to prove the first NPC problem?

Circuit-satisfiability problem.
Discussion on Poly time problems
• (n100) vs. (2n)
– Reasonable to regard a problem of (n100) as
intractable, however, very few practical problem with
(n100).
– Most poly time algorithms require much less.
– Once a poly time algorithm is found, more efficient
algorithm may follow soon.
• Poly time keeps same in many different
computation models, e.g., poly class of serial
random-access machine  poly class of abstract
Turing machine  poly class of parallel computer
(#processors grows polynomially with input size)
• Poly time problems have nice closure properties
under addition, multiplication and composition.
Encoding impact on complexity
• The problem instance must be represented in a way
the program (or machine) can understand.
• General encoding is “binary representation”.
• Different encoding will result in different
complexities.
• Example: an algorithm, only input is integer k,
running time is (k).
– If k is represented in unary: a string of k 1s, the running
time is (k) = (n) on length-n input, poly on n.
– If k is represented in binary: the input length n =  log k 
+1, the running time is (k) = (2n), exponential on n.
• Ruling out unary, other encoding methods are same.
Examples of encoding and complexity

• Given integer n, check whether n is a

composite.
• Dynamic programming for subset-sum.
Class P Problems

• Let n= the length of binary encoding of a problem

(i.e., input size), T(n) is the time to solve it.
• A problem is poly-time solvable if T(n) =O(nk) for
some constant k.
• Complexity class P=set of problems that are poly-
time solvable.
Poly Time Verification

• PATH problem: Given <G,u,v,k>, whether

exists a path from u to v with at most k
edges?
• Moreover, also given a path p from u to v,
verify whether the length of p is at most k?
• Easy or not?
Of course, very easy.
Poly Time Verification, encoding, and language

• Hamiltonian cycles
– A simple path containing every vertex.
– HAM-CYCLE={<G>: G is a Hamiltonian graph, i.e. containing
Hamiltonian cycle}.
– Suppose n is the length of encoding of G.
– HAM-CYCLE can be considered as a Language after encoding, i.e. a
subset of * where ={0,1}*.
• The naïve algorithm for determining HAM-CYCLE runs in
(m!)=(2m) time, where m is the number of vertices, m
n1/2.
• However, given an ordered sequence of m vertices (called
“certificate”), let you verify whether the sequence is a
Hamiltonian cycle. Very easy. In O(n2) time.
Class NP problems
• For a problem p, given its certificate, the
certificate can be verified in poly time.
• Call this kind of problem an NP one.
• Complement set/class: Co-NP.
– Given a set S (as a universal) and given a subset A
– The complement is that S-A.
– When NP problems are represented as languages
(i.e. a set), we can discuss their complement set,
i.e., Co-NP.
Relation among P, NP and co-NP={L: L NP where L= *-L}

P
NP-completeness and Reducibility

• A (class of) problem P1 is poly-time reducible to

P2 , written as P1p P2 if there exists a poly-time
function f: P1  P2 such that for any instance of
p1 P1, p1 has “YES” answer if and only if answer
to f(p1) ( P2) is also “YES”.
• Theorem 34.3: (page 985)
– For two problems P1, P2, if P1p P2 then P2  P implies
P1  P.
NP-completeness and Reducibility (cont.)

• A problem p is NP-complete if
1. p  NP and
2. p'p p for every p'  NP.
(if p satisfies 2, then p is said NP-hard.)
Theorem 34.4 (page 986)
if any NP-compete problem is poly-time
solvable, then P=NP. Or say: if any problem in
NP is not poly-time solvable, then no NP-
complete problem is poly-time solvable.
First NP-complete problem—Circuit
Satisfiability (problem definition)

• Boolean combinational circuit

– Boolean combinational elements, wired together
– Each element, inputs and outputs (binary)
– Limit the number of outputs to 1.
– Called logic gates: NOT gate, AND gate, OR gate.
– true table: giving the outputs for each setting of inputs
– true assignment: a set of boolean inputs.
– satisfying assignment: a true assignment causing the output
to be 1.
– A circuit is satisfiable if it has a satisfying assignment.
Circuit Satisfiability Problem: definition

• Circuit satisfying problem: given a boolean

combinational circuit composed of AND, OR, and
NOT, is it stisfiable?
• CIRCUIT-SAT={<C>: C is a satisfiable boolean circuit}
• Implication: in the area of computer-aided hardware
optimization, if a subcircuit always produces 0, then
the subcircuit can be replaced by a simpler subcircuit
that omits all gates and just output a 0.
Two instances of circuit satisfiability problems
Solving circuit-satisfiability problem
• Intuitive solution:
– for each possible assignment, check whether it
generates 1.
– suppose the number of inputs is k, then the
total possible assignments are 2k. So the
running time is (2k). When the size of the
problem is (k), then the running time is not
poly.
Circuit-satisfiability problem is NP-
complete
• Lemma 34.5:(page 990)
– CIRCUIT-SAT belongs to NP.
• Proof: CIRCUIT-SAT is poly-time verifiable.
– Given (an encoding of) a CIRCUIT-SAT problem C and a certificate,
which is an assignment of boolean values to (all) wires in C.
– The algorithm is constructed as follows: just checks each gates and
then the output wire of C:
• If for every gate, the computed output value matches the value of the
output wire given in the certificate and the output of the whole circuit
is 1, then the algorithm outputs 1, otherwise 0.
• The algorithm is executed in poly time (even linear time).
• An alternative certificate: a true assignment to the inputs.
Circuit-satisfiability problem is NP-
complete (cont.)

• Lemma 34.6: (page 991)

– CIRCUIT-SAT is NP-hard.
• Proof: Suppose X is any problem in NP
– construct a poly-time algorithm F maps every
problem instance x in X to a circuit C=f(x) such that
the answer to x is YES if and only if CCIRCUIT-SAT
(is satisfiable).
Circuit-satisfiability problem is NP-hard (cont.)

• Since XNP, there is a poly-time algorithm A

which verifies X.
• Suppose the input length is n and Let T(n)
denote the worst-case running time. Let k be
the constant such that T(n)=O(nk) and the
length of the certificate is O(nk).
Circuit-satisfiability problem is NP-hard (cont.)

• Idea is to represent the computation of A as a

sequence of configurations, c0, c1,…,ci,ci+1,…,cT(n),
each ci can be broken into
– (program for A, program counter PC, auxiliary machine state, input x,
certificate y, working storage) and
– ci is mapped to ci+1 by the combinational circuit M
implementing the computer hardware.
– The output of A: 0 or 1– is written to some designated
location in working storage. If the algorithm runs for at
most T(n) steps, the output appears as one bit in cT(n).
– Note: A(x,y)=1 or 0.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Circuit-satisfiability problem is NP-hard (cont.)

• The reduction algorithm F constructs a single

combinational circuit C as follows:
– Paste together all T(n) copies of the circuit M.
– The output of the ith circuit, which produces ci, is directly
fed into the input of the (i+1)st circuit.
– All items in the initial configuration, except the bits
corresponding to certificate y, are wired directly to their
known values.
– The bits corresponding to y are the inputs to C.
– All the outputs to the circuit are ignored, except the one
bit of cT(n) corresponding to the output of A.
Circuit-satisfiability problem is NP-hard (cont.)
• Two properties remain to be proven:
– F correctly constructs the reduction, i.e., C is
satisfiable if and only if there exists a certificate y,
such that A(x,y)=1.
Suppose there is a certificate y, such that A(x,y)=1. Then
if we apply the bits of y to the inputs of C, the output of
C is the bit of A(x,y), that is C(y)= A(x,y) =1, so C is
satisfiable.
Suppose C is satisfiable, then there is a y such that
C(y)=1. So, A(x,y)=1.
– F runs in poly time.
Circuit-satisfiability problem is NP-hard (cont.)
– F runs in poly time.
• Poly space:
– Size of x is n.
– Size of A is constant, independent of x.
– Size of y is O(nk).
– Amount of working storage is poly in n since A runs at most O(nk).
– M has size poly in length of configuration, which is poly in O(nk), and
hence is poly in n.
– C consists of at most O(nk) copies of M, and hence is poly in n.
– Thus, the C has poly space.
• The construction of C takes at most O(nk) steps and each step takes
poly time, so F takes poly time to construct C from x.
CIRCUIT-SAT is NP-complete

• In summary
– CIRCUIT-SAT belongs to NP, verifiable in poly time.
– CIRCUIT-SAT is NP-hard, every NP problem can be
reduced to CIRCUIT-SAT in poly time.
– Thus CIRCUIT-SAT is NP-complete.
NP-completeness proof basis
• Lemma 34.8 (page 995)
– If X is a problem (class) such that P'p X for some P' 
NPC, then X is NP-hard. Moreover, if X NP, then X
NPC.
• Steps to prove X is NP-complete
– Prove X NP.
• Given a certificate, the certificate can be verified in poly time.
– Prove X is NP-hard.
• Select a known NP-complete P'.
• Describe a transformation function f that maps every instance x
of P' into an instance f(x) of X.
• Prove f satisfies that the answer to xP' is YES if and only if the
answer to f(x)X is YES for all instance x P'.
• Prove that the algorithm computing f runs in poly-time.
 NPC proof –Formula Satisfiability (SAT)
• SAT definition
– n boolean variables: x1, x2,…, xn.
– M boolean connectives: any boolean function with one
or two inputs and one output, such as ,,,,,…
and
– Parentheses.
• A SAT  is satisfiable if there exists an true
assignment which causes  to evaluate to 1.
• SAT={< >:  is a satifiable boolean formula}.
• The historical honor of the first NP-complete
problem ever shown.
SAT is NP-complete

• Theorem 34.9: (page 997)

– SAT is NP-complete.
• Proof:
– SAT belongs to NP.
• Given a satisfying assignment, the verifying algorithm
replaces each variable with its value and evaluates the
formula, in poly time.
– SAT is NP-hard (show CIRCUIT-SATp SAT).
 SAT is NP-complete (cont.)
• CIRCUIT-SATp SAT, i.e., any instance of circuit
satisfiability can be reduced in poly time to an instance
of formula satisfiability.
• Intuitive induction:
– Look at the gate that produces the circuit output.
– Inductively express each of gate’s inputs as formulas.
– Formula for the circuit is then obtained by writing an
expression that applies the gate’s function to its input
formulas.
•Unfortunately, this is not a poly reduction
–Shared formula (the gate whose output is fed to 2 or more inputs of
other gates) cause the size of generated formula to grow exponentially.
SAT is NP-complete (cont.)
• Correct reduction:
– For every wire xi of C, give a variable xi in the formula.
– Every gate can be expressed as xo(xi1 xi2…  xil)
– The final formula  is the AND of the circuit output variable
and conjunction of all clauses describing the operation of
each gate. (example Figure 34.10)
• Correctness of the reduction
– Clearly the reduction can be done in poly time.
– C is satisfiable if and only if  is satisfiable.
• If C is satisfiable, then there is a satisfying assignment. This means
that each wire of C has a well-defined value and the output of C is 1.
Thus the assignment of wire values to variables in  makes each
clause in  evaluate to 1. So  is 1.
• The reverse proof can be done in the same way.
 Example of reduction of CIRCUIT-SAT to SAT
= x10(x10(x7 x8 x9))
(x9(x6  x7))
(x8(x5  x6))
(x7(x1 x2 x4))
(x6 x4))
(x5(x1  x2))
(x4x3)

INCORRECT REDUCTION: = x10= x7 x8 x9=(x1 x2 x4)  (x5  x6) (x6  x7)
=(x1 x2 x4)  ((x1  x2)  x4) (x4  (x1 x2 x4))=….
NPC Proof –3-CNF Satisfiability

• 3-CNF definition
– A literal in a boolean formula is an occurrence of a variable
or its negation.
– CNF (Conjunctive Nornal Form) is a boolean formula
expressed as AND of clauses, each of which is the OR of
one or more literals.
– 3-CNF is a CNF in which each clause has exactly 3 distinct
literals (a literal and its negation are distinct)
• 3-CNF-SAT: whether a given 3-CNF is satiafiable?
 3-CNF-SAT is NP-complete
• Proof: 3-CNF-SAT NP. Easy.
– 3-CNF-SAT is NP-hard. (show SAT p3-CNF-SAT)
• Suppose  is any boolean formula, Construct a binary ‘parse’ tree,
with literals as leaves and connectives as internal nodes.
• Introduce a variable yi for the output of each internal nodes.
• Rewrite the formula to ' as the AND of the root variable and a
conjunction of clauses describing the operation of each node.
• The result is that in ', each clause has at most three literals.
• Change each clause into conjunctive normal form as follows:
– Construct a true table, (small, at most 8 by 4)
– Write the disjunctive normal form for all true-table items evaluating to 0
– Using DeMorgan law to change to CNF.
• The resulting '' is in CNF but each clause has 3 or less literals.
• Change 1 or 2-literal clause into 3-literal clause as follows:
– If a clause has one literal l, change it to (lpq)(lpq) (lpq)
(lpq).
– If a clause has two literals (l1 l2), change it to (l1 l2 p)  (l1 l2 p).
Binary parse tree for =((x1 x2) ((x1 x3)  x4))x2

'= y1(y1  (y2x2))

(y2  (y3  y4))
(y4  y5)
(y3  (x1  x2))
(y5  (y6  x4))
(y6  (x1  x3))
Example of Converting a 3-literal clause to CNF format

Disjunctive Normal Form:

i'=(y1y2x2)(y1y2x2)
(y1y2x2) (y1y2x2)

Conjunctive Normal Form:

i''=(y1y2x2)(y1y2x2)
(y1y2x2)(y1y2x2)
3-CNF is NP-complete

•  and reduced 3-CNF are equivalent:

– From  to ' , keep equivalence.
– From ' to '' , keep equivalence.
– From '' to final 3-CNF, keep equivalence.
• Reduction is in poly time,
– From  to ' , introduce at most 1 variable and 1 clause per
connective in .
– From ' to '' , introduce at most 8 clauses for each clause in '.
– From '' to final 3-CNF, introduce at most 4 clauses for each clause in
''.
NP-completeness proof structure
 NPC proof -- CLIQUE
• Definition: a clique in an undirected graph G=(V,E) is a
subset V'V of vertices, each pair of which is connected
by an edge in E, i.e., a clique is a complete subgraph of G.
• Size of a clique is the number of vertices in the clique.
• Optimization problem: find the maximum clique.
• Decision problem: whether a clique of given size k exists
in the graph?
• CLIQUE={<G,k>: G is a graph with a clique of size k.}
• Intuitive solution: ???
 CLIQUE is NP-complete
• Theorem 34.11: (page 1003)
– CLIQUE problem is NP-complete.
• Proof:
– CLIUEQE NP: given G=(V,E) and a set V'V as a
certificate for G. The verifying algorithm checks for
each pair of u,vV', whether <u,v> E. time:
O(|V'|2|E|).
– CLIQUE is NP-hard:
• show 3-CNF-SAT pCLUQUE.
• The result is surprising, since from boolean formula to
graph.
CLIQUE is NP-complete

• Reduction from 3-CNF-SAT to CLUQUE.

– Suppose =C1 C2… Ck be a boolean formula in 3-CNF
with k clauses.
– We construct a graph G=(V,E) as follows:
• For each clause Cr =(l1r l2r l3r), place a triple of v1r, v2r, v3r into V
• Put the edge between two vertices vir and vjs when:
– rs, that is vir and vjs are in different triples, and
– Their corresponding literals are consistent, i.e, lir is not negation of ljs .
– Then  is satisfiable if and only if G has a clique of size k.
=(x1x2x3)(x1x2x3)(x1x2x3) and its reduced graph G

C1=x1x2 x3
 CLIQUE is NP-complete
• Prove the above reduction is correct:
– If  is satisfiable, then there exists a satisfying assignment,
which makes at least one literal in each clause to evaluate to
1. Pick one this kind of literal in each clause. Then consider
the subgraph V' consisting of the corresponding vertex of
each such literal. For each pair vir,vjs V', where rs. Since
lir,ljs are both evaluated to 1, so lir is not negation of ljs, thus
there is an edge between vir and vjs. So V' is a clique of size k.
– If G has a clique V' of size k, then V' contains exact one
vertex from each triple. Assign all the literals corresponding
to the vertices in V' to 1, and other literals to 1 or 0, then
each clause will be evaluated to 1. So  is satisfiable.
• It is easy to see the reduction is in poly time.
• The reduction of an instance of one problem to a
specific instance of the other problem.
Traveling-salesman problem is NPC
• TSP={<G,c,k>:
G=(V,E) is a complete graph,
c is a function from VVZ,
kZ, and G has a traveling salesman
tour with cost at most k.}
• Theorem 34.14: (page 1012)
– TSP is NP-complete.
 TSP is NP-complete
• TSP belongs to NP:
– Given a certificate of a sequence of vertices in the tour, the
verifying algorithm checks whether each vertex appears once,
sums up the cost and checks whether at most k. in poly time.
• TSP is NP-hard (show HAM-CYCLEpTSP)
– Given an instance G=(V,E) of HAM-CYCLE, construct a TSP
instance <G',c,0) as follows (in poly time):
• G'=(V,E'), where E'={<i,j>: i,j V and ij} and
• Cost function c is defined as c(i,j)=0 if (i,j) E, 1, otherwise.
– If G has a hamiltonian cycle h, then h is also a tour in G' with
cost at most 0.
– If G' has a tour h' of cost at most 0, then each edge in h' is 0, so
each edge belong to E, so h' is also a hilmitonian cycle in G.
Subset Sum is NPC
• SUNSET-SUM={<S,t>: S is a set of integers
and there exists a S'S such that t=sS's.}
• Theorem 34.15: (page 1014)
– SUBSET-SUM is NP-complete.
SUBSET-SUM is NPC
• SUBSET-SUM belongs to NP.
– Given a certificate S', check whether t is sum of
S' can be finished in poly time.
• SUBSET-SUM is NP-hard (show 3-CNF-
SATpSUBSET-SUM).
 SUBSET-SUM is NPC
• Given a 3-CNF formula =C1 C2… Ck with literals x1, x2,…, xn. Construct a SUBSET-
SUM instance as follows:
– Two assumptions: no clause contains both a literal and its negation, and either a literal or
its negation appears in at least one clause.
– The numbers in S are based on 10 and have n+k digits, each digit corresponds to (or is
labeled by) a literal or a clause.
– Target t=1…1||4…4 (n 1’s and k 4’s)
– For each literal xi, create two integers:
• vi=0…01(i)0…0||0…01(l)0…01(w)0…0, where xi appears in Cl,…,Cw.
• vi'=0…01(i)0…0||0…1(m)0……01(p)0…0, where xi appears in Cm,…,Cp. .
• Clearly, vi and vi' can not be equal in right k digits, moreover all vi and vi' in S are distinct.
– For each clause Cj, create two integers:
• sj=0…0||0…01(j)0…0,
• sj'=0…0||0…02(j)0…0.
• all sj and sj' are called “slack number”. Clearly, all sj and sj' in S are distinct.
– Note: the sum of digits in any one digit position is 2 or 6, so when there is no carries when adding any
subset of the above integers.
SUBSET-SUM is NPC
• The above reduction is done in poly time.
• The 3-CNF formula  is satisfiable if and only if there is a subset S'
whose sum is t.
– suppose  has a satisfying assignment.
• Then for i=1,…,n, if xi=1 in the assignment, then vi is put in S', otherwise, then
vi' is put in S'.
• The digits labeled by literals will sum to 1.
• Moreover, for each digit labeled by a clause Cj and in its three literals, there
may be 1, 2, or 3 assignments to be 1. correspondingly, both sj and sj' or sj', or
sj is added to S' to make the sum of the digit to 4.
• So S' will sum to 1…14…4.
– Suppose there is a S' which sums to 1…14…4. then S' contains exact one
of vi and vi' for i=1,…,n. if vi S', then set xi=1, otherwise, vi' S', then set
xi=0. It can be seen that this assignment makes each clause of  to
evaluate to 1. so  is satisfiable.