‫طراحی ا لگور یتم ها‬

‫مبحث دوازدهم‪ :‬برنامه نویسی پو یا‬

‫ﺳﺠﺎد ﺷﯿﺮﻋیل ﴲﺮﺿﺎ‬

‫ﲠﺎر ‪1401‬‬
‫ﯾﮏ ﺷﻨﺒﻪ‪ 15 ،‬ﻓﺮوردﯾﻦ ‪1402‬‬ ‫‪1‬‬
‫‪Based on Stanford CS161 Slides from Summer 2021 by Karey Shi‬‬
 ‫اﻃﻼع رﺳﺎ ﮲ﻰ‬
‫ه‪15 :‬‬ ‫اب برای این‬ ‫●‬
‫● یادآوری امت ان دوم‪ :‬ی ش به دو ه ته دی ‪ ،‬ی ش به ‪ 27‬روردین ‪1402‬‬

‫‪2‬‬
‫مقدمه ای بر برنامه نویسی پو یا‬

‫ﯾﮏ روش ﻃﺮاﺣﯽ اﻟﮕﻮرﯾﱲ‬

‫‪3‬‬
DYNAMIC PROGRAMMING

Dynamic programming (DP) is an algorithm design paradigm.

It’s often used to solve optimization problems (e.g. shortest path).

4
DYNAMIC PROGRAMMING

Dynamic programming (DP) is an algorithm design paradigm.

It’s often used to solve optimization problems (e.g. shortest path).

We’ll see an example of DP today:

Longest Common Subsequence (LCS)
We will see two more examples of DP next week for shortest path
problems: Bellman-Ford and Floyd-Warshall algorithms

But first, an overview of DP!

5
DYNAMIC PROGRAMMING

Elements of dynamic programming:

Optimal substructure: the optimal solution of a problem can be
expressed in terms of optimal solutions to smaller sub-problems.
e.g. d(k)[b] = min{ d(k–1)[b], mina{d(k–1)[a] + w(a, b)} }

Overlapping sub-problems: the subproblems overlap a lot!

This means we can save time by solving a sub-problem once & cache the answer.
(this is sometimes called “memoization”)
e.g. Lots of different entries in the row d(k) may ask for d(k-1)[v]

6
DYNAMIC PROGRAMMING

Elements of dynamic programming:

Optimal substructure: the optimal solution of a problem can be
expressed in terms of optimal solutions to smaller sub-problems.
e.g. d(k)[b] = min{ d(k–1)[b], mina{d(k–1)[a] + w(a, b)} }

Overlapping sub-problems: the subproblems overlap a lot!

This means we can save time by solving a sub-problem once & cache the answer.
(this is sometimes called “memoization”)
e.g. Lots of different entries in the row d(k) may ask for d(k-1)[v]

7
DYNAMIC PROGRAMMING

Elements of dynamic programming:

Optimal substructure: the optimal solution of a problem can be
expressed in terms of optimal solutions to smaller sub-problems.
e.g. d(k)[b] = min{ d(k–1)[b], mina{d(k–1)[a] + w(a, b)} }

Overlapping sub-problems: the subproblems overlap a lot!

This means we can save time by solving a sub-problem once & cache the answer.
(this is sometimes called “memoization”)
e.g. Lots of different entries in the row d(k) may ask for d(k-1)[v]

8
DYNAMIC PROGRAMMING

Elements of dynamic programming:

Optimal substructure: the optimal solution of a problem can be
expressed in terms of optimal solutions to smaller sub-problems.
e.g. d(k)[b] = min{ d(k–1)[b], mina{d(k–1)[a] + w(a, b)} }

Overlapping sub-problems: the subproblems overlap a lot!

This means we can save time by solving a sub-problem once & cache the answer.
(this is sometimes called “memoization”)
e.g. Lots of different entries in the row d(k) may ask for d(k-1)[v]

9
DYNAMIC PROGRAMMING

Elements of dynamic programming:

Optimal substructure: the optimal solution of a problem can be
expressed in terms of optimal solutions to smaller sub-problems.
e.g. d(k)[b] = min{ d(k–1)[b], mina{d(k–1)[a] + w(a, b)} }

Overlapping sub-problems: the subproblems overlap a lot!

This means we can save time by solving a sub-problem once & cache the answer.
(this is sometimes called “memoization”)
e.g. Lots of different entries in the row d(k) may ask for d(k-1)[v]

10
DYNAMIC PROGRAMMING
Two approaches for DP
(2 different ways to think about and/or implement DP algorithms)

Bottom-up: iterates through problems by size and solves the small

problems first (kind of like taking care of base cases first & building up).
e.g. Bellman-Ford (as we will see shortly!) computes d(0), then d(1), then d(2), etc.

Top-down: instead uses recursive calls to solve smaller problems, while

using memoization/caching to keep track of small problems that you’ve
already computed answers for (simply fetch the answer instead of
re-solving that problem and waste computational effort)
We will see a way later to implement Bellman-Ford using a top-down approach.

11
DYNAMIC PROGRAMMING
Two approaches for DP
(2 different ways to think about and/or implement DP algorithms)

Bottom-up: iterates through problems by size and solves the small

problems first (kind of like taking care of base cases first & building up).
e.g. Bellman-Ford (as we will see shortly!) computes d(0), then d(1), then d(2), etc.

Top-down: instead uses recursive calls to solve smaller problems, while

using memoization/caching to keep track of small problems that you’ve
already computed answers for (simply fetch the answer instead of
re-solving that problem and waste computational effort)
We will see a way later to implement Bellman-Ford using a top-down approach.

12
DYNAMIC PROGRAMMING
Two approaches for DP
(2 different ways to think about and/or implement DP algorithms)

Bottom-up: iterates through problems by size and solves the small

problems first (kind of like taking care of base cases first & building up).
e.g. Bellman-Ford (as we will see next week) computes d(0), then d(1), then d(2), etc.

Top-down: instead uses recursive calls to solve smaller problems, while

using memoization/caching to keep track of small problems that you’ve
already computed answers for (simply fetch the answer instead of
re-solving that problem and waste computational effort)
We will see a way later to implement Bellman-Ford using a top-down approach.

13
DYNAMIC PROGRAMMING
Two approaches for DP
(2 different ways to think about and/or implement DP algorithms)

Bottom-up: iterates through problems by size and solves the small

problems first (kind of like taking care of base cases first & building up).
e.g. Bellman-Ford (as we will see next week) computes d(0), then d(1), then d(2), etc.

Top-down: instead uses recursive calls to solve smaller problems, while

using memoization/caching to keep track of small problems that you’ve
already computed answers for (simply fetch the answer instead of
re-solving that problem and waste computational effort)
We will see a way later to implement Bellman-Ford using a top-down approach.

14
DYNAMIC PROGRAMMING
Two approaches for DP
(2 different ways to think about and/or implement DP algorithms)

Bottom-up: iterates through problems by size and solves the small

problems first (kind of like taking care of base cases first & building up).
e.g. Bellman-Ford (as we will see next week) computes d(0), then d(1), then d(2), etc.

Top-down: instead uses recursive calls to solve smaller problems, while

using memoization/caching to keep track of small problems that you’ve
already computed answers for (simply fetch the answer instead of
re-solving that problem and waste computational effort)
We will see a way later to implement Bellman-Ford using a top-down approach.

15
DYNAMIC PROGRAMMING

Why “dynamic programming”?

Richard Bellman invented the term in the 1950’s. He was working for
the RAND corporation at the time, which was employed by the Air
Force, and government projects needed flashy non-mathematical
non-researchy names to get funded and approved.

“It’s impossible to use the word dynamic in a pejorative sense…

I thought dynamic programming was a good name.
It was something not even a Congressman could object to.”

16
DIVIDE & CONQUER vs DP
DIVIDE-AND-CONQUER DYNAMIC PROGRAMMING

big problem big problem

sub- sub- sub- sub- sub-

problem problem problem problem problem

sub-sub sub-sub sub-sub sub-sub sub-sub sub-sub sub-sub sub-sub

problem problem problem problem problem problem problem problem
‫ﺳﻮال؟‬
‫‪18‬‬
RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

19
RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

20
RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

21
RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

22
RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

23
‫ﺳﻮال؟‬
‫‪24‬‬
‫پیداکردن بزرگ ترین‬
‫زیر رشته مشترک‬

‫‪25‬‬
LONGEST COMMON SUBSEQUENCE
A sequence Z is a SUBSEQUENCE of X if Z can be obtained from X by deleting symbols

BDFH is a subsequence of ABCDEFGH

C is a subsequence of ABCDEFGH
ABCDEFGH is a subsequence of ABCDEFGH

26
LONGEST COMMON SUBSEQUENCE
A sequence Z is a SUBSEQUENCE of X if Z can be obtained from X by deleting symbols

BDFH is a subsequence of ABCDEFGH

C is a subsequence of ABCDEFGH
ABCDEFGH is a subsequence of ABCDEFGH

A sequence Z is a LONGEST COMMON SUBSEQUENCE (LCS) of X and Y

if Z is a subsequence of both X and Y
and any sequence longer than Z is not a subsequence of at least one of X or Y.

ABDFGH is the LCS of

ABCDEFGH and ABDFGHI
27
LONGEST COMMON SUBSEQUENCE
A sequence Z is a SUBSEQUENCE of X if Z can be obtained from X by deleting symbols

BDFH is a subsequence of ABCDEFGH

C is a subsequence of ABCDEFGH
ABCDEFGH is a subsequence of ABCDEFGH

TASK: Given sequences X and Y, find the length of their LCS, Z.

A sequence Z is a LONGEST COMMON SUBSEQUENCE (LCS) of X and Y
if Z is
(Later, we’ll a subsequence
also ofstart
output Z, but we’ll bothoffXwith
andtheY length)
and any sequence longer than Z is not a subsequence of at least one of X or Y.

ABDFGH is the LCS of

ABCDEFGH and ABDFGHI
28
APPLICATIONS OF LCS

Computational
the diff unix
Bioinformatics! linguistics!
command!

Detect similarities Extract similarities in

Identify differences
between DNA or words/word-forms
between the contents
protein sequences and determine how
of two files
words are related

and so much more...

29
‫ﺳﻮال؟‬
‫‪30‬‬
LCS: RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

31
STEP 1: OPTIMAL SUBSTRUCTURE
SUBPROBLEM: Find the length of LCS’s of prefixes to X and Y.

X A C G G A T

Y A C T A T

Notation: denote this prefix ACT by Y3

32
STEP 1: OPTIMAL SUBSTRUCTURE
SUBPROBLEM: Find the length of LCS’s of prefixes to X and Y.

X A C G G A T

Y A C T A T
Examples:
C[2,3] = 2 (LCS of X2 and Y3 is AC)
Notation: denote this prefix ACT by Y3 C[5,4] = 3 (LCS of X5 and Y4 is ACA)

Let C[i, j] = length_of_LCS(Xi, Yj)

33
STEP 1: OPTIMAL SUBSTRUCTURE
SUBPROBLEM: Find the length of LCS’s of prefixes to X and Y.

X A C G G A T

Y A C T A T
Examples:
C[2,3] = 2 (LCS of X2 and Y3 is AC)
Notation: denote this prefix ACT by Y3 C[5,4] = 3 (LCS of X5 and Y4 is ACA)

Let C[i, j] = length_of_LCS(Xi, Yj)

Why is this a good choice?

34
STEP 1: OPTIMAL SUBSTRUCTURE
Let C[i, j] = length_of_LCS(Xi, Yj)
Consider the ends of our prefixes, X[i] and Y[j]. We have two cases:

Case 1: X[i] = Y[j] Case 2: X[i] ≠ Y[j]

35
STEP 1: OPTIMAL SUBSTRUCTURE
Let C[i, j] = length_of_LCS(Xi, Yj)
Consider the ends of our prefixes, X[i] and Y[j]. We have two cases:

Case 1: X[i] = Y[j] Case 2: X[i] ≠ Y[j]

i i

Xi A C G G A Xi A C G G T

Yj A C T A Yj A C T A

j These two are the same!

j
These two are not the same!

36
STEP 1: OPTIMAL SUBSTRUCTURE
Let C[i, j] = length_of_LCS(Xi, Yj)
Consider the ends of our prefixes, X[i] and Y[j]. We have two cases:

Case 1: X[i] = Y[j] Case 2: X[i] ≠ Y[j]

i i

Xi A C G G A Xi A C G G T

Yj A C T A Yj A C T A

j These two are the same!

j
These two are not the same!

What is C[i, j]? What is C[i, j]?

37
STEP 1: OPTIMAL SUBSTRUCTURE
Let C[i, j] = length_of_LCS(Xi, Yj)
Consider the ends of our prefixes, X[i] and Y[j]. We have two cases:

Case 1: X[i] = Y[j] Case 2: X[i] ≠ Y[j]

i i

Xi A C G G A Xi A C G G T

Yj A C T A Yj A C T A

j These two are the same!

j
These two are not the same!

Then, C[i, j] = 1 + C[i–1, j–1]

because LCS(Xi, Yj) = LCS(Xi–1, Yj-1) followed by A .

38
STEP 1: OPTIMAL SUBSTRUCTURE
Let C[i, j] = length_of_LCS(Xi, Yj)
Consider the ends of our prefixes, X[i] and Y[j]. We have two cases:

Case 1: X[i] = Y[j] Case 2: X[i] ≠ Y[j]

i i

Xi A C G G A Xi A C G G T

Yj A C T A Yj A C T A

j These two are the same!

j
These two are not the same!

Then, C[i, j] = 1 + C[i–1, j–1] Then, C[i, j] = max{ C[i–1, j], C[i, j–1] }
because LCS(Xi, Yj) = LCS(Xi–1, Yj-1) followed by A . Give A a chance to “match”: LCS(Xi, Yj) = LCS(Xi–1, Y )
Give T a chance to “match”: LCS(Xi, Yj) = LCS(Xi, Yj–1)
39
‫ﺳﻮال؟‬
‫‪40‬‬
LCS: RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

41
STEP 2: RECURSIVE FORMULATION
Our recursive formulation:

0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

42
STEP 2: RECURSIVE FORMULATION
Our recursive formulation:

0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

CASE 0 (base case)

X0
Yj A C T A
43
STEP 2: RECURSIVE FORMULATION
Our recursive formulation:

0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

CASE 0 (base case) CASE 1

X0 Xi A C G G A

Yj A C T A Yj A C T A
44
STEP 2: RECURSIVE FORMULATION
Our recursive formulation:

0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

CASE 0 (base case) CASE 1 CASE 2

X0 Xi A C G G A Xi A C G G T

Yj A C T A Yj A C T A Yj A C T A
45
‫ﺳﻮال؟‬
‫‪46‬‬
LCS: RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

47
STEP 3: WRITE A DP ALGORITHM

We’ll store answers to our subproblems C[i, j] in a table (this is our cache)!

Now that we’ve defined our recursive formulation, translating to appropriate

pseudocode is straightforward: establish your base cases & define your cases!

We’ll do this in a bottom-up fashion. Why?

We know we need answers to shorter prefixes first, and it’s pretty easy to iterate in
order and fill out answers to smaller prefixes before building up to longer prefixes
(ultimately getting our final answer C[m, n] where |X| = m, and |Y| = n)

48
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
C[i,j] = max{ C[i,j–1], C[i–1,j] }
return C[m,n]

49
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
Make sure that C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
our base cases
are set up
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
C[i,j] = max{ C[i,j–1], C[i–1,j] }
return C[m,n]

50
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
Make sure that C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
our base cases
are set up
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1 Case 1

else:
C[i,j] = max{ C[i,j–1], C[i–1,j] }
return C[m,n]

51
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
Make sure that C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
our base cases
are set up
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1 Case 1

else:
C[i,j] = max{ C[i,j–1], C[i–1,j] } Case 2
return C[m,n]

52
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
Make sure that C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
our base cases
are set up
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1 Case 1

else:
Final answer
C[i,j] = max{ C[i,j–1], C[i–1,j] } Case 2
return C[m,n]

53
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
Make sure that C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
our base cases
are set up
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1 Case 1

else:
Final answer
C[i,j] = max{ C[i,j–1], C[i–1,j] } Case 2
return C[m,n]

Runtime: O(mn)
? 54
STEP 3: WRITE A DP ALGORITHM
0 if i = 0 or j = 0
C[ i, j ] = C[i–1, j–1] + 1 if X[i] = Y[j] and i, j > 0
max{ C[i–1, j], C[i, j–1] } if X[i] ≠ Y[j] and i, j > 0

LCS(X,Y): len(X) = m & len(Y) = n

Initialize an (m+1) x (n+1) 0-indexed array C
Make sure that C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n
our base cases
are set up
for i = 1,...,m and j = 1,...,n:
if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1 Case 1

else:
Final answer
C[i,j] = max{ C[i,j–1], C[i–1,j] } Case 2
return C[m,n]
Constant amount of work to fill
Runtime: O(mn) out each of the mn entries in C 55
‫ﺳﻮال؟‬
‫‪56‬‬
EXAMPLE
Y
A C T G Initialize an (m+1) x (n+1) 0-indexed array C
C[i,0] = C[0,j] = 0 for all i=0,...,m and j=0,...,n

0 0 0 0 0
A 0
C 0 Fill in our base
X G 0 cases first
G 0
A 0
57
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1
C 0
X[i] = Y[j]
X G 0 C[i,j] = C[i–1,j–1] + 1

G 0
A 0
58
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1
C 0
X[i] ≠ Y[j]
X G 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
59
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1
C 0
X[i] ≠ Y[j]
X G 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
60
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0
X[i] ≠ Y[j]
X G 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
61
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1
X[i] ≠ Y[j]
X G 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
62
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2
X[i] = Y[j]
X G 0 C[i,j] = C[i–1,j–1] + 1

G 0
A 0
63
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2
X[i] ≠ Y[j]
X G 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
64
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
65
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
66
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
67
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0
A 0
68
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] = Y[j]
X G 0 1 2 2 3 C[i,j] = C[i–1,j–1] + 1

G 0
A 0
69
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 3 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0 1
A 0
70
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 3 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0 1 2
A 0
71
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 3 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0 1 2 2
A 0
72
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] = Y[j]
X G 0 1 2 2 3 C[i,j] = C[i–1,j–1] + 1

G 0 1 2 2 3
A 0
73
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] = Y[j]
X G 0 1 2 2 3 C[i,j] = C[i–1,j–1] + 1

G 0 1 2 2 3
A 0 1
74
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 3 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0 1 2 2 3
A 0 1 2
75
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 3 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0 1 2 2 3
A 0 1 2 2
76
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
X[i] ≠ Y[j]
X G 0 1 2 2 3 C[i,j] = max{ C[i,j–1], C[i–1,j] }

G 0 1 2 2 3
A 0 1 2 2 3
77
EXAMPLE
Y
for i = 1,...,m and j = 1,...,n:
A C T G if X[i] = Y[j]:
C[i,j] = C[i–1,j–1] + 1
else:
0 0 0 0 0 C[i,j] = max{ C[i,j–1], C[i–1,j] }
A 0 1 1 1 1
C 0 1 2 2 2
So the LCM of X and Y
X G 0 1 2 2 3
has length 3.
G 0 1 2 2 3
A 0 1 2 2 3
78
‫ﺳﻮال؟‬
‫‪79‬‬
LCS: RECIPE FOR APPLYING DP

1. Identify optimal substructure. What are your overlapping subproblems?

2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.

80
STEP 4: FIND ACTUAL LCS
Y
A C T G Suppose we want to
recover the actual LCS.
0 0 0 0 0 How can we construct the actual LCS
A 0 1 1 1 1 given this table C that we just filled out?

C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3
A 0 1 2 2 3
81
STEP 4: FIND ACTUAL LCS
Y
A C T G Suppose we want to
recover the actual LCS.
0 0 0 0 0 How can we construct the actual LCS
A 0 1 1 1 1 given this table C that we just filled out?

C 0 1 2 2 2 Weʼll start at C[m,n] and work backwards to trace

out how we ended up with a 3 as our answer!
X G 0 1 2 2 3 If we see that the last character in X matches the
character in Y, then we mark that character as part
G 0 1 2 2 3 of our LCS and take a diagonal step backwards.
Otherwise, if the characters donʼt match, we just
A 0 1 2 2 3 simply take a step towards the larger adjacent entry
82
STEP 4: FIND ACTUAL LCS
Y
A C T G RECOVER_LCS(X, Y, C):
// C is already filled out
L = []
0 0 0 0 0 i = m
j = n
A 0 1 1 1 1 while i > 0 and j > 0:
if X[i] = Y[j]:
append X[i] to the beginning of L
C 0 1 2 2 2 i = i–1
j = j–1
X G 0 1 2 2 3 else if C[i,j] = C[i,j–1]:
j = j–1
G 0 1 2 2 3 else:
i = i–1
A 0 1 2 2 3 return L
83
STEP 4: FIND ACTUAL LCS
Y
A C T G RECOVER_LCS(X, Y, C):
// C is already filled out
L = []
0 0 0 0 0 i = m
j = n
A 0 1 1 1 1 while i > 0 and j > 0:
if X[i] = X[j]:
append X[i] to the beginning of L
C 0 1 2 2 2 i = i–1
j = j–1
X G 0 1 2 2 3 else if C[i,j] = C[i,j–1]:
j = j–1
G 0 1 2 2 3 else: Runtime?
i = i–1
A 0 1 2 2 3 return L
84
STEP 4: FIND ACTUAL LCS
Y
A C T G RECOVER_LCS(X, Y, C):
// C is already filled out
L = []
0 0 0 0 0 i = m
j = n
A 0 1 1 1 1 while i > 0 and j > 0:
if X[i] = X[j]:
append X[i] to the beginning of L
C 0 1 2 2 2 i = i–1
j = j–1
X G 0 1 2 2 3 else if C[i,j] = C[i,j–1]:
j = j–1
G 0 1 2 2 3 else: This extra subroutine
i = i–1 takes O(m+n) time!
A 0 1 2 2 3 return L
85
STEP 4: FIND ACTUAL LCS
Y
A C T G RECOVER_LCS(X, Y, C):
// C is already filled out
L = []
0 Note:
0 Sometimes,
0 0 0 you don’t need i = m
j = n to track more info in your
A 0 original
1 DP 1 algorithm
1 1 from Stepwhile 3 (alli we
ifyou
> 0did
X[i]may
andhere
= X[j]:
was do some
j > 0:
reverse engineering). Other times, want to augment
append X[i] to the beginning of L
C 0 your1 Step
2 3 algorithm
2 2 to keep track iof=extra i–1 information (e.g.
info that might tell you which subproblem
j = j–1 contributed the
X G 0 1 2most,2 hints 3 to help you reverse
else ifengineer,
C[i,j] = etc.)
C[i,j–1]:
j = j–1
G 0 1 2 2 3 else: This extra subroutine
i = i–1 takes O(m+n) time!
A 0 1 2 2 3 return L
86
STEP 4: FIND ACTUAL LCS
Y
A C T G

0 0 0 0 0
A 0 1 1 1 1
C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y:

A 0 1 2 2 3
87
STEP 4: FIND ACTUAL LCS
Y
A C T G
X[5] ≠ Y[4]
0 0 0 0 0 We don’t add anything to our LCS.
A 0 1 1 1 1 But we can go up → C[4,4]

C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y:

A 0 1 2 2 3
88
STEP 4: FIND ACTUAL LCS
Y
A C T G
X[4] = Y[4]
0 0 0 0 0 We can add “G” to our LCS
A 0 1 1 1 1 We go diagonally back → C[3,3]

C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y: G
A 0 1 2 2 3
89
STEP 4: FIND ACTUAL LCS
Y
A C T G
X[3] ≠ Y[3]
0 0 0 0 0 We don’t add anything to our LCS.
A 0 1 1 1 1 But we can go up → C[2,3]
(Going left is okay too since it’s a tie. How you choose to break ties might
C 0 1 2 2 2 result in different LCS’s when there are multiple. In this example, there’s
actually only one LCS so we we’ll end up with the same LCS either way)

X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y: G
A 0 1 2 2 3
90
STEP 4: FIND ACTUAL LCS
Y
A C T G
X[2] ≠ Y[3]
0 0 0 0 0 We don’t add anything to our LCS.
A 0 1 1 1 1 But we can go left → C[2,2]

C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y: G
A 0 1 2 2 3
91
STEP 4: FIND ACTUAL LCS
Y
A C T G
X[2] = Y[2]
0 0 0 0 0 We can add “C” to our LCS.
A 0 1 1 1 1 We go diagonally back → C[1,1]

C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y: C G
A 0 1 2 2 3
92
STEP 4: FIND ACTUAL LCS
Y
A C T G
X[1] = Y[1]
0 0 0 0 0 We can add “A” to our LCS.
A 0 1 1 1 1 We’re done!

C 0 1 2 2 2
X G 0 1 2 2 3
G 0 1 2 2 3 LCS of X and Y: A C G
A 0 1 2 2 3
93
‫ﺳﻮال؟‬
‫‪94‬‬
LCS: RECIPE FOR APPLYING DP
1. Identify optimal substructure. What are your overlapping subproblems?
2. Define a recursive formulation. Recursively define your optimal solution in terms
of sub-solutions. Always write down this formulation.
3. Use dynamic programming. Turn the recursive formulation into a DP algorithm.
4. If needed, track additional information. You may need to solve a related
problem, e.g. step 3 finds you an optimal value/cost, but you need to recover the
actual optimal solution/path/subset/substring/etc. Go back and modify your
algorithm in step 3 to make this happen.
5. Can we do better? Any wasted space? Other things to optimize?
(We won’t focus on this step too much in lecture/assignments/exams, but in practice, this is definitely a very important step to always consider)!

95
STEP 5: CAN WE DO BETTER?
● If we only care about the length of the LCS, then we don’t need to store the entire
table (we can just store 2 rows at a time).
● If we want to recover the entire LCS, we do need to keep the whole table.
● Can we improve the runtime of O(mn)?
○ If you have a bounded alphabet size, you can reduce the running time of the DP algorithm
by a logarithmic factor (using the Method of Four Russians).
○ The general LCS problem is NP-hard, so performing much better is an open problem!

96
STEP 5: CAN WE DO BETTER?
● If we only care about the length of the LCS, then we don’t need to store the entire
table (we can just store 2 rows at a time).
● If we want to recover the entire LCS, we do need to keep the whole table.
● Can we improve the runtime of O(mn)?
○ If you have a bounded alphabet size, you can reduce the running time of the DP algorithm
by a logarithmic factor (using the Method of Four Russians).
○ The general LCS problem is NP-hard, so performing much better is an open problem!

97
STEP 5: CAN WE DO BETTER?
● If we only care about the length of the LCS, then we don’t need to store the entire
table (we can just store 2 rows at a time).
● If we want to recover the entire LCS, we do need to keep the whole table.
● Can we improve the runtime of O(mn)?
○ If you have a bounded alphabet size, you can reduce the running time of the DP algorithm
by a logarithmic factor (using the Method of Four Russians).
○ The general LCS problem is NP-hard, so performing much better is an open problem!

98
STEP 5: CAN WE DO BETTER?
● If we only care about the length of the LCS, then we don’t need to store the entire
table (we can just store 2 rows at a time).
● If we want to recover the entire LCS, we do need to keep the whole table.
● Can we improve the runtime of O(mn)?
○ If you have a bounded alphabet size, you can reduce the running time of the DP algorithm
by a logarithmic factor (using the Method of Four Russians).
○ The general LCS problem is NP-hard, so performing much better is an open problem!

99
STEP 5: CAN WE DO BETTER?
● If we only care about the length of the LCS, then we don’t need to store the entire
table (we can just store 2 rows at a time).
● If we want to recover the entire LCS, we do need to keep the whole table.
● Can we improve the runtime of O(mn)?
○ If you have a bounded alphabet size, you can reduce the running time of the DP algorithm
by a logarithmic factor (using the Method of Four Russians).
○ The general LCS problem is NP-hard, so performing much better is an open problem!

100
STEP 5: CAN WE DO BETTER?
● If we only care about the length of the LCS, then we don’t need to store the entire
table (we can just store 2 rows at a time).
● If we want to recover the entire LCS, we do need to keep the whole table.
● Can we improve the runtime of O(mn)?
○ If you have a bounded alphabet size, you can reduce the running time of the DP algorithm
by a logarithmic factor (using the Method of Four Russians).
○ The general LCS problem is NP-hard, so performing much better is an open problem!

101
‫ﺳﻮال؟‬
‫‪102‬‬