Scribd
Upload
48 views

DP and Edit Dist

This document discusses dynamic programming approaches to calculating edit distance between strings. It introduces the concept of using a cache or memoization to store previously calculated subproblem solutions, in order to avoid recomputing them, and thereby improve the efficiency of the recursive edit distance algorithm from exponential to linear time. An example dynamic programming algorithm is presented that builds up the edit distance values in a 2D table.

Uploaded by

jagadeeswara71
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
48 views

DP and Edit Dist

This document discusses dynamic programming approaches to calculating edit distance between strings. It introduces the concept of using a cache or memoization to store previously calculated subproblem solutions, in order to avoid recomputing them, and thereby improve the efficiency of the recursive edit distance algorithm from exponential to linear time. An example dynamic programming algorithm is presented that builds up the edit distance values in a 2D table.

Uploaded by

jagadeeswara71
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 30

Dynamic programming

and edit distance


Ben Langmead

Department of Computer Science

You are free to use these slides. If you do, please sign the
guestbook (www.langmead-lab.org/teaching-materials), or email
me ([email protected]) and tell me briefly how you’re
using them. For original Keynote files, email me.
Beyond approximate matching: sequence similarity
In many settings, Hamming and edit distance are too simple. Biologically-relevant
distances require algorithms. We will expand our tool set accordingly.

Score = 248 bits (129), Expect = 1e-63


Identities = 213/263 (80%), Gaps = 34/263 (12%)
Strand = Plus / Plus

Query: 161 atatcaccacgtcaaaggtgactccaactcca---ccactccattttgttcagataatgc 217


||||||||||||||||||||||||||||| | | | || ||||||||||||||
Sbjct: 481 atatcaccacgtcaaaggtgactccaact-tattgatagtgttttatgttcagataatgc 539

Query: 218 ccgatgatcatgtcatgcagctccaccgattgtgagaacgacagcgacttccgtcccagc 277


||||||| ||||||||||||||||||||| || | ||||||||||||
Sbjct: 540 ccgatgactttgtcatgcagctccaccgattttg-g------------ttccgtcccagc 586

Query: 278 c-gtgcc--aggtgctgcctcagattcaggttatgccgctcaattcgctgcgtatatcgc 334


| || | | ||||||||||||||||||||||||||||||||||||||| |||||||||
Sbjct: 587 caatgacgta-gtgctgcctcagattcaggttatgccgctcaattcgctgggtatatcgc 645

Query: 335 ttgctgattacgtgcagctttcccttcaggcggga------------ccagccatccgtc 382


||||||||||||||||||||||||||||||||||| |||||||||||||
Sbjct: 646 ttgctgattacgtgcagctttcccttcaggcgggattcatacagcggccagccatccgtc 705

Query: 383 ctccatatc-accacgtcaaagg 404


|||||||| ||||||||||||| Example BLAST alignment
Sbjct: 706 atccatatcaaccacgtcaaagg 728
Approximate string matching

A mismatch is a single-character substitution:


X: G T A G C G G C G
||| |||||
Y: G T A A C G G C G

An edit is a single-character substitution or gap (insertion or deletion):


X: G T A G C G G C G
||| |||||
Y: G T A A C G G C G
Gap in X
X: G T A G C - G C G
||||| ||| AKA insertion in Y or deletion in X
Y: G T A G C G G C G

X: G T A G C G G C G
|| |||||| AKA insertion in X or deletion in Y
Y: G T - G C G G C G
Gap in Y
Alignment

X: G C G T A T G A G G C T A - A C G C
|| |||| ||||| ||||
Y: G C - T A T G C G G C T A T A C G C

Above is an alignment: a way of lining up the characters of x and y

Could include mismatches, gaps or both

Vertical lines are drawn where opposite characters match


Hamming and edit distance

Finding Hamming distance between 2 strings is easy:


def  hammingDistance(x,  y):
       assert  len(x)  ==  len(y)
       nmm  =  0 GAGGTAGCGGCGTTTAAC
       for  i  in  xrange(0,  len(x)): | |||| ||| |||||||
               if  x[i]  !=  y[i]: GTGGTAACGGGGTTTAAC
                       nmm  +=  1
       return  nmm

Edit distance is harder:


def  editDistance(x,  y): GCGTATGCGGCTA-ACGC
|| |||||||||| |||
       ??? GC-TATGCGGCTATACGC
Edit distance

def  editDistance(x,  y): GCGTATGCGGCTA-ACGC


|| |||||||||| ||||
       return  ??? GC-TATGCGGCTATACGC

If strings x and y are same length, what can we say about


editDistance(x, y) relative to hammingDistance(x, y)?

editDistance(x, y) ≤ hammingDistance(x, y)

If strings x and y are different lengths, what can we say about


editDistance(x, y)?

editDistance(x, y) ≥ | | x | - | y | |

Python example: http://bit.ly/CG_DP_EditDist


Edit distance
Can think of edits as being introduced by an optimal editor working left-to-right.
Edit transcript describes how editor turns x into y.

x: GCGTATGCGGCTAACGC Operations:
M = match, R = replace,
y: GCTATGCGGCTATACGC I = insert into x, D = delete from x

x: GCGTATGCGGCTAACGC
|| MMD
y: G C - T A T G C G G C T A T A C G C

x: GCGTATGCGGCTA-ACGC
|| |||||||||| MMDMMMMMMMMMMI
y: G C - T A T G C G G C T A T A C G C

x: GCGTATGCGGCTA-ACGC
|| |||||||||| |||| MMDMMMMMMMMMMIMMMM
y: G C - T A T G C G G C T A T A C G C
Edit distance

Edit transcripts with


Alignments:
respect to x:

x: GCGTATGCGGCTA-ACGC
|| |||||||||| |||| MMDMMMMMMMMMMIMMMM Distance = 2
y: G C - T A T G C G G C T A T A C G C

x: GCGTATGAGGCTA-ACGC
|| |||| ||||| |||| MMDMMMMRMMMMMIMMMM Distance = 3
y: G C - T A T G C G G C T A T A C G C

x: the longest----
||||||| DDDDMMMMMMMIIII Distance = 8
y: - - - - l o n g e s t d a y
Edit distance
D[i, j]: edit distance between length-i prefix of x and length-j prefix of y
i
x:
y:
j

Think in terms of edit transcript. Optimal transcript for D[i, j] can be built
by extending a shorter one by 1 operation. Only 3 options:

Append D to transcript for D[i-1, j]


Append I to transcript for D[i, j-1]
Append M or R to transcript for D[i-1, j-1]

D[i, j] is minimum of the three, and D[|x|, |y|] is the overall edit distance
Edit distance

Let D[0, j] = j, and let D[i, 0] = i


8 D
< D[i 1, j] + 1 I
Otherwise, let D[i, j] = min D[i, j 1] + 1 M or R
:
D[i 1, j 1] + (x[i 1], y[j 1])
(a, b) is 0 if a = b, 1 otherwise
Edit distance
Let D[0, j] = j, and let D[i, 0] = i
8
< D[i 1, j] + 1
Otherwise, let D[i, j] = min D[i, j 1] + 1
:
D[i 1, j 1] + (x[i 1], y[j 1])
(a, b) is 0 if a = b, 1 otherwise

A simple recursive algorithm: prefixes of x and y currently


under consideration
def  edDistRecursive(x,  y):
       if  len(x)  ==  0:  return  len(y)
       if  len(y)  ==  0:  return  len(x)
       delt  =  1  if  x[-­‐1]  !=  y[-­‐1]  else  0
       diag  =  edDistRecursive(x[:-­‐1],  y[:-­‐1])  +  delt  
       vert  =  edDistRecursive(x[:-­‐1],  y)  +  1
Recursively solve
       horz  =  edDistRecursive(x,  y[:-­‐1])  +  1 smaller problems
       return  min(diag,  vert,  horz)

Python example: http://bit.ly/CG_DP_EditDist


Edit distance
def  edDistRecursive(x,  y):
       if  len(x)  ==  0:  return  len(y)
       if  len(y)  ==  0:  return  len(x)
       delt  =  1  if  x[-­‐1]  !=  y[-­‐1]  else  0
       diag  =  edDistRecursive(x[:-­‐1],  y[:-­‐1])  +  delt  
       vert  =  edDistRecursive(x[:-­‐1],  y)  +  1
       horz  =  edDistRecursive(x,  y[:-­‐1])  +  1
       return  min(diag,  vert,  horz)

>>>  import  datetime  as  d


>>>  st  =  d.datetime.now();  \
...  edDistRecursive("Shakespeare",  "shake  spear");  \
...  print  (d.datetime.now()-­‐st).total_seconds()
3
31.498284

Simple, but takes >30 seconds for a small problem


Edit distance: dynamic programming
Subproblems (D[i, j]s) can be reused instead of being recalculated:
def  edDistRecursive(x,  y):
       if  len(x)  ==  0:  return  len(y)
       if  len(y)  ==  0:  return  len(x)
       delt  =  1  if  x[-­‐1]  !=  y[-­‐1]  else  0
       diag  =  edDistRecursive(x[:-­‐1],  y[:-­‐1])  +  delt  
       vert  =  edDistRecursive(x[:-­‐1],  y)  +  1
       horz  =  edDistRecursive(x,  y[:-­‐1])  +  1
       return  min(diag,  vert,  horz)

def  edDistRecursiveMemo(x,  y,  memo=None):


       if  memo  is  None:  memo  =  {}
       if  len(x)  ==  0:  return  len(y)
Reusing Return        if  len(y)  ==  0:  return  len(x)
solutions to memoized        if  (len(x),  len(y))  in  memo:
               return  memo[(len(x),  len(y))]
subproblems is answer, if        delt  =  1  if  x[-­‐1]  !=  y[-­‐1]  else  0
memoization: avaialable        diag  =  edDistRecursiveMemo(x[:-­‐1],  y[:-­‐1],  memo)  +  delt
       vert  =  edDistRecursiveMemo(x[:-­‐1],  y,  memo)  +  1
       horz  =  edDistRecursiveMemo(x,  y[:-­‐1],  memo)  +  1
       ans  =  min(diag,  vert,  horz)
Memoize D[i, j]        memo[(len(x),  len(y))]  =  ans
       return  ans

Python example: http://bit.ly/CG_DP_EditDist


Edit distance: dynamic programming
def  edDistRecursiveMemo(x,  y,  memo=None):
       if  memo  is  None:  memo  =  {}
       if  len(x)  ==  0:  return  len(y)
       if  len(y)  ==  0:  return  len(x)
       if  (len(x),  len(y))  in  memo:
               return  memo[(len(x),  len(y))]
       delt  =  1  if  x[-­‐1]  !=  y[-­‐1]  else  0
       diag  =  edDistRecursiveMemo(x[:-­‐1],  y[:-­‐1],  memo)  +  delt
       vert  =  edDistRecursiveMemo(x[:-­‐1],  y,  memo)  +  1
       horz  =  edDistRecursiveMemo(x,  y[:-­‐1],  memo)  +  1
       ans  =  min(diag,  vert,  horz)
       memo[(len(x),  len(y))]  =  ans
       return  ans

>>>  import  datetime  as  d


>>>  st  =  d.datetime.now();  \
...  edDistRecursiveMemo("Shakespeare",  "shake  spear");  \
...  print  (d.datetime.now()-­‐st).total_seconds()
3
0.000593

Much better
Edit distance: dynamic programming

edDistRecursiveMemo is a top-down dynamic programming approach


Alternative is bottom-up. Here, bottom-up recursion is pretty intuitive and
interpretable, so this is how edit distance algorithm is usually explained.
Fills in a table (matrix) of D(i, j)s:
import  numpy numpy: package for matrices, etc

def  edDistDp(x,  y):


       """  Calculate  edit  distance  between  sequences  x  and  y  using
               matrix  dynamic  programming.    Return  distance.  """
       D  =  numpy.zeros((len(x)+1,  len(y)+1),  dtype=int)
       D[0,  1:]  =  range(1,  len(y)+1)
       D[1:,  0]  =  range(1,  len(x)+1) Fill 1st row, col
       for  i  in  xrange(1,  len(x)+1):
               for  j  in  xrange(1,  len(y)+1):
                       delt  =  1  if  x[i-­‐1]  !=  y[j-­‐1]  else  0 Fill rest of matrix
                       D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,  D[i-­‐1,  j]+1,  D[i,  j-­‐1]+1)
       return  D[len(x),  len(y)]
Edit distance: dynamic programming

ϵ is empty y
string
ϵ G C T A T G C C A C G C Let n = | x |, m = | y |
ϵ
G D: (n+1) x (m+1) matrix
C D[i, j] = edit distance b/t
G length-i prefix of x and
T length-j prefix of y
A
D: x T
G
C
A
C
G
C
Edit distance: dynamic programming
y
ϵ G C T A T G C C A C G C
ϵ
G D[6, 5]
C D[5, 5]
G
D[5, 6]
T
A
D: x T D[6, 6]
G Value in a cell depends upon its upper,
C left, and upper-left neighbors
A 8
C < D[i 1, j] + 1 upper
left upper-left
G D[i, j] = min
:
D[i, j 1] + 1
D[i 1, j 1] + (x[i 1], y[j 1])
C
Edit distance: dynamic programming
       D  =  numpy.zeros((len(x)+1,  len(y)+1),  dtype=int)
First few lines
       D[0,  1:]  =  range(1,  len(y)+1)
of edDistDp:        D[1:,  0]  =  range(1,  len(x)+1)

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12 Initialize D[0, j] to j,
G 1 D[i, 0] to i
C 2
G 3
T 4
A 5
T 6
G 7
C 8
A 9
C 10
G 11
C 12
Edit distance: dynamic programming
       for  i  in  xrange(1,  len(x)+1):
Loop from                for  j  in  xrange(1,  len(y)+1):
edDistDp:                        delt  =  1  if  x[i-­‐1]  !=  y[j-­‐1]  else  0
                       D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,  D[i-­‐1,  j]+1,  D[i,  j-­‐1]+1)

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12 Fill remaining cells from
G 1 top row to bottom and
C 2 from left to right
G 3
T 4
A 5 etc
T 6
G 7
C 8
A 9
C 10
G 11
C 12
Edit distance: dynamic programming
       for  i  in  xrange(1,  len(x)+1):
Loop from                for  j  in  xrange(1,  len(y)+1):
edDistDp:                        delt  =  1  if  x[i-­‐1]  !=  y[j-­‐1]  else  0
                       D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,  D[i-­‐1,  j]+1,  D[i,  j-­‐1]+1)

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12 Fill remaining cells from
G 1 ? top row to bottom and
C 2 from left to right
G 3
T 4 What goes here in i=1,j=1?
A 5 x[i-­‐1] = y[j-­‐1] = ‘G ‘,
T 6 so delt =  0
G 7
C 8 D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,
                           D[i-­‐1,  j]+1,
A 9                            D[i,  j-­‐1]+1)
               =  min(0  +  0,  1  +  1,  1  +  1)
C 10                =  0
G 11
C 12
Edit distance: dynamic programming
       for  i  in  xrange(1,  len(x)+1):
Loop from                for  j  in  xrange(1,  len(y)+1):
edDistDp:                        delt  =  1  if  x[i-­‐1]  !=  y[j-­‐1]  else  0
                       D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,  D[i-­‐1,  j]+1,  D[i,  j-­‐1]+1)

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12 Fill remaining cells from
G 1 0 1 2 3 4 5 6 7 8 9 10 11 top row to bottom and
C 2 1 0 1 2 3 4 5 6 7 8 9 10 from left to right
G 3 2 1 1 2 3 3 4 5 6 7 8 9
T 4 3 2 1 2 2 3 4 5 6 7 8 9
A 5 4 3 2 1 2 3 4 5 5 6 7 8
T 6 5 4 3 2 1 2 3 4 5 6 7 8
G 7 6 5 4 3 2 1 2 3 4 5 6 7
C 8 7 6 5 4 3 2 1 2 3 4 5 6
A 9 8 7 6 5 4 3 2 2 2 3 4 5
C 10 9 8 7 6 5 4 3 2 3 2 3 4 Edit distance for x, y
G 11 10 9 8 7 6 5 4 3 3 3 2 3
C 12 11 10 9 8 7 6 5 4 4 3 3 2
Edit distance: dynamic programming
       for  i  in  xrange(1,  len(x)+1):
Loop from                for  j  in  xrange(1,  len(y)+1):
edDistDp:                        delt  =  1  if  x[i-­‐1]  !=  y[j-­‐1]  else  0
                       D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,  D[i-­‐1,  j]+1,  D[i,  j-­‐1]+1)

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12 Could we have filled the
G 1 cells in a different order?
C 2
G 3
T 4
A 5 etc
T 6
G 7
C 8
A 9
C 10
G 11
C 12
Edit distance: dynamic programming
       for  j  in  xrange(1,  len(y)+1):  
Switched                for  i  in  xrange(1,  len(x)+1):
                       delt  =  1  if  x[i-­‐1]  !=  y[j-­‐1]  else  0
                       D[i,  j]  =  min(D[i-­‐1,  j-­‐1]+delt,  D[i-­‐1,  j]+1,  D[i,  j-­‐1]+1)

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 Yes: e.g. invert the loops
C 2
G 3
T 4
A 5
T 6 etc
G 7
C 8
A 9
C 10
G 11
C 12
Edit distance: dynamic programming

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 Or by anti-diagonal
C 2
G 3
T 4 etc
A 5
T 6
G 7
C 8
A 9
C 10
G 11
C 12
Edit distance: dynamic programming

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 Or blocked
C 2
G 3
T 4
A 5
T 6
G 7
C 8
A 9
C 10
G 11 etc
C 12
Edit distance: getting the alignment

Full backtrace path corresponds to an optimal alignment / edit transcript:


Start at end; at each step, ask: which predecessor gave the minimum?

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 0 1 2 3 4 5 6 7 8 9 10 11
C 2 1 0 1 2 3 4 5 6 7 8 9 10
G 3 2 1 1 2 3 3 4 5 6 7 8 9
T 4 3 2 1 2 2 3 4 5 6 7 8 9
A 5 4 3 2 1 2 3 4 5 5 6 7 8
T 6 5 4 3 2 1 2 3 4 5 6 7 8
G 7 6 5 4 3 2 1 2 3 4 5 6 7
C 8 7 6 5 4 3 2 1 2 3 4 5 6
A 9 8 7 6 5 4 3 2 2 2 3 4 5
C 10 9 8 7 6 5 4 3 2 3 2 3 4 A: From here
G 11 10 9 8 7 6 5 4 3 3 3 2 3 Q: How did I get here?
C 12 11 10 9 8 7 6 5 4 4 3 3 2
Edit distance: getting the alignment

Full backtrace path corresponds to an optimal alignment / edit transcript:


Start at end; at each step, ask: which predecessor gave the minimum?

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 0 1 2 3 4 5 6 7 8 9 10 11
C 2 1 0 1 2 3 4 5 6 7 8 9 10
G 3 2 1 1 2 3 3 4 5 6 7 8 9
T 4 3 2 1 2 2 3 4 5 6 7 8 9
A 5 4 3 2 1 2 3 4 5 5 6 7 8
T 6 5 4 3 2 1 2 3 4 5 6 7 8
G 7 6 5 4 3 2 1 2 3 4 5 6 7
C 8 7 6 5 4 3 2 1 2 3 4 5 6
A 9 8 7 6 5 4 3 2 2 2 3 4 5 A: From here
C 10 9 8 7 6 5 4 3 2 3 2 3 4 Q: How did I get here?
G 11 10 9 8 7 6 5 4 3 3 3 2 3
C 12 11 10 9 8 7 6 5 4 4 3 3 2
Edit distance: getting the alignment

Full backtrace path corresponds to an optimal alignment / edit transcript:


Start at end; at each step, ask: which predecessor gave the minimum?

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 0 1 2 3 4 5 6 7 8 9 10 11
C 2 1 0 1 2 3 4 5 6 7 8 9 10
G 3 2 1 1 2 3 3 4 5 6 7 8 9
T 4 3 2 1 2 2 3 4 5 6 7 8 9
A 5 4 3 2 1 2 3 4 5 5 6 7 8
T 6 5 4 3 2 1 2 3 4 5 6 7 8 A: From here
G 7 6 5 4 3 2 1 2 3 4 5 6 7 Q: How did I get here?
C 8 7 6 5 4 3 2 1 2 3 4 5 6
A 9 8 7 6 5 4 3 2 2 2 3 4 5
C 10 9 8 7 6 5 4 3 2 3 2 3 4
G 11 10 9 8 7 6 5 4 3 3 3 2 3
C 12 11 10 9 8 7 6 5 4 4 3 3 2
Edit distance: getting the alignment

Full backtrace path corresponds to an optimal alignment / edit transcript:


Start at end; at each step, ask: which predecessor gave the minimum?

ϵ G C T A T G C C A C G C
ϵ 0 1 2 3 4 5 6 7 8 9 10 11 12
G 1 0 1 2 3 4 5 6 7 8 9 10 11
Alignment:
C 2 1 0 1 2 3 4 5 6 7 8 9 10
G 3 2 1 1 2 3 3 4 5 6 7 8 9 GCGTATG-CACGC
T 4 3 2 1 2 2 3 4 5 6 7 8 9 || |||| |||||
GC-TATGCCACGC
A 5 4 3 2 1 2 3 4 5 5 6 7 8
T 6 5 4 3 2 1 2 3 4 5 6 7 8
G 7 6 5 4 3 2 1 2 3 4 5 6 7
C 8 7 6 5 4 3 2 1 2 3 4 5 6 Edit transcript:
A 9 8 7 6 5 4 3 2 2 2 3 4 5 MMDMMMMIMMMMM
C 10 9 8 7 6 5 4 3 2 3 2 3 4
G 11 10 9 8 7 6 5 4 3 3 3 2 3
C 12 11 10 9 8 7 6 5 4 4 3 3 2
Edit distance: summary

Matrix-filling dynamic programming algorithm is O(mn) time and space

FillIng matrix is O(mn) space and time, and yields edit distance

Backtrace is O(m + n) time, yields optimal alignment / edit transcript

You might also like