What is dynamic programming?

Sean R. Eddy
Howard Hughes Medical Institute & Department of Genetics,
Washington University School of Medicine
4444 Forest Park Blvd., Box 8510
Saint Louis, Missouri 63108 USA
[email protected]
June 8, 2004

Sequence alignment methods often use something called a “dynamic program-

ming” algorithm. What is dynamic programming, and how does it work?

Dynamic programming algorithms are a very good place to start understand-

ing what’s really going on inside computational biology software. The heart of
many well-known programs is a dynamic programming algorithm, or a fast ap-
proximation of one, including sequence database search programs like BLAST
and FASTA, multiple sequence alignment programs like CLUSTALW, profile
search programs like HMMER, genefinding programs like GENSCAN, and even
RNA folding programs like MFOLD and phylogenetic inference programs like
PHYLIP.
Don’t expect much enlightenment from the etymology of the term “dynamic
programming”, though. Dynamic programming was formalized in the early 1950’s
by mathematician Richard Bellman, who was working at RAND Corporation on
optimal decision processes. He wanted to concoct an impressive name that would
shield his work from U.S. Secretary of Defense Charles Wilson, a man known
to be hostile to mathematics research. His work involved time series and plan-
ning – hence “dynamic” and “programming” (note, nothing particularly to do
with computer programming). Bellman especially liked “dynamic” because “it’s
impossible to use the word dynamic in a pejorative sense”; he figured dynamic
programming was “something not even a Congressman could object to” [1].

1
 The best way to understand how dynamic programming works is to see an
example. Conveniently, optimal sequence alignment provides an example that is
both simple and biologically relevant.

The biological problem: pairwise sequence alignment

We have two DNA or protein sequences, and we want to infer if they are ho-
mologous or not. To do this, we will calculate a score that reflects how similar
the two sequences are (that is, how likely they are to be derived from a common
ancestor). Since sequences will differ not just by substitution, but also by inser-
tion and deletion, we want to optimally align the two sequences to maximize their
similarity.
Why do we need a fancy algorithm? Can we just score all possible alignments
2N
and pick the best one? This isn’t practical, because there are about √22πN different
alignments for two sequences of length N ; for two sequences of length 300, that’s
about 10179 different alignments.
Let’s set up the problem with some notation. Call the two sequences x and
y. They are of length M and N residues, respectively. Call the i’th residue of
x xi , and the j’th residue of y yj . We need some parameters for how to score
alignments: we’ll use a scoring matrix σ(a, b) for aligning two residues a, b to
each other (e.g. a 4x4 matrix for scoring any pair of aligned DNA nucleotides,
or simply a match and a mismatch score), and a gap penalty γ for every time we
introduce a gap character.
A dynamic programming algorithm consists of four parts: a recursive defini-
tion of the optimal score; a dynamic programming matrix for remembering opti-
mal scores of subproblems; a bottom-up approach of filling the matrix by solving
the smallest subproblems first; and a traceback of the matrix to recover the struc-
ture of the optimal solution that gave the optimal score. For pairwise alignment,
those steps are the following:

Recursive definition of the optimal alignment score.

There are only three ways the alignment can possibly end: 1) residues xM and
yN are aligned to each other; 2) residue xM is aligned to a gap character, and yN
appeared somewhere earlier in the alignment; or 3) residue yN is aligned to a gap
character and xM appeared earlier in the alignment. The optimal alignment will
be the highest scoring of these three cases.

2
 Crucially, our scoring system allows us to define the score of these three cases
recursively, in terms of optimal alignments of the preceding subsequences. Let
S(i, j) be the score of the optimum alignment of sequence prefix x1 ..xi to prefix
y1 ..yj . The score for case (1) above is the score σ(xM , yN ) for aligning xM to yN ,
plus the score S(M − 1, N − 1) for an optimal alignment of everything else up to
this point. Case (2) is the gap penalty γ plus the score S(M − 1, N ); case (3) is
the gap penalty γ plus the score S(M, N − 1).
This works because the problem breaks into independently optimizable pieces,
since the scoring system is strictly local to one aligned column at a time. That
is, for instance, the optimal alignment of x1 ..xM −1 to y1 ..yN −1 is unaffected by
adding on the aligned residue pair xM , yN , and likewise, the score σ(xM , yN ) we
add on is independent of the previous optimal alignment.
So, to calculate the score of the three cases, we will need to know three more
alignment scores for three smaller problems:
S(M − 1, N − 1), S(M − 1, N ), S(M, N − 1).
And to calculate those, we need the solutions for nine still smaller problems:
S(M − 2, N − 2), S(M − 2, N − 1), S(M − 1, N − 2),
S(M − 2, N − 1), S(M − 2, N ), S(M − 1, N − 1),
S(M − 1, N − 2), S(M − 1, N − 1), S(M, N − 2).
and so on, until we reach tiny alignment subproblems with obvious solutions (the
score S(0, 0) for aligning nothing to nothing is zero).
Thus, we can write a general recursive definition of all our partial optimal
alignment scores S(i, j):


 S(i − 1, j − 1) + σ(xi , yj )

S(i, j) = max  S(i − 1, j) + γ (1)

S(i, j − 1) + γ.

The dynamic programming matrix.

The problem with a purely recursive alignment algorithm may already be ob-
vious, if you looked carefully at that list of nine smaller subproblems we’d be
solving in the second round of the top-down recursion. Some subproblems are
already occurring more than once, and this wastage gets exponentially worse as
we recurse deeper. Clearly, the sensible thing to do is to somehow keep track of
which subproblems we are already working on. This is the key difference between
dynamic programming and simple recursion: a dynamic programming algorithm

3
memorizes the solutions of optimal subproblems in an organized, tabular form (a
dynamic programming matrix), so that each subproblem is solved just once.
For the pairwise sequence alignment algorithm, the optimal scores S(i, j) are
tabulated in a two-dimensional matrix, with i running from 0..M and j running
from 0..N , as showing in Figure 1. As we calculate solutions to subproblems
S(i, j), their optimal alignment scores are stored in the appropriate (i, j) cell of
the matrix.

A bottom-up calculation to get the optimal score.

Once the dynamic matrix programming matrix S(i, j) is laid out – either on a
napkin or in your computer’s memory – it is easy to fill it in in a “bottom-up” way,
from smallest problems to progressively bigger problems. We know the boundary
conditions in the leftmost column and topmost row (S(0, 0) = 0; S(i, 0) = γi;
S(0, j) = γj): for example, the optimum alignment of the first i residues of
sequence x to nothing in sequence y has only one possible solution, which is to
align x1 ..xi to gap characters and pay i gap penalties. Once we’ve initialized the
top row and left column, we can fill in the rest of the matrix by using the recursive
definition of S(i, j) to calculate any cell where we already know the values we
need for the three adjoining cells to the upper left (i − 1, j − 1), above (i − 1, j)
and to the left (i, j − 1). There are several different ways we can do this; one is to
iterate two nested loops, i = 1 to M and j = 1 to N , so we’re filling in the matrix
left to right, top to bottom.

A traceback to get the optimal alignment.

Once we’re done filling in the matrix, the score of the optimal alignment of the
complete sequences is the last score we calculate, S(M, N ). We still don’t know
the optimal alignment itself, though. This, we recover by a recursive “traceback”
of the matrix. We start in cell M, N , determine which of the three cases we used
to get here (by repeating the same three calculations, for example), record that
choice as part of the alignment, and then follow the appropriate path for that case
back into the previous cell on the optimum path. We keep doing that, one cell in
the optimal path at a time, until we reach cell (0, 0), at which point the optimal
alignment is fully reconstructed.

4
Fine. But what do I really need to know?
Dynamic programming is guaranteed to give you a mathematically optimal (high-
est scoring) solution. Whether that corresponds to the biologically correct align-
ment is a problem for your scoring system, not for the algorithm.
Similarly, the dynamic programming algorithm will happily align unrelated
sequences. (The two sequences in Figure 1 might look well-aligned; but in fact,
they are unrelated, randomly generated DNA sequences!) The question of when a
score is statistically significant is also a separate problem, requiring clever statis-
tical theory.
Dynamic programming is surprisingly computationally demanding. You can
see that filling in the matrix takes time proportional to M N . Alignment of two
200-mers will take four times as long as two 100-mers. This is why there is so
much research devoted to finding good, fast approximations to dynamic program-
ming alignment, like the venerable workhorses BLAST and FASTA, and newer
programs like BLAT and FLASH.
Only certain scoring systems are amenable to dynamic programming. The
scoring system has to allow the optimal solution to be broken up into independent
parts, or else it can’t be dealt with recursively. The reason that programs use
simple alignment scoring systems is that we’re striking a reasonable compromise
between biological realism and efficient computation.

Further study.
To study a working example, you can download a small, bare-bones C imple-
mentation of this algorithm from http://blah-blah.blah. I used this C
program to generate Figure 1.

References
[1] R. E. Bellman. Eye of the Hurricane: An Autobiography. World Scientific,
1984.

5
Figure legend
The filled dynamic programming matrix for two randomly generated sequences,
x = TTCATA and y = TGCTCGTA, for a scoring system of +5 for a match, −2
for a mismatch, and −6 for each insertion or deletion. The cells in the optimum
path are shown in red. Arrowheads are “traceback pointers”, indicating which of
the three cases were optimal for reaching each cell. (Some cells can be reached by
two or three different optimal paths of equal score: whenever two or more cases
are equally optimal, dynamic programming implementations usually choose one
case arbitrarily. In this example, though, the optimal path is unique.)

6
dynamic programming matrix:
j (sequence y)
0 1 2 3 4 5 6 7 8 =N
T G C T C G T A
i 0 0 -6 -12 -18 -24 -30 -36 -42 -48

1 T -6 5 -1 -7 -13 -19 -25 -31 -37

2 T -12 -1 3 -3 -2 -8 -14 -20 -26

(sequence x)

3 C -18 -7 -3 8 2 3 -3 -9 -15

4 A -24 -13 -9 2 6 0 1 -5 -4

5 T -30 -19 -15 -4 7 4 -2 6 0

M= 6 A -36 -25 -21 -10 1 5 2 0 11

optimum alignment scores 11:

T - - T C A T A
T G C T C G T A
+5 -6 -6 +5 +5 -2 +5 +5