Scribd
Upload
72 views6 pages

1 Intro Dynamic Programming

The document introduces dynamic programming and uses the shortest path problem as an example. It explains that dynamic programming provides a framework to analyze dynamic models more tractably. It defines the cost-to-go and Bellman equations, which characterize the minimum cost path from any node to the final node. The standard algorithm for solving the Bellman equation iteratively updates the cost-to-go function until convergence.

Uploaded by

PAOLA NELLY ROJAS VALERO
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
72 views6 pages

1 Intro Dynamic Programming

The document introduces dynamic programming and uses the shortest path problem as an example. It explains that dynamic programming provides a framework to analyze dynamic models more tractably. It defines the cost-to-go and Bellman equations, which characterize the minimum cost path from any node to the final node. The standard algorithm for solving the Bellman equation iteratively updates the cost-to-go function until convergence.

Uploaded by

PAOLA NELLY ROJAS VALERO
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

Macroeconomics I

Basic tools: Introduction to Dynamic Programming

Carlos Lizama
March 14, 2022

1
Dynamic Programming

• We will use dynamic programming a lot!

• We aim to analyze dynamic models, so dynamic programming provides a framework to


make our problems more tractable.

• We will look more deeply at dynamic programming in the future. Today, we only cover
basics to get some intuition.

• Let us discuss one classic example: The shortest path problem.

2
The shortest path

• What is the shortest path from A to G?

• Arrows (edges) indicate the movements


we can take.

• Numbers on edges indicate the cost of


traveling that edge.

• Interpretation?

3
Find the least-cost path

• Let J(v) denote the minimum cost-to-go from node v. i.e., the total cost from v if we
take the best route.

• Note that J(G) = 0

• The best path can be solved as follows:


• Start at A.
• From node v, move to any node that solves

min {c(v, w) + J(w)}


w∈Fv

where Fv is the set of nodes that can be reached from v in one step and c(v, w) is the cost
of traveling from v to w.

4
The Bellman Equation

• If we knew the function J, the problem seems pretty simple. How to find J?

• Notice that the function J solves

J(v) = min {c(v, w) + J(w)}


w∈Fv

• This is known as the Bellman Equation.

• The function J is also called the value function, v is the state. The solution

w∗ (v) = arg min {c(v, w) + Jn (w)}


w∈Fv

is called the policy function.

5
Solving for J

• The standard algorithm is to start with a guess J0 . A good guess is J(v) = 0 if v = G


(final node), and J(v) = M otherwise, where M is a large number.

• Then we can update Jn as follows.


1. Set n = 0.
2. Set Jn+1 (v) = {minw∈Fv c(v, w) + Jn (w)}
3. If Jn+1 = Jn , stop. Otherwise n ← n + 1 and go to 2.

• The proof of convergence is omitted.

You might also like