Scribd
Upload
52 views

1 Intro Dynamic Programming

The document introduces dynamic programming and uses the shortest path problem as an example. It explains that dynamic programming provides a framework to analyze dynamic models more tractably. It defines the cost-to-go and Bellman equations, which characterize the minimum cost path from any node to the final node. The standard algorithm for solving the Bellman equation iteratively updates the cost-to-go function until convergence.

Uploaded by

PAOLA NELLY ROJAS VALERO
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
52 views

1 Intro Dynamic Programming

The document introduces dynamic programming and uses the shortest path problem as an example. It explains that dynamic programming provides a framework to analyze dynamic models more tractably. It defines the cost-to-go and Bellman equations, which characterize the minimum cost path from any node to the final node. The standard algorithm for solving the Bellman equation iteratively updates the cost-to-go function until convergence.

Uploaded by

PAOLA NELLY ROJAS VALERO
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 6

Macroeconomics I

Basic tools: Introduction to Dynamic Programming

Carlos Lizama
March 14, 2022

1
Dynamic Programming

• We will use dynamic programming a lot!

• We aim to analyze dynamic models, so dynamic programming provides a framework to


make our problems more tractable.

• We will look more deeply at dynamic programming in the future. Today, we only cover
basics to get some intuition.

• Let us discuss one classic example: The shortest path problem.

2
The shortest path

• What is the shortest path from A to G?

• Arrows (edges) indicate the movements


we can take.

• Numbers on edges indicate the cost of


traveling that edge.

• Interpretation?

3
Find the least-cost path

• Let J(v) denote the minimum cost-to-go from node v. i.e., the total cost from v if we
take the best route.

• Note that J(G) = 0

• The best path can be solved as follows:


• Start at A.
• From node v, move to any node that solves

min {c(v, w) + J(w)}


w∈Fv

where Fv is the set of nodes that can be reached from v in one step and c(v, w) is the cost
of traveling from v to w.

4
The Bellman Equation

• If we knew the function J, the problem seems pretty simple. How to find J?

• Notice that the function J solves

J(v) = min {c(v, w) + J(w)}


w∈Fv

• This is known as the Bellman Equation.

• The function J is also called the value function, v is the state. The solution

w∗ (v) = arg min {c(v, w) + Jn (w)}


w∈Fv

is called the policy function.

5
Solving for J

• The standard algorithm is to start with a guess J0 . A good guess is J(v) = 0 if v = G


(final node), and J(v) = M otherwise, where M is a large number.

• Then we can update Jn as follows.


1. Set n = 0.
2. Set Jn+1 (v) = {minw∈Fv c(v, w) + Jn (w)}
3. If Jn+1 = Jn , stop. Otherwise n ← n + 1 and go to 2.

• The proof of convergence is omitted.

You might also like