Integrated Intelligent Research (IIR) International Journal of Computing Algorithm

Volume: 03 Issue: 03 December 2014 Pages: 277-279

ISSN: 2278-2397

Bellman Equation in Dynamic Programming

K.Vinotha
Associate Professor, Sri ManakulaVinayagar College of Engineering, Madagadipet, Puducherry
E-mail :[email protected]

Abstract - The unifying purpose of this paper to introduces (2)

basic ideas and methods of dynamic programming. It sets out making the change of variable k = s - 1, (2) rewrites
the basic elements of a recursive optimization problem,
describes Bellman's Principle of Optimality, the Bellman
equation, and presents three methods for solving the Bellman
equation with example. or

Key words:Dynamic Programming, Bellmen’s equation,

Contraction Mapping Theorem, Blackwell's Sufficiency
Conditions. Note that, by definition, we have

I. PRELIMINARIES - DYNAMIC PROGRAMMING …

Dynamic programming is a mathematical optimization method. (3)

It refers to simplifying a complicated problem by breaking it Such that (3) rewrites as
down into simpler subproblems in a recursive manner. While
some decision problems cannot be taken apart this way,
decisions that span several points in time do often break apart …(4)
recursively; Bellman called this the "Principle of Optimality".
If subproblems can be nested recursively inside larger This is the socalledBellman equation that lies at the core of the
problems, so that dynamic programming methods are dynamicprogramming theory. With this equation are
applicable, then there is a relation between the value of the associated, in each and everyperiodt, a set of optimal policy
larger problem and the values of the subproblems. In the functions for y and x, which are defined by
optimization literature this relationship is called the Bellman
equation.
…(5)
II. THE BELLMAN EQUATION AND RELATED
THEOREMS Our problem is now to solve (4) for the function . This
problem isparticularly complicated as we are not solving for
A derivation of the Bellman equation just a point that wouldsatisfy the equation, but we are
Let us consider the case of an agent that has to decide on the interested in finding a function that satisfiesthe equation. A
path of a set of control variables, in order to maximize simple procedure to find a solution would be the following
1. Make an initial guess on the form of the value function
the discounted sum of its future payoffs, where xt is
state variables assumed to evolve accordingto 2. Update the guess using the Bellman equation such that
given.We may assume that
the model is markovian, a mathematical framework for
modeling decision making in situations where outcomes are
partly random and partly under the control of a decision maker. 3. If , then a fixed point has been found
The optimal value our agent can derive from this maximization and the problem is solved, if not we go back to 2, and iterate on
process isgiven by the value function the process untilconvergence.
In other words, solving the Bellman equation just amounts to
find the fixedpoint of the bellman equation, or introduction an
(1) operator notation, findingthefixed point of the operator T, such
where D is the set of all feasible decisions for the variables of that
choice. Notethat the value function is a function of the state
variable only, as since themodel is markovian, only the past is
necessary to take decisions, such that allthe path can be whereT stands for the list of operations involved in the
predicted once the state variable is observed. Therefore, computation of theBellman equation. The problem is then that
thevalue in tis only a function of xt(1) may now be rewritten as of the existence and the uniqueness of this fixed-point.
Luckily, mathematicians have provided conditions forthe
existence and uniqueness of a solution.
Existence and uniqueness of a solution
277
Integrated Intelligent Research (IIR) International Journal of Computing Algorithm
Volume: 03 Issue: 03 December 2014 Pages: 277-279
ISSN: 2278-2397

Definition 1 A metric space is a set S, together with a metric we have assume that S is complete, Vn converges to .
, such that for all We now have to show that V = T V in order to complete the
1. , with if and only if , proof of the first part. Note that, for each , and for
2. the triangular inequality implies
3.
Definition 2 A sequence in S converges to , if But since is a Cauchy sequence, we have

for each there exists an integer such that for large enough
n, therefore V = T V .
Hence, we have proven that T possesses a fixed point and
Definition 3 A sequence in S is a Cauchy sequence if therefore have established its existence. We now have to prove
uniqueness. This can be obtained by contradiction. Suppose,
for each there exists an integer such that there exists another function, say W S that satisfies W = T
W . Then, the definition of the fixed point implies

Definition 4 A metric space is complete if every but the contraction property implies
Cauchy sequence inS converges to a point in S.
Definition 5 Let be a metric space and be which, as implies and so V = W . The
function mappingS into itself. T is a contraction mapping (with limit is then unique.
modulus ) if for Proving 2.is straightforward as
, for all .
We then have the following remarkable theorem that but we have such that
establishes the existence and uniqueness of the fixed point of a
contraction mapping.
which completes the proof
Theorem 1 (Contraction Mapping Theorem) If is a This theorem is of great importance as it establishes that any
complete metric space and is a contraction operator that possesses the contraction property will exhibit a
mapping with modulus , then unique fixed-point, which therefore provides some rationale to
the algorithm we were designing in the previous section. It also
1. T has exactly one fixed point such that V = T V ,
insure that whatever the initial condition for thisalgorithm, if
2. for any , with n = 0, the value function satisfies a contraction property, simple
1, 2 … iterations will deliver the solution. It therefore remains to
Since we are endowed with all the tools we need to prove the provide conditions for the value function to be a contraction.
theorem, we shall do it. These are provided by the following theorem.
Proof: In order to prove 1., we shall first prove that if we select Theorem 2(Blackwell's Sufficiency Conditions) Let `

any sequence , such that for each n, and

and B(X) be the space of bounded functions with
Vn+1 = T Vnthis sequence converges and that it converges to
the uniform metric. Let be an operator
. In order to show convergence of , we shall
satisfying
prove that is a Cauchy sequence. First of all, note that 1. (Monotonicity) Let
, then
the contraction property of T implies that
2. (Discounting) There exists some constant such
and therefore
that for all and , we have
Now
consider two terms of the sequence, VmandVn, m > n. The thenT is a contraction with modulus
triangle inequality implies that
Proof: Let us consider two functions satisfying
theref
1. and 2., and such that
ore, making use of the previous result, we have

Since Monotonicity first implies that

as , we have that for each , and discounting

there exists such that . Hence is

since plays the same roles as a . We therefore get
a Cauchy sequence and it therefore converges. Further, since
278
Integrated Intelligent Research (IIR) International Journal of Computing Algorithm
Volume: 03 Issue: 03 December 2014 Pages: 277-279
ISSN: 2278-2397

Likewise, if we now consider that , we end up REFERENCES

[1] Sergio Verdu, H.Vincent Poor, “Abstract Dynamic Programme Models
with under Community Conditions”, Siam J. Control and Optimization,
Vol.25, No.4, July 1987.
Consequently, we have [2] Wikipedia, “Bellman Equation”,
http://en.wikipedia.org/wiki/Bellman_equation
[3] R Bellman, On the Theory of Dynamic Programming, Proceedings of
so that the National Academy of Sciences, 1952
[4] S. Dreyfus (2002), 'Richard Bellman on the birth of dynamic
programming' Operations Research 50 (1), pp. 48–51.
Which defines a contraction. This completes the proof.
This theorem is extremely useful as it gives us simple tools to
check whether a problem is a contraction and therefore permits
to check whether the simple algorithm we were defined is
appropriate for the problem we have in hand.
As an example, let us consider the optimal growth model, for
which the Bellman equation writes

withkt+1 = F (kt) - ct. In order to save on notations, let us drop

the time subscript and denote the next period capital stock by
, such that the Bellman equation rewrites, plugging the law
of motion of capital in the utility function

Let us now define the operator T as

We would like to know if T is a contraction and therefore if

there exists a unique function V such that
V (k) = (T V )(k)
In order to achieve this task, we just have to check whether T is
monotonic and satisfies the discounting property.
Monotonicity: Let us consider two candidate value functions, V
and W , such that for all . What we
want to show is that . In order to do that, let
us denote by the optimal next period capital stock, that is

But now, since for all , we have

such that it should be clear that

Hence we
have shown that implies and
therefore established monotonicity.

III. DISCOUNTING: LET US CONSIDER A

CANDIDATE VALUE FUNCTION, V , AND A
POSITIVE CONSTANT A.

=
Therefore, the Bellman equation satisfies discounting in the
case of optimal growth model.
Hence, the optimal growth model satisfies the Blackwell's
sufficient conditions for a contraction mapping, and therefore
the value function exists and is unique.

279