Introduction To Optimization
Introduction To Optimization
Optimization
R. Clark Robinson
Copyright
c 2013 by R. Clark Robinson
Department of Mathematics
Northwestern University
Evanston Illinois 60208 USA
Contents
Preface v
Chapter 1. Linear Programming 1
1.1. Basic Problem 1
1.2. Graphical Solution 2
1.3. Simplex Method 5
1.3.1. Slack Variables 5
1.3.2. Simplex Method for Resource Requirements 7
1.3.3. General Constraints 10
1.4. Duality 17
1.4.1. Duality for Non-standard Linear Programs 19
1.4.2. Duality Theorems 21
1.5. Sensitivity Analysis 27
1.5.1. Changes in Objective Function Coefficients 29
1.6. Theory for Simplex Method 31
Chapter 2. Unconstrained Extrema 39
2.1. Mathematical Background 39
2.1.1. Types of Subsets of Rn 39
2.1.2. Continuous Functions 41
2.1.3. Existence of Extrema 43
2.1.4. Differentiation in Multi-Dimensions 44
2.1.5. Second Derivative and Taylor’s Theorem 47
2.1.6. Quadratic Forms 50
2.2. Derivative Conditions 55
2.2.1. First Derivative Conditions 56
2.2.2. Second Derivative Conditions 56
Chapter 3. Constrained Extrema 61
3.1. Implicit Function Theorem 61
3.2. Extrema with Equality Constraints 69
3.2.1. Interpretation of Lagrange Multipliers 73
iii
iv Contents
v
Chapter 1
Linear Programming
1
2 1. Linear Programming
Notations
We denote the transpose of a matrix (or a vector) A by A> .
For two vectors v and w in Rn , v ≥ w means that vi ≥ wi for all 1 ≤ i ≤ n. In
the same way, v w means that vi > wi for all 1 ≤ i ≤ n. We also consider two positive
“quadrants” of Rn ,
Rn+ = { x ∈ Rn : x ≥ 0 } = { x ∈ Rn : xi ≥ 0 for 1 ≤ i ≤ n } and
Rn++ = { x ∈ Rn : x 0 } = { x ∈ Rn : xi > 0 for 1 ≤ i ≤ n }.
Definition. The rank of an m × n matrix A is used at various times in this course. The rank
of A, rank(A), is the dimension of the column space of A, i.e., the largest number of linearly
independent columns of A. This integer is the same as number of pivots in the row reduced
echelon form of A. (See Lay [9].)
It is possible to determine the rank by determinants of submatrices: rank(A) = k iff
k ≥ 0 is the largest integer with det(Ak ) 6= 0, where Ak is any k × k submatrix of A
formed by selecting k columns and k rows. To see this is equivalent, let A0 be submatrix of
A with k columns that are linearly independent and span column space, so rank(A0 ) = k .
The dimension of the row space of A0 is k , so there are k rows of A0 forming the k × k
submatrix Ak with rank(Ak ) = k , so det(Ak ) 6= 0. The submatrix Ak is also a submatrix
of pivot columns and rows. Note in linear algebra, the pivot columns are usually chosen with
the matrix in echelon form with zeroes in the row to the left of a pivot positions. We often
choose other columns as will be since in the examples given in the rest of the chapter.
I3
2
1.6
(0.8, 1.2)
(1, 1)
I2
I1
(0, 0) 1.5 2 3.2
The values of the objective function at the vertices are f (0, 0) = 0, f (1.5, 0) = 12,
f (1, 1) = 14, f (0.8, 1.2) = 13.6, and f (0, 1.6) = 9.6. Therefore, the maximizer on F is
(1, 1) and the maximal value is 14.
Definition. A resource constraint is one of the form ai1 x1 + · · · + ain xn ≤ bi with bi ≥ 0.
The name refers to the upper limit on the combination of the variables.
A standard maximization linear programming problem has only resource constraints:
Maximize: f (x) = c · x = c1 x1 + · · · + cn xn ,
Subject to: a11 x1 + · · · + a1n xn ≤ b1 ,
.. .. ..
. . .
am1 x1 + · · · + amn xn ≤ bm ,
xj ≥ 0 for 1 ≤ j ≤ n.
The data specifying the problem includes an m × n matrix A = (aij ), c = (c1 , . . . , cn )> ∈
Rn , and b = (b1 , . . . , bm )> ∈ Rm+ with all the bi ≥ 0.
Using matrix notation, the constraints can be written as Ax ≤ b and x ≥ 0. The feasible
set is the set F = { x ∈ Rn+ : Ax ≤ b }.
Since the objective is to maximize f (x), the function f (x) is called the objective function.
A nonstandard linear program allows other types of inequalities for the constraints. See the
exercises of this section and Section 1.3.3 for examples of these other types of constraints.
(0, 4)
(1, 2)
(2, 1)
(4, 0)
Figure 1.2.2. Feasible set for the minimization example
The feasible set that is shaded in Figure 1.2.2 is unbounded but f (x) ≥ 0 is bounded
below, so f (x) has a minimum. The vertices are (4, 0), (2, 1), (1, 2), and (0, 4), with values
4 1. Linear Programming
f (4, 0) = 12, f (2, 1) = 8, f (1, 2) = 7, and f (0, 4) = 12. Therefore, the minimum value
is 7 which is attained at (1, 2).
Graphical Solution Method for a Linear Programming Problem
1. Determine or draw the feasible set F . If F = ∅, then the problem has no
optimal solution, and it is said to be infeasible.
2. The problem is called unbounded and has no optimal solution provided that
F 6= ∅ and the objective function on F has
a. arbitrarily large positive values for a maximization problem, or
b. arbitrarily large negative values for a minimization problem.
3. A problem is called bounded provided that it is neither infeasible nor un-
bounded. In this case, an optimal solution exists. Determine all the vertices of
F and values of the objective function at the vertices. Choose the vertex of
F producing the maximal or minimal value of the objective function.
1.2. Exercises
1.2.1. Consider the following maximization linear programming problem.
Maximize 4x + 5y
Subject to: x + 2y ≤ 12
x+y ≤7
3x + 2y ≤ 18
0 ≤ x, 0 ≤ y .
a. Sketch the feasible set.
b. Why must a maximum exist?
c. Find the maximal feasible solution and the maximal value using the geometric
method.
1.2.2. Consider the following linear programming problem:
Minimize: 3x + 2y
Subject to: 2x + y ≥ 4
x+y ≥3
x ≥ 0, y ≥ 0.
1. Sketch the feasible set.
2. Why must a minimum exist?
3. Find the optimal feasible solution and the optimal value using the geometric
method.
1.2.3. Consider the following minimization linear programming problem.
Minimize 3x + 2y
Subject to: 2x + y ≥ 4
x+y ≥3
3x + 2y ≤ 18
0 ≤ x, 0 ≤ y .
a. Sketch the feasible set.
b. Why must a minimum exist?
c. Find the minimal feasible solution and the minimal value using the geometric
method.
1.3. Simplex Method 5
1.2.4. Show graphically that the following linear program does not have a unique solution:
Maximize 30x + 40y
Subject to: 3x + 4y ≤ 48
x + y ≤ 14
0 ≤ x, 0 ≤ y .
1.2.5. Consider the following maximization linear programming problem.
Maximize 4x + 5y
Subject to: x + 2y ≥ 0
2x − y ≥ 3
0 ≤ x, 0 ≤ y .
a. Sketch the feasible set.
b. Explain why the problem is unbounded and a maximizer does not exists.
The last three columns of the coefficient matrix for the slack variables can be considered as
pivot columns and are linearly independent. Thus, the rank of the matrix is 3 and a solution to
the nonhomogeneous system has two free variables. The starting solution p0 = (0, 0, 2, 16, 3)
in the feasible set is formed by setting the free variables x1 and x2 equal to zero and solving for
the values of the three slack variables. The value is f (p0 ) = 0.
We proceed to change pivots to make a different pair the free variables equal to zero and a
different triple of positive variables, in a process called “pivoting”.
s3
p3
p2
s2
s1
p0 p1
If we leave the vertex (x1 , x2 ) = (0, 0), or p0 = (0, 0, 2, 16, 3), making x1 > 0 the
entering variable while keeping x2 = 0, the first slack variable to become zero is s3 when
x1 = 1.5. We have arrived at the vertex (x1 , x2 ) = (1.5, 0) of the feasible set by moving
along one edge. Therefore we have a new solution with two zero variables and three nonzero
variables, p1 = (1.5, 0, 0.5, 8.5, 0) with f (p1 ) = 8(1.5) = 12 > 0 = f (p0 ). The point p1
is a better feasible solution than p0 . At this vertex, all of input one is used, s3 = 0.
We repeat leaving p1 by making x2 > 0 the entering variable while keeping s3 = 0;
the first other variable to become zero is s1 . We have arrived at p2 = (1, 1, 0, 1, 0) with
f (p2 ) = 80(50) + 60(50) = 14 > 12 = f (p1 ). The point p2 is a better feasible solution
than p1 . In the x-plane, we leave (x1 , x2 ) = (1.5, 0) and to arrive at (x1 , x2 ) = (1, 1). At
this point, all of inputs one and three are used, s1 = 0 and s3 = 0, and s2 = 1 units of input
two remains.
If we were to leave p2 by making s3 > 0 the entering variable while keeping s1 = 0,
the first variable to become zero is s2 , and we arrive at the point p3 = (0.8, 1.2, 0, 0, 0.2).
The objective function f (p3 ) = 8(0.8) + 6(1.2) = 13.6 < 14 = f (p2 ), and p3 is a worse
feasible solution than p2 .
Let z ∈ F r {p2 }, v = z − p2 , and vj = pj − p2 for j = 1, 3. Then
f (vj ) = f (pj ) − f (p2 ) < 0 for j = 1, 3.
2
The vectors { v1 , v3 } form a basis of R , so v = y1 v1 + y3 v3 for some yj . The vector v
points into the feasible set so y1 , y3 ≥ 0 and either y1 > 0 or y2 > 0. Therefore,
f (z) = f (p2 ) + f (v) = f (p2 ) + y1 f (v1 ) + y3 f (v3 ) < f (p2 ).
Since f cannot increase by moving along either edge going out from p2 , f (p2 ) is the max-
imum among the values at all the points of the feasible set and p2 is an optimal feasible
solution.
In this example, the points p0 = (0, 0, 2, 16, 3), p1 = (1.5, 0, 0.5, 8.5, 0), p2 =
(1, 1, 0, 1, 0), p3 = (0.8, 1.2, 0, 0, 10), and p4 = (0, 1.6, 0.4, 0, 1.4) are called basic
solutions since at most 3 variables are positive, where 3 is the rank of the coefficient matrix for
the constraint equations.
Definition. Consider a standard maximization linear programming problem with only resource
constraints: Maximize f (x) = c · x subject to Ax ≤ b and x ≥ 0 for c = (c1 , . . . , cn )>
in Rn , A = (aij ) an m × n matrix, and b = (b1 , . . . , bm )> with bi ≥ 0 for 1 ≤ i ≤ m.
The corresponding standard maximization linear programming problem with slack variables
s1 ,. . . , sm added, called slack-variable form, is the following:
1.3. Simplex Method 7
Maximize: f (x) = c · x
Subject to: a11 x1 + · · · + a1n xn + s1 = b1 ,
.. .. ..
. . .
am1 x1 + · · · + amn xn + sm = bm ,
xj ≥ 0 for 1 ≤ j ≤ n, and
si ≥ 0 for 1 ≤ i ≤ m.
Using matrix notation with I the m × m identity matrix, the problem is to maximize f (x) =
c · x subject
" # to
x
[ A, I ] = b with x ≥ 0 and s ≥ 0.
s
When solving this matrix equation by row reduction, we indicate the partition of the aug-
mented matrix into the columns of (i) the original variables, (ii) the slack variables, and (iii)
the constant vector b by inserting extra vertical lines between A and I and between I and b,
[ A | I | b ].
Example 1.4 (Linear Algebra Solution of the Production Linear Program). The augmented
matrix forthe original system is
x1 x2 s1 s2 s3
1 1 1 0 0 2
with free variables x1 and x2 and basic variables
5 10 0 1 0 16
2 1 0 0 1 3
s1 , s2 , and s3 .
If we make x1 > 0 while keeping x2 = 0, x1 becomes a new basic variable, called the
entering variable.
8 1. Linear Programming
2 1 0 0 1 3 1 .5 0 0 .5 1.5
Setting the free variables x2 and s3 equal zero, gives p1 = (75, 0, 25, 425, 0)> as the new
basic solution. The entries in the furthest right (augmented) column give the values of the basic
variables and are all positive.
In order to keep track of the value of the objective function (or variable), f = 80x1 +60x2 ,
or −80x1 − 60x2 + f = 0, we include in the the augmented matrix a row for this equation
and a column for the variable f ,
x1 x2 s1 s2 s3 f
1 1 1 0 0 0 2
5 10 0 1 0 0 16
2 1 0 0 1 0 3
8 6 0 0 0 1 0
The entry in the column for f is one in the last objective function row and zero elsewhere. The
entries for the xi in this objective function row often start out negative as is the case in this
example.
Performing the row reduction on this this further augmented matrix, we get
x1 x2 s1 s2 s3 f x1 x2 s1 s2 s3 f
1 1 1 0 0 0 2 0 .5 1 0 .5 0 0.5
5 10 0 1 0 0 16 ∼ 0 7.5 0 1 2.5 0 8.5 .
2 1 0 0 1 0 3 1 .5 0 0 .5 0 1.5
8 6 0 0 0 1 0 0 2 0 0 4 1 12
Since f is the pivot for the objective function row, the bottom right entry gives that the new
value f = 12 for x2 = s3 = 0, x1 = 1.5 > 0, s1 = 0.5 > 0, and s2 = 8.5 > 0.
If we pivot back to make s3 > 0, the value of f becomes smaller, so we select x2 as the
next entering variable, keeping s3 = 0.
(i) x1 becomes zero when x2 = 1.5 0.5
= 3,
(ii) s1 becomes zero when x2 = 0.5 0.5
= 1, and
(iii) s2 becomes zero when x2 = 8.5 7.5
≈ 1.13 .
Since the smallest positive value of x1 comes from s1 , s1 is the departing variable and we make
the entry in the first row second column the new pivot.
x1 x2 s1 s2 s3 f x1 x2 s1 s2 s3 f
0 .5 1 0 .5 0 0.5 0 1 2 0 1 0 1
0 7.5 0 1 2.5 0 8.5 ∼ 0 0 15 1 5 0 1 .
1 .5 0 0 .5 0 1.5 1 0 1 0 1 0 1
0 2 0 0 4 1 12 0 0 4 0 2 1 14
1.3. Simplex Method 9
In the pivoting process used in the example, the variable f remains the pivot variable for
the objective function row. This explains why the value in the bottom right of the augment
matrix is the value of the objective function at every step. However, since it does not play a role
in the row reduction, we drop this column for the variable f from the augmented matrices.
Definition. The augmented matrix with a row augmented for the objective function but without
a column for the objective function variable is called the tableau. The row in the tableau for the
objective function is called the objective function row.
x1 x2 s1 s2 s3 r1 r2 r3
7 4 7 4
0 0 1 1 16
5 5 5 5
0 1 1 0 1 1 0 1 25
∼
1 2 1 2
1 0 5
0 5 5
0 5
2
0 0 4 0 1 4 0 1 205
0 0 0 0 0 1 1 1 0
After these three steps, the artificial variables are no longer pivots and are zero, so they
can be dropped. The values (x1 , x2 , s1 , s2 , s3 ) = (2, 25, 0, 16, 0) form an initial feasible basic
solution. Finally, pivoting on a15 yields the optimal solution:
x1 x2 s1 s2 s3
x1 x2 s1 s2 s3
7 5
0 0 7
1 4
16
0 0 4 4
1 20
5 5
3 5
0 1 0 5
0 1 1 0 1 25 ∼ 4 4 .
1 2 1 1
1 0 0 2 1 0 0 10
5 5
2 2
0 0 4 0 1 205 0 0 9 5
0 185
4 4
At this point, all the entries in the objective function are positive, so this is the optimal solution:
(x1 , x2 , s1 , s2 , s3 ) = (10, 5, 0, 0, 20) with an value of 185 for f .
For the original problem, the minimal solution has a value of 185.
1.3. Exercises
1.3.1. Solve the following problem (a) graphically and (b) by the simplex method:
Maximize: 3 x1 + 2 x2
Subject to: x1 + 2 x2 ≤ 70
x1 + x2 ≤ 40
3 x1 + x2 ≤ 90
x1 ≥ 0, x2 ≥ 0.
1.3.5. Solve the linear program by the simplex algorithm with artificial variables:
Maximize: x1 + 2x2 + 3x3
Subject to: x1 + 2x2 + x3 = 36
2x1 + x2 + 4x3 ≥ 12
0 ≤ x1 , 0 ≤ x2 , 0 ≤ x3 .
1.3.6. Solve the linear program by the simplex algorithm with artificial variables:
Maximize: 4x1 + 5x2 + 3x3
Subject to: x1 + 2x2 + 3x3 ≤ 20
x1 + 2x2 ≥ 2
x1 − x2 + x3 = 7
x1 , x 2 , x 3 ≥ 0.
16 1. Linear Programming
1.3.7. Solve the linear program by the simplex algorithm with artificial variables:
Maximize: 4x1 + 5x2 + 3x3
Subject to: x1 + 2x2 + 3x3 ≤ 20
x1 + 2x2 ≥ 2
x1 − x2 + x3 = 7
x1 , x2 , x3 ≥ 0.
1.3.8. Solve the linear program by the simplex algorithm with artificial variables:
Maximize: −x1 − 2 x2 + 2x3
Subject to: x1 + 2 x3 ≤ 12
2 x1 + 3 x2 + x3 ≥ 4
x1 + 2 x2 − x3 ≥ 6
x1 , x2 , x3 ≥ 0.
1.3.9. Solve the linear program by the simplex algorithm with artificial variables:
Maximize: x1 + x2 + x3
Subject to: x1 + x2 ≥ 3
x1 + 2 x2 + x3 ≥ 4
2 x1 + x2 + x3 ≤ 2
x1 , x 2 , x 3 ≥ 0.
1.3.10. Use artificial variables to determine whether there are any vectors satisfying
x1 + x2 + ≤ 40
2 x1 + x2 ≥ 70
x1 + 3 x2 ≤ 90
x1 , x2 , x3 ≥ 0.
1.3.11. Show that the following problem is unbounded.
Maximize 2x + y
Subject to: x − y ≤ 3
−3x + y ≤ 1
0 ≤ x, 0 ≤ y .
1.3.12. Consider the following maximization linear program:
Maximize 2 x1 − 5 x2
Subject to: x1 + 2 x2 ≤ 2
x1 − 4 x2 ≥ 4
0 ≤ x1 , 0 ≤ x2 .
a. Use the simplex method with artificial variables to show that there are no fea-
sible solutions.
b. Plot the constraints and argue why the feasible set is empty.
1.4. Duality 17
1.4. Duality
A linear program can be associated with a two-person zero-sum game. A Nash equilibrium in
mixed strategies for this game gives an optimal solution to not only the original game but also
to an associated dual linear programming problem. If the original problem is a maximization
problem, then the dual problem is a minimization problem and vice versa. We do not discuss
this zero-sum game, but introduce this dual minimization problem using an example.
Example 1.8. Consider the production maximization linear programming problem that was
given previously:
MLP: Maximize: z = 8x1 + 6x2 (profit)
subject to: x1 + x2 ≤ 2, (I 1 )
5x1 + 10x2 ≤ 16, ( I2 )
2x1 + x2 ≤ 3, (I3 ),
x1 , x2 ≥ 0
Assume that excess inputs can be sold and shortfalls can be purchased for prices of y1 , y2 ,
and y3 , each of which is nonnegative, yj ≥ 0 for j = 1, 2, 3. The prices yj are called shadow
prices or imputed values of the resources or inputs and are set by the market (the potential for
competing firms). With either purchase or sale of the input resource allowed, the profit for the
firm is
0 = (2 − x1 − x2 )y1 ,
0 = (16 − 5x1 − 10x2 )y2 , and (1)
0 = (3 − 2x1 − x2 )y3 .
For such shadow prices of inputs, the firm’s profit is P = 8x1 + 6x2 and the situation from
the firm’s perspective yields the original maximization problem.
For a feasible choice of (x1 , x2 ), the coefficient of each yj in P is nonnegative, 2 − x1 −
x2 ≥ 0, 16 − 5x1 − 10x2 ≥ 0, 3 − 2x1 − x2 ≥ 0, and the competitive market is forcing to
zero the shadow price of any resource with a surplus; therefore, the market is minimizing P as
a function of (y1 , y2 , y3 ).
Regrouping the profit function P from the markets perspective yields
Example 1.9. A manufacturing company produces x1 amount of the regular type and x2
amount of the super type with a profit of P = 12x1 + 15x2 . The assembly time is 20 minutes
per regular unit and 30 minutes per super unit with a total of 2400 minutes available, so 2 x1 +
3 x2 ≤ 240. The painting time is 15 minutes per regular unit and 40 minutes per super unit with
a total of 3000 minutes available, so 15 x1 + 40 x2 ≤ 3000 or 3 x1 + 8 x2 ≤ 600. Finally, the
inspection time is 12 minutes per unit of each type with 1200 minutes available. Thus, we have
the following maximization problem.
1.4. Duality 19
x1 x2 s1 s2 s3 x1 x2 s1 s2 s3
2 3 1 0 0 240
0 1 1 0 2 40
3 8 0 1 0 600 ∼ 0 5 0 1 3 300
1 1 0 0 1 100 1 1 0 0 1 100
12 15 0 0 0 0 0 3 0 0 12 1200
x1 x2 s1 s2 s3
x1 x2 s1 s2 s3
0 1 1 0 2 40 0 1 1 0 2 40
0 0 5 1 7 100
0 5 0 1 3 300 ∼ .
1 1 0 0 1 100
1 0 1 0 3 60
0 3 0 0 12 1200 0 0 3 0 6 1320
The optimal solutions has x1 = 60 regular type, x2 = 40 super type, with a profit of $1320.
The optimal values of the yi can be read off in the columns of the slack variables in the
objective function row: y1 = 3 profit per 10 minutes of assembly time or $0.30 per minute,
y2 = 0 profit per 5 minutes of painting time, and y3 = 2 profit per 12 minutes of inspection
or $0.16 per minute . The values of the yi correspond to the marginal value of each additional
unit of assembly, painting, and inspection time. Additional units of the exhausted resources,
assembly and inspection times, contribute to the profit but not from painting time, which is
slack.
Note that the standard conditions for the MLP, Pj aij xj ≤ bi or 0 ≤ xj , corresponds
P
Example 1.10.
Minimize : 10 y1 + 4 y2 + 8 y3
Subject to : 2 y1 − 3 y2 + 4 y3 ≥ 2
3 y1 + 5 y2 + 2 y3 ≤ 15
2 y1 + 4 y2 + 6 y3 = 4
y1 ≥ 0, y2 ≥ 0, y3 ≤ 0
Using the table on the relations starting from the minimization, we get the dual maximization
problem as follows:
By making the change of variables x2 = z2 and x3 = z3 − v3 , all the variables are now
restricted to be greater than or equal to zero. Summarizing the new maximization problem:
Maximize : 2 x1 − 15 z2 + 4 z3 − 4 v3
Subject to : 2 x1 − 3 z2 + 2 z3 − 2 v3 ≤ 10
3 x1 − 5 z2 + 4 z3 − 4 v3 ≤ 4
4 x 1 − 2 z2 + 6 z3 − 6 v 3 ≥ 8
x1 ≥ 0, z2 ≥ 0, z3 ≥ 0, v3 ≥ 0.
The tableau for the maximization problem with variables x1 , z2 , z3 , v3 , with slack variables
s1 and s2 , and with surplus and artificial variables s3 and r3 is
1.4. Duality 21
x 1 z2 z3 v3 s1 s2 s3 r3
2 3 2 2 1 0 0 0 10
3 5 4 4 0 1 0 0 4
4 2 6 6 0 0 1 1 8
2 15 4 4 0 0 0 0 0
4 2 6 6 0 0 1 0 8
x1 z2 z3 v3 s1 s2 s3 r3
1 1
0 2 1 1 1 0 6
2 2
13 17 17 3 3
0 2 2 2
0 1 4 4
10
∼ .
1 3 3 1 1
1 2 2 2
0 0 4 4
2
1 1
0 14 1 1 0 0 2 2
4
0 0 0 0 0 0 0 1 0
The artificial objective function is no longer needed but we keep the artificial variable to deter-
mine the value of its dual variable.
x1 z2 z3 v3 s1 s2 s3 r3
47 2 7 7 122
0 0 0 1 17
17 17 17 17
13 2 3 3 20
∼ 0 17 1 1 0 17
34 34 17
11 3 2 2 4
1 0 0 0 17
17 17 17 17
225 2 10 10 88
0 17
0 0 0 17 17 17 17
4
The solution of the maximization problem is x1 = 17 , x2 = z2 = 0, x3 = z3 − v3 =
20 20 122 88
17
− 0 = 17
, s1 = 17
, and s 2 = s3 = r3 = 0, with a maximal value of 17 .
According to Theorem 1.16, the optimal solution for the original minimization problem
2
can be also read off from the slack and artificial columns of the final tableau, y1 = 0, y2 = 17 ,
10 2 10 88
y3 = 17 , and optimal value 10 (0) + 4 17 + 8 17 = 17 .
Alternatively, we could first write the minimization problem in standard form by setting
y3 = u3 ,
Minimize : 10y1 + 4y2 − 8y3
Subject to : 2y1 − 3y2 − 4u3 ≥ 2
3y1 + 5y2 − 2u3 ≤ 15
2y1 + 4y2 − 6u3 = 4
y1 ≥ 0, y2 ≥ 0, u3 ≥ 0.
The dual maximization problem will now have a different tableau than before but the same
solution for the original problem.
Maximize: f (x) = c · x
j aij xj ≤ bi , ≥ bi , or = bi for 1 ≤ i ≤ m and
P
Subject to:
xj ≥ 0, ≤ 0, or unrestricted for 1 ≤ j ≤ n.
The (dual) minimization linear programming problem mLP with feasible set F m is
Minimize: g(y) = b · y
i aij yi ≥ cj , ≤ cj , or = cj for 1 ≤ j ≤ n and
P
Subject to:
yi ≥ 0, ≤ 0, or unrestricted for 1 ≤ i ≤ m.
Theorem 1.11 (Weak Duality Theorem). Assume that x ∈ F M is a feasible solution for a
primal maximization linear programming problem MLP and y ∈ F m is a feasible solution
for its dual minimization linear programming problem mLP.
a. Then c · x ≤ b · y. Thus, the optimal value M to either problem must satisfy
c · x ≤ M ≤ y · b.
b. Further, c · x = b · y iff
0 = y · (b − Ax) and
0 = x · A> y − c .
Remark. The equations of part (b) of the theorem are known as complementary slackness.
They imply that for 1 ≤ j ≤ m either
yj = 0 or 0 = bj − aj1 x1 − · · · − ajn xn ;
similarly for 1 ≤ i ≤ n, either
xi = 0 or 0 = a1i y1 + · · · + ami ym − ci .
In the material on nonlinear equations, we have a similar result that gives the necessary
conditions for a maximum, called the Karush-Kuhn-Tucker equations. We usually solve linear
programming problems by the simplex method, i.e., by row reduction. For nonlinear program-
ming with inequalities, we often solve them using the Karush-Kuhn-Tucker equations.
j aij xj ≤ bi then yi ≥ 0, so
P
Proof. (1) If
X
yi (Ax)i = yi aij xj ≤ yi bi .
j
y> (b − Ax) ≥ 0.
(2) By same type of argument as (1),
>
c · x ≤ x · A> y = A> y x = y> (Ax) = y · Ax or
>
A> y − c x ≥ 0.
Combining gives part (a),
c · x ≤ y · Ax ≤ y · b.
1.4. Duality 23
Also,
>
y · b − c · x = y> (b − Ax) + A> y − c x=0 iff
0 = A> y − c · x and 0 = y · (b − Ax) .
This proves part (b).
Corollary 1.12 (Feasibility/Boundedness). Assume that MLP and mLP both have feasible
solutions. Then, MLP is bounded above and has an optimal solution. Also, mLP is bounded
below and has an optimal solution.
Proof. If mLP has a feasible solution y0 ∈ F m , then for any feasible solution x ∈ F M of
MLP, f (x) = c · x ≤ b · y0 so f is bounded above and has an optimal solution.
Similarly, if MLP has a feasible solution x0 ∈ F M , then for any feasible solution y ∈
F m of mLP, g(y) = b · y ≥ c · x0 so g is bounded below and has an optimal solution.
Proposition 1.13 (Necessary Conditions). If x̄ is an optimal solution for MLP, then there is a
feasible solution ȳ ∈ F m of the dual mLP that satisfies complementary slackness equations,
0 = ȳ · (b − Ax̄) and
0 = A> ȳ − c · x̄.
Similarly, if ȳ is an optimal solution for mLP, then there is a feasible solution x̄ ∈ F M
of the Dual MLP that that satisfies complementary slackness equations,
Proof. Let E be the set of i such that bi ≥ j aij x̄j is tight or effective. The ith -row of A,
P
R> 0
i , is the gradient of this constraint. Let E the set of i such that xi = 0, so this constraint is
tight. The gradient of this constraint is the standard unit vector ei = (0, . . . , 1, . . . 0)> , with a
1 only in the ith -coordinate.
We assume that this solution is nondegenerate so that the gradients of the constraints
{R> i
i }i∈E ∪ { e }i∈E0 = {wj }j∈E00 are linearly independent. (Otherwise we have to take
an appropriate subset in the following argument.)
The objective function f has a local maximum on the level set for the tight constraints
at x̄. By Lagrange multipliers, ∇(f ) = c is a linear combination of the gradients of the tight
constraints, X X
c= ȳi R>i − z̄i ei .
i∈E i∈E0
By setting ȳi = 0 for i ∈
/ E and z̄i = 0 for i ∈/ E0 , the sum can be extended to all the
appropriate i, X X
c= ȳi R>
i − z̄i ei = A> ȳ − z̄. (*)
1≤i≤m 1≤i≤n
Since ȳi = 0 for bi − j aij x̄j 6= 0 and z̄i = 0 for xi 6= 0,
P
X
0 = y¯i bi − aij x̄j for 1 ≤ i ≤ m and
j
To finish the proof, we need to show that (i) ȳi ≥ 0 for a resource constraint (≤ bi ), (ii)
ȳi ≤ 0 for a requirement constraint (≥ bi ), (iii) ȳi is unrestricted for an equality constraint
− ≥ ≥ i − cj ≤ 0 for xj ≤ 0,
P P
(= bi ), (iv) z̄ j = i a ij ȳ i cj 0 for x j 0 , (iv) z̄ j = i a ij ȳ
(iv) z̄j = i aij ȳi − cj = 0 for xj unrestricted,
P
Theorem 1.15 (Duality Theorem). Consider dual problems MLP and mLP. Then, MLP has
an optimal solution iff the dual mLP has an optimal solution.
Proof. If MLP has an optimal solution x̄, then mLP has a feasible solution ȳ that satisfies
the complementary slackness equations by the necessary conditions of Proposition 1.13. By
Corollary 1.14, ȳ is an optimal solution of mLP.
c> 0 0 0 0 0
" #
M1 A1 + M2 A2 + M3 A3 M1 M2 M2 M3 Mb
= .
ȳ> > >
1 A1 + ȳ2 A2 + ȳ3 A3 − c
>
ȳ>
1 ȳ>
2 ȳ>
2 ȳ>
3 ȳ> b
In the optimal tableau, the entries in the objective function row are nonnegative (except for
artificial variable columns), so ȳ> A − c> ≥ 0, ȳ> >
1 ≥ 0, ȳ2 ≤ 0. Thus, ȳ is a feasible
solution of the mLP. If x̄ is a maximizer for the MLP, the value ȳ> b = b · ȳ in the lower
right position of the optimal tableau is the optimal value of the MLP, so equals c · x̄. By the
Optimality Corollary 1.14, ȳ is an optimal solution of mLP.
Note that A> ȳ i = Li · ȳ where Li is the ith -column of A.
Li · ȳ ≥ ci and
Li · ȳ ≤ ci , or
Li · ȳ = ci ,
1.4. Exercises
1.4.1. Determine the dual of the linear program:
Minimize: 4y1 + 3y2 + 8y3
Subject to: y1 + y2 + y3 ≥ 12
5y1 − 2y2 + 4y3 ≤ 20
2y1 + 3y2 − y3 = 12
0 ≤ y1 , 0 ≤ y2 , y3 unrestricted.
1.4.4. Use the dual problem MLP to discover the nature of the solution to the following
minimization linear programming problem, mLP:
Minimize: −y1 + 2 y2
Subject to: −5y1 + y2 ≥ 2
4y1 − y2 ≥ 3
y1 ≥ 0, y2 ≥ 0.
1.5. Sensitivity Analysis 27
x1 x2 s1 s2 s3
1 3
0 1 0 60
7 14
1 1
1 0 0 28
∼
35 7 .
22 9
0 0 1 36
35 14
2 25
0 0 7 7
0 1720
The optimal solution is
x1 = 28, x2 = 60, s1 = 0, s2 = 0, s3 = 36,
2
with optimal value of 1720. The values of an increase of the constrained quantities are y1 = 7
and y2 = 25
7
, and y3 = 0 for the quantity that is not tight.
Resource = 344
Resource = 400
Resource = 680
(20,72)
(28,60)
(40,0) (68,0)
We begin discussing the affect of its change to the limitation on fasteners, b2 + δ2 , while
keeping the same basic variables and free variables s1 = s2 = 0. The starting form of the
constraint is
10x1 + 2x2 + s2 = 400 + δ2 .
The slack variable s2 and δ2 play similar roles (and have similar units). So in the new optimal
tableau, δ2 times the s2 -column is added to the right hand column, the column for the constraint
28 1. Linear Programming
a. How much can each constant bi increase and decrease, bi + δi , and keep the
same set of basic variables in the optimal solution?
What is the change in the optimal value of the objective function with an al-
lowable change δi in each bi ?
What is the marginal value per additional unit of a small amount of each re-
source?
b. Determine the range of values of the objective function coefficient of x1 such
that the optimal basis remains unchanged?
1.5.2. A farmer can grow x1 acres of corn and x2 acres of potatoes. The linear program
to maximize the profit is the following:
Maximize: 50 x1 + 40 x2 Profit
Subject to: x1 + x2 ≤ 50 Acres of land
3 x1 + 2 x2 ≤ 120 Days of labor
10 x1 + 60 x2 ≤ 1200 Dollars of capital
20 x1 + 10 x2 ≤ 800 Pounds of fertilizer
0 ≤ x1 , 0 ≤ x2 .
The optimal tableau is the following:
x1 x2 s 1 s2 s3 s4
5 1
0 0 1 0 5
16 160
0 1 3
1 0 16 160
0 15
0 55 1
0 0 8 16
1 50
3 1
1 0 0 0 30
8 80
65 1
0 0 0 4 8
0 2100
a. What is the value each additional acre of land, each additional day of labor,
each additional dollar of capital, and each additional pound of fertilizer?
b. What range of the number of days of labor is the the value given in part (a)
valid?
1.5.3. For the linear program in Exercise 1.5.1, what range of each of the coefficients ck
leaves the set of basic variables the same? In each case, what is the change in maximal
value for the changes ∆k ?
1.5.4. For the linear program in Exercise 1.5.2, what range of each of the coefficients ck
leaves the set of basic variables the same? In each case, what is the change in maximal
value for the changes ∆k ?
(b) Each closed half-space and hyperplane is convex so the intersection is convex.
Theorem 1.20. If S is a convex set, and pi ∈ S for 1 ≤ i ≤ k , then any convex combination
Pk
i=1 ti pi ∈ S.
Proof. The proof is by induction on the number of points. For k = 2, it follows from the
definition of a convex set.
Assume the result is true for k − 1 ≥ 2 points. If tk = 1, then pk ∈ S, so true. If
tk < 1, then k−1
Pk−1 ti Pk−1 ti
i=1 ti = 1 − tk > 0 and i=1 1−tk pi ∈ S by
P
i=1 1−tk = 1. The sum
induction hypothesis. So,
k k−1
X X ti
ti pi = (1 − tk ) pi + tk pk ∈ S.
i=1 i=1
1 − tk
Definition. A point p in a nonempty convex set S is called an extreme point provided that
whenever p = (1 − t)x0 + tx1 for some 0 < t < 1 with x0 and x1 in S then p = x0 = x1 .
An extreme point for a polyhedral set is also called a vertex.
An extreme point of a set must be a boundary point. The disk D = { x ∈ R2 : kxk ≤ 1 }
is convex and each point on its boundary circle is an extreme point.
For the rest of the section, we consider a standard linear program in slack-variable form
with both slack and surplus variables included in x. The feasible set is given by
F = { x ∈ Rn+m
+ : Ax = b }.
Theorem 1.21. Assume x ∈ F = { x ∈ Rn+m + : Ax = b } is a feasible solution to a linear
programming problem. Then x is a point of F if and only if x is a basic feasible solution,
i.e., if and only if the columns of A with xj > 0 form a linearly independent set of vectors.
1.6. Theory for Simplex Method 33
Theorem 1.22. Assume that the feasible set is nonempty for a bounded standard maximization
linear program in slack-variable form. Then the following hold.
a. If x0 is a feasible solution to a bounded linear program, then there exists a basic feasi-
ble solution xb such that f (xb ) = c · xb ≥ c · x0 = f (x0 ).
b. There is at least one optimal basic solution.
c. If two or more basic solutions are optimal, then any convex combination of them is also
an optimal solution.
Proof. (a) If x0 is already a basic feasible solution, then we are done.
Otherwise, the columns A corresponding to x0i 6= 0 are linearly dependent. Let A0 be
the matrix with only these columns. Since the columns of A0 are linearly dependent, there
is a nonzero vector y0 such that A0 y0 = 0. Adding zeroes in the other entries, we obtain a
nonzero vector y such that Ay = 0. Since, A( y) = 0, we can assume that y has c · y ≥ 0
by replacing y by y if necessary. Also,
A [x0 + t y] = b,
so x0 + t y is a solution. Remember that by construction, if x0j = 0, then yj = 0, so if
yj 6= 0, then x0j > 0.
Case 1. Assume that c · y > 0 and some component yi < 0. Then x0i > 0 and
0
x0i + t yi = 0 for ti = −xi/yi > 0. As t increases from 0 to ti , the objective function
increases from c · x0 to c · [x0 + ti y]. If more than one yi < 0, then we select the one with
the smallest value of ti . In this way, we have constructed a new feasible solution x1 with one
more component of the vector zero, fewer components of yi < 0, and a greater value of the
objective function.
34 1. Linear Programming
We can continue in this manner until either the columns are linearly independent or all the
components of the y are nonnegative.
Case 2. If c · y > 0 y ≥ 0, then x + t y is a feasible solution for all t > 0, and
c · [x + t y] = c · x + t c · y
is arbitrarily large. Thus the linear program is unbounded and has no maximum, which is a
contradiction.
Case 3. Assume c · y = 0. Some yk 6= 0. Considering y and y, can assume yk < 0.
Then there exists first tk > 0 such that x0k +tk yk = 0. The value of the objective function does
not change. There are few positive components of new x0 . Eventually we get the corresponding
columns linearly independent, and we are at a basic solution as claimed in part (a).
(b) There are only a finite number of basic feasible solutions {pj }N
j=1 since there is a finite
number of equations. By part (a),
Thus, a maximum can be found among these finite set of values of the basic solutions.
(c) Assume that f (pji ) = M is the maximum for some collection of basic feasible
solutions i = 1, . . . , `. Then any convex combination is also a maximizer:
X
` X` X`
A tji pji = tji Apji = tji b = b,
i=1 i=1 i=1
X`
tji pji ≥ 0 is feasible,
i=1
X
` X` X`
f tji pji = tji f (pji ) = tji M = M.
i=1 i=1 i=1
If there are degenerate basic feasible solutions with fewer than m nonzero basic variables,
then the simplex method can cycle by row reduction to matrices with the same nonzero basic
variables but different zero basic variables so different sets of pivots. Interchanging a basic
variable equal to zero for a zero non-basic variable corresponds to the same vertex of the feasible
set. See the following example. There are ways to programs computers to avoid repeating a set
of basic variables, so avoid cycling. See Jongen et al [8]. Humans avoid cycling naturally.
Example 1.23. The following maximization problem has a degenerate basic solution.
Maximize: 8 x1 + 7 x2 + 2 x3
Subject to: 2 x1 + x2 + x3 ≤ 15,
14 x1 + 13 x2 − 2 x3 ≤ 105,
2 x1 + 4 x2 + 4 x3 ≤ 30,
x1 ≥ 0, and x2 ≥ 0.
1.6. Theory for Simplex Method 35
are different: (x1 , s2 , s3 ) are basic variables for the second tableau and (x1 , x2 , s3 ) are basic
variables for the third tableau. When leaving a basic solution, the variable which becomes
positive must be a free variable (non-basic variable) and not a zero basic (pivot) variable. The
first pivot operation at this degenerate solution interchanges a basic variable equal to 0 and a
free variable, so this new free variable made positive with the next pivoting when the value
of the objective function is increased; this first pivoting results in all the same values of the
variables, so the same point in F . At a degenerate solution, the value will increase after a
finite number of pivoting steps unless there is a cycle of pivoting sets (all staying at the same
point of the feasible set) that results back with with the original set of pivots. The difficulty of
a degenerate vertex and the potential of cycling is a matter of how row reduction relates to a
movement on the feasible set.
Theorem 1.24. If a maximal solution exists for a linear programming problem and the simplex
algorithm does not cycle among degenerate basic feasible solutions, then the simplex algorithm
locates a maximal solution in finitely many steps.
Proof. Pivoting corresponds to changing from one basic feasible solution to another as in The-
orem 1.22(a). Assume that never reach a degenerate basic feasible solution during the steps of
the simplex method. Since there are only a finite number of vertices, the process must termi-
nate with a basic feasible solution p0 for which pivoting to any of the nearby p1 ,. . . ,pk has
f (pj ) ≤ f (p0 ). (Usually, it will strictly decrease the value.) Complete to the set of all basic
feasible solutions (vertices) p0 , p1 ,. . . , p` .
The set of all convex combinations is a bounded polyhedron
( ` `
)
ti = 1 ⊂ F .
X X
H= ti pi : ti ≥ 0,
i=0 i=0
Then,
k
X
f (q) − f (p) = yi [f (pi ) − f (p∗ )] ≤ 0.
i=1
1.6. Exercises
1.6.1. This exercise corresponds to Case 1 in the proof of Theorem 1.22(a).
Consider the linear program
Maximize: f (x1 , x2 , x3 ) = 9 x1 + 2 x2 + x3 ,
Subject to: 4 x1 + 5 x2 + 7 x3 + s1 = 20,
x1 + 3 x2 + 2 x3 + s2 = 7,
x1 ≥ 0, x2 ≥ 0, x3 ≥ 0.
Consider the vectors x0 = (1, 2, 0, 6, 0) and y = ( 3, 1, 0, 7, 0) with coordinates
(x1 , x2 , x3 , s1 , s2 ).
a. Show that x0 is a nonnegative solution of the linear program. Is it a basic
solution? Why or why not?
b. Show that y is a solution of the corresponding homogeneous equation.
c. Determine a value of t such that x0 + t y is a basic solution with
f (a + t b) ≥ f (a).
1.6.2. This exercise corresponds to Case 1 in the proof of Theorem 1.22(a).
Consider the linear program
Maximize: f (x1 , x2 , x3 ) = 9 x1 + 2 x2 + x3 ,
Subject to: x1 + 3 x2 + 7 x3 + s1 = 9,
2 x1 + x2 + 5 x3 + s2 = 12,
x1 ≥ 0, x2 ≥ 0, x3 ≥ 0.
a. Show that (x1 , x2 , x3 , s1 , s2 ) = (2, 0, 1, 0, 3) is a non-basic nonzero solution
of the linear program.
b. Let x0 = (2, 0, 1, 0, 3) as in part (a). Find a vector y that is a solution
of the homogeneous equations and has only nonzero components in the same
coordinates as x0 .
c. Determine a value of t such that x0 + t y is a basic solution with
f (a + t b) ≥ f (a).
1.6.3. This exercise gives an unbounded problems and corresponds to Case 2 in the proof
of Theorem 1.22(a).
Consider the maximization with the nonhomogeneous system of equations
Maximize: f (x1 , x2 , x3 ) = 2 x1 + 5 x2 + 3 x3 ,
Subject to: 6 x1 − x2 + 5 x3 + x4 = 6,
−4 x1 + x2 + 3 x3 − x5 + x6 = 2,
x ≥ 0.
1.6. Theory for Simplex Method 37
a. Set up the tableau (without adding any more artificial variables). Apply the
simplex method. You should come to the situation where an entry in the ob-
jective function row is negative and all the entries above it are negative. You
can use this to give a feasible nonnegative solution.
b. Take the last tableau obtained in part (a) and write out the general solution tak-
ing all the free variables to the right hand size. Explain why the variable with
the negative entry in the objective function row can be increased arbitrarily
large and still give feasible nonnegative solutions. Why does this show that the
problem is unbounded?
1.6.4. Consider the maximization with the nonhomogeneous system of equations
Maximize: f (x1 , x2 , x3 , x4 ) = 75 x1 − 250 x2 + 50 x3 − 100 x4 ,
Subject to: x1 − 4 x2 − 4 x3 + 6 x4 ≤ 1,
x1 − 3 x2 − x3 + x4 ≤ 1,
x3 ≤ 1,
x1 ≥ 0, x2 ≥ 0, x3 ≥ 0, x4 ≥ 0.
a. Row reduce the tableau with slack variables added by choosing the following
sequence of new pivots: (i) row 1 and column 1, (ii) row 2 and column 3, then
(iii) row three and column 5.
b. For the second and third tableau answer the following: (i) What are the basic
variables? (ii) What is the basic feasible solution? (iii) Why are these degen-
erate basic feasible solutions?
38 1. Linear Programming
e. The basic feasible solutions correspond to the extreme points of the feasible
region.
f. The bottom entry in the right column of a simplex tableau gives the maximal
value of the objective function.
g. For a tableau for a maximization linear program, if there is a column with all
negative entries including the one in the row for the objective function, then
the linear programming problem has no feasible solution.
h. The value of the objective function for a MLP at any basic feasible solution is
always greater than the value at any non-basic feasible solution.
i. If a standard maximization linear program MLP has nonempty feasible set,
then it has an optimal basic solution.
j. In the two-phase simplex method, if an artificial variable is positive for the
optimal solution for the artificial objective function, then there is no feasible
solution to the original linear program.
k. The dual mLP problem is to minimize y ∈ Rm subject to Ay ≥ c and
y ≥ 0.
l. If x̄ is an optimal solution to the primal MLP and ŷ is a feasible solution to
the dual mLP, then f (x̄) = g(ŷ).
m. If a slack variable is s̄i > 0 in an optimal solution, then the addition to the
objective function that would be realized by one more unit of the resource
corresponding to its inequality is positive.
n. If a maximization linear program MLP and its dual minimization linear prob-
lem mLP each have nonempty feasible sets (some feasible point), then each
problem has an optimal solution.
o. If the optimal solution of a standard MLP has a slack variable si = 0, then
the ith resource has zero marginal value, i.e., one unit of the ith resource would
add nothing to the value of the objective function.
Chapter 2
Unconstrained Extrema
We begin our treatment of nonlinear optimization problems in this chapter with consideration
of unconstrained problems where the variables are free to move in any direction in a Euclidean
space. The first section presents the mathematical background from calculus and linear algebra
and summarizes some terminology and results from real analysis that is used in the rest of the
book. The second section gives the standard first and derivative conditions for an extremizer.
A standard multi-dimensional calculus course certainly discusses the unconstrained extrema
of functions of two variables and three variables and some calculus courses treat most of the
material of this chapter. However, we present a unified treatment in any dimension that forms
the foundation for the remaining chapters.
39
40 2. Unconstrained Optima
and x ∈ B(p, r)c . Thus, B(x0 , r0 ) ⊂ B(p, r)c and B(p, r)c is open and B(p, r) is closed.
Definition. The interior of S ⊂ Rn , denoted by int(S), is the set with its boundary removed,
int(S) = S r ∂(S). It is the largest open set contained in S. It is also the set of all points
p ∈ S for which there exists an r > 0 such that B(p, r) ⊂ S.
The closure of S ⊂ Rn , denoted by cl(S) or S, is the union of S and its boundary ∂(S),
cl(S) = S ∪ ∂(S). It is the smallest closed set containing S.
Notice that the boundary of a set equals its closure minus its interior, ∂(S) = cl(S) r
int(S).
Example 2.7. For intervals in R, int([0, 1]) = (0, 1), cl((0, 1)) = [0, 1], and ∂([0, 1]) =
∂((0, 1)) = {0, 1}. Also, cl (Q ∩ (0, 1)) = [0, 1], int (Q ∩ (0, 1)) = ∅, and ∂ (Q ∩ (0, 1)) =
[0, 1].
The analogous object in Rn to an interval in R is an open or closed ball. For these sets,
int B(a, r) = B(a, r) and cl(B(a, r) = B(a, r). The boundary is the same for both of these
balls, ∂B(a, r) = ∂B(a, r) = { x ∈ Rn : kx − ak = r }.
To guarantee that a maximum of a real valued function exists, the domain or feasible set
must be closed and bounded: the domain must contain its boundary and cannot “go off to
infinity”.
Definition. A set S ⊂ Rn is bounded provided that there exists a sufficiently large r > 0
such that S ⊂ B(0, r), i.e., kxk ≤ r for all x ∈ S.
Definition. A set S ⊂ Rn is called compact provided that it is closed and bounded.
Remark. In a course in real analysis, a compact set is defined differently: either in terms of
sequences or covers of the set by open sets. With one of these definitions, the above definition
is a theorem about a compact set in Rn .
f g
The function f does not have a limit as (x, y) goes to (0, 0) since it approaches different
values as (x, y) approaches the origin from different directions:
y2
lim f (0, y) = lim =1 and
y→0 y→0 y 2
t2 x2 t2
lim f (x, tx) = lim 2 = 6= 1.
x→0 x→0 x + t2 x2 1 + t2
Definition. A function f : S ⊂ Rn → Rm is continuous at a ∈ S provided that
limx→a f (x) = f (a), i.e., for all > 0 there exists a δ > 0 such that kf (x) − f (a)k <
whenever kx − ak < δ and x ∈ S.
We say that f is continuous on a set S provided that it is continuous at all points in S.
Continuity at a means that given a tolerance > 0 in the values, there is a tolerance
δ > 0 in the input such that all points within δ of a have values within of f (a).
In terms of sequences, f is continuous at a ∈ S provided that for any sequence of points
xk in S that converge to a, the values f (xk ) converge to f (a).
Example 2.10. We return to the function g(x) = sin 1/x for x > 0 and g(x) = 0 for
x < 0. For any small δ > 0, there are some points xn = 1/(n2π) > 0 for which g(xn ) =
0 = g(0), but there are other points x0n = 2/(n4π + π) such that g(x0n ) = 1 that all have
|g(x0n ) − g(0)| = 1 > 1/2. Thus, g(x) is not continuous at x = 0.
Example 2.11. Let f : (0, ∞) → R be defined by f (x) = 1/x. We claim that f is continuous
at all points x > 0. Fix a > 0 and > 0. If |x − a| < δ , then
1 |a − x|
1 δ
|f (x) − f (a)| = − = < .
x a |xa| |xa|
If δ < a/2, then x = a − (a − x) > a/2, 1/x < 2/a, and
2δ
|f (x) − f (a)| < .
a2
2
Thus,n if δ is also a /2, then |f (x) − f (a)| < . Therefore, if we take δ <
2
o less than
min a/2, a /2 , then we get that |f (x) − f (a)| < as desired.
2.1. Mathematical Background 43
Theorem 2.14 (Extreme Value Theorem). Assume that F ⊂ Rn is a nonempty compact set.
Assume that f : F → R is a continuous function. Then f attains a maximum and a minimum
on F , i.e., there exist points xm , xM ∈ F such that
f (xm ) ≤ f (x) ≤ f (xM ) for all x ∈ F , so
f (xm ) = min f (x), and
F
x∈F
f (xM ) = max f (x).
F
x∈F
For a proof see a book on real analysis such as the ones by Wade [15] or Rudin [10].
Example 2.15. We give some examples that illustrate why it is necessary to assume that the
domain of a function is compact in order to be certain that the function attains a maximal or
minimal value.
On the unbounded set F = R, the function f (x) = x3 is unbounded above and below
and has no maximum nor minimum. The function is continuous, but the domain is not bounded.
The same function f (x) = x3 on ( 1, 1) is bounded above and below, but it does not
attain a maximum or minimum on ( 1, 1); the set is bounded but not closed.
Similarly, g(x) = tan(x) on the interval ( π/2, π/2) is unbounded and does not have a
minimum or maximal value; again, this interval is bounded but not closed.
The function h(x) = arctan(x) on R is bounded but the function does not attain a maxi-
mal or minimal value. In this case, the domain is closed but is not bounded and the set of values
attained is bounded but not closed.
Example 2.16. Consider
1
(
x if x 6= 0
f (x) =
0 if x = 0
This function is not continuous at x = 0 and has no maximum nor minimum on [ 1, 1] even
though the domain is compact.
In Section 4.2.1, we define the least upper bound or supremum for a function that is
bounded above but does not attain its maximum.
nonlinear function f (x) near p. See Figure 2.1.1. (An affine function is a constant plus a
linear function.)
f (x)
f (p) + [f 0 (p) + ](x − p) f (p) + f 0 (p) (x − p)
f (p) + [f 0 (p) − ](x − p)
p
lim R
e 1 (p, x) = 0.
x→p
When this limit is satisfied, the the matrix Df (p) is called the derivative of f at p.
If m = 1, then Df (p)> = ∇f (p) is the gradient, which is a column vector. Then
P ∂f
Df (p)(x − p) = ∇f (p) · (x − p) = j (p) (xj − pj ).
∂xj
Remark. The fact that the remainder R1 (p, x) = R e 1 (p, x) kx − pk goes to zero faster than
kx − pk means that f (p) + Df (p)(x − p) is a good affine approximation of the nonlinear
function f (x) near p for all small displacements just as in the one dimensional case.
Since we cannot divide by a vector, the first limit in the definition of differentiability divides
by the length of the displacement.
Note that the rows of the derivative matrix Df (p) are determined by the coordinate func-
tions and the columns by the variable of the partial derivative. This choice is important so that
the correct terms are multiplied together in the matrix product Df (p)(x − p).
Proof. For x a point near p, let pj = (p1 , . . . , pj , xj+1 , . . . , xn ) and rj (t) = (1−t) pj−1 +
t pj−1 . The Mean Value Theorem for a function of one variable can be applied to each coordi-
nate function fi along the paths rj (t):
dfi (rj (t))
j j−1 j j
fi (p ) − fi (p ) = fi (r (1)) − fi (r (0)) = (1 − 0)
dt
t=tij
∂fi j
= (r (tij )) (xj − pj ).
∂xj
If we add these up, the sum telescopes,
n n
X X ∂fi
fi (x) − fi (p) = fi (pj ) − fi (pj−1 ) = (rj (tij )) (xj − pj ).
j=1 j=1
∂xj
Then,
∂fi j ∂fi
(r (tij )) − (r) (xj − pj )
fi (x) − fi (p) − Dfi (p)(x − p)
X
= ∂xj ∂xj
kx − pk kx −
j pk
X ∂fi
j ∂fi
≤ ∂x (r (tij )) − ∂x (r) .
j j j
This last term goes to zero because the partial derivatives are continuous. Since this is true for
each coordinate function, it is true for the vector valued function.
The following result corresponds to the usual chain rule for functions of one variable and
the chain rule for partial derivatives for functions of several variables.
Theorem 2.18 (Chain Rule). Assume that f : Rn → Rm and g : Rm → Rk are C 1 ,
p ∈ Rn and q = f (p) ∈ Rm . Then the composition g ◦ f : Rn → Rk is C 1 , and
D(g ◦ f )(p) = Dg(q) Df (p).
This form of the chain rule agrees with the usual chain rule for functions of several variables
written in terms of partial derivatives: Assume that w = g(x) and x(t), then
Proof. We let v = x − p and w = y − q. In the limits for the derivative, we write R e f (v)
for R
e f (p, p + v) and Re g (v) for R
e g (q, q + w).
g ◦ f (p + v) = g p + Df (p) v + R e f (v) kvk
h i
= g(q) + Dg(q) Df (p) v + R e f (v)kvk
+R e g Df (p) v + R e f (v)kvk
Df (p) v + R e f (v)kvk
h i
= g(q) + Dg(q) Df (p) v + Dg(q) R e f (v) kvk
+R e g Df (p) v + R e f (v)kvk .
Df (p) v + Re f (v)kvk
2.1. Mathematical Background 47
e f (v) goes to zero as needed and is multiplied by kvk. For the second
The first term Dg(q) R
term,
Df (p) v + R e f (v)kvk
v
=
Df (p)
+ Rf (v)
e
kvk kvk
is bounded as kvk goes to zero, and R
e g Df (p) v + R
e f (v)kvk goes to zero. Thus,
where R e g◦f (v) goes to zero as kvk goes to zero. It follows that Dg(q) Df (p) must be the
derivative.
Definition. Let D ⊂ Rn be an open set and f : D → R. If all the second partial derivatives
∂ 2f
(p) exists and are continuous for all p ∈ D then f is said to be twice continuously
∂xi ∂xj
differentiable or C 2 .
∂ 2f
The matrix of second partial derivatives (p) is called the second derivative at p
∂xi ∂xj
and is denoted by D 2f (p). Some authors call it the Hessian matrix of f .
In more formal treatments of calculus, D 2f (p) can be understood as follows. The matrix
Df (x) can be considered as a point in Rn , and the map Df : x 7→ Df (x) is a function from
Rn to Rn . The derivative of the map Df at x0 would be an n × n matrix, which is the second
derivative D 2f (x0 ). This matrix can be applied to two vectors v1 , v2 ∈ Rn ; the result is a
real number, v> 2 n
1 D f (p) v2 . The second derivative is a bilinear map from R to R, i.e., it takes
two vectors and gives a number and is linear in each of the vectors separately.
∂ 2f ∂ 2f
(p) = (p),
∂xi ∂xj ∂xj ∂xi
>
X ∂ 2f
(x − p) D 2f (p) (x − p) = (p)(xj − pj )(xi − pi ) for x ∈ Rn ,
i,j
∂xi ∂xj
If g is C r+1 , then the remainder can also be given by either of the following expressions:
1 x
Z
Rr (p, x) = (x − t)r g (r+1) (t) dt
r! p
1
= g (r+1) (c)(x − p)r+1 ,
(r + 1)!
where c is between p and x.
Proof. The second form of the remainder can be proved by induction using integration by parts.
The other two forms of the remainder can be proved from the second. We refer the reader to a
book on calculus.
To estimate the remainder we use the first form of the remainder. Let
n o
Cr (x) = sup g (r) (t) − g (r) (p) : t is between p and x .
We treat the case with x > p and leave the details for x < p to the reader. Then,
Z x i
1 h
(x − t)r−1 g (r) (t) − g (r) (p) dt
|Rr (p, x)| =
(r − 1)! p
Z x
1
≤ (x − t)r−1 g (r) (t) − g (r) (p) dt
(r − 1)! p
Z x
1
≤ (x − t)r−1 Cr (x) dt
(r − 1)! p
1
= (x − p)r Cr (x).
r!
Since Cr (x) goes to zero as x goes to p, we get the desired result.
Theorem 2.21 (Taylor’s Theorem for a Multi-variable Function). Assume that F : D ⊂
D ) and p ∈ int(D
Rn → R is C 2 on int(D D ). Then,
>
F (x) = F (p) + DF (p) (x − p) + 21 (x − p) D 2F (p) (x − p) + R2 (p, x)
where
R2 (p, x)
lim = 0,
x→p kx − pk2
e 2 (p, x)kx − pk2 , then limx→p R
i.e., if R2 (p, x) = R e 2 (p, x) = 0.
Proof. Define
xt = p + t(x − p) and
g(t) = F (xt ),
2.1. Mathematical Background 49
so g(0) = F (p) and g(1) = F (x). For x near enough to p, xt ∈ D for 0 ≤ t ≤ 1. The
derivatives of g in terms of F are
n
X ∂F
g 0 (t) = (xt )(xi − pi )
i=1
∂xi
g 0 (0) = DF (p)(x − p),
X ∂ 2F
g 00 (t) = (xt )(xi − pi )(xj − pj )
i=1,...,n
∂xj ∂xi
j=1,...,n
Applying usual Taylor’s Theorem for a function of one variable to g gives the result including
the estimate on the remainder.
Remark. For a 3 × 3 symmetric matrix A = (aij ), a21 = a12 , a31 = a13 , and a32 = a23 ,
so
v> Av = a11 v12 + a22 v22 + a33 v32 + 2 a12 v1 v2 + 2 a13 v1 v3 + 2 a23 v2 v3 .
F (x) = F (p + Df (p)(x − p)
∂ 2F 2
2
1 ∂ F 2
2
1 ∂ F
+ 1
(p)(x 1 − p1 ) + (p)(x 2 − p 2 ) + (p)(x3 − p3 )2
∂x21
2 2
∂x22 2
∂x23
∂ 2F ∂ 2F
+ (p)(x1 − p1 )(x2 − p2 ) + (p)(x1 − p1 )(x3 − p3 )
∂x1 ∂x2 ∂x1 ∂x3
∂ 2F
+ (p)(x2 − p2 )(x3 − p3 ) + R2 (p, x).
∂x2 ∂x3
Example 2.22. Find the second order Taylor expansion of F (x, y, z) = 3x2 y + y 3 − 3x2 −
3y 2 + z 3 − 3z , about the point p = (1, 2, 3).
F (p) = 8. The first order partial derivatives are
∂F ∂F
= 6xy − 6x, (p) = 18,
∂x ∂x
∂F ∂F
= 3x2 + 3y 2 − 6y, (p) = 27,
∂y ∂y
∂F ∂F
= 3z 2 − 3, (p) = 24.
∂z ∂z
50 2. Unconstrained Optima
Proof of Theorem 2.23(a). Lay [9] discusses some aspects of the proof of this theorem. A
more complete reference is [13] by Gilbert Strang.
In the proof, we need to add the following intermediate steps.
(v) For each 1 ≤ k ≤ n, the quadratic form associated to Ak is positive definite.
(vi) All the eigenvalues of Ak are positive for 1 ≤ k ≤ n.
(vii) By completion of the squares, Q(x) can be represented as a sum of squares, with all
positive coefficients,
Q(x1 , . . . , xn ) = (x1 , . . . , xn )U> DU(x1 , . . . , xn )>
= p1 (x1 + u1,2 x2 + · · · + u1,n xn )2
+ p2 (x2 + u2,3 x3 + · · · + u2,n xn )2
+ · · · + pn x2n .
(i ⇔ ii) The fact the positive definite is equivalent to all positive eigenvalues is proved in
Lay [9].
(i ⇒ v) Assume Q is positive definite. Then, for any 1 ≤ k ≤ n and any (x1 , . . . , xk ) 6=
0,
0 < Q(x1 , . . . , xk , 0, . . . , 0)
= (x1 , . . . , xk , 0, . . . , 0)A(x1 , . . . , xk , 0, . . . , 0)>
= (x1 , . . . , xk )Ak (x1 , . . . , xk )> .
This shows that (i) implies (v).
(v ⇔ vi) This is the same as the result (i ⇔ ii) applied to Ak . Notice that the eigenvalues
of Ak are not necessarily eigenvalues of A.
(vi ⇒ iii) For any k , det(Ak ) is positive since it is the product of its eigenvalues of Ak ,
(iii ⇒ iv) Assume (iii). If A cannot be
" row#reduced without row exchanges, then there is a
0 a
subblock down the diagonal of the form and some det(Ak ) = 0 and det(Ak+1 ) =
a b
det(Ak−1 ). If we can row reduce A without row exchanges, then it also row reduces all
the Ak . Therefore the pivots of the Ak are pivots of A. Also, the determinant of Ak is the
product of the first k pivots, det(Ak ) = p1 . . . pk . Therefore
pk = (p1 . . . pk )/(p1 . . . pk−1 ) = det(Ak )/ det(Ak−1 > 0,
for each k . This proves (iv).
(iv ⇒ vii) Assume (iv). Row reduction can be realized by matrix multiplication on the left
by a lower triangular matrix with ones on the diagonal. Therefore, L - 1 A = U1 = DU, where
U1 and U are upper triangular, D is a diagonal matrix with the pivots on the diagonal, and U
has ones on the diagonal. (See §2.5 in [9].) Since A is symmetric, LDU = A = A> =
U> DL> . It can then be shown that the factorization is unique and U> = L, so
Q(x1 , . . . , xn ) = (x1 , . . . , xn )U> DU(x1 , . . . , xn )>
= p1 (x1 + u1,2 x2 + · · · + u1,n xn )2
+ p2 (x2 + u2,3 x3 + · · · + u2,n xn )2
+ · · · + pn x2n .
Thus, we can “complete the squares”, expressing Q as the sum of squares with the pivots as
the coefficients. If the pivots are all positive, then all the coefficients pi are positive. Thus (iv)
implies (vii). Note that z = Ux is a non-orthonormal change of basis that makes the quadratic
form diagonal.
54 2. Unconstrained Optima
(vii ⇒ i) If Q(x) can be written as the sum of squares of the above form with positive
coefficients, then the quadratic form must be positive. Thus, (vii) implies (i).
2.1. Exercises
2.1.1. Consider the sets
S1 = { (x, y) ∈ R2 : 1 < x < 1 }
S2 = { (x, y) ∈ R2 : x ≥ 1, y ≥ 0 }.
a. For the sets S1 and S2 , discuss which points are in the boundary and which
points are not using the definition of the boundary.
b. Discuss why S1 is open in two ways: (i) S1 ∩ ∂(S1 ) = ∅ and (ii) for every
point p ∈ S1 , there is an r > 0 such that B(p, r) ⊂ S1 .
c. Discuss why S2 is closed in two ways: (i) ∂(S2 ) ⊂ S2 and (ii) for every
point p ∈ Sc2 , there is an r > 0 such that B(p, r) ⊂ Sc2 .
2.1.2. Consider the sets
S1 = {(x, y) : x > 0, y > 0, xy < 1 } ,
S2 = {(x, y) : x ≥ 0, y ≥ 0, xy ≤ 1 } ,
S3 = {(x, y) : x ≥ 0, y ≥ 0, 2x + 3y ≤ 7 } .
a. For each of the sets S1 , S2 , and S3 , discuss which points are in the boundary
and which points are not using the definition of the boundary.
b. Discuss why S1 is open in two ways: (i) S1 ∩ ∂(S1 ) = ∅ and (ii) for every
point p ∈ S1 , there is an r > 0 such that B(p, r) ⊂ S1 .
c. Discuss why S2 and S3 are closed in two ways: (i) ∂(S) ⊂ S and (ii) for
every point p ∈ Sc , there is an r > 0 such that B(p, r) ⊂ Sc .
2.1.3. Which of the following sets are open, closed, and or compact? Explain why your
answer is correct.
a. Let g1 (x, y, z) = x2 + y 2 , g2 (x, y, z) = x2 + z 2 , and
S1 = g1−1 ([0, 9]) ∩ g2−1 ([0, 4])
= (x, y, z) ∈ R3 : x2 + y 2 ≤ 9, x2 + z 2 ≤ 4 .
It has a strict local maximum at xM provided that there exists an r > 0 such that
f (x) < f (xM ) for all x ∈ F ∩ B(xM , r) r {xM }.
If xM ∈ int(FF ) is a (local) maximizer, then f is said to have an unconstrained (local)
maximum at xM .
The minimum, global minimum, minimizer, local minimum, strict local minimum, and un-
constrained local minimum can be defined in a similar manner.
We say that a function f has an extremum at a point x∗ provided that f has either a
maximum or a minimum at x∗ . In the same way, we say that a function f has a local extremum
at a point x∗ provided that f has either a local maximum or local minimum at x∗ .
Proof. We prove the contrapositive: We assume that the point x∗ is not a critical point and
prove that the function does not have a maximum nor a minimum at x∗ . We are assuming that
the column vector (the gradient) exists and v = Df (x∗ )> 6= 0. We consider the values of f
along the line xt = x∗ + tv (in the direction of the gradient) using the remainder of the affine
approximation:
f (xt ) = f (x∗ ) + Df (x∗ ) (tv) + R
e 1 (x∗ , xt ) ktvk
This proves that x∗ is neither a maximum nor a minimum, i.e., it is not an extreme point. What
we have shown is that if the gradient is nonzero, then the function is decreasing in the direction
of the negative gradient and increasing in the direction of the gradient.
For x near x∗ , R
e 2 is small, and the term involving the second derivative dominates and deter-
mines whether the right hand side of the last equation is positive or negative.
F ) and x∗ ∈ int(F
Theorem 2.27. Suppose that f : F ⊂ Rn → R is C 2 on int(F F ).
2.2. Derivative Conditions 57
a. If f has a local minimum (resp. local maximum) at x∗ , then D 2f (x∗ ) is positive semi-
definite (resp. negative semidefinite).
b. If Df (x∗ ) = 0 and D 2f (x∗ ) is positive definite (resp. negative definite), then f has
a strict local minimum (resp. strict local maximum) at x∗ .
c. If Df (x∗ ) = 0 and D 2f (x∗ ) is indefinite, then x∗ is not an extreme point for f .
Proof. (a) If f has a local minimum at x∗ , then it is a critical point. Use the second order
Taylor’s expansion:
>
f (x) = f (x∗ ) + 12 (x − x∗ ) D 2f (x∗ ) (x − x∗ ) + R
e 2 (x∗ , x) kx − x∗ k2 .
Assume the conclusion is false and there is a direction v such that v> D 2f (x∗ ) v < 0. Let
xt = x∗ + t v, so xt − x∗ = t v. Then
t2 > 2 ∗
f (x) = f (x∗ ) + v D f (x ) v + R e 2 (x∗ , xt ) t2 kvk2
2h i
= f (x∗ ) + t2 12 v> D 2f (x∗ ) v + R e 2 (x∗ , xt )kvk2
1 >
< f (x∗ ) e 2 (x∗ , xt )kvk2 <
for t 6= 0 small enough so that R 2
v D 2f (x∗ ) v.
This implies that f would not have a local minimum at x∗ . Thus, D 2f (x∗ ) must be positive
semidefinite.
(b) Assume that D 2f (x∗ ) is positive definite. The set { u : kuk = 1 } is compact, so
Example 2.28. For the function F (x, y, z) = 3x2 y + y 3 − 3x2 − 3y 2 + z 3 − 3z , find the
critical points and classify them as local maximum, local minimum, or neither.
58 2. Unconstrained Optima
Therefore, (0, 0, 1) is a local maximum, (0, 2, 1) is a local minimum, and the other points are
neither.
2.2. Exercises
2.2.1. Find the points at which each of the following functions attains a maximum and
minimum on the interval 0 ≤ x ≤ 3. For parts (a) and (b), also find the maximal and
minimal values. Remember to consider the end points of the interval [0, 3].
a. f (x) = x2 − 2x + 2.
b. g(x) = −x2 + 2x + 4.
c. The function h(x) satisfies h0 (x) > 0 for all 0 ≤ x ≤ 3.
2.2. Derivative Conditions 59
maximizer must be among the points found in parts (a), (c), (d), and the origin
(0, 0).
2.2.7. A firm produces a single output Q determined by the Cobb-Douglas production func-
1 1
tion of two inputs q1 and q2 , Q = q13 q22 . Let p1 be the price of q1 , and p2 be the
price of q2 ; let the price of the output Q be one. (Either the price of Q is taken as
the unit of money or the pj are the ratios of the prices of qj to the price of Q.) The
profit is given by π = Q − p1 q1 − p2 q2 .
a. Considering the inputs as variables, (q1 , q2 ) ∈ R2++ , show that there is a
unique critical point q1∗ , q2∗ ).
b. Show that the critical point is a local maximum.
Constrained Extrema
This chapter concerns the problem of find the extrema of a function on a feasible set defined by
a number of constraints. When there are equality constraints, the Implicit Function Theorem
gives conditions on the derivatives to ensure that some of the variables locally can be considered
as functions of the other variables. This theorem makes use of the derivative as a matrix in an
explicit manner and is the mathematical basis for implicit differentiation, which we illustrate
with examples from comparative statics within economics. The Implicit Function Theorem
is also used in the proofs of subsequent topics, including in the derivation of the Lagrange
Multiplier Theorem that concerns optimization problems for which the constraint functions are
set equal to constants. The last two sections concern nonlinear optimization problems where
the feasible set is defined by inequality constraints.
0 = ∇g(x∗ ) · (x − x∗ )
∂g ∗ ∂g ∂g ∗
= (x )(x1 − x∗1 ) + · · · + (x∗ )(xn−1 − x∗n−1 ) + (x )(xn − x∗n ), so
∂x1 ∂xn−1 ∂xn
∂g
∂g ∗
∗
(x ) (x )
(x1 − x∗ ) − · · · − ∂xn−1
∂x1
xn = x∗n − (xn−1 − x∗ ).
∂g 1 ∂g n−1
(x∗ ) (x∗ )
∂xn ∂xn
The implicit function says that the nonlinear level set is also locally a graph, with xn deter-
∂g
mined by the other variables. If a different variable xm has (x∗ ) 6= 0, then the same
∂xm
type of argument shows that the nonlinear level set locally determines xm as a function of the
other variables.
61
62 3. Constrained Extrema
Now we turn the case of a vector constraint or several scalar constraints. Consider a C 1
g : Rn → Rk for k > 1 with coordinate functions gi . For a constant b ∈ Rk , the level set
where is the function takes on the values b is denoted by
g - 1 (b) = { x ∈ Rn : g (x) = b for i = 1, . . . , k }.
i i
For x∗ ∈ g - 1 (b), the assumption of nonzero gradient for a scalar function is replaced by
the assumption is that the rank of Dg(x∗ ) is equal to k , i.e., the gradients {∇gi (x∗ )}ki=1 are
linearly independent. Then, it is possible to select k variables xm1 , . . . , xmk such that
∂g1 ∗ ∂g1 ∗
∂xm (x ) · · · ∂xmk
(x )
1
.. ..
det
.. 6= 0.
(3)
. . .
∂gk ∂gk
(x∗ ) · · · (x∗ )
∂xm1 ∂xmk
We show below that if (3) holds, then the null space of Dg(x∗ ) is the graph of the vari-
ables z = (xm1 , . . . , xmk ) in terms of the other n − k variables w = x`1 , . . . , x`n−k .
The Implicit Function states that the nonlinear level set is also locally a graph with the z =
(xm1 , . . . , xmk ) determined implicitly as functions of the other variables w = x`1 , . . . , x`n−k .
Theorem 3.1 (Implicit Function Theorem). Assume there are k C 1 constraint functions,
>
gi : Rn → R for 1 ≤ i ≤ k with g(x) = (g1 (x), . . . , gk (x)) , such that g(x∗ ) = b and
∗ ∗ k
rank(Dg(x )) = k , i.e., the gradients {∇gi (x )}i=1 are linearly independent.
Then, the nonlinear level set g - 1 (b) is locally a graph near x∗ .
>
If z = (xm1 , . . . , xmk ) are k variables such that inequality (3) holds, then they are
>
locally determined implicitly as functions of the other n−k variables, w = x`1 , . . . , x`n−k .
This implicitly defined function z = h(w) is as differentiable as g. The partial derivatives of
∂hq ∂xmq
the coordinate functions of z = h(w), = , can be calculated by the chain rule
∂x`j ∂x`j
and satisfy
k
∂gi X ∂gi ∂xmq
0= + ,
∂x`j q=1 ∂xmq ∂x`j
or in matrix notation
∂g1 ∂g1 ∂g1 ∂g1 ∂xm1 ∂xm1
∂x ... ... ···
`1 ∂x`n−k
∂xm1 ∂xmk
∂x
`1 ∂x`n−k
. .
.. + ... .. .. ..
. .. ..
0 = .. .. . (ImD)
. .
.
.
∂gk ∂gk ∂gk ∂gk
∂xmk ∂xmk
... ... ···
∂x`1 ∂x`n−k ∂xm1 ∂xmk ∂x`1 ∂x`n−k
Remark. The hard part of the proof is showing that the function h exists and is differentiable,
even though there is no explicit formula for it. Once this is known, its derivative can be calcu-
lated by the chain rule and solving the matrix equation (ImD). For a proof, see Wade [15] or
Rudin [10].
>
Take the grouping of variables as given in the theorem: z = (xm1 , . . . , xmk ) are k vari-
>
ables such that equation (3) holds, and w = x`1 , . . . , x`n−k are the other n − k variables.
We write x = (w, z), as if the submatrix of the last k columns has full rank k . (We could
3.1. Implicit Function Theorem 63
also relabel the variables so this was true.) Use the following notation for the matrix of partial
derivatives with respect to only the w variables or only the w variables,
∂gi ∗
Dw g(x∗ ) = (x ) and
∂x`j 1≤i≤k, 1≤j≤n−k
∂gi
Dz g(x∗ ) = (x∗ )
∂xmj 1≤i≤k, 1≤j≤k
Using this notation, rank(Dz g(x∗ )) = k or det (Dz g(x∗ )) 6= 0 so Dz g(x∗ ) is invertible.
Using this notation, the matrix equation (ImD) in the theorem becomes
0 = Dw g(x∗ ) + Dz g(x∗ ) Dh(w∗ ), so
-1
Dh(w∗ ) = (Dz g(x∗ )) Dw g(x∗ ).
In a given example, we write down the equation (ImD), and then solve it for the derivative
Dh(z∗ ), or just the desired partial derivatives. Notice that (i) the matrix Dz g(x∗ ) includes
all the partial derivatives with respect to the dependent variable used to calculate the nonzero
determinant and (ii) the matrix Dw g(x∗ ) includes all the partial derivatives with respect to the
independent (other) variables.
Before giving examples, we want to note that the assumption of the Implicit Function
Theorem is exactly what makes the null space of Dg(x∗ ) a graph of the z-coordinates in
terms of the w-coordinates. The null space of Dg(x∗ ) is given by
null (Dg(x∗ )) = {v ∈ Rn : Dg(x∗ )v = 0 } = {v ∈ Rn : v · ∇gi (x∗ ) = 0 for 1 ≤ i ≤ k } .
If v is in this
! null space, then we can split it up into the components in the w and z directions,
vw
v= , where we use notation as if the z-variables were in the last k components.
vz
!
∗ ∗ ∗ vw
0 = Dg(x )v = [Dw g(x ), Dz g(x )] = Dw g(x∗ )vw + Dz g(x∗ )vz and
vz
-1
vz = (Dz g(x∗ )) Dw g(x∗ )vw .
The Implicit Functions says that if this is possible at the linear level, then the nonlinear level set
is also locally a differentiable graph of these same variable in terms of the other variables.
∇g1
∇g2
Example 3.2 (Changing Technology for Production). This example is based on Section 6.6
of [4]. A firm uses two inputs to produce a single output. Assume the amounts of the inputs
are x and y with p the price of x and q the price of y . The amount produced Q is assumed to
be determined by the Cobb-Douglas production function Q = xa y b , where the technology
determines the exponents a and b. By changing the technology, the firm can vary the
exponent b while keeping a fixed or vary a while keeping b fixed. It wants to keep the
amount produced fixed, Q0 , and the cost of the inputs fixed, px + qy = 125. What is the rate
of change of the amounts of inputs as functions of a and b at x = 5, y = 50, p = 5, q = 2,
a = 1/3, and b = 2/3?
Rather than use the equation Q0 = xa y b , we take its logarithm and obtain the two equations
These two equations define x and y as functions of a, b, p, and q since equation (3) is
∂g1 ∂g1
" #
∂x ∂y p q
det
∂g2 ∂g2 = det a
b
x y
∂x ∂y
pb qa pbx − qay 5 · 2 · 5 − 2 · 1 · 50 1
= − = = = 6= 0.
y x xy 3 · 5 · 50 15
so
∂x ∂x ∂x ∂x
b
q " #
xy y
∂a ∂b ∂p ∂q
0 0 x y
=−
pbx − qay xa
∂y ∂y ∂y ∂y p ln(x) ln(y) 0 0
∂a ∂b ∂p ∂q
bx
" #
xy q ln(x) q ln(y) y b
= ay
qay − pbx p ln(x) p ln(y) a x
xyq ln(x) xyq ln(y) bx2 bxy
qay − pbx qay − pbx qay − pbx qay − pbx
= xyp ln(x) xyp ln(y) ay 2 ln(y)
.
axy
qay − pbx qay − pbx qay − pbx qay − pbx
3.1. Implicit Function Theorem 65
100−50 50
At the point in question, qay − pbx = 3
= 3
and
Example 3.3 (Marginal Inputs of Prices). This example is based on Section 7.3 of [4]. A
firm produces a single output Q determined by the Cobb-Douglas production function of two
1 1
inputs q1 and q2 , Q = q13 q22 . Let p1 be the price of q1 , and p2 be the price of q2 ; let the price of
the output Q be one. (Either the price of Q is taken as the unit of money or the pj are the ratios
of the prices of qj to the price of Q.) The profit is given by π = Q − p1 q1 − p2 q2 . The inputs
that are a critical point satisfy
∂π 1 2 1
0= = q1 3 q22 − p1 = g1 (p1 , p2 , q1 , q2 ) and (5)
∂q1 3
∂π 1 1 1
0= = q13 q2 2 − p2 = g2 (p1 , p2 , q1 , q2 ),
∂q2 2
where the function g is defined by the last two equations.
We have two equations and four variables q1 , q2 , p1 , and p2 which clearly define the prices
in terms of the inputs. We want to show that it also implicitly determines the two inputs q1 and
q2 in terms of the prices p1 and p2 . The derivative of g with respect to q is
2 q 53 q 12 1 q 23 q 12
9 1 2 6 1 2
Dq g = 1 2 1
1 1 3
with
q
6 1
3
q 2
2
q 3
4 1 2 q 2
4
1 2 1
∆2 = det(Dq g) = q1 q2 3
−
36 36
1 4
= q1 3 q2 1 > 0.
36
The second derivative D 2π = Dq g is negative definite, so this critical point maximize profits.
Because det(Dg) 6= 0, the Implicit Function Theorem implies that these two equations (5)
implicitly determine the two inputs q1 and q2 in terms of the prices p1 and p2 . Also, the partial
66 3. Constrained Extrema
derivatives of q1 and q2 with respect to p1 and p2 satisfy the following matrix equation (ImD):
" # ∂g1 ∂g1 ∂g1 ∂g1 ∂q1 ∂q1
0 0 ∂p
∂p2 ∂q1 ∂q2 ∂p1 ∂p2
= 1 +
0 0
∂g2 ∂g2 ∂g2 ∂g2 ∂q2 ∂q2
∂p1 ∂p2 ∂q1 ∂q2 ∂p1 ∂p2
5 1 2 1
∂q1 ∂q1
2q1 q2 1q
6 1 q2
" # 3 2 3 2
1 0 9 ∂p ∂p2
= + 1 2 1
1 3
1
1 q 3 q 2 ∂q2
,
0 1 ∂q2
6 q1 q2
3 2
4 1 2
∂p1 ∂p2
2 1
∂q1 ∂q1 5 1 1 23 2 1
9 q1 q2 q q
3 2
∂p1
∂p2 6 1 2
=
∂q2 1 q 23 q 12 1 31 32
∂q2
1 2 q q
∂p1 ∂p2 6 4 1 2
∂q1 ∂q1
1 13 32 1 23 12
q q q q
∂p1
∂p2
4 4 1 2 6 1 2
= 36 q13 q2 1 23 12 2 53 12
∂q2 ∂q2
q q q1 q2
∂p1 ∂p2 6 1 2 9
5 1 2 1
9 q13 q2 2
6 q13 q22
= 2 1 1 3
,
6 q1 q2
3 2
8 q1 q2
3 2
∂qi
and all the partial derivatives are negative, i.e., the inputs decrease with an increase in
∂pj
either price.
Because the sum of the exponents in the production function is less than one, 13 + 12 < 1,
the production has decreasing return to scale. This property causes D 2π = Dg to be negative
definite, the critical points to be maximum, and the equations to define the inputs as implicit
functions of the prices.
Example 3.4 (Nonlinear Keynesian Model IS-LM Model for National Income). See §15.3
in [11] or §8.6 in [5] for more economic discussion. The variables are as follows:
Y Gross domestic product (GDP) T Taxes
C Consumption r Interest rate
I Investment expenditure M Money supply
G Government spending
The GDP is assumed to be the sum of the consumption, investment, and government spending,
Y = C + I + G. The consumption is a function of after taxes income or the difference Y − T ,
C = C(Y − T ); the investment is a function of the interest rate, I = I(r); and the money
supply is a function (called the liquidity function) of GDP and the interest rate, M = L(Y, r).
We can think of the assumptions as yielding the following two functional relationships between
the five variables:
0 = −Y + C(Y − T ) + I(r) + G and (6)
0 = L(Y, r) − M.
3.1. Implicit Function Theorem 67
3.1. Exercises
3.1.1. A firm uses two inputs, q1 and q2 to produce a single output Q, given by the produc-
2/ 1/
tion function Q = kq1 5 q2 5 . Let P be the price of the output Q, p1 be the price of
2/ 1/
q1 , and p2 be the price of q2 . The profit is given by π = P kq1 5 q2 5 − p1 q1 − p2 q2 .
The inputs that maximize profits satisfy
2P k −3/5 1/5
0= q q2 − p 1 and
5 1
P k 2/5 −4/5
0= q q − p2 .
5 1 2
a. Show that this two equations can be used to determine the amounts of inputs
q1 and q2 in terms of the prices p1 , p2 , and P . Show that the relevant matrix
has nonzero determinant.
b. Write the matrix equation for the partial derivatives of q1 and q2 with respect
to p1 , p2 and P in terms of the variables.
c. Solve for the matrix of partial derivatives of q1 and q2 in terms of p1 , p2 and
P.
3.1.2. Assume that the two equations that balance supply and demand for a single product
are given by
0 = Q − S0 1 − eT −P
0 = Q − Y e-P .
Here Q is the quantity sold, P is the price, T are the taxes, Y is the income of the
consumers, and S0 is a constant.
a. Use the implicit function theorem to show that these two equations define Q
and P as functions of T and Y (keeping S0 fixed).
∂P ∂P ∂Q ∂Q
b. Solve the matrix of partial derivatives of , , , . You may leave
∂Y ∂T ∂Y ∂T
your answer in matrix form.
3.1.3. A nonlinear Keynesian IS-LM model for national income involves the following
quantities:
Y Gross domestic product (GDP)
G Government spending
r Interest rate
M Money supply
In addition, there are three quantities which are functions of the other variables (inter-
mediate variables). Investment expenditure I is a function of the interest rate given
I0
by I(r) = . The consumption is a function of Y given by C(Y ) =
r+1
C0 + 65 Y + 16 e - Y with C0 a constant. The gross domestic product is the sum of
consumption, investment expenditure, and government spending, Y = C +I +G =
I0
C0 + 56 Y + 61 e - Y + + G. The money supply equals the liquidity func-
r+1
Y
tion, M = . With these assumptions, the model yields the following two
r+1
3.2. Extrema with Equality Constraints 69
equations:
1 1 I0
0 = C0 − Y + e-Y + +G and
6 6 r+1
Y
0= − M.
r+1
a. Using the Implicit Function Theorem, show that these two equations define Y
and r as dependent variables which are determined by the independent vari-
ables G and M , i.e., these equations define Y and r as functions of G and
M.
b. Write the matrix equation that the partial derivatives of Y and r with respect
to G and M must satisfy.
c. Solve for the matrix equation for the partial derivatives of Y and r with respect
to G and M .
3.1.4. Consider the three equations
xyz + u + v 2 = 6,
xy − zy 2 + u2 + v + w = 6,
xy 3 − zx2 + u2 + w = 4.
a. Show that these equations implicitly define (u, v, w) in terms of (x, y, z).
b. What is the system of equations (ImD) for the partial derivatives near the point
(x, y, z, u, v, w) = (1, 1, 1, 1, 2, 3)?
A similar justification can be given with two constraints. The tangent vectors to the level
set g1 (x) = b1 and g2 (x) = b2 are all vectors perpendicular to both ∇g1 (x∗ ) and ∇g2 (x∗ ).
This is the same as the null space of Dg(x∗ ). Again, the directional derivative of f in the
direction of v must be zero, ∇f (x∗ ) · v = 0. Since ∇f (x∗ ) is perpendicular to all these
vectors v, it must be a linear combination of ∇g1 (x∗ ) and ∇g2 (x∗ ).
Example 3.6. It is possible for the constraint qualification to fail at a maximum. Let g(x, y) =
x3 + y 2 = 0 and f (x, y) = y + 2x.
f - 1( 1
2)
g - 1 (0)
f - 1 ( 1) f - 1 (0)
Figure 3.2.1. Maximum at a singular point
A point x∗ satisfies the first order Lagrange multiplier conditions (LM) with multipliers λ∗ iff
λ∗ , x∗ ) is a critical point of L with respect to all its variables,
(λ
∂L ∗ ∗
λ , x ) = bi − gi (x∗ ) = 0
(λ for 1 ≤ i ≤ k and
∂λi
k
X
λ∗ , x∗ ) = Df (x∗ ) −
Dx L(λ λ∗i Dgi (x∗ ) = 0.
i=1
To ensure that the constraint qualification does not fail, we need that
rank Lλi ,xj (λ λ∗ , x∗ ) ij = rank (Dg(x∗ )) = k.
These conditions on the Lagrangian are not a proof but merely a mnemonic device.
3.2. Extrema with Equality Constraints 73
as a function of b as well as x and λ . Also, assume the second derivative of the Lagrangian
as a function of λ and x is nondegenerate, i.e., it has a nonzero determinant. Then,
∂
λ∗i (b) = f (x∗ (b)).
∂bi
λ∗ , x∗ ) = (λ
Proof. For fixed b, (λ λ∗ (b), x∗ (b)) satisfy
! !
Dλ L> b − g(x)
0= = λ, x, b).
= G(λ
Dx L> Df (x)> − j λj Dgj (x)>
P
The derivative of G with respect to λ and x is the second derivative of L with respect to λ
and x and is the bordered Hessian
" #
0k Dg
Dλ ,x G(λλ, x, b)) = (Dλ G(λ
λ, x, b), Dx G(λ λ, x, b)) = .
Dg> Dx2L
This derivative satisfies det (Dλ ,x G(λ λ∗ (b), x∗ (b), b)) 6= 0, since this second derivative
of the Lagrangian is assumed to be nondegenerate. See Section 3.5 for a discussion of bor-
dered Hessians and nondegenerate extrema on a level set. Therefore, the variables (λ λ, x) =
λ∗ (b), x∗ (b) are implicitly determined differentiable functions of b by the equation G(λ
(λ λ, x, b) =
0.
At these points on the level sets, f (x∗ (b)) = L(λ λ∗ (b), x∗ (b), b). Taking the partial
derivative of this equality with respect to bi gives
∂ ∂ ∂ ∗ ∂L ∗
f (x∗ (b)) = Dλ L λ ∗ (b) + Dx L x (b) + (x (b), λ ∗ (b), b).
∂bi ∂bi ∂bi ∂bi
Because bj − gj (x∗ (b)) = 0 for all j ,
∂ X ∂
Dλ L λ ∗ (b) = b∗j − gj (x∗ (b)) λj (b) = 0.
∂bi j
∂bi
Because the point satisfies the (LM) conditions,
k
∂ ∗ ∗
X
∗ ∗ ∂ ∗
Dx L x (b) = Df (x (b)) − λj (b)Dgj (x (b)) x (b) = 0.
∂bi j=1
∂bi
∂ ∂ ∗
f (x∗ (b)) = Df (x∗ (b)) x (b)
∂bi ∂bi
X ∂ ∗ ∗
= λ∗j (b) Dgj (x∗ (b)) x (x (w))
j
∂bi
X ∂
= λ∗j (b) gj (x∗ (b))
j
∂bi
= λ∗i (b).
3.2. Exercises
3.2.1. Find the points satisfying the first order conditions for a constrained extrema. Then
compare the values and argue which points are global maxima and which are global
minima.
a. f (x, y, z) = xyz and g(x, y, z) = 2x + 3y + z = 6.
b. f (x, y, z) = 2x + y 2 − z 2 , g1 (x, y, z) = x − 2y = 0, and g2 (x, y, z) =
x + z = 0.
3.2.2. For each of the following objective and constraint functions, find the maximizer and
minimizers.
a. f (x, y, z) = x2 + y 2 + z 2 , subject to g(x, y, z) = x + y + z = 12 and
h(x, y, z) = x2 + y 2 − z = 0.
b. f (x, y, z) = x+y+z , subject to g(x, y, z) = x2 +y 2 = 2 and h(x, y, z) =
x + z = 1.
c. Minimize f (x, y, z) = x2 +y 2 +z 2 , subject to g(x, y, z) = x+2y+3z = 6
and h(x, y, z) = y + z = 0.
x≤0
(0, 1)
x + (y − 1)3 ≤ 0
At (0, 1), E(0, 1) = {1, 2}, gE(0,1) (x, y) = (x + (y − 1)3 , x)> , and
" # " #
1 3(y − 1)2 1 0
rank(DgE(0,1) (0, 1)) = rank = rank = 1 < 2 = |E(0, 1)|.
1 0 1 0
y=1
Therefore, the constraint qualification fails at (0, 1). Note that the level sets g1- 1 (0) and
g2- 1 (0) are tangent at (0, 1) and the gradients ∇g1 (0, 1) and ∇g2 (0, 1) are parallel.
Theorem 3.12 (Karush-Kuhn-Tucker). Suppose that f, gi : Rn → R are C 1 functions for
1 ≤ i ≤ m, and f attains a local extremum at x∗ on F g,b .
Then either (a) the constraint qualification fails at x∗ with rank(DgE(x∗ ) (x∗ )) < |E(x∗ )|,
or
(b) there exist λ ∗ = (λ∗1 , . . . , λ∗m ) such that KKT-1,2,3 or KKT-1,2,30 hold:
76 3. Constrained Extrema
Df (x∗ ) = m ∗ ∗
P
KKT-1. i=1 λi Dgi (x ).
KKT-2. λ∗i ( bi − gi (x∗ ) ) = 0 for 1 ≤ i ≤ m ( so λ∗i = 0 for i ∈
/ E(x∗ ) ).
KKT-3. If x∗ is a maximizer, then λ∗i ≥ 0 for 1 ≤ i ≤ m.
KKT-30 . If x∗ is a minimizer, then λ∗i ≤ 0 for 1 ≤ i ≤ m.
Remark. This theorem is the nonlinear version of Proposition 1.13 in Chapter 1 for linear pro-
grams. The proof is similar with the Implicit Function Theorem replacing part of the argument
done by linear algebra for the linear program.
Remark. Exercise 3.3.5 gives an example for which the maximum occurs at a point where the
constraint qualification fails.
Remark. We call KKT-1,2,3 the first order Karush-Kuhn-Tucker conditions for a maximum.
Condition KKT-1 implies that ∇f (x∗ ) is perpendicular to the tangent space of the level
- 1 (b
set of the tight constraints at x∗ , gE(x ).
∗) E(x∗ )
Condition KKT-2 is called complementary slackness: If a constraint is slack, gi (x∗ ) < bi ,
then the corresponding multiplier λ∗i = 0; if λ∗i 6= 0, then the constraint is tight, gi (x∗ ) = bi .
Condition KKT-3, λ∗i ≥ 0 for a maximum, implies that the gradient of f points out of
the feasible set at x∗ . The inequalities, gi (x) ≤ bi , are resource type and signs λ∗i ≥ 0 are
compatible with maximization for a linear programming problem.
Condition KKT-30 , λ∗i ≤ 0 for a minimum, implies that the gradient ofPf points into the
m
feasible set at x∗ . The point x∗ is a maximizer of f (x) and ∇f (x∗ ) = i=1 ( λ∗i ) ∇gi (x∗ )
∗ ∗
with λi ≥ 0. So the signs λi ≤ 0 are compatible with minimization for a linear program-
ming problem.
Steps to Find an Optimal Solution using the KKT Theorem 3.12
1. Verify that a maximum (resp. minimum) exists by showing either that the
feasible set is compact or that f (x) takes on smaller values (resp. larger
values) near infinity.
2. Find all the possible extremizers: (i) Find all the points on the boundary of the
feasible set where the constraint qualification fails; (ii) find all the points that
satisfy KKT-1,2,3 (resp. KKT-1,2,30 ).
3. Compare the values of f (x) at all the points found in 2(i) and 2(ii).
Proof of Theorem 3.12. Assume the constraint qualification holds at x∗ and |E(x∗ )| = k .
We can rearrange the indices of the gj so that E(x∗ ) = { 1, . . . , k }, i.e., gi (x∗ ) = bi for
∗
1 ≤ i ≤ k and gi (x ) < bi for k + 1 ≤ i ≤ m. Also, we can rearrange the indices of the xj
∂gi ∗
so that det (x ) 6= 0.
∂xj 1≤i,j≤k
Set λ∗i = 0 for i ∈/ E(x∗ ), i.e., for k + 1 ≤ i ≤ m. We essentially drop these ineffective
constraints from consideration in the proof. The function f also attains a extremum at x∗ on
{ x : gi (x) = bi for i ∈ E(x∗ ) }, so by the Lagrange Multiplier Theorem, there exist λ∗i for
1 ≤ i ≤ k so that X
Df (x∗ ) = λ∗i Dgi (x∗ ).
1≤i≤k
Since λ∗i = 0 for k + 1 ≤ i ≤ m, we can change the summation to 1 to m and obtain condition
KKT-1. Also, either λ∗i = 0 or bi − gi (x∗ ) = 0, so condition KKT-2 holds.
The question remains: For a maximum, why are λ∗` ≥ 0 for ` ∈ E(x∗ )? (The case of a
minimum is similar.)
We apply the Implicit Function Theorem to show that there is a curve r(t) in F such
that, for small t > 0, (i) g` (r(t)) < b` , (ii) gi (r(t)) = bi for i 6= ` and 1 ≤ i ≤ k , and (iii)
ri (t) = x∗i for k + 1 ≤ i ≤ n.
3.3. Extrema with Inequality Constraints: Necessary Conditions 77
gi- 1 (0)
T
i6=`
r(t)
∗
x
g`- 1 (0)
Figure 3.3.2. Curve where only one more constraint becomes slack
0 ··· 0 0 ··· 1
∂gi ∗
= det (x ) 6= 0.
∂xj 1≤i,j≤k
By the Implicit Function Theorem, there exists x = r(t) such that r(0) = x∗ and G(r(t), t) ≡
0, i.e., gi (r(t)) = bi − t δi` for 1 ≤ i ≤ k and xi = x∗i for k + 1 ≤ i ≤ n. This curve r(t)
has the desired properties:
∗ 0 d
Dgi (x r (0) = gi ◦ r(t) = δi` for 1 ≤ i ≤ k.
dt t=0
The function f has a maximum at x , so f (x∗ ) ≥ f (r(t)) for t ≥ 0, and
∗
d X
= Df (x∗ ) r0 (0) = λ∗i Dgi (x∗ ) r0 (0) = λ∗` ,
0 ≥ f ◦ r(t)
dt t=0 1≤i≤k
where the second equality holds by the first order Lagrange Multiplier condition KKT-1 and the
third equality holds by the calculation of Dgi (x∗ )r0 (0). We have shown that that λ∗` ≥ 0 for
1 ≤ i ≤ k , so we have KKT-3.
Example 3.13. Let f (x, y) = x2 − y and g(x, y) = x2 + y 2 ≤ 1. (Note that the feasible set
is compact.)
The derivative of the constraint is Dg(x, y) = (2x, 2y), which has rank one at all the
points of the boundary of the feasible set. (At least one variable is nonzero at each of the
points.) Therefore, the constraint qualification is satisfied at all the points in the feasible set.
78 3. Constrained Extrema
Notice that although the multiplier is positive at (0, 1), it is not a maximizer. The function
f decreases as it moves into the interior of the region at (0, 1), but it is a local minimum along
the boundary so this point is a type of saddle point on the feasible set.
Example 3.14. Maximize f (x, y, z) = x2 + 2y 2 + 3z 2 , on the constraint set 1 = x + y + z =
g0 (x, y, z), 0 ≥ x = g1 (x, y, z), 0 ≥ y = g2 (x, y, z), and 0 ≥ z = g3 (x, y, z).
Because the 0th -equation involves an equality, λ0 can have any sign. For 1 ≤ i ≤ 3, we need
λi ≥ 0.
We want to check that the constraint qualification is satisfied at all points of the feasible
set. On the face where g0 (x, y, z) = 1 and gi (x, y, z) < 0 for i = 1, 2, 3,
h i
rank DgE (x, y, z) = rank (Dg0 (x, y, z)) = rank 1 1 1 = 1.
There are several aspects that make KKT Theorem 3.12 difficult to apply. First, it is nec-
essary to show that a maximum or minimum exists. Second, it is not easy to show that the
constraint qualification holds at all points of F g,b or find all the points on the boundary where
it fails. Thus, the possibility of the constraint qualification failing makes the theorem difficult
to apply in applications. Finally, it is necessary to find all the points that satisfy KKT-1,2,3 or
where the constraint qualification fails and compare the values of the objective function at these
points.
The deficiencies of Theorem 3.12 are overcome by means of convexity and concavity of
constraints and objective function as developed in the next section.
3.3. Exercises
3.3.1. Maximize the revenue
1/2 1/2 1/3
π = p 1 x1 + p2 x1 x2
subject to a wealth constraint on the inputs
w1 x1 + w2 x2 ≤ C > 0, x1 ≥ 0 x2 ≥ 0.
a. Write down the constraint functions and the KKT-1,2,3 equations that must be
satisfied for the Karush-Kuhn-Tucker Theorem.
b. Take w1 = w2 = 2, p1 = p2 = 1, and C = 8, and find explicit values of x1
and x2 that attains the maximum.
3.3.2. Given H > 0, p1 > 0, p2 > 0, and w > 0. Let u(x1 , x2 , y) = xα1 1 xα2 2 − (H − y)2 ,
for α1 > 0 and α2 > 0. Consider the feasible set
Φ(p, q, w, H) = { (x1 , x2 , y) ∈ R3+ : p1 x1 + p2 x2 + wy ≤ wH, 2y ≤ H }.
a. What are the constraint functions gi which define the inequality constraints in
the form of the Kuhn-Tucker Theorem?
wHαi
Which constraints are effective at y ∗ = 21 H , and x∗i = for
2pi (α1 + α2 )
i = 1, 2?
Do the constraints satisfy the rank condition?
b. Why does the feasible set satisfy the Slater condition?
c. What are the KKT-1,2,3 equations?
3.3.3. Consider the following problem:
Maximize: f (x, y) = x2 + y 2 + 2y
Subject to: x2 + y 2 ≤ 5
x + 2y ≤ 4
0 ≤ x, 0 ≤ y .
a. Explain why f (x, y) must have a maximum on the feasible set.
b. Using the fact that the points (x, y) = (2, 1), 6/5, 7/5 , and (0, 2) are the
only points that satisfy the first order KKT equations for a maximum with the
correct signs of the multipliers, what is the maximal value of f (x, y) on the
feasible set and what is the point that is the maximizer? Explain why this must
be the maximizer, including explaining how the theorems apply and what other
conditions need to be satisfied.
the constraint qualification is satisfied at the three points (2, 1),
c. Verify that
6/5, 7/5 , and (0, 2).
3.4. Extrema with Inequality Constraints: Sufficient Conditions 81
3.3.4. Assuming the parameters p > 1, w0 > 0, 0 < x̄1 < w0/p, and 0 < x̄2 < w0 ,
consider the following problem:
Maximize: U (x1 , x2 ) = x1 x2
Subject to: p x1 + x2 ≤ w0 ,
0 ≤ x1 ≤ x̄1 ,
0 ≤ x2 ≤ x̄2 .
a. Show that the constraint qualification is satisfied on the feasible set.
b. Why must U attain a maximum on the feasible set?
c. What are the KKT-1,2,3 equations?
d. What conditions on the parameters need to be satisfied for U to have a maxi-
mum at X1 = x̄1 and x2 = x̄2 ?
3.3.5. Consider the problem
Maximize: f (x, y) = 2 − 2 y
Subject to g1 (x, y) = y + (x − 1)3 ≤ 0
g2 (x, y) = x ≤ 0,
g3 (x, y) = y ≤ 0.
Carry out the following steps to show that the maximizer is a point at which the
constrain qualification fails.
a. By drawing a figure, show that the feasible set is a three sided (nonlinear)
region with vertices at (0, 0), (1, 0), and (0, 1).
b. Plot several level curves f - 1 (C) of the objective function to your figure from
part (a) and conclude geometrically that (0, 1) is a maximizer and (1, 0) is a
minimizer of f (x, y) on the feasible set.
c. Show that the constraint qualification fails at (0, 1). Also, show that Df (0, 1)
cannot be written as a linear combination of the derivatives Dgi (0, 1) of the
effective constraints.
f (xt ) f (y)
x xt y
function these conditions can be expressed in terms of the function being above or below the
tangent plane at all points.
Definition. A function f : D ⊂ Rn → R is concave on D provided that for all x, y ∈ D
and 0 ≤ t ≤ 1,
f (xt ) ≥ (1 − t) f (x) + t f (y) for xt = (1 − t) x + t y.
This is equivalent to assuming that the set of points below the graph,
{(x, y) ∈ D × R : y ≤ f (x) },
is a convex subset of Rn+1 .
A function f : D → R is strictly concave provided that for all x, y ∈ D with x 6= y and
0 < t < 1,
f (xt ) > (1 − t) f (x) + t f (y) for xt = (1 − t) x + t y.
A function f : D ⊂ Rn → R is convex on D provided that for all x, y ∈ D and
0 ≤ t ≤ 1,
f (xt ) ≤ (1 − t) f (x) + t f (y) for xt = (1 − t) x + t y.
This is equivalent to assuming that the set of points above the graph,
{(x, y) ∈ D × R : y ≥ f (x) },
n+1
is a convex subset of R .
A function f : D → R is strictly convex provided that for all x, y ∈ D with x 6= y and
0 < t < 1,
f (xt ) < (1 − t) f (x) + t f (y) for xt = (1 − t) x + t y.
Remark. If f is either concave or convex on D then D is convex. Also, the condition is on
the graph of f and not on the domain of f .
Theorem 3.15. Let f : D ⊂ Rn → R be a concave or convex function on D . Then in the
interior of D , f is continuous and possesses all directional derivatives (which can possibly be
infinite).
3.4. Extrema with Inequality Constraints: Sufficient Conditions 83
See [14] or [2] for a proof. Since concave/convex functions are continuous, they are rea-
sonable functions to maximize or minimize.
Theorem 3.16. Assume that D ⊂ Rn is an open convex subset, and gi : D → R are C 1
convex functions for 1 ≤ i ≤ m. Then F g,b = { x ∈ D : gi (x) ≤ bi for 1 ≤ i ≤ m } is a
convex set for any b ∈ Rm ,
Proof. Take x, y ∈ F g,b and let xt = (1 − t)x + ty for 0 ≤ t ≤ 1. For any 1 ≤ i ≤ m,
gi (xt ) ≤ (1 − t)gi (x) + tgi (y) ≤ (1 − t)bi + tbi = bi , so xt ∈ F g,b . Thus, F g,b is
convex.
The assumption that a function is convex or concave can be verified by a second-derivative
condition at all points of the feasible set as shown in the following theorem.
Theorem 3.17 (Second-Derivative Test). Let D ⊂ Rn be open and convex and f : D → R
be a C 2 function.
a. The function f is convex (respect. convex) on D iff D 2f (x) is positive (respect.
negative) semidefinite for all x ∈ D .
b. If D 2f (x) is positive (respect. negative) definite for all x ∈ D , then f is strictly
convex (respect. concave) on D .
We follow the proof in Sundaram [14]. The general theorem can be adapted from the
special case when D is all of Rn . The idea is that if D 2f (x) is positive definite (resp. negative
definite), then locally the graph of f lies above (resp. below) the tangent plane. The proof
makes this global.
We start by proving a lemma that says it is enough to show f is convex along straight lines
in the domain.
Lemma 3.18. Let f : Rn → R. For any x, h ∈ Rn , let gx,h (t) = f (x + t h) for t ∈ R.
Then the following hold.
a. f is convex iff gx,h is convex for each fixed x, h ∈ Rn .
b. If gx,h is strictly convex for each fixed x, h ∈ Rn with h 6= 0, then f is strictly
convex.
Proof. First suppose that f is convex. Fix x, h ∈ Rn . For any t1 , t2 ∈ R and any 0 ≤ β ≤ 1,
gx,h (β t1 + (1 − β) t2 ) = f (x + β t1 h + (1 − β) t2 h)
= f (β (x + t1 h) + (1 − β) (x + t2 h) )
≤ β f (x + t1 h) + (1 − β) f (x + t2 h)
= β gx,h (t1 ) + (1 − β) gx,h (t2 ).
This shows that gx,h is convex.
Next, suppose that gx,h is convex for any x, h ∈ Rn . Pick any x, y ∈ Rn , and let
h = y − x, and xt = (1 − t) x + t y = x + t h for 0 ≤ t ≤ 1. Then,
f ( (1 − t) x + t y ) = gx,h (t) + gx,h ( (1 − t)0 + t 1)
≤ (1 − t) gx,h (0) + t gx,h (1)
= (1 − t) f (x) + t f (y).
Since this is true for any x, h ∈ Rn , f is convex.
The proof of (b) is similar.
Proof of Theorem 3.17. We consider part (b) in the case when D 2f (x) is positive definite and
show that f is strictly convex. Pick any x, h ∈ Rn with h 6= 0, and define g(t) = gx,h (t)
as in the lemma. In the notes for Chapter 6, we showed that
00
gx,h (t) = h> D 2f (xt )h,
84 3. Constrained Extrema
Remark. In most of the examples we use part a.i (or b.i). Note that for this use it is not nec-
essary to verify Slater’s condition or that a maximum must exist (by compactness or similar
argument). The feasible set F g,b is shown convex by conditions on constraint function. We
have shown that if all the gi (x) are convex then it is true. Later, we allow for rescaling of con-
vex functions. Finally, once one point is found that satisfies KKT-1,2,3, then it is automatically
a maximizer; we do not need to verify separately that a maximum exists.
We delay the proof of the Karush-Kuhn-Tucker Theorem under Convexity until Section
3.4.5. In the rest of this section, we give examples of convex/concave functions and applications
of the Karush-Kuhn-Tucker Theorem.
Proposition 3.20. For a ∈ Rn and b ∈ R, the affine function on Rn given by
g(x) = a · x + b = a1 x1 + · · · + an xn + b is both concave and convex.
Proof. For p0 , p1 ∈ Rn , set pt = (1 − t) p0 + t p1 . Then,
g(pt ) = a · pt + b
= a · [(1 − t) p0 + t p1 ] + b
= (1 − t) [a · p0 + b] + t [a · p1 + b]
= (1 − t) g(p0 ) + t g(p1 ).
Thus, we have equality and not just an inequality and g is both concave and convex.
Example 3.21.
Minimize : f (x, y) = x4 + y 4 + 12 x2 + 6 y 2 − xy − x + y ,
Subject to : g1 (x, y) = −x − y ≤ 6
g2 (x, y) = −2x + y ≤ 3
x ≤ 0, y ≤ 0.
The constraints are linear and so are convex. The objective function has second derivative
" #
2 12 x2 + 24 1
D f (x, y) = .
1 12 y 2 + 12
The determinant is greater than 24(12) − 1 > 0, so this is positive definite and f is convex.
Let λ1 and λ2 be the multipliers for the first two inequalities and µ1 and µ2 be the
multiplier for x ≤ 0 and y ≤ 0. Condition KKT-1 is
0 = 4x3 + 24x − y − 1 + λ1 + 2λ2 + µ1
0 = 4y 3 + 12y − x + 1 + λ1 − λ2 + µ2
If x = 0, then y ≤ 3, so y > 0, which is not feasible. Therefore this constraint cannot
be tight and µ1 = 0. We consider the slackness of the other constraints in cases.
If y = 0, then x = g1 (x, 0) ≥ 6, so g2 (x, 0) = −2x ≤ 12 < 3, and λ2 = 0.
The function f (x, 0) = x4 + 12 x2 − x, is minimized for x ≥ 6 at x = 6. For that value, the
second equation gives
0 = −6 + 1 + λ1 + µ2 , or
5 = λ 1 + µ2 .
Both these multipliers cannot be negative, so it is not a minimum.
Finally assume that x, y > 0, so µ2 = 0. Points where both g1 and g2 are tight satisfies
6 = g1 (x, y) = −x − y
3 = g2 (x, y) = −2x + y
86 3. Constrained Extrema
(ii) Although we do not need to do so, we check that there is no minimizer where the
constraint is not effective and λ = 0.
1600
x2 = = 64, x=8 and
25
y 2 = 54, y ≈ 7.34.
Since 8 + 7.34 < 20, they do not satisfy the constraint.
Proposition 3.23 (Cobb-Douglas in R2 ). Let f (x, y) = xa y b with a, b > 0. If a + b ≤ 1,
then f is concave on R2+ (and xa y b is convex). If a + b > 1, then f is neither concave
nor convex.
See Figure 3.4.3.
xy
1 1
x y 3 3
y y
x x
(a) (b)
Proof.
!
2 a(a − 1)xa−2 y b
abxa−1 y b−1
det D f (x, y) = det a−1 1−b
abx y b(b − 1)xa y b−2
> 0
if a + b < 1
= ab(1 − a − b)x2a−2 y 2b−2 =0 if a + b = 1
<0 if a + b > 1.
If a + b < 1 (so a < 1), D 2f (x, y) is negative definite and f is strictly concave on R2++ ;
since f is continuous on R2+ , it is also concave on R2+ . If a + b = 1, D 2f (x, y) is negative
semidefinite and f is concave on R2+ ; if a + b > 1, the D 2f (x, y) is indefinite and f is neither
concave nor convex. For example, xy is neither concave nor convex.
Proposition 3.24 (Cobb-Douglas in Rn ). Assume that a1 + · · · + an < 1 and ai > 0 for
1 ≤ i ≤ n. Then the function f (x) = xa1 1 xa2 2 · · · xann is concave on Rn+ (and xa1 1 · · · xann is
convex). If a1 + · · · + an > 1, then f is neither concave nor convex.
Proof. The reader should try to carry out the calculations directly for the case n = 3.
On Rn++ , the partial derivatives of f are as follows for i 6= j :
fxi = ai xa1 1 · · · xai i −1 · · · xann
fxi xi = ai (ai − 1)xa1 1 · · · xai i −2 · · · xann = ai (ai − 1)xi- 2 f
= a a xa1 · · · xai −1 · · · x j · · · xan = a a x - 1 x - 1 f.
a −1
f xi xj i j 1 i j n i j i j
88 3. Constrained Extrema
Using linearity on rows and columns, the determinant of the k th -principal submatrix is as fol-
lows:
a1 (a1 − 1)x1- 2 f · · · a1 ak x1- 1 xk- 1
.. .. ..
∆k = det . . .
-
ak a1 xk x1 f1 -1
· · · ak (ak − 1)xk f - 2
a1 − 1 · · · ak
. ..
= a1 · · · ak x1- 2 · · · xk- 2 f k det ..
..
. .
a1 · · · ak − 1
= a1 · · · ak x - 2 · · · x - 2 f k ∆
1 k
¯ k,
where the last equality defines ∆ ¯ k as the determinant of the previous matrix. Below, we show
¯ k
by induction that ∆k = ( 1) (1 − a1 − · · · − ak ). Once this is established, since signs of
the ∆k alternate as required, D 2f is negative definite on Rn++ , and f is strictly concave on
Rn++ . Since f is continuous, it is concave on the closure of Rn++ , i.e., on Rn+ .
We show that ∆ ¯ k = ( 1)k + ( 1)k−1 (a1 + · · · + ak ) by induction on k , using linearity
of the determinant on the last column and column operations on the subsequent matrix:
a1 1 a2 ··· ak−1 ak
a
1 a2 1 ··· ak−1 ak
. .. .. .. ..
¯ k = det .
∆ . . . . .
···
a1 a2 ak−1 1 ak
a1 a2 ··· ak−1 ak 1
a1 1 a2 ··· ak−1
ak a1 1 a2 ··· ak−1 0
a
1 a2 1 ··· ak−1
ak
a1
a2 1 ··· ak−1 0
. .. ....
.. + det .. .. .. .. ..
= det ..
. . . . . . . . .
··· ···
a1 a2 ak−1 1 ak a1 a2 ak−1 1 0
a1 a2 ··· ak−1 ak a1 a2 ··· ak−1 1
1 0 ··· 0 1
0
1 ··· 0 1
¯ k = ak .. .. .. .. .. ¯
∆ . . . . . − ∆k−1
0 ···
0 1 1
0 0 ··· 0 1
k−1
ak − ( 1)k−1 + ( 1)k−2 (a1 + · · · + ak−1 )
= ( 1)
= ( 1)k + ( 1)k−1 (a1 + · · · + ak ).
This proves the claim by induction and completes the proof.
following Theorem 3.32.) Rather than focusing on quasi-convex and pseudo-convex functions,
we consider rescalings of convex functions. The next theorem shows that constraints that are
rescaled convex functions have a convex feasible set and so are quasi-convex. Then Corol-
lary 3.26 shows that we can use more general exponents in Cobb-Douglas functions than the
preceding two proposition allowed.
Definition. A function g : D ⊂ Rn → R is a rescaling of ĝ : D → R provided that there
is an increasing function φ : R → R such that g(x) = φ ◦ ĝ(x). Note that since φ has an
inverse, ĝ(x) = φ - 1 ◦ g(x).
We say that φ is a C 1 rescaling provided that φ is C 1 and φ0 (y) > 0 for all y ∈ R.
If g : D ⊂ Rn → R is a rescaling of a convex (resp. concave) function ĝ : D → R, then
we say that g(x) is a rescaled convex function (resp. rescaled concave function). Similarly, if
g : D ⊂ Rn → R is a C 1 rescaling of a convex (resp. concave) function ĝ : D → R, then
we say that g(x) is a C 1 rescaled convex function (resp. C 1 rescaled concave function).
Theorem 3.25 (Rescaling). Assume that g : D ⊂ Rn → R is a rescaling of a convex function
ĝ : D → R, g(x) = φ ◦ ĝ(x). Then F g,b is convex for any b ∈ R, and g is quasi-convex.
The proof follows because F = { x ∈ D : ĝ(x) ≤ φ - 1 (b) } is convex.
g,b
Example 3.27. The Cobb-Douglas function f (x, y) = xy is a rescaled concave function, but
not concave. See Figure 3.4.4.
xy
1 1
x3 y3
y
y
x x
(a) Non-concave (b) Concave
Example 3.28. The function f (x) = x6 − 2.9 x4 + 3 x2 has its graph given in Figure 3.4.5.
The derivative is f 0 (x) = x[6 x4 − 11.6 x2 + 6], and f (x) has a single critical point at
x = 0. The second derivative f 00 (x) = 30 x4 − 34.8 x2 + 6 has zeroes at ±0.459 and
±0.974, and f 00 (x) < 0 for x ∈ [ 0.974, 0.459] ∪ [0.459, 0.974]. Thus, f (x) is not
convex. The function fˆ(x) = x6 is convex. Using the inverse of f (x) for positive values of
x, φ(y) = [f - 1 (y)] satisfies φ ◦ f (x) = fˆ(x) and is a rescaling of f (x) to fˆ(x). Thus,
6
f (x) x6
x x
1.3 1.3 1.3 1.3
Figure 3.4.5. A rescaled convex function of Example 3.28
The following corollary of the KKT Theorem 3.19 allows us to rescale the objective func-
tion as well as the constraints. Thus the general Cobb-Douglas functions considered in the last
proposition can be used as objective functions even though they are not concave, and the KKT
Theorem 3.19 does not apply.
Corollary 3.29 (KKT for Rescaled Functions). Assume that gi : D ⊂ Rn → R are C 1 for
1 ≤ i ≤ m, each of the constraints gi is a C 1 rescaled convex function, and
x∗ ∈ F g,b = { x ∈ D : gi (x) ≤ bi for 1 ≤ i ≤ m }.
a. Assume that f : Rn → R is a C 1 rescaled concave function.
i. If (x∗ , λ∗ ) satisfies KKT-1,2,3 with all the λ∗i ≥ 0, then f has a maximum on
F g,b at x∗ .
ii. If f has a maximum on F g,b at x∗ , and F g,b satisfies the Slater condition,
then there exist λ∗ = (λ∗1 , . . . , λ∗m ) such that (x∗ , λ∗ ) satisfies conditions KKT-
1,2,3 with all the λ∗i ≥ 0.
b. Assume that f is a C 1 rescaled convex function.
i. If (x∗ , λ∗ ) satisfies KKT-1,2,30 with all the λ∗i ≤ 0, then f has a minimum on
F g,b at x∗ .
ii. If f has a minimum on F g,b at x∗ , and F g,b satisfies the Slater condition,
then there exist λ∗ = (λ∗1 , . . . , λ∗m ) such that (x∗ , λ∗ ) satisfies conditions KKT-
1,2,30 with all the λ∗i ≤ 0.
Proof. Assume ĝi (x) = φi ◦ gi (x) with ĝ a convex C 1 function, φi : R → R C 1 , and
φ0i (bi ) > 0 for all bi ∈ R. Similarly, assume fˆ, fˆ(x) = T ◦ f (x) with fˆ a concave (respect.
convex) C 1 function, T : R → R C 1 , and T 0 (y) > 0 for all y = f (x) with x ∈ F g,b .
Let b0i = φi (bi ). If gi (x∗ ) = bi is tight, then ĝi (x∗ ) = b0i ,
Dĝi (x∗ ) = φ0 (b0 )Dgi (x∗ )
i i and Dfˆ(x∗ ) = T 0 (f (x∗ )) Dfˆ(x∗ ).
(a.i) F g,b = { x ∈ U : gi (x) ≤ bi } = { x ∈ U : gˆi (x) ≤ b0i } is convex. If f satisfies
KKT-1. then
Dfˆ(x∗ ) = T 0 (f (x∗ )) Df (x∗ ) = T 0 (f (x∗ ))
X
λi Dgi (x∗ ),
i
where we have use the fact that λi = 0 unless gi (x∗ ) = bi . Since, T 0 (f (x∗ )) > 0 and
φ0i (bi ) > 0 for all effective i, we get that conditions KKT-1.2 hold for f and the gi with
0
multipliers λi Ti (bi )/T 0 (f (x∗ )) > 0.
3.4. Extrema with Inequality Constraints: Sufficient Conditions 91
which have a solution λ1 = λ2 = 1/3 > 0. (It can be checked that there is no other solution
of the first order KKT conditions on R3++ ∩ F , i.e., when one or two of the constraints are
effective, but this is not necessary.)
We have shown that f is a rescaling of a concave function, all the gi are all convex
functions on R3+ , and that p∗ = (1, 1, 1) multipliers λ1 = λ2 = 1/3 > 0 satisfy satisfies
conditions KKT-1,2,3. By the Karush-Kuhn-Tucker Theorem under Convexity, f must have a
maximum on R3++ ∩ F at p∗ . But as we remarked earlier, this must be the maximum on all
of F since f (x, y, z) = 0 when one or more variable is zero.
Remark. Although we do not need it, the feasible set F satisfies the Slater condition since
the point (0.5, 0.5, 0.5) in D = { (x, y, z) ∈ R3+ : gi (x, y, z) ≤ 0 for 1 ≤ i ≤ 5 } has all the
gi positive, the constraint functions satisfy the Slater condition.
We could also check that the constraint qualification is indeed satisfied on all of F . How-
ever, if we add another constraint, x + y + z − 3 ≤ 0, then p∗ can be shown to be a solution of
KKT-1,2,3 in R3++ . By the Karush-Kuhn-Tucker Theorem under Convexity, p∗ is a maximizer.
For this example, there are three effective constraints at p∗ , but the rank is still 2. Therefore,
this system does not satisfy the constraint qualification.
92 3. Constrained Extrema
The next theorem shows that for C 1 functions, a function is convex if and only if a condition
on the relationship between the function and its tangent plane is satisfied. This first order
condition is not used to check convexity or concavity of a function but is used in the succeeding
theorem to give first order conditions for a maximizer.
Theorem 3.32. Let D ⊂ Rn be open and convex. Assume f : D → R is C 1 .
a. f is convex iff f (x) ≥ f (p) + Df (p)(x − p) for all x, p ∈ D (the graph of f (x)
lies above the tangent plane at p.)
b. f is concave iff f (x) ≤ f (p) + Df (p)(x − p) for all x, p ∈ D (the graph of f (x)
lies below the tangent plane at p.)
Proof. We consider part (a) only since (b) is similar. (⇒) Assume that f is convex, x, p ∈ D ,
and xt = p + t (x − p) = (1 − t)p + tx. Then, f (xt ) ≤ (1 − t)f (p) + tf (x), so
f (xt ) − f (p)
Df (p)(x − p) = lim
t→0+ t
(1 − t) f (p) + t f (x) − f (p)
≤ lim
t→0+ t
t [f (x) − f (p)]
= lim
t→0+ t
= f (x) − f (p).
(⇐) Assume that f satisfies the first derivative condition of the theorem. Let xt = (1 − t) p +
t x and wt = x − xt = (1 − t)(x − p), so p − xt = − t/1 − t wt . Then,
Multiplying the first inequality by (1 − t), the second by t, and adding these two inequalities
together, we get
(1 − t) f (p) + t f (x) − f (xt ) ≥ 0, or
(1 − t) f (p) + t f (x) ≥ f (xt ).
This proves that the function is convex.
The first derivative condition of the previous theorem inspires the following definition.
Definition. A function f is pseudo-concave on a set D provided that if f (x) > f (p) for
x, p ∈ D , then Df (p)(x − p) > 0. A direct check shows that a C 1 rescaling of a concave
function is pseudo-concave.
The following generalizes the condition of being a critical point for a point that is on the
boundary.
Theorem 3.33. Assume that f : F ⊂ Rn → R is concave or pseudo-concave and x∗ ∈ F .
Then, x∗ maximizes f on F iff Df (x∗ )v ≤ 0 for all vectors v that point into F at x∗ .
F ), f attains a maximum at x∗ on F iff x∗ is a critical
In particular for x∗ ∈ int(F
point of f .
Proof. Since the directional derivatives exist for a concave function, we do not need to assume
that the function is C 1 .
(⇒) Assume x∗ maximizes f over F . Take a vector v that points into F at x∗ . Then
for small t ≥ 0, x∗ + tv ∈ F and f (x∗ + tv) ≤ f (x∗ ). Taking the derivative with respect
to t, we get that
f (x∗ + tv) − f (x∗ )
Df (x∗ )v = lim ≤ 0.
t→0+ t
This proves the desired inequality on these directional derivatives.
(⇐) Now, assume that Df (x∗ )v ≤ 0 for all vectors v that point into F at x∗ . If
f does not have a maximum at x∗ , there exists a point z ∈ F such that f (z) > f (x∗ ).
Let v = z − x∗ . If f is pseudo-concave, this leads to the contradiction that Df (x∗ )v =
Df (x∗ )(z−x∗ ) > 0. If f is concave, then v points into F at x∗ . Also, for xt = x∗ +t v =
(1 − t) x∗ + t z,
f (xt ) ≥ (1 − t) f (x∗ ) + t f (z) = f (x∗ ) + t [f (z) − f (x∗ )] so
∗ ∗
f (xt ) − f (x ) t [f (z) − f (x )]
Df (x∗ )v = lim ≥ lim = f (z) − f (x∗ ) > 0.
t→0+ t t→0+ t
This contradicts the inequality on the directional derivatives and shows that f must have a
maximum at x∗ .
Since all the λ∗i ≥ 0, λ∗i Dgi (x∗ )v ≤ 0. By the first order condition KKT-1
P
i
X
Df (x∗ )v = λ∗i Dgi (x∗ )v ≤ 0.
i
Since the directional derivative is negative for any vector pointing into F g,b , f has a maxi-
mum at x∗ by Theorem 3.33.
We claim that
X = { (w, z) ∈ R × Rm : w ≤ f (x) & z ≤ b − g(x) for some x ∈ Rn }
is also convex. Note that X is defined using Rn and not F g,b . Let (w0 , z0 ), (w1 , z1 ) ∈ X
have corresponding points x0 , x1 ∈ Rn . Set wt = (1 − t)w0 + tw1 , zt = (1 − t)z0 + tz1 ,
and xt = (1 − t)x0 + tx1 . Then,
f (xt ) ≥ (1 − t) f (x0 ) + t f (x1 ) ≥ (1 − t)w0 + tw1 = wt
g(xt ) ≤ (1 − t) g(x0 ) + t g(x1 ) ≤ (1 − t)(b − z0 ) + t(b − z1 ) = b − zt .
This shows that (wt , zt ) ∈ X and X is convex.
We next claim that X ∩ Y = ∅. Assume that there exists a (w, z) ∈ X ∩ Y . Because
the point is in Y , z 0 and w > f (x∗ ). Because the pair is in X , there exists x ∈ Rn
with w ≤ f (x) and z ≤ b − g(x). The n b − g(x) ≥ z 0, so x ∈ F g,b . Combining,
f (x) ≥ w > f (x∗ ) for x ∈ F g,b , which contradicts the fact that f has a maximum at x∗ on
F g,b . This contradiction shows that they have empty intersection.
By the separation theorem for disjoint convex sets (Theorem 1.68 in [14]), there exist a
nonzero vector (p, q) ∈ R × Rm such that
pw + q · z ≤ pu + q · v for all (w, z) ∈ X , (u, v) ∈ Y . (7)
We claim that (p, q) ≥ 0. Assume one of the components is negative. Fix (w, z) ∈ X
with corresponding point x. By taking the corresponding coordinate of (u, v) ∈ Y large and
positive, p u + q · v can be made arbitrarily negative, contradicting the separation inequality
(7). Thus, (p, q) ≥ 0.
Taking any x ∈ Rn , setting w = f (x) and z = b − g(x), and letting (u, v) ∈ Y
converge to (f (x∗ ), 0), we get that
p f (x) + q · (b − g(x)) ≤ p f (x∗ ) for all x ∈ Rn . (8)
3.4. Extrema with Inequality Constraints: Sufficient Conditions 95
For x = x∗ , this gives that λ∗i (bi − gi (x∗ )) ≤ 0. But λ∗i ≥ 0 and bi − gi (x∗ ) ≥ 0, so
P
i
λ∗i (bi − gi (x∗ )) = 0 for 1 ≤ i ≤ m, or condition KKT-2.
∗ ∗ ∗
λ , x ) = f (x ), so substituting into (9),
Also L(λ
λ∗ , x) ≤ L(λ
L(λ λ ∗ , x∗ ) for all x ∈ Rn .
Thus with λ ∗ fixed, L(λ
λ∗ , x) has an interior maximum at x∗ and
X
0 = Dx L(λ λ∗ , x∗ ) = Df (x∗ ) − λ∗i Dgi (x∗ ),
i
or condition KKT-1.
3.4. Exercises
3.4.1. Which of the following functions are convex, concave, or neither? Why?
a. f (x, y) = 2x2 − 4xy − 7x + 5y .
b. f (x, y) = xe−x−5y .
c. f (x, y, z) = −x2 + 2xy − 3y 2 + 9x − 7y .
d. f (x, y, z) = 2x2 + y 2 + 2z 2 + xy − 3xz .
e. f (x, y, z) = −2x2 − 3y 2 − 2z 2 + 2xy + 3xz + yz .
3.4.2. Let f : Rn++ → R be defined by f (x1 , . . . , xn ) = ln(xα1 1 · · · xαnn ), where all the
αi > 0. Is f convex or concave?
3.4.3. Let D be a convex set and h : D → R a concave function.
a. Show that
F = { (x, y) ∈ D : 0 ≤ x, 0 ≤ y, 0 ≤ h(x, y) }
is convex.
b. Assume f : D → R is convex and f (y) < f (x) for x, y ∈ D . Let
xt = (1 − t)x + ty.
Show that f (xt ) < f (x) for 0 < t ≤ 1.
3.4.4. Consider the following problem:
Minimize: f (x1 , x2 , x3 ) = 2 x21 + 5 x22 + 3 x23 − 2 x1 x2 − 2 x2 x3
Subject to: 25 ≤ 4 x1 + 6 x2 + x3
xi ≥ 0 for i = 1, 2, 3.
a. What are the KKT-1,2,30 equations for this problem to have a minimum? Be
sure an list all the equations that must be satisfied. Then, solve these equations
for a solution (x∗1 , x∗2 , x∗3 ).
b. Explain why the objective function and constraints satisfy the assumptions for
a minimum of the Karush-Kuhn-Tucker Theorem under Convexity.
Note that the function f is a positive definite quadratic function.
c. Explain why (x∗1 , x∗2 , x∗3 ) must be a minimizer, including explaining how the
theorems apply and what other conditions need to be satisfied.
96 3. Constrained Extrema
a. Write down the KKT-1,2,30 equations for a minimum on the feasible set.
b. Find a solution to these equations where both constraints are tight.
c. Why must the solution found in part (b) be a minimizer of f on the feasible
set?
3.4.9. A firm produces a single output q with two inputs x and y , with production function
q = xy . The output must be at least q0 units, xy ≥ q0 > 0. The firm is obligated
to use at least one unit of x, x ≥ 1. The prices of x and y are p and 1 respectively.
Assume that the firm wants to minimize the cost of the inputs f (x, y) = px + y .
a. Is the feasible set closed? Compact? Convex?
b. Write down the KKT-1,2,30 equation for a minimum.
c. Find the minimizer by solving the KKT-1,2,30 equations.
Hints: (i) Note that one of the equations for KKT-1 implies that the multiplier
for 0 ≥ q0 −xy is nonzero and so this constraint must be effective at a solution.
(ii) If 0 ≥ 1 − x is tight, then q ≤ p because both multiplier must be less
than or equal to zero.
(iii) If the multiplier for 0 ≥ 1 − x is zero, then q ≥ p because x ≥ 1.
3.4.11. Let T > 1 be a fixed integer and 0 < δ < 1. Consider the following maximization
problem.
>
X 1
Maximize: δ j xj2 ,
j=1
n
j=1 xj ≤ 1,
P
Subject to:
xj ≥ 0 for j = 1, . . . T .
a. Write down the KKT-1,2,3 equations.
b. Consider the case when all the xj > 0. Solve the KKT-1,2,3 equations for a
solution. Hint: Why must the multiplier be nonzero, so the constraint tight?
How must xj be related to x1 ? Using the tight constraint, what must x1
equal?
c. Why must the solution found in part (b) be a maximizer?
and
n n
d2
X d ∂f 0
X ∂f
(r(0)) ri00 (0)
f (r(t)) = (r(t)) ri (0) +
dt2
t=0 i=1
dt ∂xi
t=0 i=1
∂xi
∂ 2g`
X
= (x∗ ) ri0 (0)rj0 (0) + D(g` )(x∗ )r00 (0), so
i=1,...,n
∂xj ∂x i
j=1,...,n
λ∗` D(g` )(x∗ )r00 (0) = −λ∗` (r0 (0))> D2 (g` )(x∗ )r0 (0).
Substituting this equality into the expression for the second derivative of f (r(t)),
" k
#
d2
X
> 2 ∗ ∗ 2 ∗
f (r(t)) = v D f (x ) − λ` D g` (x ) v,
dt2 t=0 `=1
The next theorem uses the above lemma to derive conditions for local maxima and minima
in terms of the second derivative of the Lagrangian on the null space null (Dg(x∗ )).
Theorem 3.35. Assume f, gi : Rn → R are C 2 for 1 ≤ i ≤ k . Assume that x∗ ∈ Rn and
λ ∗ = (λ∗1 . . . . , λ∗k ) satisfy the first-order conditions of the Theorem of Lagrange with
rank(Dg(x∗ )) = k . Set Dx2L∗ = Dx2L(λ λ∗ , x∗ ) = D 2f (x∗ ) − k`=1 λ∗` D 2g` (x∗ ).
P
a. If f has a local maximum on g - 1 (b) at x∗ , then v> Dx2L∗ v ≤ 0 for all v ∈ null(Dg(x∗ )).
b. If f has a local minimum on g - 1 (b) at x∗ , then v> D 2L∗ v ≥ 0 for all v ∈ null(Dg(x∗ )).
x
c. If v> Dx2L∗ v < 0 for all v ∈ null(Dg(x∗ )) r {0}, then f has a strict local maximum
on g - 1 (b) at x∗ .
d. If v> Dx2L∗ v > 0 for all v ∈ null(Dg(x∗ )) r {0}, then f has a strict local minimum
on g - 1 (b) at x∗ .
e. If v> Dx2L∗ v is positive for some vector v ∈ null(Dg(x∗ )) and negative for another
such vector, then f has is neither a local maximum nor a local minimum of f on g - 1 (b)
at x∗ .
100 3. Constrained Extrema
Proof. (b) We consider the case of minima. (The case of maximum just reverses the direction
of the inequality.) Lemma 3.34 shows that
d2
= v> Dx2L∗ v,
2
f (r(t))
dt
t=0
d2
2
f (r(t)) ≥0
dt t=0
for any curves r(t) in g - 1 (b) with r(0) = x∗ . Thus, v> Dx2 L∗ v ≥ 0 for any vector v
in Tg (x∗ ). But we had by Proposition 3.8 that Tg (x∗ ) = null(Dg(x∗ )), so part (b) of the
theorem is proved.
(d) If v> Dx2 L∗ v > 0 for all vectors v 6= 0 in null(Dg(x∗ )), then by Proposition 3.8 and
Lemma 3.34,
d2
= r0 (0)> Dx2 L∗ r0 (0) > 0
2
f (r(t))
dt
t=0
for any curves r(t) in g - 1 (b) with r(0) = x∗ and r0 (0) 6= 0. This latter condition implies that
x∗ is a local minimizer on g - 1 (b).
For part (e), if v> Dx2 L∗ v is both positive and negative, then there are some curves where
the value of f is greater than at x∗ and others on which the value is less.
The preceding theorem shows that we need to consider the quadratic form x> Dx2 L∗ x on
the null space null(Dg(x∗ )). The next theorem shows that this restricted quadratic form can
λ ∗ , x∗ ) .
be shown to be positive or negative definite by determinants of a submatrices of DL(λ
λ, x) = f (x)+ 1≤i≤k λi (bi − gi (x)) be the Lagrangian. The derivative
P
Definition. Let L(λ
of L with respect to all its variables is
!
∗
0 k Dg(x )
Hn = Dx2 L(λ λ ∗ , x∗ ) = ,
Dg(x∗ )> Dx2 L∗
which is obtained by “bordering” the n × n matrix Dx2 L∗ with the k × n matrix Dg(x∗ ).
We assume that the rank of Dg(x∗ ) is k < n, so there is a k × k submatrix with
nonzero determinant. To form the correct submatrices, we need to assume that the the variables
have been rearranged so the first k columns suffice and this k × k submatrix has nonzero
determinant. For 1 ≤ ` ≤ n, the bordered Hessians H` are (k + `) × (k + `) submatrices
of Hn = DL(λ λ∗ , x∗ ) given as follows:
∂g1 ∂g1
0 ... 0 ∂x1 . . . ∂x`
. .. .. ..
. .. ..
. . . . . .
∂gk ∂gk
0 ... 0 . . .
∂x1 ∂x`
H` = ∂g1 . . . ∂gk ∂ 2 L∗ 2 ∗ .
. . . ∂x1L
∂
∂x1 ∂x1 ∂x21 ∂x`
.. .. .. .. .. ..
. . . . . .
2 ∗ 2 ∗
∂g1 ∂gk ∂ L ∂ L
∂x` . . . ∂x` ∂x` ∂x1 . . . 2
∂x`
3.5. Second-Order Conditions for Extrema of Constrained Functions 101
Theorem 3.36. Assume that f, gi : Rn → R areC 2 for 1 ≤ i ≤ k and (λλ∗ , x∗ ) is a critical
∂gi ∗
point of L satisfying (LM) with det (x ) 6= 0 and bordered Hessians H` .
∂xj 1≤i,j≤k
Let w = (x1 , . . . , xk )> be the first k coordinates and z = (xk+1 , . . . , xn )> be the last n − k
coordinates. Then the null space can be expressed in terms of the z variables:
0 = B1 w + B2 z
w = −B - 1 B z = Jz
1 2
" #
A11 A12
where J = −B1- 1 B2 . Partitioning Dx2 L∗ = A = into blocks, where A11 is
A>12 A22
k × k , A12 is k × (n − k), and A22 is (n − k) × (n − k), the quadratic form on the null space
has the following symmetric matrix E:
" #" # " #
h i A A12 J h i A J+A
> 11 11 12
E= J I = J> I
A>
12 A22 I A>
12 J + A 22
Here the matrix E induces the quadratic form on the null space as we showed above. Since the
determinant of the change of basis matrix is one, this change of basis preserves the determinant
of Hn , and also the determinants of H` for 2 + 1 ≤ ` ≤ n.
102 3. Constrained Extrema
0 B1 0 B>1 A11 C12
> k
det(Hn ) = det B1 A11 C12 = ( 1) det 0 B1 0
0 C>
12 E 0 C>12 E
" #
B1 0
= ( 1)k
det(B>
1 ) det
C>
12 E
This calculation carries over to all the H` , ( 1)k det(H` ) = det(B1 )2 det(E`−k ). Therefore,
we can use the signs of the determinants of the H` for k + 1 ≤ ` ≤ n to check the signs of the
determinants of the principal submatrices Ej−k with size ranging from 1 to n − k .
The quadratic form Q for A is positive definite on the null space iff the quadratic form for
E is positive definite iff
For the negative definite case, the quadratic form Q for A is negative definite on the null
space iff the quadratic form for E is negative definite iff
Example 3.37. We check the second order conditions for the critical points of Example 3.9:
f (x, y, z) = z on the set given by x + y + z = 12 and z = x2 + y 2 .
∗ ∗ ∗ ∗ ∗
We considered this example earlier andfound the two critical points (λ , µ , x , y , z ) =
4/5, 1/5, 2, 2, 8, and 6/5, 1/5, 3, 3, 18 .
For this example, n = 3 and k = 2, so n − k = 1 determinant. The Lagrangian is
L = z + λ(12 − x − y − z) − µ(z − x2 − y 2 ). The bordered Hessian is
0 0 1 1 1
0 0 2x 2y 1
H3 = D2 L = 1 2x µ2 0 0 .
1 2y 0 µ2 0
1 1 0 0 0
3.5. Second-Order Conditions for Extrema of Constrained Functions 103
0 0 1 1 1 1 1 0 0 0
0 0 4 4 1 0 0 4 4 1
2 2
det(H3 ) = det 1 4 0 0= det 1 4 0 0
5 5
1 2 2
4 0 5
0 1 4 0 5
0
1 1 0 0 0 0 0 1 1 1
1 1 0 0 0 1 1 0 0 0
0 2
0 4 4 1
0
5 0 5
0
2 2
= det 0 5 0 0 = det 0 5 0 0
5 5
0 2
5 0 5
0 0 0 4 4 1
0 0 1 1 1 0 0 1 1 1
1 1 0 0 0 1 1 0 0 0
0 2 2
5 0 5
0
0
5 0 5
0
2 2 2 2
= det 0 0 0 = det 0 0 0
5 5 5 5
0 0 4 4 1 0 0 0 8 1
0 0 1 1 1 0 0 0 2 1
1 1 0 0 0
0 2
5 0 0
5 2 5
2 2
= det 0 0 0 = 1( 5) ( 8) = 20 > 0.
5 5
0
5 4
0 0 8 1
0 0 0 0 54
Since ( 1)k = ( 1)2 = 1i, k+1 = n = 3, and ( 1)k det(D2 L∗ ) > 0, (λ∗ , µ∗ , x∗ , y ∗ , z ∗ ) =
4/5, 1/5, 2, 2, 8, is a local minimum.
At (λ∗ , µ∗ , x∗ , y ∗ , z ∗ ) = 6/5, 1/5, 3, 3, 18 ,
0 0 1 1 1 1 1 0 0 0
0 0 6 6 1 0 0 6 6 1
2 2
det(H3 ) = det 1 6 0 0= det 1 6 0 0
5 5
1 2 2
6 0 5
0 1 6 0 5
0
1 1 0 0 0 0 0 1 1 1
1 1 0 0 0 1 1 0 0 0
0 2
0 6 6 1
0
5 0 5
0
2 2
= det 0 5 0 0 = det 0 5 0 0
5 5
0 2
5 0 5
0 0 0 6 6 1
0 0 1 1 1 0 0 1 1 1
1 1 0 0 0 1 1 0 0 0
0 2 2
5 0 5
0
0
5 0 5
0
2 2 2 2
= det 0 0 0 = det 0 0 0
5 5 5 5
0 0 6 6 1 0 0 0 12 1
0 0 1 1 1 0 0 0 2 1
104 3. Constrained Extrema
1 1 0 0 0
0 2
5 0 5
0
2 2
det(H3 ) = det 0 0 0 = 20 < 0.
5 5
0 0 0 12 1
5
0 0 0 0 6
local maximum.
These answers are compatible with the values of f (x, y, z) at the two critical points:
on the constraint set, (λ∗ , µ∗ , x∗ , y ∗ , z ∗ ) = 4/5, 1/5, 2, 2, 8, is a global minimum, and
(λ∗ , µ∗ , x∗ , y ∗ , z ∗ ) = 6/5, 1/5, 3, 3, 18, is a global maximum.
Example 3.38. Consider the problem of finding the extreme point of f (x, y, z) = x2 +y 2 +z 2
on 2 = z − xy . The points that satisfy the first order conditions for the method of Lagrange are
Since n = 3 and k = 1 and both ( 1)k det(H3 ) < 0 and ( 1)n det(H3 ) < 0, this fails the
test for either a local minimum or a local maximum, so the point is not a local extremum.
The calculation at the other two points is similar, so we consider the point (λ∗ , x∗ , y ∗ , z ∗ ) =
(2, 1, 1, 1). The partial derivative gx (1, 1, 1) = ( 1) 6= 0, so
0 y x 0 1 1
H2 = y 2 λ = 1 2 2 .
x λ 2 1 2 2
3.5. Exercises
3.5.1. Find the points satisfying the first order conditions for a constrained extrema and
then apply the second order test to determine whether they are local maximum or
local minimum.
a. f (x, y, z) = xyz and g(x, y, z) = 2x + 3y + z = 6.
b. f (x, y, z) = 2x + y 2 − z 2 , g1 (x, y, z) = x − 2y = 0, and g2 (x, y, z) =
x + z = 0.
106 3. Constrained Extrema
Dynamic Programming
This chapter focuses on maximizing over more than one time period where unused resources
are carried forward to the next time period. The easier case considers a finite number of time
periods, which is said to have a finite horizon. This type of problem can be solved as a Karush-
Kuhn-Tucker problem with many variables. However, a simpler method is to solve the problem
recursively, one period at a time. This latter approach is called dynamic programming. The
harder case considers an infinite number of time periods, or infinite horizon. Although the
equation to be solved looks similar to the finite-horizon case, the infinite-horizon case requires
solving an equation for a function rather than just the value of the function in some Rn . Also,
proving that a solution exists is much harder.
In both the finite-horizon and infinite-horizon dynamic programs, the function maximized
is restricted to a domain that usually depends on the state, i.e., depends on a parameter. Such
an assignment of a set by a parameter is called a correspondence. For each parameter, there
can be more than one maximizer, so the set of maximizers in general is also a correspondence
and not a function. We start the chapter by considering correspondences and various types of
continuity as the parameter varies. The Parametric Maximization Theorem indicates how the
set of maximizers and the maximal value varies with the parameter. This theorem is at the heart
of the solution method of dynamic programming problems.
107
108 4. Dynamic Programming
2 2
1 1
s s
0 0
0.5 C 2 1.5 0.5 C 3 1.5
Figure 4.1.2. Graphs of correspondences in Example 4.2
has a single point for each parameter s ∈ R and is neither bounded nor locally bounded near
s = 0. See Figure 4.1.3. This correspondence is (i) closed-graphed and (ii) compact-valued
since it is a single point for each s.
4.1. Parametric Maximization and Correspondences 109
Our main use of correspondences occurs in maximization problems with a parameter that
have a general setup is as follows. As s varies over a parameter space S, assume that the
feasible set F (s) ⊂ Rn can vary with the parameter and is compact for each s. Thus, F :
S → P (Rn ) is a compact-valued correspondence. Assume that f : Gr(F ) ⊂ S × Rn → R
is continuous. For each s ∈ S, denote the maximal value of f (s, x) subject to x ∈ F (s) by
f ∗ (s) = max{ f (s, x) : x ∈ F (s) }
and the set of feasible maximizers by
F ∗ (s) = { x ∈ F (s) : f (s, x) = f ∗ (s) }.
Since F (s) and F ∗ (s) are sets for each s, the assignments from s to F (s) or F ∗ (s)
are examples of correspondences. On the other hand, f ∗ (s) is a real-valued function. The
following examples illustrate how f ∗ (s) and F ∗ (s) can vary with s.
Example 4.4. Consider the example with f1 (s, x) = (s − 1/3) x for s ∈ [0, 1] = S1 and
x ∈ [0, 1] = F 1 (s). (The feasible set does not vary with s in this example.) Since
1
∂f1 < 0 for s < /3 and for all x ∈ [0, 1],
(s, x) = (s − 1/3) ≡ 0 for s = 1/3 and for all x ∈ [0, 1],
∂x
> 0 for s > 1/3 and for all x ∈ [0, 1],
{0} for s < 1/3 (
for s ≤ 1/3
0
F 1 (s) = [0, 1] for s = /3
∗ 1 ∗
and f1 (s) =
s − /3 for s > 1/3.
1
{1} for s > 1/3.
See Figure 4.1.4. The set-valued correspondence F ∗1 (s) changes from {0} to {1} as s
crosses 1/3, while the maximal value f1∗ (s) is continuous. Also, note that F ∗1 (s) is
(i) bounded, (ii) compact-valued, and (iii) closed-graphed. In a strategic game from game
theory, if f1 (s, x) is your payoff for mixed strategies of s by other player and x by you, then
F ∗1 (s) is called the best response correspondence.
x x
s s
F ∗1 (s) f1∗ (s)
Figure 4.1.4. Graphs of maximizer set and maximal-value function for Example 4.4
110 4. Dynamic Programming
x−
s x+
s x−
s x+
s x−
s x+
s
Rather than calculate f2 (s, x± s ) as a function of s, we determine the sign of its (total)
derivative with respect to s at these points:
±
d ± dxs
3 3
f2 (s, x±
s ) = f 2x (s, xs ) + 31 x±
s = 13 x±
s .
ds ds
This derivative has the same sign as x± s , so
f2 (s, x− +
s ) > f (0, ±1) > f2 (s, xs ) for s < 0,
f2 (s, x−
s ) < f (0, ±1) < f2 (s, x+
s ) for s > 0.
Thus,
−
{xs }
for s < 0
F ∗2 (s) = {x−
0 , x+
0 } = { 1, 1 } for s = 0
+
{xs } for s > 0.
1
1 x+
s
1
4
x−
s 1
Figure 4.1.6 shows the numerically calculated graphs of F ∗2 and f2∗ . The correspondence
F ∗2 switches the maximizer from x− +
s to xs at s = 0 but is (i) compact-valued, (ii) locally
bounded, and (iii) closed-graphed. See Theorem 4.8. Theorem 4.8 also shows that f2∗ must be
continuous as can be seen in Figure 4.1.6(b).
4.1. Parametric Maximization and Correspondences 111
For correspondences, we consider not only continuity but also two weaker properties called
hemicontinuity. The precise conditions involve the neighborhood of a set that we define first.
Definition. For a set A ⊂ Rn , the -neighborhood of A is the set
B(A, ) = { x ∈ Rn : there is a y ∈ A with kx − yk < }.
A B(A, )
A function f (x) is continuous at x0 provided that limx→x0 f (x) = f (x0 ), i.e., for all
> 0, there exists a δ > 0 such that if |x−x0 | < δ , then |f (x)−f (x0 )| < . The conclusion
can be written as f (x) ∈ B(f (x0 ), ). If we apply a similar condition to a correspondence, it
restricts the extent that the correspondence can expand or implode (vacate regions).
Definition. A compact-valued correspondence C : S ⊂ R` → P (X) is upper-hemi-
continuous (uhc) at s0 ∈ S provided that C (s) must remain inside a small neighborhood
of C (s0 ) for small changes of s away from s0 , i.e., any > 0 there exists δ > 0 such that if
s ∈ B(s0 , δ) ∩ S then C (s) ⊂ B(C C (s0 ), ). This restricts the amount the correspondence
can expand or go into new regions.
We say that C is upper-hemicontinuous on S if it is upper-hemicontinuous at each s ∈ S.
Definition. A compact-valued correspondence C : S → P (X) is lower-hemicontinuous
(lhc) at s0 ∈ S provided that for each x0 ∈ C (s0 ), there is a point of C (s) nearby, i.e., for
any point x0 ∈ C (s0 ) and any > 0, there exists δ > 0 such that if s ∈ B(s0 , δ) ∩ S then
B(x0 , ) ∩ C (s) 6= ∅ or x0 ∈ B(C C (s), ). This is equivalent to saying that for any > 0,
there exists δ > 0 such that if s ∈ B(s0 , δ) ∩ S then C (s0 ) ⊂ B(C C (s), ). This restricts the
amount the correspondence can implode or vacate the region C (s0 ).
We say that C is lower-hemicontinuous on S if it is lower-hemicontinuous at each s ∈ S.
Definition. A compact-valued correspondence is said to be continuous provided that it is both
upper- and lower-hemicontinuous. Thus, it is continuous provided that for any > 0, there
exists δ > 0 such that if s ∈ B(s0 , δ) ∩ S then C (s0 ) ⊂ B(C C (s), ) and C (s) ⊂
B(CC (s0 ), ). Thus for small s − s0 and for each point in C (s0 ) or C (s), there is a point
nearby in the other set and the two sets are close to each other.
Although we give examples of lhc correspondences, we mainly use uhc and continuous
correspondences.
Example 4.6. The continuity of the correspondences given in the preceding examples is as
follows: C 1 is continuous; C 2 , F ∗1 , and F ∗2 are upper-hemicontinuous but not lower-
hemicontinuous nor continuous (C 2 at s = 0.5 or 1.5, F ∗1 at s = 1/3, and F ∗2 at s = 0);
C 3 is lower-hemicontinuous but not upper-hemicontinuous nor continuous at s = 0.5 or 1.5;
finally, C 4 is neither lower-hemicontinuous nor upper-hemicontinuous at s = 0,
1
C 4 (s) = * B(CC 4 (0), ) = ( , ) and
s
1 1
C 4 (s), ) =
C 4 (0) = {0} * B (C + , + .
s s
112 4. Dynamic Programming
Remark. The example C 4 given above shows why the correspondence must be locally bounded
in this proposition.
We can now state the principal result of the section which will be used in the rest of the
chapter.
Theorem 4.8 (Parametric Maximization Theorem). Assume that the feasible set F : S →
P (X) is a compact-valued and continuous correspondence and f : Gr(F F) ⊂ S × X → R
is a continuous function.
Then, f ∗ (s) = max{ f (s, x) : x ∈ F (s) } is a continuous function and
F (s) = { x ∈ F (s) : f (s, x) = f ∗ (s) } is a compact-valued upper-hemicontinuous
∗
correspondence on S.
If F ∗ (s) is a single point for each s, then this correspondence is continuous and defines
a continuous function.
Remark. If f (s, x) is strictly concave as a function of x for each s, then each F ∗ (s) is a
single point and so F ∗ is continuous.
Remark. Examples 4.4 and 4.5 both satisfy the assumptions of the theorem and have sets of
maximizers that are upper-hemicontinuous but not continuous.
Example 4.9. Let S = X = R+ , F (s) = [0, s], and h : R+ × R+ → R+ be defined by
1 1
h(s, x) = x 2 + (s − x) 2 .
The function h is continuous and the feasible set F (s) = [0, s] is a continuous correspon-
dence, so the Parametric Maximization Theorem applies. The function h is differentiable for
positive values of x and s, and the critical point satisfies
∂h 1 1
= x - 2 − (s − x) - 2 ,
1 1
0=
∂x 2 2
x - 2 = (s − x) - 2 ,
1 1
s − x = x,
s = 2x,
x̄ = 21 s ∈ [0, s].
4.1. Parametric Maximization and Correspondences 113
∂2h
= 14 x - 2 − 14 (s − x) - 2 < 0 for all x ≥ 0, h(x, s) is a concave function of
3 3
Since
∂x2
x, and x̄ is the unique maximizer on [0, s]. Also note that F ∗ (s) = 12 s is a continuous
1 1 1 1
correspondence and h∗ (s) = (s/2) 2 + (s/2) 2 = 2 2 s 2 is continuous.
B (p, w) = { x ∈ Rn+ : p · x ≤ w },
where the prices pi > 0 for 1 ≤ i ≤ n, the wealth w > 0, and the parameter space is
++ }. Then, B is a continuous, compact-valued correspondence.
S = { (p, w) ∈ Rn+1
The following corollary follows immediately from the previous theorem and the Parametric
Maximization Theorem.
Corollary 4.11. Assume u : Rn+ → R is a continuous utility function. Let v : Rn+1 ++ → R be
the indirect utility function that is the maximal value of utility on the budget constraint,
v(p, w) = u∗ (p, w) = max{ u(x) : x ∈ B (p, w) },
++ → P (R+ ) be the demand correspondence which achieve this maximum,
and let d : Rn+1 n
The functions U1 and U2 are continuous functions on S1 × S2 . Denote the maximal payoff
for the ith player in response to a mixed strategy by of s - i for the other player by
mi (s - i ) = max{ Ui (si , s - i ) : si ∈ Si },
and the best response correspondence for player i by
bi (s - i ) = { si : Ui (si , s - i ) = mi (s - i ) }.
A Nash equilibrium is a pair of mixed strategies (s∗1 , s∗2 ) such that s∗1 ∈ b1 (s∗2 ) is a best
response to s∗2 and s∗2 ∈ b2 (s∗1 ) is a best response to s∗1 , (s∗1 , s∗2 ) ∈ b1 (s∗2 ) × b2 (s∗1 ).
Given s - i , there are a finite number of pure strategies that realize mi (s - i ) and bi (s - i ) is
the set of all convex combinations of these pure strategies. Therefore, the correspondence
(s1 , s2 ) ∈ S1 × S2 7→ b(s1 , s2 ) = b1 (s2 ) × b2 (s1 ) ⊂ S1 × S2
is convex valued. Since U1 and U2 are continuous and the feasible set is the same for all strate-
gies, b1 s2 ) and b2 (s1 ) are each upper-hemicontinuous and so is b(s1 , s2 ). The existence of
a Nash equilibrium in mixed strategies then follows from the Kakutani Fixed Point Theorem.
Theorem 4.14 (Kakutani). Let S be a non-empty, compact, and convex subset of some Eu-
clidean space Rn and C : S → P(S) be a upper-hemicontinuous and convex valued corre-
spondence. Then, there exists a p∗ ∈ S such that p∗ ∈ C (p∗ ).
See [14] for more discussion of the Kakutani Threorem. The book [1] by Arrow and Hahn
has a proof and applications to economics.
4.1. Exercises
4.1.1. Let S = [0, 1] and S = R. For each of the following correspondences
C : S → P (R), (i) draw its graph and (ii) determine whether it is uhc, and/or
continuous. Hint: By Proposition 4.7, the correspondence is upper-hemicontinuous
if and only if it is closed-graphed. (They satisfy the other assumptions of the propo-
sition.)
4.1. Parametric Maximization and Correspondences 115
(
for s ∈ 0, 1/2 ,
[0, 2s]
a. C (s) =
[0, 2 − 2s] for s ∈ 1/2, 1 .
(
[0, 1 − 2s] for s ∈ 0, 1/2 ,
b. C (s) =
[0, 2 − 2s] for s ∈ 1/2, 1 .
(
[0, 1 − 2s] for s ∈ 0, 1/2 ,
c. C (s) =
[0, 2 − 2s] for s ∈ 1/2, 1 .
4.1.4. Let f (s, x) = sin(x) + sx, for s ∈ S = [ 1, 1] and x ∈ F (s) = [0, 3 π].
a. Discuss why the Maximum Theorem applies.
b. Without finding explicit values, sketch the graph of f ∗ and F ∗ . Discuss why
these graphs look as they do and how they satisfy the conclusion of the Maxi-
mum Theorem.
Hint: Draw the graph of f (s, x) as a function of x for three cases of s:
(i) s < 0, (ii) s = 0, and (iii) s > 0.
4.1.5. Let S = [0, 2], X = [0, 1], the function f : S × X → R be defined by f (s, x) =
−(x + s − 1)2 , and the feasible correspondence by F (s) = [0, s]. Find f ∗ (s) and
F ∗ (s) for each value of s. Draw the graphs of f ∗ and F ∗
116 4. Dynamic Programming
1+s 1+s
1 + 2s 1 1 1 + 2s
1 1 − 2s 1 − 2s 1
1−s 1−s
s<0 s>0
a. Sketch the graph of f for s > 0. Sketch the graph of F . Do f and F meet all
the conditions of the Maximum Theorem? If yes, justify your claim. If no, list
all the conditions you believe are violated and explain why you believe each
of them is violated. (Is f continuous at s = 0?)
b. For each s, determine the value of f ∗ (s) and the set F ∗ (s), and sketch the
graphs of f ∗ and F ∗ .
Hint: You may have to split up S into subintervals where F (s) contains the
point that maximizes f (s, x) on [0, 2] and where it does not.
c. Is f ∗ continuous? Is F ∗ (s) 6= ∅ for each s? If so, determine whether F ∗ is
uhc and/or continuous on S.
The value function at the period-T is the maximal payoff at the period-T and is given by
1
VT (wT ) = rT (σT∗ (wT )) = δ T wT2 .
(T 1) Let w = wT 1 be the wealth at period t = T 1. For an action c, the imme-
diate payoff is rT - 1 (c) = δ T - 1 c 2 . The wealth carried forward to the next period is wT =
1
1 1
f (w, c) = k(w −c), with maximal payoff at the period-T of VT (k(w −c)) = δ T k 2 (w −c) 2 .
Thus, the for any choice of consumption 0 ≤ c ≤ w, the sum of immediate payoff and the
optimal pay of the wealth carried forward to the period-T is
w − c = δ 2 k c,
w = (1 + δ 2 k) c,
wT - 1
c̄ = σT∗ - 1 (wT - 1 ) = .
1 + δ 2k
∂ 2 hT - 1
Since < 0 and 0 ≤ c̄ = σT∗ - 1 (wT - 1 ) ≤ wT - 1 , this critical point is a maximum.
∂c2
wT - 1
Thus, the optimal strategy is the choice c̄T - 1 = σT∗ - 1 (wT - 1 ) = . The value function
1 + δ 2k
at period T − 1 is the maximal payoff for periods T − 1 ≤ t ≤ T ,
1
= δ T - 1 c̄ 2 + δ T δ 2 k 2 c̄ 2
1
= δ T - 1 (1 + δ 2 k) c̄ 2
1
T -1 2 wT2 - 1
=δ (1 + δ k) 1
(1 + δ 2 k) 2
1
= δ T - 1 (1 + δ 2 k) 2 wT2 - 1 .
1
payoff for periods t + 1 to T is Vt+1 (k(w − c)). The critical point satisfies
∂ht 1 1 1 1
0= = δ t 21 c− 2 + δ t+1 1 + · · · + δ 2T −2t−2 k T −t−1 2 k 2 1
2
(w − c)− 2 ( 1),
∂c
1
c - 2 = δ k 2 1 + · · · + δ 2T −2t−2 k T −t−1 2 (w − c) - 2 ,
1 1 1
w − c = δ 2 k 1 + · · · + δ 2T −2t−2 k T −t−1 c,
w = 1 + δ 2 k + · · · + δ 2T −2t k T −t c,
w
c̄ = = σt∗ (wt ) ≤ w.
1 + · · · + δ 2T −2t k T −t
∂ 2 ht
Since < 0 and 0 ≤ σt∗ (wt ) ≤ wt , this critical point is the maximizer. Thus, the optimal
∂c2
wt
strategy is c̄t = σt∗ (wt ) = < wt . In the calculation of the maximal
1 + · · · + δ 2T −2t k T −t
value, we use that
1 21 1
[k(wt − c̄)] 2 = δ k 1 + · · · + δ 2T −2t−2 k T −t−1 c̄ 2 .
The maximal payoff from period t onward is
1 12 1
Vt (wt ) = h∗t (wt ) = δ t c̄ 2 + δ t+1 1 + · · · + δ 2T −2t−2 k T −t−1 [k(wt − c̄)] 2
1 21 12 1
= δ t c̄ 2 + δ t+1 1 + · · · + δ 2T −2t−2 k T −t−1 δ k 1 + · · · + δ 2T −2t−2 k T −t−1 c̄ 2
1 1
= δ t c̄ 2 + δ t δ 2 k + · · · + δ 2T −2t k T −t c̄ 2
1
= δ t 1 + δ 2 k + · · · + δ 2T −2t k T −t c̄ 2
δ t (1 + δ 2 k + · · · + δ 2T −2t k T −t ) 12
= 1 wt
(1 + δ 2 k + · · · + δ 2T −2t k T −t ) 2
1 1
= δ t (1 + δ 2 k + · · · + δ 2T −2t k T −t ) 2 wt2 .
This verifies the induction hypothesis for period t, so valid for all T ≥ t ≥ 0.
By induction, for each t with 0 ≤ t ≤ T , the optimal strategy is
wt
c̄t = σt∗ (wt ) = ,
1 + δ k + · · · + δ 2T −2t k T −t
2
∂hT −1 δT - 1 δT β
0= = − β ,
∂c c w −c
wβ − c = δ β c,
wβ = (1 + δ β) c,
wβ
c̄T −1 = σT −1 (w) = ≤ wβ .
1+δβ
The value function is given by the maximal value,
where the constant vT - 1 includes all the terms not involving w only involving the parameters
δ , β , and T .
For the induction hypothesis, assume that
Vj (w) = δ j β 1 + δβ + · · · + δ T −j β T −j ln(w) + vj ,
where vj is a constant involving only the parameters δ , β , T , and j . Assume this is valid for
j = t + 1 and verify it for t. Let
∂ht
The critical point, = 0, satisfies
∂c
δt 1
0= − δ t+1 β [1 + · · · + δ T −t−1 β T −t−1 ] β
c w −c
wβ − c = [δ β + · · · + δ T −t β T −t ] c
wβ = [1 + · · · + δ T −t β T −t ] c
wβ
c̄t = σt∗ (w) = ≤ wβ .
1 + · · · + δ T −t β T −t
The value function is given by the maximal value,
= δ t ln(c̄) + δ t δ β + · · · + δ T −t β T −t ln(c̄) + ln δ β + · · · + δ T −t β T −t
+ vt+1
t T −t T −t T −t T −t
= δ 1 + ··· + δ β β ln(w) − ln(1 + · · · + δ β )
t T −t T −t T −t T −t
+ δ δβ + ··· + δ β ln δ β + · · · + δ β + vt+1
t T −t T −t
= δ β 1 + ··· + δ β ln(w) + vt ,
where the constant vT - 1 includes all the terms involving only the parameters. This proves the
induction step.
4.2. Finite-Horizon Dynamic Programming 121
We show that the total reward for various choice of actions does in fact attain a finite max-
imal value. The problem is to find this maximal value and actions that realize this maximum.
Definition. A Markovian strategy profile is a collection of choice functions σ = (σ0 , . . . , σT )
with σt : S → A so at = σt (st ) ∈ F t (st ) for 0 ≤ t ≤ T . So, each σt is a function
of only st . For a non-Markovian strategy profile, each σt can be a function of (s0 , . . . , st )
and not just st . For a Markovian strategy profile σ and initial state s0 , the actions and states
at all periods are determined by induction as follows: s0 (s0 , σ ) = s0 ; for 0 ≤ t ≤ T , given
st = st (s0 , σ ),
at = at (s0 , σ ) = σt (st ),
rt (s0 , σ ) = rt (st , at ), and
st+1 = st+1 (s0 , σ ) = ft (st , at ).
The total reward for a strategy profile σ and initial state s0 is given by
XT
W (s0 , σ ) = rt (s0 , σ ).
t=0
V (s0 ) for all s0 ∈ S, i.e., it attains the maximal value of the value function.
Theorem 4.18 (FHDP Bellman Equation and Optimal Strategy). If a FHDP satisfies
FH1 – FH4, then the following hold.
a. For 0 ≤ t ≤ T , Vt attains a finite maximal value Vt (st ) < ∞ for each st ∈ S, is
continuous, and satisfies
Vt (s) = max { rt (s, a) + Vt+1 (ft (s, a)) : a ∈ F t (s) } . (10)
We take VT +1 (fT (s, a)) ≡ 0, so the equation for t = T becomes
VT (x) = max { rT (s, a) : a ∈ F T (s) }.
b. There exists a Markovian optimal strategy profile σ ∗ = (σ0∗ , . . . , σT∗ ) such that
W (s0 , σ ∗ ) = V (s0 ) for all s0 .
Remark. Equation (10) is call the Bellman equation and determines the solution method for a
FHDP. First, the strategy a∗T = σT∗ (sT ) is determined that maximizes rT (sT , aT ). This action
determines the value function VT (sT ) = rT (sT , σT∗ (sT )). By backward induction, once the
strategies σj∗ and value functions Vj have been determined for T ≥ j ≥ t + 1, then the strat-
egy a∗t = σt∗ (st ) is determined that maximizes ht (st , at ) = rt (st , at ) + Vt+1 (ft (st , at )) and
the next value function is set equal to the maximal value Vt (st ) = h∗t (st ) = rt (st , σt∗ (st )) +
Vt+1 (ft (st , σt∗ (st ))). By induction, we get back to V (s0 ) = V0 (s0 ).
Proof. We prove the theorem by backward induction, starting at t = T and going down to
t = 0.
For t = T , there is only one term in the definition of the value function and VT (sT ) =
max{ rT (s, a) : a ∈ F T (s) }. The function rT is continuous and F T is continuous and
compact-valued, so by the Parametric Maximization Theorem,
VT (sT ) = max{rT (sT , a) : a ∈ F T (sT ) } < ∞
exists for each sT ∈ S and is continuous, and F ∗T (sT ) = { a ∈ F T (sT ) : f (st , a) =
VT (sT ) } is nonempty upper-hemicontinuous correspondence. Pick σT∗ (sT ) ∈ F ∗T (sT ) for
each sT ∈ S, σT∗ : ST → A. Then, rT (sT , σT∗ (sT )) = W (s, σT∗ ) = VT (s), so σT∗ is an
optimal strategy. This proves the result for t = T .
The following lemma is the induction step.
4.2. Finite-Horizon Dynamic Programming 123
Lemma 4.19 (Induction Step). For 0 ≤ t < T , suppose that the value function Vt+1 is a
continuous function of st+1 and takes a finite value for each st+1 and that the continuation
∗
FHDP starting a period t + 1 admits a Markovian optimal strategy profile (σt+1 , . . . , σT∗ ),
∗ ∗
so that Vt+1 (st+1 ) = W (st+1 , (σt+1 , . . . , σT )). Then the following hold.
a. For each st ∈ S, the value function Vt (st ) attains a finite maximal value, is continu-
ous, and satisfies
Vt (st ) = max{ rt (st , at ) + Vt+1 (ft (st , at )) : at ∈ F t (st ) }.
b. There exists a strategy σt∗ , such that (σt∗ , . . . , σT∗ ) is a Markovian optimal strategy
profile for the continuation FHDP starting at period t, W (st , (σt∗ , . . . , σT∗ )) = Vt (st )
for all st .
Proof. We start by considering the right hand side of the Bellman equation ht (st , at ) =
rt (st , at ) + Vt+1 (ft (st , at )). Since ft and rt are continuous by assumptions of the theo-
rem and Vt+1 is continuous by the induction assumption of the lemma, ht (st , at ) is con-
tinuous. The set correspondence F t is continuous and compact-valued. By the Parametric
Maximization Theorem, the maximal value h∗t (st ) is continuous and set of points that realized
the maximum F ∗ (st ) is a nonempty set. If σt∗ (st ) is any selection of a point in F ∗ (st ), then
h(st , σt∗ (st )) = h∗t (st ) is a Markovian strategy that we show satisfies the lemma.
For any st and any allowable sequence with ai ∈ F (si ) and si+1 = fi (si , ai ) for
i ≥ t,
T
X T
X
ri (si , ai ) = rt (st , at ) + ri (si , ai )
i=t i=t+1
X
T
≤ rt (st , at ) + max ri (s0i , a0i ) : s0t+1 = st+1 , a0i ∈ F i (s0i ),
i=t+1
4.2. Exercises
1
4.2.1. Consider the Consumption-Savings FHDP with δ = 1, rt (w, c) = c 3 , transition
function ft (c, w) = (w − c), F t (wt ) = [0, wt ], and T = 2. Find the value
functions and optimal strategy for each stage.
4.2.2. Consider the Consumption-Savings FHDP with T > 0, rt (w, c) = ln(c) (δ = 1),
transition function ft (w, c) = w − c, and F t (w) = [0, w] for all periods. Find the
value functions and optimal strategy for each stage. Remark: The reward function
equals minus infinity for c = 0, but this just means that it is very undesirable.
Hint: Compute, VT (wT ) and VT −1 (wT −1 ). Then guess the form of Vj , and prove
it is valid by induction.
4.2.3. Consider the Consumption-Savings FHDP with T > 0, r(w, c) = 1 − e - c , transi-
tion function ft (wt , c) = wt − c, and F t (wt ) = [0, wt ]. Find the value functions
and optimal strategy for each stage.
1
4.2.4. Consider the FHDP with δ = 1, rt (s, c) = 1− , transition function ft (s, c) =
1+c
(s − c), F t (st ) = [0, st ], and T ≥ 2.
a. Find the value function and optimal strategy for t = T and T − 1.
(1 + t)2
b. Using backward induction, verify that Vt (s) = 1 + t − . Also,
1+t+s
determine the optimal strategy for each t.
4.2.5. Consider the Consumption-Savings FHDP with T > 0, rt (w, c) = δ t ln(c) with
0 < δ ≤ 1, transition function ft (w, c) = A wβ − c with A > 0 and β > 0, and
F t (w) = [0, A wβ ] for all periods. Verify that the value function is
Vj (w) = δ j ln(w) β(1 + βδ + · · · + β T −j δ T −j ) + vj for correctly chosen constants
vj (that can depend on δ , β , and other parameters). Also find the optimal strategy
for each stage.
Remark: The reward function equals minus infinity for c = 0, but this just means
that small values of c are very undesirable.
Definition. A stationary dynamic programming problem with infinite horizon, SDP, is specified
as follows.
SD1. S ⊂ Rn is the state space with st the state at period-t.
SD2. A ⊂ Rk is the action space with the action at at period-t.
SD3. There is a feasible action correspondence F : S → P (A) that is a compact-valued,
nonempty, continuous correspondence. For each s ∈ S, the set F (s) ⊂ A specifies
the allowable actions.
SD4. There is a continuous transition function f : S × A → S that specifies the state at the
next period in terms of the current state and action taken, st+1 = f (st , at ) for t ≥ 0.
SD5. There is a continuous one-period reward function r : S × A → R that specifies an
immediate reward r(s, a) for an action a taken at state s.
SD6. There is a discount factor δ ∈ (0, 1), so that δ t r(st , at ) is the reward at period-t
discounted back to period-0. This psychological factor represents the impatience for the
reward.
For an initial state s0 , an allowable sequence of actions {at }∞ t=0 is one with at ∈ F (st ) and
states st+1 = f (st , at ) for t ≥ 0. The total reward for such an allowable sequence of actions
is ∞
X
W (s0 , {at }∞
t=0 ) = δ t r(st , at ).
t=0
Remark. The dynamic program is called stationary because the same r, f , and F are used
for every period t. The discount factor allows the possibility that the total reward is finite.
Definition. The value function V : S → R is defined as the supremum of the total reward over
all possible sequences of allowable actions,
V (s0 ) = sup { W (s0 , {at }) : {at } is an allowable sequence } .
Problem: The problem is to find an allowable sequence of actions that realizes the value func-
tion as a maximum, i.e., that maximizes the total reward W (s0 , {at }) for allowable sequences
of actions {at } and st+1 = f (st , at ) t ≥ 0.
Definition. A stationary strategy σ is a choice of an action a = σ(s) ∈ F (s) ⊂ A for each
s ∈ S that is the same for all periods.
Definition. For a SDP, given a stationary strategy σ and an initial state s0 , we can determine
the actions and the states at all periods by induction, and so the total reward: at = at (s0 , σ) =
σ(st ), st+1 = st+1 (s0 , σ) = f (st , at ), and
∞
X
W (s0 , σ) = δ t r(st , at ).
t=0
∗
Definition. An optimal stationary strategy σ is a stationary strategy such that
W (s0 , σ ∗ ) = V (s0 ) for all s0 ∈ S.
The two solution methods that we present for finding an optimal solution strategy both use
the Bellman equation given in the following theorem.
Theorem 4.20 (SDP Bellman Equation). For a SDP, the value function V (s) satisfies the
following equation, called the Bellman equation:
V (s) = sup{ r(s, a) + δ V (f (s, a)) : a ∈ F (s) }. (11)
Remark. Note that for a SDP, the same function V is on both sides of Bellman equation.
Thus, it is not possible to just find the maximum value of the right hand side as is done for
finite-horizon dynamic programming. Instead, it is necessary to solve the equation for a value
function that is the same on both sides of the equation.
126 4. Dynamic Programming
Proof. Define h(s, a) = r(s, a) + δ V (f (s, a)) to be the function on the right-hand side of
Bellman’s equation. To show that the Bellman equation holds, we first show that V (s) is less
than or equal to the right-hand side of the Bellman equation. Fix an s0 , and take any allowable
sequence at ∈ F (st ) and st+1 = ft (st , at ) for t ≥ 0.
∞ ∞
δ t - 1 r(st , at )
X X
δ t r(st , at ) = r(s0 , a0 ) + δ
t=0 t=1
nX∞
≤ r(s0 , a0 ) + δ sup δ t - 1 r(s0t+1 , a0t+1 ) : s01 = s1 , a0t ∈ F (s0t ),
t=0
s0t = ft (s0t−1 , a0t−1 ) for t ≥ 0
= r(s0 , a0 ) + δ V (f (s0 , a0 ))
≤ sup { h(s0 , a00 ) : a00 ∈ F (s0 ) } = h∗ (s0 ).
Here we define h∗ using a supremum like we did earlier for the maximum. Since the total
reward for any allowable sequence is less than or equal to h∗ (s00 ), taking the supremum over
all allowable choices yields
( ∞ )
δ t r(st , at ) : at ∈ F (st ), st+1 = f (st , at ) for t ≥ 0 ≤ h∗ (s0 ).
X
V (s0 ) = sup
t=0
4.3.1. Examples
We delay the precise theorems and proofs until after giving examples using two methods of
using the Bellman equation to determine the value function and an optimal strategy. The first
method involves construction a sequence of functions by iteratively maximizing the right-hand
of the Bellman equation. The proof of Theorem 4.24 the sequence of function converge to the
true value function. The second method involves guessing the form of the value function in a
form that involves unknown parameters. Then the Bellman equation is used to determine the
value of these parameters and so of the true value function.
Example 4.21. This example is a special case of the optimal growth of a one-sector economy
considered later. Let
S = R+ ,
A = R+ ,
F (s) = [0, s],
f (s, a) = k(s − a), with k ≥ 1
1
r(s, a) = u(a) = a , 2 1.0inand
0 < δ < 1, with k δ 2 < 1.
The reward function is not bounded on R+ , but given an initial state s0 , s1 ≤ ks0 , s2 ≤
k s1 ≤ k 2 s0 , st ≤ k t s0 . If k δ 2 < 1, the total reward is bounded as follows:
1
t 1
δ t r(st , at ) ≤ δ t u(st ) ≤ δ t u(k t s0 ) = δ k 2 s02 ,
1
∞ ∞
X
t
X
t t 1s02
δ r(st , at ) ≤ δ k s0 ≤
2 2
1 .
t=0 t=0 1 − δk 2
Thus, the value function is finite for each s0 .
128 4. Dynamic Programming
Solution Method 1: We construct a sequence of functions Vj (s), that converge to the value
function V (s). We prove that the Vj (s) are continuous by induction. Start with the zero
function V0 (s) = 0 for all s. Assume that Vj (s) in our sequence is continuous, and let
1
hj+1 (s, a) = r(s, a) + δ Vj (f (s, a)) = a 2 + δ Vj (k(s − a))
be the function that is used in the supremum on the right hand-side of the Bellman equation
1
using the value function Vj . The functions r(s, a) = u(a) = a 2 , Vj (s), and f (s, a) =
k (s − a) are all continuous, so hj+1 (s, a) is a continuous function. The feasible action
correspondence F (s) = [0, s] is a continuous compact-valued correspondence. So, we can
apply the Parametric Maximization Theorem to get a continuous maximal value as a function
of s,
Vj+1 (s) = h∗1 (s).
None of these functions are the value function, because we will see that Vj+1 (s) 6= Vj (s).
However in this example, we note that this sequence of functions converges to the true value
function. Later in the Proof of Theorem 4.24 proving the continuity of the value function, we
indicate a little more of why this works.
Since we start with V0 (s) = 0 for all s,
1 1
h1 (s, a) = a 2 + δ V0 (k(s − k)) = a 2 .
This function is increasing on the intervals F (s) = [0, s], so h1 (s, a) has a maximum at the
right end point, ā = s. Then,
1
V1 (s) = h∗1 (x) = h1 (s, ā) = s 2 .
Note that V1 (s) is the maximum over the single period t = 0.
1 1 1 1
For the next step, h2 (s, a) = a 2 + δ V1 (k(s − a)) = a 2 + δ k 2 (s − a) 2 . The critical
point satisfies
∂h2
= 21 a - 2 − 12 δ k 2 (s − a) - 2 ,
1 1 1
0=
∂a
a - 2 = k 2 δ (s − a) - 2 ,
1 1 1
s − a = k δ 2 a,
s = (1 + k δ 2 ) a,
s
ā = .
1 + kδ2
∂ 2 h2
Since < 0 for all a ∈ [0, s], ā is a maximizer.
∂a2
1 1 1
V2 (s) = h∗2 (s) = h2 (s, ā) = ā 2 + δ k 2 (s − ā) 2
1 1 1 1
= ā 2 + δ k 2 δ k 2 ā 2
1
= (1 + k δ 2 ) ā 2
1
s2
= (1 + k δ 2 ) 1
(1 + k δ 2 ) 2
1 1
= (1 + k δ 2 ) 2 s 2 .
Note that V2 (s) is the maximum over the two periods t = 0, 1.
Our induction hypothesis is that
h i 21 1
Vj (s) = 1 + k δ 2 + · · · + k j - 1 δ 2(j - 1) s 2 .
4.3. Infinite-Horizon Dynamic Program 129
We have verified the formula for j = 1, 2. Assume this is true for j = t and show is is true
for j = t + 1. Let
h i 21 1
ht+1 (s, a) = a 2 + δ 1 + k δ 2 + · · · + k t - 1 δ 2(t - 1) k 2 (s − a) 2 .
1 1
s − a = k δ 2 + · · · + k t δ 2t a
s = 1 + k δ 2 + · · · + k t δ 2t a
s ∗
ā = = σt+1 (s).
1 + k δ 2 + · · · + k t δ 2t
∂ 2 ht+1
Again, this is a maximizer because < 0 for all a ∈ [0, s]. Then
∂a2
h i 21
Vt+1 (s) = h∗t+1 (s) = ht+1 (s, ā) = ā 2 + δ 1 + k δ 2 + · · · + k t - 1 δ 2(t - 1)
1 1 1
k 2 (s − ā) 2
h i 21 12
= ā 2 + δ 1 + k δ 2 + · · · + k t - 1 δ 2(t - 1)
1 1
k 2 k δ 2 ā
12 1
= 1 + k δ 2 + · · · + k t δ 2t
ā 2
= [1 + k δ 2 + · · · + k t δ 2t ] 2 [1 + k δ 2 + · · · + k t δ 2t ] - 1 s
1
21 1
= 1 + k δ 2 + · · · +, k t δ 2t
s2
This verifies the induction hypothesis for the form of Vt+1 (s).
Taking the limit as t goes to infinity,
h i 21 1
V∞ (s) = lim Vt (s) = lim 1 + k δ 2 + · · · + k t - 1 δ 2(t - 1) s 2
t→∞ t→∞
- 1 1
= 1 − δ 2k 2 s 2 .
Also, if we take the limit in the inductive equation defining the Vj (s),
1
V∞ (s) = lim Vj+1 (s) = lim max{ a 2 + δ Vj (k(s − a)) : 0 ≤ a ≤ s }
t→∞ t→∞
1
= max{ a 2 + δ V∞ (k(s − a)) : 0 ≤ a ≤ s }. (Bellman equation).
Since the value function is the unique locally bounded solution of Bellman equation, V (s) =
V∞ (s) and
- 1 1
V (s) = V∞ (s) = 1 − δ 2 k 2 s 2 .
The optimal strategy is also the limit of the strategies σt∗ (s),
σ ∗ (s) = lim σt∗ (s) = lim (1 + k δ 2 + · · · + k t−1 δ 2t−2 ) - 1 s = (1 − k δ 2 ) s.
t→∞ t→∞
If V0 (s) ≡ 0, then V1 (s) maximum over the single period t = 1; V2 (s) is the maximum
over the two periods t = 0, 1; by induction, Vj (s) is the maximum over the j periods
t = 0, . . . , j − 1. Taking the limit, V∞ (s) is the maximum over all the periods t ≥ 0, which
is the true value function.
130 4. Dynamic Programming
Solution Method 2: In this method, the way we find the optimal solutions is to guess the
form of the value function V with unspecified parameters. Next, we use the Bellman equation
to determine the unspecified parameters in the guess. In the process, the optimal strategy is
determined.
For the present problem, an outline of the solutions method is as follows. (1) Based on the
1
reward function, we guess that it is of the form V (s) = M s 2 where M is to be determined.
We could also use the first few Vj (s) calculated by Method 1 to motivate a guess of the form
of the true value function. (2) Next, determine the critical point ā of
1 1 1
h(s, a) = r(s, a) + δ V (f (s, a)) = a 2 + δ M k 2 (s − a) 2 .
This is the sum of the immediate payoff plus the payoff for what is carried forward to the future
calculated by the value function. Verify that these are a maximum of h(s, a) for a ∈ F (s) =
[0, s]. This value ā can depend on the unspecified parameters of V as well as the data for the
problem. (3) Calculate h∗ (s) = h(s, ā). (4) Use the Bellman equation, V (s) = h∗ (s), to
solve for the unspecified parameters of the guess of V . Finally, (5) substitute the parameters
into s̄ to determine the optimal strategy.
1
(1) Using the guess is that the value function has the form V (s) = M s 2 , define
1 1 1
h(s, a) = a 2 + δ M k 2 (s − a) 2 .
∂
h(s, a) = 12 a - 2 − k 2 δ M (s − a) - 2
1 1 1
1
0= 2
∂a
a - 2 = k 2 δ M (s − a) - 2
1 1 1
1 1 1
(s − a) 2 = k 2 δ M a 2 ,
s − a = k δ 2 M 2 a,
s = (1 + k δ 2 M 2 ) a, and
s
ā = ≤ s.
1 + k δ 2M 2
∂2
Since h(s, a) < 0, the critical point ā indeed maximizes h and is an optimal strategy.
∂a2
4.3. Infinite-Horizon Dynamic Program 131
21
s
2 2
= 1+kδ M
1 + k δ 2M 2
12 1
= 1 + k δ 2M 2 s2 .
(4) The value function must satisfy the Bellman equation (11), V (s) = h∗ (s), so
1 1 1
M s 2 = 1 + k δ 2M 2 2 s 2 ,
M 2 = 1 + k δ 2M 2,
M 2 (1 − k δ 2 ) = 1,
1
M2 = , and
1 − kδ2
21
1
M̄ = .
1 − kδ2
Because the solution of the Bellman equation is unique and this function using M̄ satisfies it,
it must be the value function,
21
s
V (s) = .
1 − kδ2
(5) The optimal strategy is
s s
σ ∗ (s) = ā = 2 2
= 2
1+kδ M M
= (1 − k δ 2 )s.
Note that we need k δ 2 < 1 for σ ∗ (s) ≥ 0 and V (s) to be defined.
Example 4.22. This is an example attributed to Weitzman. On each day, a vintner can split
his time between baking bread or squeezing grapes: The amounts of effort are bt and 1 − bt
respectively with both bt and wt+1 elements of [0, 1]. In the next period, the amount of wine
1 1
available is wt+1 = 1 − bt . The reward or utility at each period is u(wt , bt ) = wt2 bt2 . There
P∞ 1 1
is a discount factor 0 < δ < 1. The quantity to be maximized is t=0 δ t wt2 bt2 . We consider
132 4. Dynamic Programming
wt as the state variable and bt as the action with wt+1 = 1 − bt the transition function. The
Bellman equation is
√
V (w) = max{ wb + δ V (1 − b) : b ∈ [0, 1] }.
∂h2
= 21 w 2 b - 2 − 12 δ (1 − b) - 2
1 1 1
0=
∂b
w b - 1 = δ 2 (1 − b) - 1
w(1 − b) = δ 2 b
w = (w + δ 2 ) b
w
b̄ = .
w + δ2
∂ 2 h2
The second derivative < 0, so b̄ is a maximizer. Then, V2 (w) = h∗2 (w) = h2 (w, b̄), so
∂a2
1 21
w δ2
2
1
V2 (w) = w 2 + δ
w + δ2 w + δ2
2
w+δ 2 12
= 2 1 = (w + δ ) .
(w + δ ) 2
Rather than calculate more terms, we turn to Method 2. In this treatment, we will derive
formulas that allows to determine the rest of the sequence of functions for Method 1.
1
Method 2: (1) We look for a solution of the form V (w) = A(w + C) 2 , where A and C
are parameters to be determined. (2) We introduce the function to be maximized: h(w, b) =
1 1 1
b 2 w 2 + δA(1 − b + C) 2 . (3) The critical point satisfies
∂h
= 21 b - 2 w 2 − 12 δA(C + 1 − b) - 2
1 1 1
0=
∂b
w(C + 1 − b) = δ 2 A2 b
w(C + 1) = b w + δ 2 A2
w(C + 1)
b̄ = .
w + δ 2 A2
∂2h
The second derivative < 0, so the critical point is a maximum.
∂b2
4.3. Infinite-Horizon Dynamic Program 133
A2 = δ 2 A2 + 1,
(1 − δ 2 ) A2 = 1,
1
A2 = , and
1 − δ2
δ2
C= .
1 − δ2
(6) Therefore, the value function is
21 12 1
1 δ2 [w(1 − δ 2 ) + δ 2 ] 2
V (w) = w+ = .
1 − δ2 1 − δ2 1 − δ2
1 δ2 δ2 + 1 − δ2 1
Using A2 and C + 1 = + 1 = = , we get the
1−δ 2 1−δ 2 1−δ 2 1 − δ2
optimal strategy
h i
1
w(C + 1) w 1−δ 2 w
∗
b̄ = σ (w) = = δ2
= .
2
w+δ A 2
w + 1−δ 2 w(1 − δ 2 ) + δ 2
1
Return to Method 1: In the consideration of Method 2, we saw that if Vj (w) = A(w + C) 2 ,
1 1 w(1 + C)
then Vj+1 (w) = (1 + C) 2 [w + δ 2 A2 ] 2 and σj+1 (w) = . Using that V2 (w) =
w + δ 2 A2
134 4. Dynamic Programming
1
(w + δ 2 ) 2 ,
12 21
V3 (w) = 1 + δ 2 w + δ2 ,
1
2 2
21
V4 (w) = 1 + δ w + δ 2 + δ4 ,
1
2j−2 2
12
V2j (w) = 1 + · · · + δ w + δ 2 + · · · + δ 2j ,
1 1
V2j+1 (w) = 1 + · · · + δ 2j w + δ 2 + · · · + δ 2j
2
2
.
This sequence of functions converges to the value function found using Method 2,
21 12 1
1 δ2 [w(1 − δ 2 ) + δ 2 ] 2
V (w) = w+ = .
1 − δ2 1 − δ2 1 − δ2
The optimal strategy at each step in the process is
w(1 + δ 2 )
σ3 (w) = ,
w + δ2
w(1 + δ 2 )
σ4 (w) = ,
w + δ2 + δ4
w(1 + δ 2 + · · · + δ 2j )
σ2j+1 (w) = ,
w + δ 2 + · · · + δ 2j
w(1 + δ 2 + · · · + δ 2j )
σ2j+2 (w) = .
w + δ 2 + · · · + δ 2j+2
This sequence of strategies converges to the optimal strategy found using Method 2,
h i
1
w 1−δ 2 w
σ ∗ (w) = δ2
= ,
w + 1−δ 2 w(1 − δ 2 ) + δ 2
The supremum over all allowable choices of all the at is bounded by this same constant, so the
value function V (s0 ) is bounded and takes on finite values.
4.3. Infinite-Horizon Dynamic Program 135
Theorem 4.24 (SDP Continuity of Value Function). Assume that a SDP satisfies SDB and
has a bounded value function V . Then the following hold.
a. There exists a unique bounded solution of Bellman equation, and unique solution is
continuous.
b. The value function V (s) is the unique continuous function that satisfies the Bellman
equation.
Proof. The continuity of the value function V : S → R cannot be proved directly from the
Bellman equation because we do not know a priori that the right hand side is continuous.
Instead the continuity is proved by means of a process that takes a bounded function and
returns another bounded function by the solution process of Method 1. In determining the value
function for examples, Method 1 finds it as the limit of a sequence of functions calculated using
the right-hand side of the Bellman equation. The theory behind these calculations involves
(i) putting a distance between two functions (a metric on a space of functions), (ii) showing
the space of functions has a property called complete (a sequence of functions that gets closer
together has to converge to a function in the space of functions), and finally, showing that
the construction of the sequence of function is really a contraction mapping of the space of
functions. We showed earlier how this process works for two examples.
Assume G : S → R is any bounded function. Let
hG (s, a) = r(s, a) + δ G(f (s, a)), and
T (G)(s) = sup{ hG (s, a) : a ∈ F (s) }.
It is shown that T (G) : S → R is a new bounded function, and if G1 and G2 are two
such functions then T (G1 ) and T (G2 ) are closer together than G1 and G2 , i.e., T is a
contraction mapping on the space of bounded functions. The set of bounded functions needs
to be shown to be complete, i.e., a sequence of functions getting closer together (is a Cauchy
sequence) must converge to a bounded function. Then it follows that there is a unique bounded
function that is taken to itself. Since the value function is one such function, it must be the
unique function satisfying the Bellman equation.
If V0 : S → R is any bounded function and inductively Vj+1 = T (Vj ), then the
sequence Vj (s) converges to the unique function fixed by T . If we start with a continuous
function V0 , then all the functions Vj (s) in the sequence are continuous by the Parametric
Maximization Theorem. Because the distance between the functions Vj (s) and V (s) goes to
zero in terms of the distance on the function space, the sequence converges uniformly to V (S).
But the uniform limit of continuous functions is continuous, so the value function V (S) must
be continuous.
See Section 4.3.4 for more details.
Remark. Assume we start with V0 (s) ≡ 0 and inductively let Vj+1 = T (Vj ). Then, V1 (s)
is the maximum over the one period t = 0; V2 (s) is the maximum over the two periods
t = 0, 1; Vj (s) is the maximum over the j periods t = 0, . . . , j − 1. The proof of the theorem
shows that Vj (s) converges to the value function V (s) that is the maximum over all periods
t ≥ 0.
Theorem 4.25 (SDP Optimal Strategy). Assume that a SDP has a continuous value function
V such that δ t V (st ) goes to zero as t goes to infinity for any allowable sequence of {at }
with st+1 = f (st , at ).
Then optimal stationary strategy σ ∗ exists with W (s, σ ∗ ) = V (s).
In fact, for h(s, a) = r(s, a) + δ V ◦ f (s, a), an optimal strategy is any choice function
σ ∗ (s) ∈ F ∗ (s) = { a ∈ F (s) : h(s, a) = h∗ (s) }.
Remark. Note that the Theorem is valid if r(s, a) is bounded (SDB) so V (s) is bounded.
136 4. Dynamic Programming
= V (s0 ).
Let σ ∗ (s) ∈ F ∗ (s) be any choice function. We next show that V (s) = W (s, σ ∗ ). For
s0 ∈ S, let a∗t = at (σ ∗ , s0 ) = σ ∗ (st ) and s∗t+1 = st+1 (σ ∗ , s0 ) = f (st , a∗t ). Also, s∗0 = s0 .
By the equation for V and σ ∗ above and the definitions of a∗t and s∗t+1 ,
V (s∗t ) = h∗ (s∗t ) = h(s∗t , σ ∗ (s∗t ))
= r(s∗t , σ ∗ (s∗t )) + δ V ◦ f (s∗t , σ ∗ (s∗t ))
= r(s∗t , a∗t ) + δ V (s∗t+1 ).
By repeated uses of this formula,
V (s0 ) = r(s0 , a∗0 ) + δ V (s∗1 )
= r(s0 , a∗0 ) + δ r(s∗1 , a∗1 ) + δ 2 V (s∗2 )
= r(s0 , a∗0 ) + δ r(s∗1 , a∗1 ) + δ 2 r(s∗1 , a∗1 ) + δ 3 V (s∗3 )
..
.
T -1
X
= δ t r(s∗t , a∗t ) + δ T V (s∗T ).
t=0
If we let T go to infinity, then δ T V (s∗T ) goes to zero by the hypothesis. Therefore, the right
hand side converges to X∞
δ t r(s∗t , a∗t ) = W (s0 , σ ∗ ),
t=0
and V (s0 ) = W (s0 , σ ∗ ). Thus, σ ∗ is an optimal strategy.
≤ A u(s) = V ∗ (s).
Let V0∗ (s) = V ∗ (s) and Vj+1∗
= T (Vj∗ ) for j ≥ 0. Since V1∗ (s) = T (V0∗ )(s) ≤
V0∗ (s) for all s, Vj+1 (s) ≤ Vj (s) for all s by induction. Thus for each s ≥ 0, Vj∗ (s) ≥ 0
∗ ∗
The function u is strictly concave and increasing and (s−x0 )−(s−x) = (s0 −x0 )−(s0 −x),
and the points on the right are larger, so
u(s − x0 ) − u(s − x) > u(s0 − x0 ) − u(s0 − x).
This contradiction proves the theorem.
Theorem 4.29. If a 1-SecE satisfies E1 – E5, then the value function V is concave on R+ .
Proof. Let s, s0 ∈ S with s < s0 . Set sτ = (1 − τ )s + τ s0 for 0 ≤ τ ≤ 1. Let st and s0t
be the optimal sequences of states and ct and c0t be the sequences of consumptions starting at
s and s0 respectively. Note that ct ≤ st and c0t ≤ s0t . For each t, let cτt = (1 − τ )ct + τ c0t .
We will show that cst is an allowable consumption for sτt . Let x∗t denote the sequence of
investment levels if we use the sequence cτt starting at sτ .
First,
sτ = (1 − τ )s + τ s0 ≥ (1 − τ )c0 + τ a00 = cτ0 .
Then, x∗0 = sτ − cτ0 . Using the concavity of f ,
f (x∗0 ) ≥ (1 − τ ) f (s − c0 ) + s f (s0 − c00 )
= (1 − τ ) s1 + τ s01 ≥ (1 − τ ) c1 + τ c01
= cτ1 .
Continuing by induction, we get that f (x∗t ) ≥ aτt+1 . Therefore, the sequence cst is an allow-
able sequence for sτ .
The sequence cτt is feasible, but not necessarily optimal. The utility function is concave,
so
X∞
V (sτ ) ≥ δ t u(cτt )
t=0
X∞
= δ t u ((1 − τ )ct + τ a0t )
t=0
X∞ X∞
≥ (1 − τ ) δ t u(ct ) + τ δ t u(a0t )
t=0 t=0
= (1 − τ )V (s) + τ V (s0 ).
This shows that V is concave.
Theorem 4.30. If a 1-SecE satisfies E1 – E5, then the correspondence F ∗ that gives the
maximizers of the Bellman equation is single-valued. Therefore, the optimal strategy σ ∗ is
uniquely determined and is a continuous function on R+ .
Proof. We are assuming that u is strictly concave and that f is concave. By Theorem 4.29,
V is concave. Combining, u(a) + δ V (f (s − a)) is a strictly concave function of a. It
follows that there can be a single point that maximizes this function, so F ∗ (s) is a single
point. It follows that the optimal strategy is unique. Since an upper-hemicontinuous single
valued correspondence is continuous, F ∗ or σ ∗ is continuous.
Theorem 4.31. If a 1-SecE satisfies E1 – E7, then for all s > 0, the optimal strategy is an
interior point of F (s) = [0, s], 0 < σ ∗ (s) < s.
Proof. Let s0 > 0, s̄t , c̄t = σ ∗ (s̄t ), and x̄t = s̄t − c̄t be optimal sequence of states,
consumption, and savings. Assume that not all the x̄t are interior. Let x̄τ be the first x̄t that
is not interior. We get a contradiction to the fact that these are optimal. Since x̄τ - 1 is interior,
s̄τ = f (x̄τ - 1 ) > 0.
First we show that it cannot happen that x̄τ = 0. If this were true, then s̄τ +1 = f (x̄τ ) = 0
and s̄t = 0 for all t ≥ τ + 1. The value function starting at s̄τ is V (s̄τ ) = u(s̄τ − x̄τ ) +
140 4. Dynamic Programming
δ V (s̄τ +1 ) = u(s̄τ ). This is greater than the payoff choosing saving z > 0 for t = τ and
savings 0 = f (z) − cτ +1 for t = τ + 1,
u(s̄τ ) ≥ u(s̄τ − z) + δ u(f (z)) + δ 2 u(0).
But
d
= u0 (s̄τ ) + δ u0 (0+) f 0 (0+) = ∞
[u(s̄τ − z) + δ u(f (z))]
dz z=0+
= −u0 (s̄τ ) + δ (∞) f 0 (0+) = ∞.
Since this derivative is positive, z = 0 cannot be a maximum.
Second, we show that if x̄τ = s̄τ , then x̄τ +1 = s̄τ +1 .
V (s̄τ ) = u(s̄τ − x̄τ ) + δ u (f (s̄τ ) − x̄τ +1 ) + δ 2 V (f (x̄τ +1 )
= u(0) + δ u (f (s̄τ ) − x̄τ +1 ) + δ 2 V (f (x̄τ +1 ) .
If also x̄τ +1 < s̄τ +1 , then savings z at period τ can be decreased from z = x̄τ = s̄τ while
keeping x̄τ +1 fixed,
u(0) + δ u (s̄τ +1 − x̄τ +1 ) + δ 2 V (f (x̄τ +1 ))
≥ u(s̄τ − z) + δ u (f (z) − x̄τ +1 ) + δ 2 V (f (x̄τ +1 ))
or
u(0) + δ u (s̄τ +1 ) − x̄τ +1 ) ≥ u(s̄τ − z) + δ u (f (z) − x̄τ +1 ) .
Since x̄τ +1 is fixed, we can keep all the choices fixed for t ≥ τ + 1. Since this must be an
optimal choice,
d
0≤ [u(s̄τ − z) + δ u (f (z) − x̄τ +1 )]
dz z=s̄τ
= u0 (0) + δ u0 (s̄τ +1 − x̄τ +1 ) f 0 (s̄τ )
= ∞ + δ u0 (s̄τ +1 − x̄τ +1 ) f 0 (s̄τ ) = ∞.
Since this derivative is negative, it cannot be a maximum. Therefore, we would need x̄τ +1 =
s̄τ +1 .
We have shown that if x̄τ = s̄τ , then x̄τ +1 = s̄τ +1 . Continuing by induction, we would
need x̄t = s̄t and c̄t = 0 for all t ≥ τ . Therefore, the value function starting at τ would be
zero. Keeping all the ct = 0 for t ≥ τ + 1 and increasing cτ , we can increase the payoff.
Therefore, this would not be an optimal sequence.
Thus, we have ruled out the possibility of x̄τ being on either end point, and so it and c̄τ
must be interior.
Theorem 4.32. If a 1-SecE satisfies E1 – E7, then for s0 > 0, the optimal strategy σ ∗
satisfies the Ramsey-Euler equation
u0 (σ ∗ (st )) = δ u0 (σ ∗ (st+1 )) f 0 (st − σ ∗ (st ))
where st+1 = f (st − σ ∗ (st )).
Proof. In the proof of the last theorem, we showed that c = ct = σ ∗ (st ) is an interior
maximum of
u(c) + δ u (f (st − c) − xt+1 ) .
The Ramsey-Euler equation is the first order condition for an interior maximum of this function.
Theorem 4.33. If a 1-SecE satisfies E1 – E7, then the optimal strategy σ ∗ is increasing on
R+ .
4.3. Infinite-Horizon Dynamic Program 141
Proof. Suppose the theorem is false and there exist s < ŝ with c = σ ∗ (s) ≥ σ ∗ (ŝ) = ĉ.
Since s − c < ŝ − ĉ, s1 = f (s − c) ≤ f (ŝ − ĉ) = ŝ1 . Let c1 = σ ∗ (s1 ) and ĉ1 = σ ∗ (ŝ1 ).
By the Ramsey-Euler equation, u0 (c) = δ u0 (c1 ) f 0 (s − c) and u0 (ĉ) = δ u0 (ĉ1 ) f 0 (ŝ − ĉ),
so
u0 (a) u0 (c1 ) f 0 (s − c)
= or
u0 (ĉ) u0 (ĉ1 ) f 0 (ŝ − ĉ)
u0 (c1 ) u0 (c) f 0 (ŝ − ĉ)
= .
u0 (ĉ1 ) u0 (ĉ) f 0 (s − c)
Since u is strictly concave u0 (c) ≤ u0 (ĉ). Also, s − c < ŝ − ĉ, so f 0 (s − c) ≥ f 0 (ŝ − ĉ).
Combining, we must have u0 (c1 ) ≤ u0 (ĉ1 ) and c1 ≥ ĉ1 . Thus, the situation is repeated at the
next period with s1 < ŝ1 and c1 ≥ ĉ1 . Continuing by induction, we get that st < ŝt and
ct ≥ ĉt for all t, so that V (s) ≥ V (ŝ). This contradicts the fact that V is increasing and
proves the theorem.
The last theorem shows that under one additional assumption, there is a positive steady
state of consumption and savings. For this theorem , we need the following assumption.
E8. The production function f is strictly concave on R+ and δ f (0+) > 1.
Note that for f (x) = kx, then it requires kδ > 1, so k > 1 and E3.a is not valid. However,
if kδ 2 < 1, then E3.b is satisfied and we still get the results.
Theorem 4.34. If a 1-SecE satisfies E1 – E8, then there is a unique x∗ such that δf 0 (x∗ ) =
1. Let s∗ = f (x∗ ) and c∗ = s∗ − x∗ be the associated state and consumption. For s0 > 0,
define ct and st inductively by ct = σ ∗ (st ) and st+1 = f (st − ct ). Then limt→∞ st = s∗
and limt→∞ ct = c∗ .
Proof. Since δ f 0 (0+) > 1, δ f 0 (x̄) < 1, and f is strictly concave, there is a unique x∗ such
that δ f 0 (x∗ ) = 1. Since the savings function ξ and production function f are nondecreasing,
the sequence st is nondecreasing function of t and has a limit s∞ . Since σ ∗ is nondecreasing,
ct is nondecreasing and has a limit c∞ with c∞ = σ ∗ (s∞ ). Since st+1 = f (st − ct ),
s∞ = f (s∞ − c∞ ).
0 0 0
Also, since u (ct ) = δ u (ct+1 ) f (st − ct ), we get that
u0 (c∞ ) = δ u0 (c∞ ) f 0 (s∞ − c∞ ) so
0
1 = δ f (s∞ − c∞ )
Thus, x∞ = s∞ − c∞ satisfies the equation for x∗ and so x∞ = x∗ , s∞ = f (x∞ ) =
f (x∗ ) = s∗ , and c∞ = s∞ − x∞ = s∗ − x∗ = c∗ . Therefore, these quantities are x∗ , s∗ ,
and c∗ .
For our specialized optimal growth Example 4.21, the hypotheses of Theorem 4.34 are not
valid: the production function f (x) = k x is not strictly concave and we cannot satisfy both
E3.b and f 0 (0) > 1/δ : We need k δ 2 ≤ 1 so we can satisfy E3.b, and f 0 (0) = k ≥ 1/δ for
E8. For that example, the conclusion of Theorem 4.34 are not valid since st and ct both go to
zero and not a nonzero limit.
so W ∗ is a fixed point of T .
The fixed point can be shown unique. Assume both W ∗ and W̄ are fixed. Then
T (W ∗ ) − T (W̄ )k0 ≤ δ kW ∗ − W̄ k0
kW ∗ − W̄ k0 = kT
(1 − δ)kW ∗ − W̄ k0 ≤ 0.
Therefore, kW ∗ − W̄ k0 must be zero and W ∗ = W̄ . Thus, the fixed point is unique.
The rest of the proof goes as before since convergence in the k · k∗ norm implies uniform
convergence on compact intervals.
4.3. Exercises
2
4.3.1. Consider the SDP problem with reward function r(s, a) = u(a) = a /3 , transition
function f (s, a) = k(s − a) with k ≥ 1, F (s) = [0, s], and 0 < δ < 1.
2
a. Using the guess that V (s) = M s /3 , find the action a = σ(s) in terms of
M that maximizes the right hand side of the Bellman equation.
b. Substitute the solution of part (a) in the Bellman equation to determine the
constant M and V (s).
c. What is the optimal strategy?
4.3.2. (Brock-Mirman growth model.) Consider the SDP problem with st the amount
of capital at period-t, F (s) = (0, s] the allowable consumption, at consumption
at period-t, reward function r(s, a) = u(a) = ln(a), discount 0 < δ < 1, and
transition (production) function f (s, a) = (s−a)β with 0 < β ≤ 1 that determines
the capital at the next period. (Note that u(a) = ln(a) is unbounded below at 0, and
the choices are open at 0, but it turns out that the Bellman equation does have a
solution.)
4.3. Infinite-Horizon Dynamic Program 145
a. Using the guess that V (s) = A + B ln(s), find the action a = σ(s) in
terms of A and B that maximizes the right hand side of the Bellman equation.
b. Substitute the solution of part (a) in the Bellman equation to determine the
constants A and B . Hint: The coefficients of ln(y) on the two sides of the
Bellman equation must be equal and the constants must be equal. Solve for B
first and then A.
c. What are the value function and optimal strategy?
4.3.3. Consider the SDP problem with S = [0, ∞), A, F (s) = [0, s], f (s, a) = 2s −
2a, r(s, a) = 2 − 2e - a , and δ = 1/2. Start with the guess that this SDP has a value
function of the form V (s) = A − B e - bs .
a. Find the action ā = σ(s) that maximizes the right hand side of the Bellman
equation. (The answer can contain the unspecified parameters A, B , and b.)
b. What equations must A, B , and b satisfy to be a solution of the Bellman
equation? Solve these equations for A, B , and b.
c. What are the value function and the optimal strategy?
4.3.4. Consider the SDP problem with discount 0 < δ < 1, S = [0, ∞), A, F (s) =
a 1
[0, s], bounded reward function r(s, a) = = 1− , and transition
1+a 1+a
function f (s, a) = k(s − a) with k > 1 and kδ = 1. Start with the guess that
this
s 1 1
SDP has a value function of the form V (s) = = B 1− .
1 + Bs 1 + Bs
a. What is the Bellman equation for this problem?
b. Find the action a = σ(s) that maximizes the right hand side of the Bellman
equation.
c. Substitute the solution of part (b) in the Bellman equation to determine the
constant B .
d. What is the optimal strategy?
4.3.5. Consider the SDP problem with st the amount of capital at period-t, F (s) = (0, s]
the allowable consumption, at consumption at period-t, reward function r(s, a) =
u(a) = ln(a), discount 0 < δ < 1, and transition (production) function f (s, a) =
s − a that determines the capital at the next period. (Note that u(a) = ln(a) is
unbounded below at 0, and the choices are open at 0, but it turns out that the Bellman
equation does have a solution.)
a. Using the guess that V (s) = A + B ln(s), find the action a = σ(s) in
terms of A and B that maximizes the right hand side of the Bellman equation.
b. Substitute the solution of part (a) in the Bellman equation to determine the
constants A and B . Hint: The coefficients of ln(y) on the two sides of the
Bellman equation must be equal and the constants must be equal. Solve for B
first and then A.
c. What are the value function and optimal strategy?
146 4. Dynamic Programming
Mathematical Language
We summarize some of the mathematical language with which the reader should be familiar.
We denote the set of real numbers by R and n-dimensional Euclidean space by Rn .
We assume that the reader can understand common notations related to sets. To express that
p is an element of a set S, we write p ∈ S, and p ∈ / S means that p is not an element of S.
We often define sets using notation like { x ∈ R : x2 ≤ 4 } for the interval [ 2, 2]. The set
{ x ∈ R : x2 < 0 } is the empty set, denoted by ∅. If every element of the set S1 is an element
of S2 , then we call S1 a subset of S2 , and write S1 ⊂ S2 . When S1 ⊂ S2 , we allow the two
sets to be equal. Given two sets S1 and S2 , S1 = S2 if and only if S1 ⊂ S2 and S2 ⊂ S1 .
We often express the domain and target space of a function by a notation f : S ⊂ Rn → R:
This means that S is the domain and a subset of Rn , and f takes values in the real numbers.
Quantifiers are very important. The phrases “for all”, “for every”, “for any”, and “for each”
are all expressions for the universal quantifier and mean essentially the same thing, but they do
have different connotations. When we say “for all x ∈ S, x2 ≥ x”, we are thinking collectively
of the whole set of x. Consider the statement “For a function g : R → R and for any b ∈ R,
we can form the set { x ∈ R : g(x) ≤ b }.” The set that follows depends on the particular
value of b, so we are taking the b one at a time so we use “for any” or “for each” and not “for
all”.
The phases “for some” and “there exists” are expressions for the existential quantifier. The
statement “for some x ∈ R, x2 ≥ 4 + x” is true. But “for all x ∈ R, x2 ≥ 4 + x” is false.
This latter is false since there is some x, for example x = 1, for which the statement is not
true. For a statement containing a universal quantifier to be true, it must be true for every object
satisfying the quantifier. Thus, we only need to find one counter-example. Also, quantities can
generally depend on things earlier in the sentence: “For any > 0, there exists a δ > 0 such
that”, means that the δ can depend on . Therefore, the order of quantifiers is important. The
sentence “There exists a δ > 0 such that for all > 0” would mean that the same δ works
for all of the .
The use of “or” is inclusive, with one or both being true. The statement “I am taking a class
in Mathematics or Economics” is true even if I am taking both classes.
“If A is true, then B is true” and “A implies B” are two ways of saying the same thing that
we sometimes denote by the notation “A ⇒ B”. ‘A’ is the hypothesis and ‘B’ is the conclusion.
Either of these statements is equivalent to its contrapositive statement: “If B is not true, then A
is not true.” We often use this type of logic in our proof without explicitly commenting on it.
The converse is a very different statement: “If B is true, then A is true” or in symbolic notation
147
148 A. Mathematical Language
“A ⇐ B.” We use “iff” to mean “if and only if”, so each implies the other, “A ⇔ B.” Notice
that the statement “If the moon is made of blue cheese then pigs can fly” is true because the
hypothesis is false.
At various places we talk about necessary conditions and sufficient conditions. In the state-
ment “If A, then B”, ‘A’ is a sufficient condition for “B” and “B” is a necessary condition for
“A”. Theorem 3.12 states that “if the feasible set satisfies the constraint qualification and a dif-
ferentiable function f attains a maximum for the feasible set at a point x∗ , then conditions
KKT-1,2,3 hold.” Thus, KKT-1,2,3 are necessary conditions for a maximum. Theorem 3.19
states that “if the feasible set is convex and a differentiable, concave function f satisfies KKT-
1,2,3 at a point x∗ , then f attains a maximum at x∗ .” Thus, when f is concave, conditions
KKT-1,2,3 are sufficient for a maximum. The Extreme Value Theorem 2.14 states that conti-
nuity of a function on a closed and bounded set are sufficient conditions for the existence of a
maximum.
The reader should be familiar with how quantifiers change when a statement is negated. The
negation of “all mathematics classes are difficult” is “at least one (or some) mathematics class
is not difficult.” The negation of “some mathematics classes are difficult” is “all mathematics
classes are not difficult.”
In definitions, we use “provided that” to define a term. What follows “provided that” is the
defining conditions for the word or phrase being defined. Thus, its meaning is “if and only if.”
Many authors use “if”, but this has a different meaning than the use of “if” in theorems, so we
prefer “provided that” which is not often used in other contexts.
In Chapter 4, we prove some equalities by induction, although it is only for a finite range
of integers.
This quick summary of some mathematical language and logic is not complete by any
means. A more complete introduction is given in a book on the foundation of higher mathemat-
ics such as Bond and Keane [3]. The main point is to be able to read definitions and theorems
that are stated using formal mathematical language. Such statements must be read carefully and
digested in order to understand their meaning. In theorems, be sure to understand what are the
assumptions or hypotheses and what is the conclusion. In a definition, be sure to understand
the key aspects of the concept presented.
Bibliography
149
Index
C 1 , 45
degenerate basic solution, 7, 34
C 1 rescaling, 89
departing variable, 8
C 2 , 47
derivative, 45
Cauchy sequence, 142
second, 47
closed, 40
differentiable
closed ball, 40
continuously, 45
closed-graphed correspondence, 108
twice continuously, 47
closed-valued correspondence, 108
discount factor, 117, 125
closure, 41
dual linear program, 18, 19
compact, 41
dynamic programming problem, 117
compact-valued correspondence, 108
complement, set, 40
complementary slackness, 18, 22, 76 effective, 75
complete, 142 entering variable, 7
concave function, 82 equality constraint, 10
constraint extreme point, 32
equality, 10 extremizer, 56
requirement, 10 extremum, 56
151
152 Index