An Introduction to Continuous

Optimization: Foundations and

Fundamental Algorithms
Niclas Andreasson, Anton Evgrafov, and Michael Patriksson

Preface

The present book has been developed from course notes written by the
third author, and continuously updated and used in optimization courses
during the past several years at Chalmers University of Technology,
G
oteborg (Gothenburg), Sweden.
A note to the instructor: The book serves to provide lecture and exercise material in a first course on optimization for second to fourth year
students at the university. The books focus lies on providing a basis for
the analysis of optimization models and of candidate optimal solutions,
especially for continuous (even differentiable) optimization models. The
main part of the mathematical material therefore concerns the analysis
and algebra that underlie the workings of convexity and duality, and
necessary/sufficient local/global optimality conditions for unconstrained
and constrained optimization problems. Natural algorithms are then
developed from these principles, and their most important convergence
characteristics analyzed. The book answers many more questions of the
form Why/why not? than How?
This choice of focus is in contrast to books mainly providing numerical guidelines as to how these optimization problems should be
solved. The number of algorithms for linear and nonlinear optimization
problemsthe two main topics covered in this bookare kept quite low;
those that are discussed are considered classical, and serve to illustrate
the basic principles for solving such classes of optimization problems and
their links to the fundamental theory of optimality. Any course based
on this book therefore should add project work on concrete optimization problems, including their modelling, analysis, solution by practical
algorithms, and interpretation.
A note to the student: The material assumes some familiarity with
linear algebra, real analysis, and logic. In linear algebra, we assume
an active knowledge of bases, norms, and matrix algebra and calculus.
In real analysis, we assume an active knowledge of sequences, the basic

Preface
topology of sets, real- and vector-valued functions and their calculus of
differentiation. We also assume a familiarity with basic predicate logic,
since the understanding of proofs require it. A summary of the most
important background topics is found in Chapter 2, which also serves as
an introduction to the mathematical notation. The student is advised
to refresh any unfamiliar or forgotten material of this chapter before
reading the rest of the book.
We use only elementary mathematics in the main development of
the book; sections of supplementary material that provide an outlook
into more advanced topics and that require more advanced methods of
presentation are kept short, typically lack proofs, and are also marked
with an asterisk.
A detailed road map of the contents of the books chapters are provided at the end of Chapter 1. Each chapter ends with a selected number
of exercises which either illustrate the theory and algorithms with numerical examples or develop the theory slightly further. In Appendix A
solutions are given to most of them, in a few cases in detail. (Those
exercises marked exam together with a date are examples of exam
questions that have been given in the course Applied optimization at
G
oteborg University and Chalmers University of Technology since 1997.)
In our work on this book we have benefited from discussions with
Dr. Ann-Brith Str
omberg, presently at the FraunhoferChalmers Research Centre for Industrial Mathematics (FCC), Goteborg, and formerly at mathematics at Chalmers University of Technology, as well as
Dr. Fredrik Altenstedt, also formerly at mathematics at Chalmers University of Technology, and currently at Carmen Systems AB. We thank
the heads of undergraduate studies at mathematics, Goteborg University
and Chalmers University of Technology, Jan-Erik Andersson and Sven
J
arner, respectively, for reducing our teaching duties while preparing
this book. We also thank Yumi Karlsson for helping us by typesetting
a main part of the first draft based on the handwritten notes; after the
fact, we now realize that having been helped with this first draft made
us confident that such a tremendous task as that of writing a text book
would actually be possible. Finally, we thank all the students who gave
us critical remarks on the first versions during 2004 and 2005.
G
oteborg, May 2005

vi

Niclas Andreasson
Anton Evgrafov
Michael Patriksson

Contents

Introduction

1 Modelling and classification

1.1 Modelling of optimization problems . . . . . . . . . . . . .
1.2 A quick glance at optimization history . . . . . . . . . . .
1.3 Classification of optimization models . . . . . . . . . . . .
1.4 Conventions . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Applications and modelling examples . . . . . . . . . . . .
1.6 Defining the field . . . . . . . . . . . . . . . . . . . . . . .
1.7 On optimality conditions . . . . . . . . . . . . . . . . . . .
1.8 Soft and hard constraints . . . . . . . . . . . . . . . . . .
1.8.1 Definitions . . . . . . . . . . . . . . . . . . . . . .
1.8.2 A derivation of the exterior penalty function . . .
1.9 A road map through the material . . . . . . . . . . . . . .
1.10 On the background of this book and a didactics statement
1.11 Illustrating the theory . . . . . . . . . . . . . . . . . . . .
1.12 Notes and further reading . . . . . . . . . . . . . . . . . .
1.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . .

II

Fundamentals

1
3
3
9
11
13
15
16
16
18
18
19
20
25
26
27
28

31

2 Analysis and algebraA summary

33
2.1 Reductio ad absurdum . . . . . . . . . . . . . . . . . . . . 33
2.2 Linear algebra . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3 Convex analysis
3.1 Convexity of sets . . . . . . . . . . . . . . . . . . . . . . .
3.2 Polyhedral theory . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 Convex hulls . . . . . . . . . . . . . . . . . . . . .

41
41
42
42

Contents

3.3
3.4
3.5
3.6

III

3.2.2 Polytopes . . . . . . . . . . . . . . . . . . . .
3.2.3 Polyhedra . . . . . . . . . . . . . . . . . . . .
3.2.4 The Separation Theorem and Farkas Lemma
Convex functions . . . . . . . . . . . . . . . . . . . .
Application: the projection of a vector onto a convex
Notes and further reading . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . .

. .
. .
. .
. .
set
. .
. .

.
.
.
.
.
.
.

Optimality Conditions

4 An introduction to optimality conditions

4.1 Local and global optimality . . . . . . . . . .
4.2 Existence of optimal solutions . . . . . . . . .
4.2.1 A classic result . . . . . . . . . . . . .
4.2.2 Non-standard results . . . . . . . . .
4.2.3 Special optimal solution sets . . . . .
4.3 Optimality in unconstrained optimization . .
4.4 Optimality for optimization over convex sets .
4.5 Near-optimality in convex optimization . . . .
4.6 Applications . . . . . . . . . . . . . . . . . . .
4.6.1 Continuity of convex functions . . . .
4.6.2 The Separation Theorem . . . . . . .
4.6.3 Euclidean projection . . . . . . . . . .
4.6.4 Fixed point theorems . . . . . . . . .
4.7 Notes and further reading . . . . . . . . . . .
4.8 Exercises . . . . . . . . . . . . . . . . . . . .

45
47
52
57
66
69
69

73
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

75
75
78
78
81
83
84
88
95
96
96
98
99
100
106
107

5 Optimality conditions
111
5.1 Relations between optimality conditions and CQs at a glance111
5.2 A note of caution . . . . . . . . . . . . . . . . . . . . . . . 112
5.3 Geometric optimality conditions . . . . . . . . . . . . . . 114
5.4 The Fritz John conditions . . . . . . . . . . . . . . . . . . 118
5.5 The KarushKuhnTucker conditions . . . . . . . . . . . . 124
5.6 Proper treatment of equality constraints . . . . . . . . . . 128
5.7 Constraint qualifications . . . . . . . . . . . . . . . . . . . 130
5.7.1 MangasarianFromovitz CQ (MFCQ) . . . . . . . 131
5.7.2 Slater CQ . . . . . . . . . . . . . . . . . . . . . . . 131
5.7.3 Linear independence CQ (LICQ) . . . . . . . . . . 132
5.7.4 Affine constraints . . . . . . . . . . . . . . . . . . . 132
5.8 Sufficiency of the KKT conditions under convexity . . . . 133
5.9 Applications and examples . . . . . . . . . . . . . . . . . . 135
5.10 Notes and further reading . . . . . . . . . . . . . . . . . . 137
viii

Contents
5.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6 Lagrangian duality
141
6.1 The relaxation theorem . . . . . . . . . . . . . . . . . . . 141
6.2 Lagrangian duality . . . . . . . . . . . . . . . . . . . . . . 142
6.2.1 Lagrangian relaxation and the dual problem . . . . 142
6.2.2 Global optimality conditions . . . . . . . . . . . . 147
6.2.3 Strong duality for convex programs . . . . . . . . . 149
6.2.4 Strong duality for linear and quadratic programs . 154
6.2.5 Two illustrative examples . . . . . . . . . . . . . . 156
6.3 Differentiability properties of the dual function . . . . . . 158
6.3.1 Subdifferentiability of convex functions . . . . . . . 158
6.3.2 Differentiability of the Lagrangian dual function . 162
6.4 Subgradient optimization methods . . . . . . . . . . . . . 164
6.4.1 Convex problems . . . . . . . . . . . . . . . . . . . 164
6.4.2 Application to the Lagrangian dual problem . . . . 170
6.4.3 The generation of ascent directions . . . . . . . . . 173
6.5 Obtaining a primal solution . . . . . . . . . . . . . . . . 174
6.5.1 Differentiability at the optimal solution . . . . . . 175
6.5.2 Everetts Theorem . . . . . . . . . . . . . . . . . . 176
6.6 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . 177
6.6.1 Analysis for convex problems . . . . . . . . . . . . 177
6.6.2 Analysis for differentiable problems . . . . . . . . . 179
6.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 181
6.7.1 Electrical networks . . . . . . . . . . . . . . . . . . 181
6.7.2 A Lagrangian relaxation of the traveling salesman
problem . . . . . . . . . . . . . . . . . . . . . . . . 185
6.8 Notes and further reading . . . . . . . . . . . . . . . . . . 189
6.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

IV

Linear Programming

195

7 Linear programming: An introduction

197
7.1 The manufacturing problem . . . . . . . . . . . . . . . . . 197
7.2 A linear programming model . . . . . . . . . . . . . . . . 198
7.3 Graphical solution . . . . . . . . . . . . . . . . . . . . . . 199
7.4 Sensitivity analysis . . . . . . . . . . . . . . . . . . . . . . 199
7.4.1 An increase in the number of large pieces available 200
7.4.2 An increase in the number of small pieces available 201
7.4.3 A decrease in the price of the tables . . . . . . . . 202
7.5 The dual of the manufacturing problem . . . . . . . . . . 203
7.5.1 A competitor . . . . . . . . . . . . . . . . . . . . . 203
ix

Contents
7.5.2
7.5.3

A dual problem . . . . . . . . . . . . . . . . . . . . 203

Interpretations of the dual optimal solution . . . . 204

8 Linear programming models

205
8.1 Linear programming modelling . . . . . . . . . . . . . . . 205
8.2 The geometry of linear programming . . . . . . . . . . . . 210
8.2.1 Standard form . . . . . . . . . . . . . . . . . . . . 211
8.2.2 Basic feasible solutions and the Representation Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 214
8.2.3 Adjacent extreme points . . . . . . . . . . . . . . . 220
8.3 Notes and further reading . . . . . . . . . . . . . . . . . . 223
8.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
9 The simplex method
9.1 The algorithm . . . . . . . . . . . . . .
9.1.1 A BFS is known . . . . . . . .
9.1.2 A BFS is not known: phase I &
9.1.3 Alternative optimal solutions .
9.2 Termination . . . . . . . . . . . . . . .
9.3 Computational complexity . . . . . . .
9.4 Notes and further reading . . . . . . .
9.5 Exercises . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

225
225
226
232
236
237
238
238
239

10 LP duality and sensitivity analysis

10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . .
10.2 The linear programming dual . . . . . . . . . . . . . .
10.2.1 Canonical form . . . . . . . . . . . . . . . . . .
10.2.2 Constructing the dual . . . . . . . . . . . . . .
10.3 Linear programming duality theory . . . . . . . . . . .
10.3.1 Weak and strong duality . . . . . . . . . . . . .
10.3.2 Complementary slackness . . . . . . . . . . . .
10.4 The dual simplex method . . . . . . . . . . . . . . . .
10.5 Sensitivity analysis . . . . . . . . . . . . . . . . . . . .
10.5.1 Perturbations in the objective function . . . . .
10.5.2 Perturbations in the right-hand side coefficients
10.6 Notes and further reading . . . . . . . . . . . . . . . .
10.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

241
241
242
243
243
247
247
250
254
257
258
259
260
261

Algorithms

. .
. .
II
. .
. .
. .
. .
. .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

265

11 Unconstrained optimization
267
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 267
x

Contents
11.2 Descent directions . . . . . . . . . . . . . . . . . . .
11.2.1 Introduction . . . . . . . . . . . . . . . . . .
11.2.2 Newtons method and extensions . . . . . . .
11.3 The line search problem . . . . . . . . . . . . . . . .
11.3.1 A characterization of the line search problem
11.3.2 Approximate line search strategies . . . . . .
11.4 Convergent algorithms . . . . . . . . . . . . . . . . .
11.5 Finite termination criteria . . . . . . . . . . . . . . .
11.6 A comment on non-differentiability . . . . . . . . . .
11.7 Trust region methods . . . . . . . . . . . . . . . . . .
11.8 Conjugate gradient methods . . . . . . . . . . . . . .
11.8.1 Conjugate directions . . . . . . . . . . . . . .
11.8.2 Conjugate direction methods . . . . . . . . .
11.8.3 Generating conjugate directions . . . . . . . .
11.8.4 Conjugate gradient methods . . . . . . . . . .
11.8.5 Extension to non-quadratic problems . . . . .
11.9 A quasi-Newton method: DFP . . . . . . . . . . . .
11.10Convergence rates . . . . . . . . . . . . . . . . . . .
11.11Implicit functions . . . . . . . . . . . . . . . . . . . .
11.12Notes and further reading . . . . . . . . . . . . . . .
11.13Exercises . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

269
269
271
275
275
276
279
281
283
284
285
286
287
288
289
292
293
296
296
297
298

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

303
303
305
308
311
317
317
319
321
322

13 Constrained optimization
13.1 Penalty methods . . . . . . . . . . . . . . . . . . . . .
13.1.1 Exterior penalty methods . . . . . . . . . . . .
13.1.2 Interior penalty methods . . . . . . . . . . . .
13.1.3 Computational considerations . . . . . . . . . .
13.1.4 Applications and examples . . . . . . . . . . .
13.2 Sequential quadratic programming . . . . . . . . . . .
13.2.1 Introduction . . . . . . . . . . . . . . . . . . .
13.2.2 A penalty-function based SQP algorithm . . .
13.2.3 A numerical example on the MSQP algorithm .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

325
325
326
330
333
334
337
337
340
345

12 Optimization over convex sets

12.1 Feasible direction methods . . . . . . . . . .
12.2 The FrankWolfe algorithm . . . . . . . . .
12.3 The simplicial decomposition algorithm . .
12.4 The gradient projection algorithm . . . . .
12.5 Application: traffic equilibrium . . . . . . .
12.5.1 Model analysis . . . . . . . . . . . .
12.5.2 Algorithms and a numerical example
12.6 Notes and further reading . . . . . . . . . .
12.7 Exercises . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

xi

Contents
13.2.4 On recent developments in SQP algorithms
13.3 A summary and comparison . . . . . . . . . . . . .
13.4 Notes and further reading . . . . . . . . . . . . . .
13.5 Exercises . . . . . . . . . . . . . . . . . . . . . . .

VI

.
.
.
.

.
.
.
.

.
.
.
.

Appendix

A Answers to the exercises

Chapter 1: Modelling and classification . . . . . . .
Chapter 3: Convexity . . . . . . . . . . . . . . . . .
Chapter 4: An introduction to optimality conditions
Chapter 5: Optimality conditions . . . . . . . . . . .
Chapter 6: Lagrangian duality . . . . . . . . . . . .
Chapter 8: Linear programming models . . . . . . .
Chapter 9: The simplex method . . . . . . . . . . .
Chapter 10: LP duality and sensitivity analysis . . .
Chapter 11: Unconstrained optimization . . . . . . .
Chapter 12: Optimization over convex sets . . . . .
Chapter 13: Constrained optimization . . . . . . . .

.
.
.
.

346
346
347
348

351
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

353
353
356
358
360
361
363
365
366
368
370
371

References

373

Index

385

xii

Part I

Introduction

Modelling and
classification

1.1

Modelling of optimization problems

The word optimum is Latin, and means the ultimate ideal; similarly,
optimus means the best. Therefore, to optimize refers to trying to
bring whatever we are dealing with towards its ultimate state, that is,
towards its optimum. Let us take a closer look at what that means in
terms of an example, and at the same time bring the definition of the
term optimization forward, as the scientific field understands and uses
it.
Example 1.1 (a staff planning problem) Consider a hospital ward which
operates 24 hours a day. At different times of day, the staff requirement
differs. Table 1.1 shows the demand for reserve wardens during six work
shifts.
Table 1.1: Staff requirements at a hospital ward.
Shift
Hours
Demand

1
04
8

2
48
10

3
812
12

4
1216
10

5
1620
8

6
2024
6

Each member of staff works in 8 hour shifts. The goal is to fulfill the
demand with the least total number of reserve wardens.
Consider now the following interpretation of the term to optimize:
To optimize = to do something as well as is possible.

Modelling and classification

We utilize this description to identify the mathematical problem associated with Example 1.1; in other words, we create a mathematical model
of the above problem.
To do something: We identify activities which we can control and influence. Each such activity is associated with a variable whose
value (or, activity level) is to be decided upon (that is, optimized).
The remaining quantities are constants in the problem.
As well as: How good a vector of activity levels is is measured by a
real-valued function of the variable values. This quantity is to be
given a highest or lowest value, that is, we minimize or maximize,
depending on our goal; this defines the objective function.
Is possible: Normally, the activity levels cannot be arbitrarily large,
since an activity often is associated with the utilization of resources
(time, money, raw materials, labour, etcetera) that are limited;
there may also be requirements of a least activity level, resulting
from a demand. Some variables must also fulfill technical/logical
restrictions, and/or relationships among themselves. The former
can be associated with a variable necessarily being integer-valued
or non-negative, by definition. The latter is the case when products
are blended, a task is performed for several types of products, or
a process requires the input from more than one source. These
restrictions on activities form constraints on the possible choices
of the variable values.
Looking again at the problem described in Example 1.1, this is then
our declaration of a mathematical model thereof:
Variables We define
xj := number of reserve wardens whose first shift is j, j = 1, 2, . . . , 6.
Objective function We wish to minimize the total number of reserve
wardens, that is, the objective function, which we call f , is to
minimize f (x) := x1 + x2 + + x6 =
Constraints There are two types of constraints:
4

6
X
j=1

xj .

Modelling of optimization problems

Demand The demand for wardens during the different shifts can
be written as the following inequality constraints:
x6 + x1 8,

x1 + x2 10,
x2 + x3 12,

x3 + x4 10,
x4 + x5 8,
x5 + x6 6.

Logical There are two physical/logical constraints:

Sign xj 0,
j = 1, . . . , 6.
Integer xj integer,
j = 1, . . . , 6.
Summarizing, we have defined our first mathematical optimization
model, namely, that to
minimize f (x) :=
x

6
X

xj ,

j=1

subject to x1 + x6 8,

(last shift: 1)

x1 + x2 10,
x2 + x3 12,

(last shift: 2)
(last shift: 3)

x5 + x6 6,

(last shift: 6)

x3 + x4 10,
x4 + x5 8,

(last shift: 4)
(last shift: 5)

xj 0,
j = 1, . . . , 6,
xj integer,
j = 1, . . . , 6.

This problem has an optimal solution, which we denote by x , that

is, a vector of decision variable values which gives the objective function
its minimal value among the feasible solutions (that is, the vectors x
that satisfy all the constraints). In fact, the problem has at least two
optimal solutions: x = (4, 6, 6, 4, 4, 4)T and x = (8, 2, 10, 0, 8, 0)T; the
optimal value is f (x ) = 28. (The reader is asked to verify that they are
indeed optimal.)
The above model is of course a crude simplification of any real application. In practice, we would have to add requirements on the individuals competence as well as other more detailed restrictions, the planning
horizon is usually longer, employment rules and other conditions apply,
etcetera, which all contribute to a more complex model. We mention a
few successful applications of staffing problems below.
5

Modelling and classification

Example 1.2 (applications of staffing optimization problems) (a) It was
reported in 1990 that following a staffing problem application for the
Montreal municipality bus company, employing 3,000 bus drivers and
1,000 metro drivers and ticket salespersons and guards, the municipality
saved some 4 million Canadian dollars per year.
(b) Together with the San Francisco police department a group of
operations research scientists developed in 1989 a planning tool based
on a heuristic solution of the staff planning and police vehicle allocation
problem. It has been reported that it gave a 20% faster planning and
savings in the order of 11 million US dollars per year.
(c) In an 1986 application, scientists collaborating with United Airlines considered their crew scheduling problem. This is a complex problem, where the time horizon is long (typically, 30 minute intervals during
7 days), and the constraints that define a feasible pattern of allocating
staff to airplanes are defined by, among others, complicated work regulations. The savings reported then was 6 million US dollars per year.
Carmen Systems AB in Gothenburg develop and market such tools; customers include American Airlines, Lufthansa, SAS, and SJ; Carmen Systems has one of the largest concentrations of optimizers in Sweden.
Remark 1.3 (on the complexity of the variable definition) The variables
xj defined in Example 1.1 are decision variables; we say that, since the
selection of the values of these variables are immediately connected to
the decisions to be made in the decision problem, and they also contain,
within their very definition, a substantial amount of information about
the problem at hand (such as shifts being eight hours long).
In the application examples discussed in Example 1.2 the variable
definitions are much more complex than in our simple example. A typical decision variable arising in a crew scheduling problem is associated
with a specific staff member, his/her home base, information about the
crew team he/she works with, a current position in time and space, a
flight leg specified by flight number(s), additional information about the
staff members previous work schedule and work contract, and so on.
The number of possible combinations of work schedules for a given staff
member is nowadays so huge that not all variables in a crew scheduling problem can even be defined! (That is, the complete problem we
wish to solve cannot be written down.) The philosophy in solving a
crew scheduling problem is instead to algorithmically generate variables
that one believes may receive a non-zero optimal value, and most of
the computational effort lies in defining and solving good variable generation problems, whose result is (part of) a feasible work schedule for
given staff members. The term column generation is the operations researchers term for this process of generating variables.
6

Modelling of optimization problems

Remark 1.4 (non-decision variables) Not all variables in a mathematical optimization model are decision variables:
In linear programming, we will utilize slack variables whose role is to
take on the difference between the left-hand and the right-hand side of
an inequality constraint; the slack variable thereby aids in the transformation of the inequality constraint to an equality constraint, which is
more appropriate to work with in linear programming.
Other variables can be introduced into a mathematical model simply
in order to make the model more easy to state or interpret, or to improve
upon the properties of the model. As an example of the latter, consider
the following simple problem: we wish to minimize over R the special
one-variable function f (x) := maximum {x2 , x + 2}. (Plot the function
to see where the optimum is.) This is an example of a non-differentiable
function: at x = 2, for example, both the functions f1 (x) := x2 and
f2 (x) := x + 2 define the value of the function f , but they have different
derivatives there. One way to turn this problem into a differentiable one
is by introducing an additional variable. We let z take on the value of
the largest of f1 (x) and f2 (x) for a given value of x, and instead write
the problem as that to minimize z, subject to z R, x R, and the
additional constraints that x2 z and x + 2 z. Convince yourself
that this transformation is equivalent to the original problem in terms
of the set of optimal solutions in x, and that the transformed problem
is differentiable.
Figure 1.1 illustrates several issues in the modelling process, which
are forthwith discussed.

Reality

Communication
Evaluation
Simplification
Quantification
Limitation Modification
Interpretation
Optimization model

Algorithms
Results
Data

Figure 1.1: Flow chart of the modelling process

Modelling and classification

The decision problem faced in the fluffy reality is turned into an
optimization model, through a process with several stages. By communicating with those who have raised the issue of solving the problem in
the first place, one reaches an understanding about the problem to be
solved. In order to identify and describe the components of a mathematical model which is also tractable, it is often necessary to simplify
and also limit the problem somewhat, and to quantify any remaining
qualitative statements.
The modelling process does not come without difficulties. The communication can often be difficult, simply because the two parties speak
different languages in terms of describing the problem. The optimization problem quite often has uncertainties in the data, which moreover
are not always easy to collect or to quantify. Perhaps the uncertainties
are there for a purpose (such as in financial decision problems), but it
may be that data is uncertain because not enough effort has been put
into providing a good enough accuracy. Further, there is often a conflict
between problem solvability and problem realism.
The problem actually solved through the use of an optimization
methodology must be supplied with data, providing model constants and
parameters in functions describing the objective function and perhaps
also some of the constraints. For this optimization problem, an optimization algorithm then yields a result in the form of an optimal value
and/or optimal solution, if an optimal solution exists. This result is then
interpreted and evaluated, which may lead to alterations of the model,
and certainly to questions regarding the applicability of the optimal solution. The optimization model can also be altered slightly in order to
answer sensitivity analysis (what if?) type questions concerning the
effect of small variations in data.
The final problems that we will mention come at this stage: it is
crucial that the interpretation of the result makes sense to those who
wants to use the solution, and, finally, it must be possible to transfer the
solution back into the fluffy world where the problem came from.
The art of forming good optimization models is as much an art as a
science, and an optimization course can only really cover the latter. On
the other hand, this part of the modelling process should not be glossed
over; it is often possible to construct more than one form of an mathematical model that represents the same problem equally accurately, and the
computational complexity can differ substantially between them. Forming a good model is in fact as crucial to the success of the application as
the modelling exercise itself.
Optimization problems can be grouped together in classes, according
to their properties. According to this classification, the staffing problem
8

A quick glance at optimization history

is a linear integer optimization problem. In Section 1.3 we present some
major distinguishing factors between different problem classes.

1.2

A quick glance at optimization history

At Chalmers, courses in optimization are mainly given at the mathematics department. Mainly is the important word here, because courses
that have a substantial content of optimization theory and/or methodology can be found also at other departments, such as computer science,
the mechanical, industrial and chemical engineering departments, and at
the Gothenburg School of Economics. The reason is that optimization
is so broad in its applications.
From the mathematical standpoint, optimization, or mathematical
programming as it is sometimes called, rests on several legs: analysis,
topology, algebra, discrete mathematics, etcetera, build the foundation
of the theory, and applied mathematics subjects such as numerical analysis and mathematical parts of computer science build the bridge to
the algorithmic side of the subject. On the other side, then, with optimization we solve problems in a huge variety of areas, in the technical,
natural, life and engineering sciences, and in economics.
Before moving on, we would just like to point out that the term
program has nothing to do with computer program; a program is
understood to be a decision program, that is, a strategy or decision
rule. A mathematical program therefore is a mathematical problem
designed to produce a decision program.
The history of optimization is very long. Many, very often geometrical or mechanical, problems (and quite often related to warfare!) that
Archimedes, Euclid, Heron, and other masters from antiquity formulated
and also solved, are optimization problems. For example, we mention
the problem of maximizing the volume of a closed three-dimensional object (such as a sphere or a cylinder) built from a two-dimensional sheet
of metal with a given area.
The masters of two millenia later, like Bernoulli, Lagrange, Euler, and
Weierstrass developed variational calculus, studying problems in applied
physics (and still often with a mind towards warfare!) such as how to
find the best trajectory for a flying object.
The notion of optimality and especially how to characterize an optimal solution, began to be developed at the same time. Characterizations
of various forms of optimal solutions are indeed a crucial part of any basic
optimization course. (See Section 1.7.)
The scientific subject operations research refers to the study of decision problems regarding operations, in the sense of controlling complex
9

Modelling and classification

systems and phenomena. The term was coined in the 1940s at the height
of World War 2 (WW2), when the US and British military commands
hired scientists from several disciplines in order to try to solve complex
problems regarding the best way to construct convoys in order to avoid,
or protect the cargo ships from, enemy (read: German) submarines,
how to best cover the British isles with radar equipment given the scarce
availability of radar systems, and so on. The multi-disciplinarity of these
questions, and the common topic of maximizing or minimizing some objective function subject to constraints, can be seen as being the defining
moment of the scientific field. A better term than operations research
is decision science, which better reflects the scope of the problems that
can be, and are, attacked using optimization methods.
Among the scientists that took part in the WW2 effort in the US
and Great Britain, some were the great pioneers in placing optimization
on the map after WW2. Among them, we find several researchers in
mathematics, physics, and economics, who contributed greatly to the
foundations of the field as we now know it. We mention just a few here.
George W. Dantzig invented the simplex method for solving linear optimization problems during his WW2 efforts at Pentagon, as well as the
whole machinery of modelling such problems.1 Dantzig was originally
a statistician and famously, as a young Ph.D. student, provided solutions to some then unsolved problems in mathematical statistics that he
found on the blackboard when he arrived late to a lecture, believing they
were (indeed hard!) home work assignments in the course. Building on
the knowledge of duality in the theory of two-person zero-sum games,
which had been developed by the world-famous mathematician John von
Neumann in the 1920s, Dantzig was very much involved in developing
the theory of duality in linear programming, together with the various
characterizations of an optimal solution that is brought out from that
theory. A large part of the duality theory was developed in collaboration
with the mathematician Albert W. Tucker.
Several researchers interested in national economics studied transportation models at the same time, modelling them as special linear
optimization problems. Two of them, the mathematician Leonid W.
Kantorovich and the statistician Tjalling C. Koopmans received The
Bank of Sweden Prize in Economic Sciences in Memory of Alfred Nobel in 1975 for their contributions to the theory of optimum allocation
of resources. They had, in fact, both worked out some of the basics
1 As Dantzig explains in [Dan57], linear programming formulations in fact can first
be found in the work of the first theoretical economists in France, such as F. Quesnay
in 1760; they explained the relationships between the landlord, the peasant and the
artisan. The first practical linear programming problem solved with the simplex
method was the famous Diet problem.

10

Classification of optimization models

of linear programming, independently of Dantzig, at roughly the same
time. (Dantzig stands out among the three especially for creating an
efficient algorithm for solving such problems, but also as being the most
important developer of the theory of linear programming.)2

1.3

Classification of optimization models

We here develop a subset of problem classes that can be set up by contrasting certain aspects of a general optimization problem. We let
n

x Rn : vector of decision variables xj ,

j = 1, 2, . . . , n;

f : R R {} : objective function;
X Rn : ground set defined logically/physically;

gi : Rn R : constraint function defining restriction on x :

gi (x) bi ,
i I;
(inequality constraints)
gi (x) = di ,

i E.

(equality constraints)

We let bi R, i I, and di R, i E, denote the right-hand sides

of these constraints; without loss of generality, we could actually let
them all be equal to zero, as any constants can be incorporated into the
definitions of the functions gi , i I E.
The optimization problem then is to
minimize
x

f (x),

subject to gi (x) bi ,
gi (x) = di ,
x X.

(1.1a)
i I,
i E,

(1.1b)
(1.1c)
(1.1d)

(If it is really a maximization problem, then we change the sign of f .)

The problem type depends on the nature of the functions f and gi ,
and the set X. Let us look at some examples.
(LP)
Objective function linear: f (x) := cT x =
Pn Linear programming
n
T
j=1 cj xj , c R ; constraint functions affine: gi (x) := ai x bi ,
n
n
ai R , bi R, i I E; X := { x R | xj 0, j = 1, 2, . . . , n }.
(NLP) Nonlinear programming Some functions f, gi , i I E, are
nonlinear.

2 Incidentally, several other laureates in economics have worked with the tools of
optimization: Paul A. Samuelson (1970, linear programming), Kenneth J. Arrow
(1972, game theory), Wassily Leontief (1973, linear transportation models), Gerard
Debreu (1983, game theory), Harry M. Markowitz (1990, quadratic programming in
finance), John F. Nash, Jr. (1994, game theory), William Vickrey (1996, econometrics), and Daniel L. McFadden (2000, microeconomics).

11

Modelling and classification

Continuous optimization f, gi , i I E, are continuous on an open
set containing X; X is closed and convex.
(IP) Integer programming X {0, 1}n (binary) or X Zn (integer).
Unconstrained optimization I E := ; X := Rn .

Constrained optimization I E =
6 and/or X Rn .
Differentiable optimization f, gi , i I E, are continuously differentiable on X; further, X is closed and convex.
Non-differentiable optimization At least one of f, gi , i I E, is
non-differentiable.
(CP) Convex programming f is convex; gi , i I, are concave;
gi , i E, are affine; and X is closed and convex. (See Section 3.3 for
definitions.)
Non-convex programming The complement of the above.
In Figure 1.3 we show how the problems NLP, IP, and LP are related.
NLP
IP
LP

Figure 1.2: The relations among NLP, IP, and LP.

That LP is a special case of NLP is clear by the fact that a linear
function is a special kind of nonlinear function; that IP is a special case
of NLP can be illustrated by the fact that the constraint xj {0, 1} can
be written as the nonlinear constraint xj (1 xj ) = 0.3
Last, there is a subclass of IP that is equivalent to LP, that is, a class
of problems for which there exists at least one optimal solution which
3 If a non-negative integer variable x is upper bounded by the integer M , it is
Q (x k) = (xj 0)(x 1) (x M ) = 0, by which we
also possible to write M
j
j
j
k=0 j
restrict a continuous variable xj to be integer-valued.

12

Conventions
automatically is integer valued even without imposing any integrality
constraints, provided of course that the problem has any optimal solutions at all. We say that such problems have the integrality property.
An important example problem belonging to this category is the linear
single-commodity network flow problem with integer data; this class of
problems in turn includes as special cases such important problems as the
linear versions of the assignment problem, the transportation problem,
the maximum flow problem, and the shortest route problem.
Among the above list of problem classes, we distinguish, roughly only,
between two of the most important ones, as follows:
LP Linear programming applied linear algebra. LP is easy, because there exist algorithms that can solve every LP problem instance efficiently in practice.
NLP Nonlinear programming applied analysis in several variables.
NLP is hard, because there does not exist an algorithm that can
solve every NLP problem instance efficiently in practice. NLP is
such a large problem area that it contains very hard problems as
well as very easy problems. The largest class of NLP problems
that are solvable with some algorithm in reasonable time is CP (of
which LP is a special case).
Our problem formulation (1.1) does not cover the following:
infinite-dimensional problems (that is, problems formulated in function spaces rather than vector spaces);
implicit functions f and/or gi , i I E: then, no explicit formula
can be written down; this is typical in engineering applications,
where the value of, say, f (x) can be the result of a simulation (see
Section 11.11 for more details);
multiple-objective optimization:

minimize {f1 (x), f2 (x), . . . , fp (x)};

optimization under uncertainty, or, stochastic programming (that

is, where some of f , gi , i I E, are only known probabilistically).

1.4

Conventions

Let us denote the set of vectors satisfying the constraints (1.1b)(1.1d)

by S Rn , that is, the set of feasible solutions to the problem (1.1).
What exactly do we mean by solving the problem to
minimize f (x)?
x S

(1.2)
13

Modelling and classification

Since there is no explicit operation involved here, the question is warranted. The following two operations are however well-defined:
f := infimum f (x)
x S

denotes the infimum value of the function f over the set S; if and only
if the infimum value is attained at some point x in S (and then both
f and x necessarily are finite) we can write that
f := minimum f (x),
x S

(1.3)

and then we of course have that f (x ) = f . (When considering maximization problems, we obtain the analogous definitions of the supremum
and the maximum.)
The second operation defines the set of optimal solutions to the problem at hand:
S := arg minimum f (x);

x S

the set S S is nonempty if and only if the infimum value f is

attained. Finding at least one optimal solution,
x arg minimum f (x),
x S

(1.4)

is a special case which moreover defines an often much more simple task.
Consider the problem instance where S = { x R | x 0 } and
(
1/x, if x > 0,
f (x) :=
+, otherwise;
here, f = 0 but S = the value 0 is not attained for a finite value of
x, so the problem has a finite infimum value but not an optimal solution.
These examples lead to our convention in reading the problem (1.2):
the statement solve the problem (1.2) means find f and an x S ,
or conclude that S = .
Hence, it is implicit in the formulation that we are interested both
in the infimum value and in (at least) one optimal solution if one exists.
Whenever we are certain that only one of them is of interest we will
state so explicitly. We are aware that the interpretation of (1.2) may be
considered vague since no operation is visible; so, to summarize and
clarify our convention, it in fact includes two operations, (1.3) and (1.4).
There is a second reason for stating the optimization problem (1.1) in
the way it is, a reason which is computational. To solve the problem, we
almost always need to solve a sequence of relaxations/simplifications of
14

Applications and modelling examples

the original problem in order to eventually reach a solution. (These manipulations include Lagrangian relaxation, penalization, and objective
function linearization, to be developed later on.) When describing the
particular relaxation/simplification utilized, having access to constraint
identifiers [such as (1.1c)] certainly makes the presentation easier and
clearer. That will become especially valuable when dealing with various
forms of duality, when (subsets of) the constraints are relaxed.
A last comment on conventions: as it is stated prior to the problem formulation (1.1) the objective function f can in general take on
both as values. Since we are generally going to study minimization problems, we will only be interested in objective functions f having
the properties that (a) f (x) > for every feasible vector x, and (b)
f (x) < + for at least one feasible vector x. Such functions are known
as proper functions (which makes sense, as it is impossible to perform
a proper optimization unless these two properties hold). We will some
times refer to these properties, in particular by stating explicitly when f
can take on the value +, but we will assume throughout that f does
not take on the value . So, in effect then, we assume implicitly that
the objective function f is proper.

1.5

Applications and modelling examples

To give but a quick view of the scope of applications of optimization,

here is a subset of the past few years of projects performed at Chalmers
University of Technology or Gothenburg University:
Planning schedules for snow removal machines, disabled persons
transportation, and school transports
Optimization of personnel planning for airlines
Allocation of fault tolerances in complex assembly
Scheduling production and distribution of electricity
Scheduling paper cutting in paper mills
Optimization of engine performance for aircraft, boats, and cars
Engineering design by derivative-free optimization
Maintenance optimization for aircraft jet engines
Portfolio optimization under uncertainty for pension funds
Policy optimization in financial planning
Analysis of investment in future energy systems
Optimal wave-length and routing in optical networks
Intensity-modulated radiation therapy (IMRT)
Optimal congestion pricing in urban traffic networks

15

Modelling and classification

1.6

Defining the field

To define what the subject area of optimization encompasses is difficult,

given that it is connected to so many scientific areas in the natural and
technical sciences.
An obvious distinguishing factor is that an optimization model always
has an objective function and a group of constraints. On the other hand
by letting f 0 and I = then the generic problem (1.1) is that of a
feasibility problem for equality constraints, covering the important topic
of solving systems of linear equations, and by instead letting I E =
we obtain an unconstrained optimization problem. Both these special
cases are classic problems in numerical analysis, which most often deal
with the solution of a linear or non-linear system of equations.
We can here identify a distinguishing element between optimization
and numerical analysisthat an optimization problem often involve inequality constraints while a problem in numerical analysis does not. Why
does that make a difference? The reason is that while in the latter
case the analysis is performed on a manifoldpossibly even a linear
subspacethe analysis of an optimization problem must deal with feasible regions residing in different dimensions because of the nature of
inequality constraints being either active or inactive. As a result, there
will always be some kind of non-differentiability present in some associated functionals, while numerical analysis typically is smooth.
As an illustration (albeit beyond the scope of this book), we ask the
reader what the extension of the famous Implicit Function Theorem is
when we replace the system F (u, x) = 0k with, say, F (u, x) 0k ?

1.7

On optimality conditions

The most important topic of the book is the analysis of the local or global
optimality of a given feasible vector x in the problem (1.2), and its links
to the construction of algorithms for finding such vectors. While locally
or globally optimal vectors are the ones preferred, the types of vectors
that one can expect to reach for a general problem are referred to as
stationary points; we define what we mean by x S being a stationary
point in the problem (1.2) in non-mathematical terms as follows:
x S is a stationary point in the problem (1.2) if, with the
use only of first-order4 information about the problem at x ,
we cannot find a feasible descent direction at x .
4 This

means that we only utilize the values of f and f at

any constraint functions defining the set S.

16

x , and the same for

On optimality conditions
In mathematical terms, this condition can be written as follows:
x S is a stationary point in the problem (1.2) if f (x )
NS (x ) holds, where NS (x ) is the normal cone to S at x .
See Definition 4.25 for the definition of the normal cone.
In applications to all model problems considered in the book this condition collapses to something that is rather easy to check.5 In the most
general case, however, its use in formulating necessary optimality conditions requires further that the point x satisfies a regularity condition
referred to as a constraint qualification (CQ).
The connection between local or global optimality, stationarity and
regularity is given by the following two implications, which constitute
the perhaps two most important ones in the entire book:

x local min in (1.2)
= x stationary point in (1.2);
x regular

(1.5)


x stationary point in (1.2)
= x global min in (1.2).
the problem (1.2) is convex

(1.6)

The logical implication A = B is equivalent to B = A and

(A B) is equivalent to (A) (B).6 Hence, the implication (1.5)
means that if x is not stationary then it is not a local minimum in
(1.2) or it is not a regular point. Since the latter case is rare, the typical
case then is that a non-stationary point is not locally optimal, and in
the process of investigating whether x is stationary we quite often are
able to generate a feasible descent direction if there is one. Investigating stationarity is therefore important for two reasons: if we are at a
stationary point x , then x is an interesting candidate for an optimal
solution (when the problem is convex then x is even guaranteed to be a
global minimum); if x is not a stationary point, then we can generate a
feasible descent direction from x in order to move on to a better feasible
solution, thus constructing an iterative sequence of improved solutions.
For the construction and analysis of optimality conditions we refer the
reader to Section 4.3 for the unconstrained case (S := Rn ), to Section 4.4
for the case where the feasible set S is assumed to be convex, and to
Chapter 5 for the most general case when S need not be convex and
regularity issues become important.
5 For example, if S := Rn then N (x) is everywhere equal to 0n ; then the stationS
arity condition simply states that f (x ) = 0n .
6 Here, means not, means and, and means or.

17

Modelling and classification

1.8
1.8.1

Soft and hard constraints

Definitions

We have not yet discussed the role of different types ofP

constraints. In
the set covering problem, the constraints are of the form nj=1 aij xj 1,
i = 1, 2, . . . , m, where aij {0, 1}. These, as well as constraints of the
form xj 0 and xj {0, 1} are hard constraints, meaning that if they
are violated then the solution does not make much sense. Typically,
such constraints are technological ones; for example, if xj is associated
with the level of production, then a negative value has no meaning,
and therefore a negative value is never acceptable. A binary variable,
xj {0, 1}, is often logical, associated with the choice between something
being on or off, such as a production facility, a city being visited by
a traveling salesman, and so on; again, a fractional value like 0.7 makes
no sense, and binary restrictions almost always are hard.
Consider now a collection of constraints that are
P associated with the
capacity of production, and which have the form nj=1 uij xij ci , i =
1, 2, . . . , m, where xij denotes the level of production of an item/product
j using the production process i, uij is a positive number associated
with the use of a resource (man hours, hours before inspection of the
machine, etcetera) per unit of production of the item, and ci is the
available capacity of this resource in the production process. In some
circumstances, it is natural to allow for the left-hand side to become
larger than the capacity, because that production plan might still be
feasible, provided however that additional resources are made available.
We consider two types of ways to allow for this violation, and which give
rise to two different types of solution.
The first, which we are not quite ready to discuss here from a technical standpoint, is connected to the Lagrangian relaxation of the capacity constraints. If, when solving the corresponding Lagrangian dual
optimization problem, we terminate the solution process prematurely,
we will typically have a terminal primal vector that violates some of
the capacity constraints slightly. Since the capacity constraints are soft,
this solution may be acceptable.7 See Chapter 6 for further details on
Lagrangian duality.
Since it is natural that additional resources come at an additional
cost, an increase in the violation of this soft constraint should have the
effect of an increased cost in the objective function. In other words,
7 One interesting application arises when making capacity expansion decisions in production and work force planning problems (e.g., Johnson and Montgomery [JoM74, Example 4-14]) and in forest management scheduling (Hauer and
Hoganson [HaH96]).

18

Soft and hard constraints

violating a constraint should come with a penalty. Given a measure of
the cost of violating the constraints, that is, the unit cost of additional
resource, we may transform the resulting problem to an unconstrained
problem with a penalty function representing the original constraint.
Below, we relate soft constraints to exterior penalties.

1.8.2

A derivation of the exterior penalty function

Consider the standard nonlinear programming problem to

minimize f (x),

(1.7a)

subject to gi (x) 0,

i = 1, . . . , m,

(1.7b)

where f and gi , i = 1, . . . , m, are real-valued functions.

Consider the following relaxation of (1.7), where > 0:
minimize f (x) +
(x ,s)

m
X

si ,

(1.8a)

i=1

subject to gi (x) si ,
si 0,

i = 1, . . . , m,
i = 1, . . . , m.

(1.8b)

We interpret this problem as follows: by allowing the variable si to

become positive, we allow for extra slack in the constraint, at a positive
cost, si , proportional to the violation.
How do we solve this problem for a given value of > 0? We specialize the following result (see, for example, [RoW97, Proposition 1.35]):
for a function : Rn Rm R {+} one has in terms of p(s) :=
infimumx (x, s) and q(x) := infimums (x, s) that
infimum (x, s) = infimum q(x) = infimum p(s).
x

(x ,s)

In other words, we can solve an optimization problem in two types of

variables x and s by eliminating one of them (in our case, s) through
optimization, and then determine the best value of the remaining one.
Suppose then that we for a moment keep x fixed to an arbitrary
value. The above problem (1.8) then reduces to that to
minimize
s

m
X

si ,

i=1

subject to si gi (x),
si 0,

i = 1, . . . , m,

i = 1, . . . , m,
19

Modelling and classification

which clearly separates into the m independent problems to
minimize si ,
si

subject to si gi (x),
si 0.
This problem is trivially solvable: si := maximum {0, gi (x)}, that is,
si takes on the role of a slack variable for the constraint. Using this
expression in the problem (1.8) we finally obtain the problem to
minimize
f (x) +
n
x R

m
X
i=1

maximum {0, gi (x)}.

(1.9)

If the constraints instead are ofPthe form gi (x) 0, then the resulting
m
penalty function is of the form i=1 maximum {0, gi (x)}.
We note that the use of the linear penalty term in (1.8a) resulted
in the penalty problem (1.9); other penalty terms than (1.8a) lead to
other penalty problems. See Section 13.1 for a thorough discussion on
and analysis of penalty functions and methods.

1.9

A road map through the material

Chapter 2 gives a short overview of some basic material from calculus

and linear algebra that is used throughout the book. Familiarity with
these topics is therefore very important.
Chapter 3 is devoted to the study of convexity, a subject known as
convex analysis. We characterize the convexity of sets and real-valued
functions and show their relations. We give an overview of the special convex sets called polyhedra, which can be described by linear constraints. Parts of the theory covered, such as the Representation Theorem, Farkas Lemma, and the Separation Theorem, build the foundation
of the study of optimality conditions in Chapter 5, the theory of strong
duality in Chapter 6 and of linear programming in Chapters 710.
Chapter 4 gives an overview of topics associated with optimality,
including the result that locally optimal solutions are globally optimal in
a convex problem. We establish results regarding the existence of optimal
solutions, including Weierstrass Theorem, and establish basic logical
relationships between locally optimal solutions and characterizations in
terms of conditions of stationarity. Along the way, we define important
concepts such as the normal cone, the variational inequality, and the
Euclidean projection of a vector onto a convex set, and outline fixed
point theorems and their applications.
20

A road map through the material

Chapter 5 collects results leading up to the central KarushKuhn
Tucker (KKT) Theorem on the necessary conditions for the local optimality of a feasible point in a constrained optimization problem. Essentially, these conditions state that a given feasible vector x can only be
a local minimum if there is no descent direction at x which simultaneously is a feasible direction. In order to state the KKT conditions in
algebraic terms such that it can be checked in practice and such that
as few interesting vectors x as possible satisfy them, we must restrict
our study to problems and vectors satisfying some regularity properties.
These properties are called constraint qualifications (CQs); among them,
the classic one is that the active constraints are linearly independent
which is familiar from the Lagrange Multiplier Theorem in differential
calculus. Our treatment however is more general and covers weaker (that
is, better) CQs as well. The chapter begins with a schematic road map
for these results to further help in the study of this material.
Chapter 6 presents a broad picture of the theory of Lagrangian duality. Associated with the KKT conditions in the previous chapter is a
vector, known as the Lagrange multiplier vector. The Lagrange multipliers are associated with an optimization problem which is referred to as
the Lagrangian dual problem.8 The role of the dual problem is to define
a largest lower bound on the optimal value f of the original (primal)
problem. We establish the basic properties of this dual problem. In particular, it is always a convex problem, and therefore appealing to solve in
order to extract the optimal solution to the primal problem. This chapter is in fact much devoted to the topic of analyzing when it is possible
to generate, from an optimal dual solution, in a rather simple manner an
optimal primal solution. The most important term in this context then
is strong duality which refers to the occasion when the optimal values
in the two problems are equalonly then can the translation be relatively easy. Some of the results established are immediately transferable
to the important case of linear programming, whose duality theory is analyzed in Chapter 10. The main difference is that in the present chapter
we must work with more general tools, while for linear programming we
have access to a more specialized analysis; therefore, proof techniques,
for example in establishing strong duality, will be quite different. Additional topics include an analysis of optimization algorithms for the
solution of the Lagrangian dual problem, and applications.
8 The dual problem was first discovered in the study of (linear) matrix games by
John von Neumann in the 1920s, but had for a long time implicitly been used also for
nonlinear optimization problems before it was properly stated and studied by Arrow,
Hurwicz, Uzawa, Everett, Falk, Rockafellar, etcetera, starting in earnest in the 1950s.
By the way, the original problem is then referred to as the primal problem, a name
given by George Dantzigs father.

21

Modelling and classification

Chapters 710 are devoted to the study of linear programming (LP)
models and methods. Its importance is unquestionable: it has been
stated that in the 1980s LP problems was the scientific problem that
ate the most computing power in the world. While the efficiency of LP
solvers have multiplied since then, so has the speed of computers, and
LP models still define the most important problem area in optimization
in practice. (Partly, this is also due to the fact that integer optimization
solvers use LP techniques.) It is not only for this reason, however, that
we devote special chapters to this topic. Their optimal solutions can be
found using quite special techniques that are not common to nonlinear
programming. As was shown in Chapter 4 linear programs have optimal
solutions at the extreme points of the polyhedral feasible set. This fact,
together with the linearity of the objective function and the constraints,
means that a feasible-direction (descent) method can be cleverly devised.
Since we know that only extreme points are of interest, we start at one
extreme point, and then only consider as candidate search directions
those that point towards another (in fact, adjacent) extreme point. We
can generate such directions efficiently by using a basis representation
of the extreme points; the move from one extreme point to the other is
associated with a simple basis change. This procedure is known as the
simplex method, which was invented by George Dantzig in the 1940s.
In Chapter 7 a manufacturing problem illustrates the basics of linear
programming. The problem is solved geometrically and shown to have an
optimal extreme point. We investigate how the optimal solution changes
if the data of the problem is changed, and the linear programming dual
to the manufacturing problem is derived by using economical arguments.
Chapter 8 begins with a presentation of the axioms underlying the
use of LP models, and a general modelling technique is discussed. The
rest of the chapter deals with the geometry of LP models. It is shown
that every linear program can be transformed into the standard form
which is the form that the simplex method requires. We introduce the
concept of basic feasible solution (BFS) and discuss its connection to
extreme points. A version of the Representation Theorem adapted to
the standard form is presented, and we show that if there exists an
optimal solution to a linear program in standard form, then there exists
an optimal solution among the basic feasible solutions. Finally, we define
adjacency between extreme points and give an algebraic characterization
of adjacency which proves that the simplex method at each iteration step
moves from one extreme point to an adjacent one.
Chapter 9 presents the simplex method. We first assume that a
BFS is known at the start of the algorithm, and then describe what
to do when a BFS is not known. Termination characteristics of the
22

A road map through the material

algorithm are discussed: it is shown that if all the BFSs of the problem
are non-degenerate, then the algorithm terminates; if, however, there
exist degenerate BFSs there is a possibility that the algorithm cycles
between degenerate BFSs and hence never terminates. We introduce
Blands rule for choosing the adjacent BFS, which eliminates cycling.
We close the chapter by discussing the computational complexity of the
simplex algorithm.
In Chapter 10 linear programming duality is studied. We discuss how
to construct the linear programming dual to a general linear program
and present duality theory. The dual simplex method is developed, and
we discuss how the optimal solution of a linear program changes if the
right-hand sides or the objective function coefficients are modified.
Chapter 11 presents basic algorithms for differentiable, unconstrained
optimization problems. The typical optimization algorithm is iterative,
which means that a solution is approached through a sequence of trial
vectors, typically such that each consecutive objective value is strictly
lower than the previous one in a minimization problem. This improvement is possible because we can generate improving search directions
descent (ascent) directions in a minimization (maximization) problem
by means of solving an approximation of the original problem or the
optimality conditions. This approximate problem (for example, the system of Newton equations) is then combined with a line search, which approximately solves the original problem over the half-line defined by the
current iterate and the search direction. This idea of combining approximation (or, relaxation) with a line search (or, coordination) is the basic
methodology also for constrained optimization problems. Also, while
our opinion is that the subject of differentiable unconstrained optimization largely is a subject within numerical analysis rather than within the
optimization field, its understanding is important because the approximations/relaxations that we utilize in constrained optimization often
result in (essentially) unconstrained optimization subproblems. We develop a class of quasi-Newton methods in detail.
Chapter 12 presents classic algorithms for differentiable nonlinear
optimization over polyhedral sets, which utilize LP techniques when
searching for an improving direction. The basic algorithm is known
as the FrankWolfe algorithm, or the conditional gradient method; it
utilizes f (xk ) as the linear cost vector at iteration k, and the direction
towards any optimal extreme point y k has already in Chapter 4 been
shown to be a feasible direction of descent whenever xk is not stationary.
We also present an improvement in which we utilize (possibly) all the
previously generated extreme points to replace the line search with a
multi-dimensional one over the convex hull of these vectors. The gradi23

Modelling and classification

ent projection method extends the steepest descent method for unconstrained optimization problem in a natural manner. The subproblems
here are Euclidean projection problems which in this case are strictly
convex quadratic programming problems that can be solved efficiently
for some types of polyhedral sets. The convergence results reached show
that convexity of the problem is crucial in reaching good convergence
resultsnot only regarding the global optimality of limit points but regarding the nature of the set of limit points as well: Under convexity, the
gradient projection algorithm converges to an optimal solution provided
that one exists, even when the set of optimal solutions is unbounded;
the result immediately specializes to the steepest descent method.
Chapter 13 begins by describing natural approaches to nonlinearly
constrained optimization problems, wherein all (or, a subset of) the constraints are replaced by penalties. The resulting penalized problem is
then possible to solve by using techniques for unconstrained problems or
problems with convex feasible sets, like those we present in Chapters 11
and 12. In order to force the penalized problems to more and more
resemble the original one, the penalties are more and more strictly enforced. There are essentially two types of penalty functions: exterior and
interior penalties. Exterior penalty methods were devised mainly in the
1960s, and are perhaps the most natural ones; they are valid for almost
every type of explicit constraints, and are therefore amenable to solving
also non-convex problems. The penalty terms are gradually enforced
by letting larger and larger weights be associated with the constraints
in comparison with the objective function. Under some circumstances,
one can show that a finite value of these penalty parameters are needed,
but in general they must tend to infinity. Interior penalty methods are
also amenable to the solution of non-convex problems, but are perhaps
most naturally associated with convex problems, where they are quite
effective. In particular, the best methods for linear programming in
terms of their worst-case complexity are interior point methods which
are based on interior penalty functions. In this type of method, the
interior penalties are asymptotes with respect to the constraint boundaries; a decreasing value of the penalty parameters then allow for the
boundaries to be approached at the same time as the original objective
function come more and more into play. For both types of methods, we
reach convergence results on the convergence to KKT points in the general caseincluding estimates of the Lagrange multipliersand global
convergence results in the convex case.
Chapter 13 also describes another popular class of algorithms for
nonlinear programming problems with (twice) differentiable objective
and constraint functions. It is called Sequential Quadratic Program24

On the background of this book and a didactics statement

ming (SQP) and is, essentially, Newtons method applied to the KKT
conditions of the problem; there are, however, some modifications necessary. For example, because of the linearization of the constraints, it is
in general difficult to maintain feasibility in the process, and therefore
convergence cannot merely be based on line searches in the objective
function; instead one must devise a measure of goodness that takes
constraint violation into account. The classic approach is to utilize a
penalty function so that a constraint violation comes with a price; as
such the SQP method ties in with the penalty methods discussed above.

1.10

On the background of this book and a

didactics statement

This books foundation is the collection of lecture notes written by the

third author and used in the basic optimization course Applied Optimization for nearly ten years at Chalmers University of Technology
and Gothenburg University. The lecture notes have developed more and
more from being based on algorithms to mainly covering the fundamentals of optimization. With the addition of the first two authors has come
a further development of these fundamentals into the present book, in
which also our didactic wishes has begun to come true; the present book
significantly expands and improves upon the initial lecture notes.
The main inspiration in shaping the lecture notes and the book came
from the excellent text book by Bazaraa, Sherali, and Shetty [BSS93].
In the book the authors separate the basic theory (convexity, polyhedral
theory, separation, optimality, etcetera) from the algorithms devised for
solving nonlinear optimization problems, and they develop the theory
based on first principles, in a natural order. (The book is however too
advanced to be used in a first optimization course, it does not cover
linear programming, and some of the algorithmic parts are getting old.)
The main focus, as the title suggests, is the foundation of optimization models and methods. Hence, we have developed the chapters on
convexity and optimality conditions in detail. On the other hand, with
the exception of the classic topic of linear programming we have strived
to keep the algorithmic chapters concise, yet rigorous; among the plentiful of possible choices of algorithms we have made those choices that
appear the most natural given the appearance of the optimality conditions. The choices have therefore also become those of classic algorithms
rather than the most advanced and modern ones; being an undergraduate text we find this to be appropriate, and our text therefore also
serves as fundamental material that paves the way for more advanced
25

Modelling and classification

text books on optimization methods. Among those we mention especially that of Nocedal and Wright [NoW99], whose excellent graduate
level book on numerical optimization also is developed through a careful
selection of algorithms.
In writing the book we have also made a few additional didactic
developments. In almost every text book on optimization the topic of
linear programming is developed before that of nonlinear and convex
optimization, and linear programming duality is developed before Lagrangian duality. Teaching in this order may however feel unnatural
both for instructors and students: since Lagrangian duality is more general, but similar, to linear programming duality, the feeling is that more
or less the same material is repeated, or (which is even worse) that linear
programming is a rather strange special topic that we develop because
we must, but not because it is interesting. We have developed the material in this book such that linear programming emerges as a natural
special case of general convex programming, and having a duality theory
which is even richer.
In keeping with this idea of developing nonlinear programming before
linear programming, we should also have covered the simplex method last
in the book. This is a possibly conflicting situation, because we believe
that the simplex method should not be described merely as a feasibledirection method; its combinatorial nature is important, and the subject
of degeneracy, for example, is more naturally treated and understood by
developing the simplex method immediately following the development
of the connections between the geometry and linear algebra of linear
programming. This has been our choice, and we have consequently also
decided that iterative algorithms for general nonlinear optimization over
convex sets, especially polyhedra, should be developed before those for
more general constraints, the reason being that linear programming is
an important basis for these algorithms.
When teaching from this book, we have decided to stick to the chapter ordering with one exception: we introduce Chapter 11 as well as
hands-on computer exercises on algorithms for unconstrained optimization immediately after teaching from Chapter 4 on optimality conditions
for problems over convex sets. The motivation for doing so is our wish
to integrate, in our teaching, algorithms with fundamental theory; the
book itself separates the two topics.

1.11

Illustrating the theory

The subject of optimization, including both its basic theory and the
natural, basic, algorithmic development that is associated with solving
26

Notes and further reading

different classes of optimization models, is special compared to many
other mathematics subjects in that the ties between analysis/algebra and
geometry are so strong. This means, particularly, that optimization can
be learned, illustrated and revised (at least partially) by using geometric
tools. We give a few such examples.
The various techniques available for checking the convexity of a set
or a function can be illustrated by examples in one or two dimensions.
All the necessary and sufficient conditions for local optimality in constrained and unconstrained optimization given in Chapters 4 and 5 can
thus be illustrated. A simple method in R2 is as follows: choose a (suitably many times) differentiable function f such that a minimum over R2
is known. If the test problem should be unconstrained, one is immediately ready to work with the corresponding instruments; if the objective
function should be minimized subject to constraints, then choose the
feasible set such that the constrained optimum is different from the
unconstrained one and use the corresponding optimality conditions to
check that the optimal solution indeed satisfies them, or that an arbitrarily chosen non-optimal vector does not. The constraint qualifications
(CQs), which play an important role for general sets, can also be investigated through such examples.
In linear programming much of the above is specialized, since duality
and the KKT conditions have their correspondence in linear programming duality and optimality. A two-dimensional polyhedron, together
with a suitable objective function, can illustrate primaldual relationships such as the complementarity conditions, based on a problem with
a known solution; it can also test ones mastering of the simplex method.
The algorithmic chapters in Part V are similar with respect to these
tests; for each problem class and algorithm, it is possible, and instrumental, to construct a two-dimensional example and check that the algorithm
in question will reach a stationary point, if the convergence conditions
are met, or disprove convergence when the conditions are not. This also
provokes a revision of the optimality conditions of Chapters 4 and 5.
The variety of examples that can be thus constructed is immense.
This is in fact one of the reasons why we have decided to limit the
number of exercises; one can in fact create ones own set of exercises, and
will benefit greatly from doing so.

1.12

Notes and further reading

Extensive collections of optimization applications and models can be

found in several basic text books in operations research, such as [Wag75,
BHM77, Mur95, Rar98, Tah03]. The optimization modelling book by
27

Modelling and classification

Williams [Wil99] is a classic, now in its fourth edition. Modelling books
also exist for certain categories of applications; for example, the book
[EHL01] concerns the mathematical modelling and solution of optimization problems arising in chemical engineering applications. Further industrial applications are found in [AvG96, Casetal02].
Several accounts have been written during the past few years on the
origins of operations research and mathematical programming, the reasons being that we recently celebrated the 50th anniversaries of the simplex method (1997), the creation of ORSA (the Operations Research Society of America) (2002), and the Operational Research Society (2003),
as well as the 90th birthday of the inventor of the simplex method, George
Dantzig (2004). The special issue of the journal Operations Research,
vol. 50, no. 1 (2002), is filled with historical anecdotes, as are the books
[LRS91, GaA05] on the history of mathematical programming and operations research.

1.13

Exercises

Exercise 1.1 (modelling, exam 980819) A new producer of perfume wish

to get a break into a lucrative market. An exclusive fragrance, Chinelle,
is to be produced and marketed. With the equipment available it is
possible to produce the perfume using two alternative processes, and
the company also consider utilizing the services of a famous model when
launching it. In order to simplify the problem, let us assume that the
perfume is manufactured by the use of two main ingredientsthe first a
secret substance called MO and the second a more well-known mixture
of ingredients. The first of the two processes available provides three
grams of perfume for every unit of MO and two units of the standard
substance, while the other process gives five grams of perfume for every
two (respectively, three) units of the two main ingredients. The company
has at its disposal manufacturing processes that can produce at most
20,000 units of MO during the planning period and 35,000 units of the
standard mixture. Every unit of MO costs three EUR (it is manufactured
in France) to produce, and the other mixture only two EUR per unit.
One gram of the new perfume will cost fifty EUR. Even without any
advertising the company thinks they can sell 1000 grams of the perfume,
simply because of the news value. A famous model can be contracted
for commercials, costing 5,000 EUR per photo session (which takes half
an hour), and the company thinks that a campaign using his image can
raise the demand by about 200 grams per half hour of his time, but not
exceeding three hours (he has many other offers).
Formulate an LP model of the best production strategy problem.
28

Exercises
Exercise 1.2 (modelling) A computer company has estimated the service hours needed during the next five months; see Table 1.2.
Table 1.2: Number of service hours per month; Exercise 1.2.
Month
January
February
March
April
May

# Service hours
6000
7000
8000
9500
11,500

The service is performed by hired technicians; their number is 50 at

the beginning of January. Each technician can work up to 160 hours
per month. In order to cover the future demand of technicians new ones
must be hired. Before a technician is hired he/she undergoes a period of
training, which takes a month and requires 50 hours of supervision by a
trained technician. A trained technician has a salary of 15,000 SEK per
month (regardless of the number of working hours) and a trainee has a
monthly salary of 7500 SEK. At the end of each month on average 5%
of the technicians quit to work for another company.
Formulate a linear integer program whose optimal solution will minimize the total salary costs during the given time period, given that the
number of available service hours are enough to cover the demand.
Exercise 1.3 (modelling, exam 010821) The advertising agency ZAP
(Zetterstr
om, Anderson, and Pettersson) is designing their new office
with an open office space. The office is rectangular, with length l meters
and width b meters. Somewhat simplified, we may assume that each
working space requires a circle of diameter d and that the working spaces
must not overlap. In addition, each working space must be connected to
the telecom and computer network at one of the two possible connection
points in the office. As the three telephones have limited cable lengths
(the agency is concerned with the possible radiation danger associated
with hands-free phones and therefore do not use cord-less phones)ai
meters, respectively, i = 1, . . . , 3the work spaces must be placed quite
near the connection points.9 See Figure 1.3 for a picture of the office.
For simplicity we assume that the phone is placed at the center of the
work place. One of the offices walls is a large panorama window and the
9 All

the money went to other interior designs of the office space, so there is no
money left to buy more cable.

29

Modelling and classification

Window

b/2
Connection

b
Connection
l/2
l

Figure 1.3: Image of the office; Exercise 1.3

three partners all want to sit as close as possible to it. Therefore, they
decide to try to minimize the distance to the window for the workplace
that is the furthest away from it.
Formulate the problem of placing the three work places so that the
maximum distance to the panorama window is minimized, subject to all
the necessary constraints.
Exercise 1.4 (modelling, exam 010523) A large chain of department
stores wants to build distribution centers (warehouses) which will supply
30 department stores with goods. They have 10 possible locations to
choose between. To build a warehouse at location i, i = 1, . . . , 10, costs
ci MEUR and the capacity of a warehouse at that location would be ki
volume units per week. Department store j has a demand of ej volume
units per week. The distance between warehouse i and department store
j is dij km, i = 1, . . . , 10, j = 1, . . . , 30, and a certain warehouse can
only serve a department store if the distance is at most D km.
One wishes to minimize the cost of investing in the necessary distribution centers.
(a) Formulate a linear integer optimization model describing the optimization problem.
(b) Suppose each department store must be served from one of the
warehouses. What must be changed in the model?

30

Part II

Fundamentals

Analysis and
algebraA summary

II

The analysis of optimization problems and related optimization algorithms requires the basic understanding of formal logic, linear algebra,
and multidimensional analysis. This chapter is not intended as a substitute for the basic courses on these subjects but rather to give a brief
review of the notation, definitions, and basic facts which will be used in
the subsequent chapters without any further notice. If you feel inconvenient with the limited summaries presented in this chapter, contact any
of the abundant number of basic text books on the subject.

2.1

Reductio ad absurdum

Together with the absolute majority of contemporary mathematicians

we accept proofs by contradiction. The proofs in this group essentially
appeal to Aristotles law of the excluded middle, which states that any
proposition is either true or false. Thus, if some statement can be shown
to lead to a contradiction, we conclude that the original statement is
false.
Formally, proofs by contradiction amount to the following:
(A = B) (A B) (B A) (B = A).
In the same spirit, when proving A B, that is, (A = B)(B =
A), we often equivalently argue according to (A = B)(A = B)
(see, for example, the proof of Farkas Lemma 3.30).

Analysis and algebraA summary

2.2

Linear algebra

We will always work with finite dimensional Euclidean vector spaces Rn ,

the natural number n denoting the dimension of the space. Elements
v Rn will be referred to as vectors, and we will always think of them
as of n real numbers stacked on top of each other, i.e., v = (v1 , . . . , vn )T ,
vi being real numbers, and T denoting the transpose sign. The basic
operations defined for two vectors a = (a1 , . . . , an )T Rn and b =
(b1 , . . . , bn )T Rn , and an arbitrary scalar R are as follows:
addition: a + b := (a1 + b1 , . . . , an + bn )T Rn ;

multiplication by a scalar: a := (a1 , . . . , an )T Rn ;

Pn
scalar product between two vectors: (a, b) := i=1 ai bi R. The
scalar product will most often be denoted by aT b in the subsequent
chapters.

A linear subspace L Rn is a set enjoying the following two properties:

for every a, b L it holds that a + b L, and
for every R, a L it holds that a L.

An affine subspace A Rn is any set that can be represented as v +L :=

{ v + x | x L } for some vector v Rn and some linear subspace
L Rn .
We associate the norm, or length, of a vector v Rn with the following scalar product:
p
kvk := (v, v).

We will sometimes write |v| in place of kvk. The CauchyBunyakowski

Schwarz inequality says that (a, b) kakkbk for a, b Rn ; thus, we
may define an angle between two vectors via cos := (a, b)/(kakkbk).
We say that a Rn is orthogonal to b Rn if and only if (a, b) = 0 (i.e,
when cos = 0). The only vector orthogonal to itself is the zero vector
0n := (0, . . . , 0)T Rn ; moreover, this is the only vector having a zero
norm.
The scalar product is symmetric and bilinear, i.e., for every a, b, c, d
Rn , , , , R it holds that (a, b) = (b, a), and (a + b, c + d) =
(a, c) + (b, c) + (a, d) + (b, d).
A collection of vectors (v 1 , . . . , v k ) is said to be linearly independent
Pk
if and only if the equality i=1 i v i = 0n , where 1 , . . . , k are arbitrary
real numbers, implies that 1 = = k = 0. Similarly, a collection of
vectors (v 1 , . . . , v k ) is said to be affinely independent if and only if the
collection (v 2 v 1 , . . . , v k v 1 ) is linearly independent.
34

Linear algebra
The largest number of linearly independent vectors in Rn is n; any
collection of n linearly independent vectors in Rn is referred to as a
basis. The basis (v 1 , . . . , v n ) is said to be orthogonal if (v i , v j ) = 0 for
all i, j = 1, . . . , n, i 6= j. If, in addition, it holds that kvi k = 1 for all
i = 1, . . . , n, the basis is called orthonormal.
Given the basis (v 1 , . . . P
, v n ) in Rn , every vector v Rn can be writn
ten in a unique way as v = i=1 i v i , and the n-tuple (1 , . . . , n )T will
be referred to as coordinates of v in this basis. If the basis (v 1 , . . . , v n )
is orthonormal, then the coordinates i are computed as i = (v, v i ),
i = 1, . . . , n.
The space Rn will typically be equipped with the standard basis
(e1 , . . . , en ), where
ei := ( 0, . . . , 0 , 1, 0, . . . , 0 )T Rn .
| {z }
| {z }
i 1 zeros

n i zeros

This basis is orthogonal, and for every vector v = (v1 , . . . , vn )T Rn

we have (v, ei ) = vi , i = 1, . . . , n, which allow us to identify vectors and
their coordinates.
Now, consider two spaces Rn and Rk . All linear functions from Rn
to Rk may be described using a linear space of real matrices Rkn (i.e.,
with k rows and n columns). Given a matrix A Rkn it will often be
convenient to view it as a row of its columns, which are thus vectors in
Rk . Namely, let A Rkn have elements aij , i = 1, . . . , k, j = 1, . . . , n,
then we write A = (a1 , . . . , an ), where ai := (a1i , . . . , aki )T Rk , i =
1, . . . , n. The addition of two matrices and scalarmatrix multiplication
are defined inPa straightforward way. For v = (v1 , . . . , vn ) Rn we
n
define Av = i=1 vi ai Rk , where ai Rk are the columns of A. We
also define the norm of the matrix A by
kAk :=

max

v Rn :kv k=1

kAvk.

Well, this is an example of an optimization problem already!

For a given matrix A Rkn with elements aij we define AT Rnk
as the matrix with elements a
ij := aji i = 1, . . . , n, j = 1, . . . , k. We
can give a more elegant, but less straightforward definition: AT is the
unique matrix, satisfying the equality (Av, u) = (v, AT u) for all v Rn ,
u Rk . From this definition it should be clear that kAk = kAT k, and
that (AT )T = A.
Given two matrices A Rkn and B Rnm
Pn , we define the product
C = AB Rkm element-wise by cij =
=1 ai bj , i = 1, . . . , k,
j = 1, . . . , m. In other words, C = AB if and only if for all v
Rn , Cv = A(Bv). By definition, the matrix product is associative
35

Analysis and algebraA summary

(that is, A(BC) = (AB)C) for matrices of compatible sizes, but not
commutative (that is, AB 6= BA) in general. It is easy (and instructive)
to check that kABk kAkkBk, and that (AB)T = B T AT . Vectors
v Rn can be (and sometimes will be) viewed as matrices v Rn1 .
Check that this embedding is norm-preserving, that is, the norm of v
viewed as a vector equals the norm of v viewed as a matrix with one
column.
Of course, no discussion about norms could escape mentioning the
triangle inequality: for all a, b Rn it holds that ka+bk kak+kbk, as
well as its consequence (check this!) that for all A, B Rkn , kA+Bk
kAk + kBk holds. It will often be used in a little bit different form: for
all a, b Rn , kbk kak kb ak holds.

For square matrices A Rnn we can discuss the existence of the

unique matrix A1 , called the inverse of A, verifying the equality that
for all v Rn , A1 Av = AA1 v = v holds. If the inverse of a given
matrix exists, we call the latter nonsingular. The inverse matrix exists
if and only if the columns of A are linearly independent; if and only if
the columns of AT are linearly independent; if and only if the system of
linear equations Ax = v has a unique solution for every v Rn ; if and
only if the homogeneous system of equations Ax = 0n has x = 0n as its
unique solution. From this definition it follows that A is nonsingular if
and only if AT is nonsingular, and, furthermore, (A1 )T = (AT )1 and
therefore will be denoted simply by AT . At last, if A and B are two
nonsingular square matrices of the same size, then AB is nonsingular
(why?) and (AB)1 = B 1 A1 .
If, for some vector v Rn and scalar R it holds that Av = v,
then we call v an eigenvector of A, corresponding to the eigenvalue
of A. Eigenvectors, corresponding to a given eigenvalue, form a linear
subspace of Rn ; two nonzero eigenvectors, corresponding to two distinct
eigenvalues are linearly independent. In general, every matrix A Rnn
has n eigenvalues (counted with multiplicity), maybe complex, which are
furthermore roots of the characteristic equation det(AI n ) = 0, where
I n Rnn is the identity matrix, characterized by the fact that for all
v Rn it holds that I n v = v. The norm of the matrix A is in fact
equal to the largest absolute value of its eigenvalues. The matrix A is
nonsingular if and only if none of its eigenvalues are equal to zero, and
in this case the eigenvalues of A1 are equal to the inverted eigenvalues
of A. The eigenvalues of AT are equal to the eigenvalues of A.
We call A symmetric if and only if AT = A. All eigenvalues of
symmetric matrices are real, and eigenvectors corresponding to distinct
eigenvalues are orthogonal.
36

Analysis
Even if A is not square, AT A as well as AAT are square and symmetric. If the columns of A are linearly independent, then AT A is
nonsingular. (Similarly, if the columns of AT are linearly independent,
then AAT is nonsingular.)
Sometimes, we will use the following simple fact: for every A Rkn
with elements aij , i = 1, . . . , k, j = 1, . . . , n, it holds that aij = (e
ei , Aej ),
where (e
e1 , . . . , e
ek ) is the standard basis in Rk , and (e1 , . . . , en ) is the
standard basis in Rn , i = 1, . . . , k, j = 1, . . . , n.
We will say that A Rnn is positive semidefinite (respectively,
positive definite), and denote this by A  0 (respectively, A 0) if
and only if for all v Rn it holds that (v, Av) 0 (respectively, for all
v Rn , v 6= 0n , it holds that (v, Av) > 0). The matrix A is positive
semidefinite (respectively, positive definite) if and only if its eigenvalues
are nonnegative (respectively, positive).
For two symmetric matrices A, B Rnn we will write A  B
(respectively, A B) if and only if AB  0 (respectively, AB 0).

2.3

Analysis

Consider a sequence {xk } Rn . We will write limk xk = x, for some

x Rn , or just xk x, if and only if limk kxk xk = 0. We will say
in this case that {xk } converges to x, or, equivalently, that x is the limit
of {xk }. Owing to the triangle inequality, every sequence might have at
most one limit. (Why?) At the same time, there are sequences that
do not converge. Moreover, an arbitrary non-converging sequence might
contain a converging subsequence (or even several subsequences). We
will refer to the limits of such converging subsequences as limit points of
a given sequence {xk }.
A subset S Rn is called bounded if there exists a constant C > 0
such that for all x S : kxk C; otherwise, the set will be called
unbounded. Now, let S Rn be bounded. An interesting and very
important fact about bounded subsets S Rn is that every sequence
{xk } S contains a convergent subsequence.
The set B (x) = { y Rn | kx yk < } is called an open ball of
radius > 0 with center x Rn . A set S Rn is called open if and
only if for all x S there exists an > 0 such that B (x) S. A set
S is closed if and only if its complement Rn \ S is open. An equivalent
definition of closedness in terms of sequences is that a set S Rn is
closed if and only if all the limit points of any sequence {xk } S belong
to S. There exist sets which are neither closed nor open. The set Rn is
both open and closed. (Why?)
The closure of a set S Rn (notation: cl S) is the smallest closed
37

Analysis and algebraA summary

set containing S; equivalently, it can be defined as the intersection of all
closed sets in Rn containing S. More constructively, the closure cl S can
be obtained by considering all limit points of all sequences in S. The
closure is a closed set, and, quite naturally, the closure of a closed set
equals the set itself.
The interior of a set S Rn (notation: int S) is the largest open set
contained in S. The interior of an open set equals the set itself.
Finally, the boundary of a set S Rn (notation: bd S, or S) is the
set difference cl S \ int S.
A neighbourhood of a point x Rn is an arbitrary open set containing x.
Consider a function f : S R, where S Rn . We say that f is
continuous at x0 S if and only if for every sequence {xk } S such
that xk x0 it holds that limk f (xk ) = f (x0 ). We say that f is
continuous on S if and only if f is continuous at every point of S.
Now, let f : S R be a continuous function defined on some open
set S. We say that f (x0 ; d) R is a directional derivative of f at x0 S
in the direction d Rn if the following limit exists:
f (x0 , d) = lim
t0

f (x0 + td) f (x0 )

,
t

and then f will be called directionally differentiable at x0 S in the

direction d. Clearly, if we fix x0 S and assume that f (x0 ; d) exists
for some d, then for every 0 we have that f (x0 ; d) = f (x0 ; d).
If further f (x0 ; d) is linear in d, then there exists a vector called the
gradient of f at x0 S, denoted by f (x0 ) Rn , such that f (x0 ; d) =
(f (x0 ), d) and f is then called differentiable at x0 S. Naturally, we
say that f is differentiable on S if it is differentiable at every point in S.
Equivalently, the gradient can be defined as follows: f (x0 ) Rn is
the gradient of f at x0 if and only if there exists a function o : R R
such that
f (x) = f (x0 ) + (f (x0 ), x x0 ) + o(kx x0 k),

(2.1)

and moreover
lim
t0

o(t)
= 0.
t

(2.2)

For a differentiable function f : S R we can go one step further

and define second derivatives of f . Namely, a differentiable function f
will be called twice differentiable at x0 S if and only if there exists a
symmetric matrix denoted by 2 f (x0 ), and referred to as the Hessian
38

Analysis
matrix, and a function o : R R verifying (2.2), such that
1
f (x) = f (x0 ) + (f (x0 ), x x0 ) + (x x0 , 2 f (x0 )(x x0 ))
2
+ o(kx x0 k2 ).
(2.3)
Sometimes it will be convenient to discuss vector-valued functions
f : S Rk . We say that f = (f1 , . . . , fk )T is continuous if every fi ,
i = 1, . . . , k is; similarly we define differentiability. In the latter case,
by f Rnk we denote a matrix with columns (f1 , . . . , fk ). Its
transpose is often referred to as the Jacobian of f .
We call a continuous function f : S R continuously differentiable [notation: f C 1 (S)] if it is differentiable on S and the gradient
f : S Rn is continuous on S. We call f : S R twice continuously
differentiable [notation: f C 2 (S)], if it is continuously differentiable
and in addition every component of f : S Rn is continuously differentiable.
The following alternative forms of (2.1) and (2.3) will be useful some
times. If f : S R is continuously differentiable on S, and x0 S, then
for every x in some neighbourhood of x0 we have
f (x) = f (x0 ) + (f (), x x0 ),

(2.4)

where = x0 + (1 )x, for some 0 1, is a point between x and

x0 . (This result is also known as the mean-value theorem.) Similarly,
for twice differentiable functions we have
1
f (x) = f (x0 ) + (f (x0 ), x x0 ) + (x x0 , 2 f ()(x x0 )), (2.5)
2
with the same notation.
If f, g : Rn R are both differentiable, then f + g and f g are, and
(f + g) = f + g, (f g) = gf + f g. Moreover, if g is never zero,
then f /g is differentiable and (f /g) = (gf f g)/g 2 .
If both F : Rn Rk and h : Rk R are differentiable, then h(F )
is, and (h(F ))(x) = (F )(x) (h)(F (x)).
Finally, consider a vector-valued function F : Rk Rn Rk . Assume
that F is continuously differentiable in some neighbourhood Nu Nx
of the point (u0 , x0 ) Rk Rn , and that F (u0 , x0 ) = 0k . If the
square matrix u F (u0 , x0 ) is nonsingular, then there exists a unique
function : Nx Nu such that F ((x), x) 0k in Nx , where Nu
Nx Nu Nx is another neighbourhood of (u0 , x0 ). Furthermore, is
differentiable at x0 , and
(x0 ) = (u F (u0 , x0 ))1 x F (u0 , x0 ).
39

Analysis and algebraA summary

The function is known as the implicit function defined by the system
of equations F (u, x) = 0k .
Now we consider two special but very important cases.
Let for some a Rn define a linear function f : Rn R via f (x) :=
(a, x). By the CauchyBunyakowskiSchwarz inequality this function is
continuous, and writing f (x) f (x0 ) = (a, x x0 ) for every x0 Rn we
immediately identify from the definitions of the gradient and the Hessian
that f = a, 2 f = 0nn .
Similarly, for some A Rnn define a quadratic function f (x) =
(x, Ax). This function is also continuous, and since f (x) f (x0 ) =
(Ax0 , xx0 )+(x0 , A(xx0 ))+(xx0 , A(xx0 )) = ((A+AT )x0 , x
x0 ) + 0.5(x x0 , (A + AT )(x x0 )), we identify f (x0 ) = (A + AT )x0 ,
and 2 f (x0 ) = A + AT . If the matrix A is symmetric, then these
expressions reduce to f (x0 ) = 2Ax0 , and 2 f (x0 ) = 2A.

40

III

Convex analysis

3.1

Convexity of sets

Definition 3.1 (convex set) Let S Rn . The set S is convex if


x1 , x2 S
= x1 + (1 )x2 S
(0, 1)
holds.
A set S is convex if, from everywhere in S, all other points of S are
visible.
Figure 3.1 illustrates a convex set.

x1 + (1 )x2

x2

x1

Figure 3.1: A convex set. (For the intermediate vector shown, the value
of is 1/2.)
Two non-convex sets are shown in Figure 3.2.
Example 3.2 (convex and non-convex sets) By using the definition of a
convex set, the following can be established:
(a) The set Rn is a convex set.
(b) The empty set is a convex set.

Convex analysis

x1
x1 + (1 )x2
S

S
x2
Figure 3.2: Two non-convex sets.

(c) The set { x Rn | kxk a } is convex for every value of a R.

[Note: k k here denotes any vector norm, but we will almost always
use the 2-norm,
v
uX
u n 2
kxk2 := t
xj .
j=1

We will not write the index 2 , but instead use the 2-norm implicitly
whenever writing k k.]
(d) The set { x Rn | kxk = a } is non-convex for every a > 0.
(e) The set {0, 1, 2} is non-convex. (The second illustration in Figure 3.2 is such a case of a set of integral points in R2 .)

Proposition 3.3 (convex intersection) Suppose that Sk , k K, is any

collection of convex sets. Then, the intersection
\
S :=
Sk
kK

is a convex set.
Proof. Let both x1 and x2 belong to S. (If two such points cannot be
found, then the result holds vacuously.) Then, x1 Sk and x2 Sk for
all k K. Take (0, 1). Then, x1 + (1 )x2 Sk , k K, by the
convexity of the sets Sk . So, x1 + (1 )x2 kK Sk = S.

3.2
3.2.1

Polyhedral theory
Convex hulls

Consider the set V := {v 1 , v 2 }, where v 1 , v 2 Rn and v 1 6= v 2 . A set

naturally related to V is the line in Rn through v 1 and v 2 [see Figure
42

Polyhedral theory
3.3(b)], that is, { v 1 + (1 )v 2 | R } = { 1 v 1 + 2 v 2 | 1 , 2
R; 1 + 2 = 1 }. Another set naturally related to V is the line segment
between v 1 and v 2 [see Figure 3.3(c)], that is, { v 1 + (1 )v 2 |
[0, 1] } = { 1 v 1 + 2 v 2 | 1 , 2 0; 1 + 2 = 1 }. Motivated by these
examples we define the affine hull and the convex hull of a set in Rn .
Definition 3.4 (affine hull) Let V := {v 1 , . . . , v k } Rn . The affine
hull of V is the set

(
)
k

X

aff V := 1 v 1 + + k v k  1 , . . . , k R;
i = 1 .

i=1

The affine hull of an arbitrary set V Rn is the smallest affine

subspace that includes V .
A pointP
1 v 1 + + k v k , where v 1 , . . . , v k V and 1 , . . . , k R
k
such that
i=1 i = 1, is called an affine combination of the points
v 1 , . . . , v k (the number k of points in the sum must be finite).

Definition 3.5 (convex hull) Let V := {v1 , . . . , v k } Rn . The convex

hull of V is the set

)
(
k

X
1
k 
conv V := 1 v + + k v  1 , . . . , k 0;
i = 1 .

i=1

The convex hull of an arbitrary set V Rn is the smallest convex

set that includes V .
A pointP
1 v 1 + + k v k , where v 1 , . . . , v k V and 1 , . . . , k 0
k
such that
i=1 i = 1, is called a convex combination of the points
1
k
v , . . . , v (the number k of points in the sum must be finite).

Example 3.6 (affine hull, convex hull) (a) The affine hull of three or
more points in R2 not all lying on the same line is R2 itself. The convex
hull of five points in R2 is shown in Figure 3.4 (observe that the corners
of the convex hull of the points are some of the points themselves).
(b) The affine hull of three points not all lying on the same line in
R3 is the plane through the points.
(c) The affine hull of an affine space is the space itself and the convex
hull of a convex set is the set itself.
From the definition of convex hull of a finite set it follows that the
convex hull equals the set of all convex combinations of points in the set.
It turns out that this also holds for arbitrary sets.
43

Convex analysis

v2

v2

v1

v2

v1
(a)

v1
(b)

(c)

Figure 3.3: (a) The set V . (b) The set aff V . (c) The set conv V .
v1
v5

v2

v4

v3
Figure 3.4: The convex hull of five points in R2 .

Proposition 3.7 Let V Rn . Then, conv V is the set of all convex

combinations of points of V .
Proof. Let Q be the set of all convex combinations of points of V .
The inclusion Q conv V follows from the definition of a convex set
(since conv V is a convex set). We next show that Q is a convex set. If
x1 , x2 Q, then x1 = 1 a1 + + k ak and x2 = 1 b1 + + m bm
1
for some
ak , b1 , . . . , bm V and 1 , . . . , k , 1 , . . . , m 0 such
Pk a , . . . ,P
m
that i=1 i = i=1 i = 1. Let (0, 1). Then
x1 + (1 )x2 = 1 a1 + + k ak

+ (1 )1 b1 + + (1 )m bm ,

44

Polyhedral theory
and since 1 + + k + (1 )1 + + (1 )m = 1, we have
that x1 + (1 )x2 Q, so Q is convex. Since Q is convex and V Q
it follows that conv V Q (from the definition of convex hull of an arbitrary set in Rn it follows that conv V is the smallest convex set that
contains V ). Therefore Q = conv V .
Proposition 3.7 shows that every point of the convex hull of a set
can be written as a convex combination of points from the set. It tells,
however, nothing about how many points that are required. This is the
content of Caratheodorys Theorem.
Theorem 3.8 (Caratheodorys Theorem) Let x conv V , where V
Rn . Then, x can be expressed as a convex combination of n + 1 or fewer
points of V .
Proof. From Proposition 3.7 it follows that x = 1P
a1 + + m am for
m
1
m
some a , . . . , a V and 1 , . . . , m 0 such that i=1 i = 1. We assume that this representation of x is chosen so that x cannot be expressed
as a convex combination of fewer than m points of V . It follows that
no two of the points a1 , . . . , am are equal and that 1 , . . . , m > 0. We
prove the theorem by showing that m n + 1. Assume that m > n + 1.
Then the set {a1 , . . . , am } must be affinely
dependent, so
exist
Pm
Pthere
m
1 , . . . , m R, not all zero, such that i=1 i ai = 0n and i=1 i = 0.
Let > 0 be such that 1 + 1 , . . . , m + m are non-negative with at
least one of them zero (such an exists since the s are
P all positive and
i
at least one of the s must be negative). Then, x = m
i=1 (i + i )a ,
and if terms with zero coefficients are omitted this is a representation of
x with fewer than m points; this is a contradiction.

3.2.2

Polytopes

We are now ready to define the geometrical object polytope.

Definition 3.9 (polytope) A subset P of Rn is a polytope if it is the
convex hull of finitely many points in Rn .
Example 3.10 (polytopes) (a) The set shown in Figure 3.4 is a polytope.
(b) A cube and a tetrahedron are polytopes in R3 .
We next show how to characterize a polytope as the convex hull of
its extreme points.
45

Convex analysis
Definition 3.11 (extreme point) A point v of a convex set P is called
an extreme point if whenever v = x1 + (1 )x2 , where x1 , x2 P
and (0, 1), then v = x1 = x2 .
Example 3.12 (extreme points) The set shown in Figure 3.3(c) has the
extreme points v 1 and v 2 . The set shown in Figure 3.4 has the extreme
points v 1 , v 2 , and v 3 . The set shown in Figure 3.3(b) does not have any
extreme points.
Lemma 3.13 Let V := {v1 , . . . , v k } Rn and let P be the polytope
conv V . Then, each extreme point of P lies in V .
Proof.
/ V is an extreme
of P . We have that
Pk Assume that w
Ppoint
k
w = i=1 i v i , for some i 0 such that i=1 i = 1. At least one of
the i s must be nonzero, say 1 . If 1 = 1 then w = v 1 , a contradiction,
so 1 (0, 1). We have that
w = 1 v 1 + (1 1 )
Pk

k
X
i=2

i
vi .
1 1

P
Since i=2 i /(1 1 ) = 1 we have that ki=2 i /(1 1 )v i P , but
w is an extreme point of P so w = v 1 , a contradiction.
Proposition 3.14 Let V := {v1 , . . . , v k } Rn and let P be the polytope conv V . Then P is equal to the convex hull of its extreme points.
Proof. Let Q be the set of extreme points of P . If v i Q for all i =
1, . . . , k we are done, so assume that v 1
/ Q. Then v 1 = u + (1 )w
Pk
for some (0, 1) and u, w P , u 6= w. Further, u = i=1 i v i and
P
P
w = ki=1 i v i , for some 1 , . . . , k , 1 , . . . , k 0 such that ki=1 i =
Pk
i=1 i = 1. Hence,
v1 =

k
X
i=1

i v i + (1 )

k
X
i=1

i v i =

k
X
i=1

(i + (1 )i )v i .

It must hold that 1 , 1 6= 1, since otherwise u = w = v 1 , a contradiction. Therefore,

v1 =

k
X
i=2

46

i + (1 )i
vi ,
1 (1 + (1 )1 )

Polyhedral theory
Pk
and since i=2 (i + (1 )i )/(1 1 (1 )1 ) = 1 it follows
that conv V = conv (V \ {v1 }). Similarly, every v i
/ Q can be removed,
and we end up with a set T V such that conv T = conv V and T Q.
On the other hand, from Lemma 3.13 we have that every extreme point
of the set conv T lies in T and since conv T = conv V it follows that Q
is the set of extreme points of conv T , so Q T . Hence, T = Q and we
are done.

3.2.3

Polyhedra

Closely related to the polytope is the polyhedron. We will show that

every polyhedron is the sum of a polytope and a polyhedral cone. In
the next subsection we show that a set is a polytope if and only if it is
a bounded polyhedron.
Definition 3.15 (polyhedron) A subset P of Rn is a polyhedron if there
exist a matrix A Rmn and a vector b Rm such that
P = { x Rn | Ax b }
holds.
The importance of polyhedra is obvious, since the set of feasible
solutions of every linear programming problem is a polyhedron.
Example 3.16 (polyhedra) (a) Figure 3.5 shows the bounded polyhedron P := { x R2 | x1 2; x1 + x2 6; 2x1 x2 4 }.
(b) The unbounded polyhedron P := { x R2 | x1 + x2 2; x1
x2 2; 3x1 x2 0 } is shown in Figure 3.6.
Often it is hard to decide whether a point in a convex set is an extreme
point or not. This is not the case for the polyhedron since there is an
algebraic characterization of the extreme points of such a set. Given
{ x Rn | Ax b } we refer to the rows of A
an x
x b that are
fulfilled with equality as the equality subsystem of A
x b, and denote
consists of the rows i {1, . . . , m} of (A, b)
x =
b)
it by A
b, that is, (A,
= bi , where Ai. is the ith row of A. The number of rows
such that Ai. x

in (A, b) is denoted by m.

Theorem 3.17 (algebraic characterization of extreme points) Let x
P = { x Rn | Ax b }, where A Rmn has rank A = n and
be the equality subsystem of A
x = b
b Rm . Further, let A
x b.
= n.

Then x is an extreme point of P if and only if rank A

47

Convex analysis
x2

6
x1 = 2

5
4

2x1 x2 = 4

3
P

x1 + x2 = 6

1
1

x1

Figure 3.5: Illustration of the bounded polyhedron P := { x R2 | x1

2; x1 + x2 6; 2x1 x2 4 }.

is an extreme point of P . If A
Proof. [=] Suppose that x
x < b
+ 1n , x
1n P if > 0 is sufficiently small. But x
=
then x
is an extreme point,
1/2(
x +1n )+1/2(
x1n ) which contradicts that x
so assume that at least one of the rows in A
x b is fulfilled with equality.
x = b
is the equality subsystem of A
n 1,
If A
x b and rank A
n
m

then there exists a w 6= 0 such that Aw = 0 , so x + w, x w P

= 1/2(
if > 0 is sufficiently small. But x
x + w) + 1/2(
x w), which
= n.
is an extreme point. Hence, rank A
contradicts that x
= n. Then, x
is the unique solution
[=] Assume that rank A
= b.
If x
is not an extreme point of P it follows that x
=
to Ax
u + (1 )v for some (0, 1) and u, v P , u 6= v. This yields
and since Au b and Av b it follows
+ (1 )Av
= b,
that Au
= Av
= b,
which contradicts that x
is the unique solution to
that Au
Therefore x
= b.
must be an extreme point.
Ax

48

Polyhedral theory

x2
3x1 x2 = 0
6
5
4
P

x1 x2 = 2

2
1
1

6
5
x1 + x2 = 2
4

x1

Figure 3.6: Illustration of the unbounded polyhedron P := { x R2 |

x1 + x2 2; x1 x2 2; 3x1 x2 0 }.
Corollary 3.18 Let A Rmn and b Rm . The number of extreme
points of the polyhedron P := { x Rn | Ax b } is finite.
Proof. The theorem implies that the number of extreme points of P
never exceeds the number of ways in which n objects can be chosen from
a set of m objects, that is, the number of extreme points is less than or
equal to
 
m!
m
=
.
n
n!(m n)!
We are done.
Remark 3.19 Since the number of extreme points is finite, the convex
hull of the extreme points of a polyhedron is a polytope.
49

Convex analysis
Definition 3.20 (cone) A subset C of Rn is a cone if x C whenever
x C and > 0.
Example 3.21 (cone) (a) The set { x Rn | Ax 0m }, where A
Rmn , is a cone. Since this set is a polyhedron, this type of cone is
usually called a polyhedral cone.
(b) Figure 3.7(a) illustrates a convex cone and Figure 3.7(b) illustrates a non-convex cone in R2 .

(a)

(b)

Figure 3.7: (a) A convex cone in R2 . (b) A non-convex cone in R2 .

We have arrived at the most important theorem of this section,

namely the Representation Theorem, which tells that every polyhedron
is the sum of a polytope and a polyhedral cone. The Representation
Theorem will have great importance in the linear programming theory
in Chapter 8.
Theorem 3.22 (Representation Theorem) Let A Rmn and b Rm .
Let Q := { x Rn | Ax b }, P denote the convex hull of the extreme
points of Q, and C := { x Rn | Ax 0m }. If rank A = n then
Q = P + C := { x Rn | x = u + v for some u P and v C }. In
other words, every polyhedron (that has at least one extreme point) is
the sum of a polytope and a polyhedral cone.
be the corresponding equality subsystem
x = b
Q and A
Proof. Let x

of A
x b. We prove the theorem by induction on the rank of A.

If rank A = n it follows from Theorem 3.17 that x is an extreme

P + C, since 0n C. Now assume that x
P + C for
point of Q, so x
50

Polyhedral theory
n, and choose x
= k 1.
Q with k rank A
Q with rank A
all x

= 0m
Then there exists a w 6= 0n such that Aw
. If || is sufficiently
+ w Q. (Why?) If x
+ w Q for all R
small it follows that x
we must have Aw = 0n which implies rank A n 1, a contradiction.
+ + w Q. Then if
Suppose that there exists a largest + such that x
+
is the equality subsystem of A(
x + w) = b
A(
x + + w) b we must
k. (Why?) By the induction hypothesis it then follows
have rank A
+ + w P + C. On the other hand, if x
+ w Q for all 0
that x
+ (w) Q for all 0
then Aw 0m , so w C. Similarly, if x
+ (w) Q
then w C, and if there exists a largest such that x

+ (w) P + C.
then x
Above we got a contradiction if none of + or existed. If only
+ + w P + C and w C, and
one of them exists, say + , then x
P + C. Otherwise, if both + and exist then
it follows that x
+
+ w P + C and x
+ (w) P + C, and x
can be written as
x
P + C. We have
a convex combination of these points, which gives x
P + C for all x
Q with k 1 rank A n and the
shown that x
theorem follows by induction.
Example 3.23 (illustration of the Representation Theorem) Figure 3.8(a)
can be written as a
shows a bounded polyhedron. The interior point x
convex combination of the extreme point x5 and the point v on the
boundary, that is, there is a (0, 1) such that
= x5 + (1 )v.
x

Further, the point v can be written as a convex combination of the

extreme points x2 and x3 , that is, there exists a (0, 1) such that
v = x2 + (1 )x3 .

This gives that

= x5 + (1 )x2 + (1 )(1 )x3 ,
x
and since , (1 ), (1 )(1 ) 0 and
+ (1 ) + (1 )(1 ) = 1
lies in the convex hull of the extreme points x2 ,
holds we have that x
3
5
x , and x .

Figure 3.8(b) shows an unbounded polyhedron. The interior point x

can be written as a convex combination of the extreme point x3 and the
point v on the boundary, that is, there exists a (0, 1) such that
= x3 + (1 )v.
x

51

Convex analysis
The point v lies on the halfline { x R2 | x = x2 + (x1 x2 ),
0 }. All the points on this halfline are feasible, which gives that if the
polyhedron is given by { x R2 | Ax b } then
A(x2 + (x1 x2 )) = Ax2 + A(x1 x2 ) b,

0.

But then we must have that A(x1 x2 ) 02 since otherwise some component of A(x1 x2 ) tends to infinity as tends to infinity. Therefore
x1 x2 lies in the cone C := { x R2 | Ax 02 }. Now there exists a
0 such that
v = x2 + (x1 x2 ),
and it follows that
= x3 + (1 )x2 + (1 )(x1 x2 ),
x
is the sum of a point in the
so since (1 ) 0 and x1 x2 C, x
convex hull of the extreme points and a point in the polyhedral cone C.
in a polyhedron is normally
Note that the representation of a vector x
not uniquely determined; in the case of Figure 3.8(a), for example, we
as a convex combination of x1 , x4 , and x5 .
can also represent x

3.2.4

The Separation Theorem and Farkas Lemma

We introduce the important concept of separation and use it to show

that every polytope is a polyhedron.
Theorem 3.24 (Separation Theorem) Suppose that the set C Rn is
closed and convex, and that the point y does not lie in C. Then there
exist a vector 6= 0n and R such that T y > and T x for
all x C.
We postpone the proof of this theorem since it requires the Weierstrass Theorem 4.7 and the first order necessary optimality condition
given in Proposition 4.23(b). Instead the proof is presented in Section 4.4.
The Separation Theorem is easy to describe geometrically: If a point
y does not lie in a closed and convex set C, then there exists a hyperplane
that separates y from C.
Example 3.25 (illustration of the Separation Theorem) Consider the convex and closed set C := { x R2 | kxk 1 } (i.e., C is the unit disc in
52

Polyhedral theory

x2
x1

x1
x

x4

x
x4

x2
x3

x5
(a)

(b)

Figure 3.8: Illustration of the Representation Theorem (a) in the

bounded case, and (b) in the unbounded case.

R2 ), and the point y := (1.5, 1.5)T . Since y

/ C the Separation Theorem implies that there exists a line in R2 that separates y from C. (This
line is however not unique: in Figure 3.9 we see that the line given by
= (1, 1)T and = 2 is a separating line, while the one constructed in
the proof of Theorem 3.24 is actually a tangent plane to C.)

Theorem 3.26 A set P is a polytope if and only if it is a bounded

polyhedron.
Proof. [=] From the Representation Theorem 3.22 we get that a
bounded polyhedron is the convex hull of its extreme points and hence
by Remark 3.19 a polytope.
[=] Let V := {v 1 , . . . , v k } Rn and let P be the polytope conv V .
In order to prove that P is a polyhedron we must show that P is the
solution set of some finite system of linear inequalities. The idea of the
proof is to define a bounded polyhedron consisting of the coefficients and
right-hand sides of all valid inequalities for P and then apply the Representation Theorem to select a finite subset of those valid inequalities.
53

Convex analysis
x2

2
y = (1.5, 1.5)T

T x = x1 + x2 = 2
C
2

x1

Figure 3.9: Illustration of the Separation Theorem: the unit disk is

separated from y by the line { x R2 | x1 + x2 = 2 }.

To carry this out, consider the set Q Rn+1 defined as


  
a 
n
n
n
T
a R ; b R; 1 a 1 ; 1 b 1; a v b, v V .
b 

Since V is a finite set, Q is a polyhedron. Further, Q is bounded, so by

the Representation Theorem we know that Q is the convex hull of its
extreme points given by
 1
 m
a
a
,...,
.
b1
bm
We will prove that the linear system
(a1 )T x b1 , . . . , (am )T x bm ,

(3.1)

defines P . We first show that P is contained in the solution set of (3.1).

P . Then x
= 1 v 1 + +k v k for some 1 , . . . , k 0
Suppose that x
54

Polyhedral theory
such that

Pk

i=1

i = 1. Thus, for each i = 1, . . . , m, we have

= (ai )T (1 v 1 + + k v k ) = 1 (ai )T v 1 + + k (ai )T v k

(ai )T x
1 bi + + k bi = bi ,
satisfies all inequalities in (3.1).
so x

In order to show that the solution set of (3.1) is contained in P , let x

be a solution to (3.1) and suppose that x

/ P . Then, by the Separation
>
Theorem 3.24 there exist a vector 6= 0n and R such that T x
and T x for all x P . By scaling T x by a positive constant
n
n
if necessary, we may assume
 that 1 1 and 1 1. That

is, we may assume that

Q. So we may write

 
 1
 m
a

a
+ + m
,
= 1
bm

b1

for some 1 , . . . , m 0 such that

Pm

i=1

i = 1. Therefore,

= 1 (a1 )T x
+ + m (am )T x
1 b1 + + m bm = .
T x
> . So x
P , which completes
But this is a contradiction, since T x
the proof.
We introduce the concept of finitely generated cones. In the proof
of Farkas Lemma below we will use that finitely generated cones are
convex and closed, and in order to show this fact we prove that finitely
generated cones are polyhedral sets.
Definition 3.27 (finitely generated cone) A finitely generated cone is
one that is generated by a finite set, that is, a cone of the form
cone {v 1 , . . . , v m } := { 1 v 1 + + m v m | 1 , . . . , m 0 },
where v 1 , . . . , v m Rn . Note that if A Rmn , then the set { y Rm |
y = Ax; x 0n } is a finitely generated cone.
Recall that a cone that is a polyhedron is called a polyhedral cone.
We show that a finitely generated cone is always a polyhedral cone and
vice versa.
Theorem 3.28 A convex cone in Rn is finitely generated if and only if
it is polyhedral.
55

Convex analysis
Proof. [=] Assume that C is the finitely generated cone
cone {v 1 , . . . , v m },
where v 1 , . . . , v m Rn . From Theorem 3.26 we know that polytopes
are polyhedral sets, so conv {0n , v 1 , . . . , v m } is the solution set of some
linear inequalities
(a1 )T x b1 , . . . , (ak )T x bk .

(3.2)

Since the solution set of these inequalities contains 0n we must have

b1 , . . . , bk 0. We show that C is the polyhedral cone A that equals
the solution set of the inequalities of (3.2) for which bi = 0. Since
v 1 , . . . , v m A we have C A. In order to show that A C, assume
that w A. Then w is in the solution set of (3.2) if > 0 is sufficiently
small. Hence there exists a > 0 such that
w { x Rn | (a1 )T x b1 , . . . , (ak )T x bk }
= conv {0n , v 1 , . . . , v m } C,

so w (1/)C = C. Hence A C, and C = A.

[=] Suppose that C is a polyhedral cone in Rn . Let P be a polytope
in Rn such that 0n int P (that is, 0n lies in the interior of P ). Then
C P is a bounded polyhedron and hence the Representation Theorem
gives that C P = conv {v 1 , . . . , v m }, where v 1 , . . . , v m is the set of
extreme points of C P . We show that C is the finitely generated cone
cone {v 1 , . . . , v m }. Since v 1 , . . . , v m C and C is a polyhedral cone we
get that cone {v 1 , . . . , v m } C. If c C, then, since 0n int P , there
exists a > 0 such that c P . Thus,
c C P = conv {v 1 , . . . , v m } cone {v 1 , . . . , v m },

and so c (1/)cone {v 1 , . . . , v m } = cone {v 1 , . . . , v m }. Hence it follows that C cone {v 1 , . . . , v m }, and C = cone {v 1 , . . . , v m }.

Corollary 3.29 Finitely generated cones in Rn are convex and closed.
Proof. Halfspaces, that is, sets of the form { x Rn | aT x b } for
some vector a Rn and b R, are convex and closed. (Why?) By
the theorem a finitely generated cone is the intersection of finitely many
halfspaces and thus the corollary follows from Proposition 3.3 and the
fact that intersections of closed sets are closed.
We close this section by proving the famous Farkas Lemma by using
the Separation Theorem 3.24 and the fact that finitely generated cones
are convex and closed.
56

Convex functions
Theorem 3.30 (Farkas Lemma) Let A Rmn and b Rm . Then,
exactly one of the systems
Ax = b,

(I)

x0 ,
and
AT 0n ,

(II)

b > 0,

has a feasible solution, and the other system is inconsistent.

Proof. Let C := { y Rm | y = Ax; x 0n }. If (I) is infeasible then
b
/ C. The set C is a finitely generated cone. Hence, by Corollary 3.29,
it follows that C is convex and closed so by the Separation Theorem 3.24
there exist a vector 6= 0m and R such that bT > and y T
for all y C, that is,
xT AT ,

x 0n .

(3.3)

Since 0m C it follows that 0, so bT > 0, and if there exists an

0n such that x
T AT > 0, then (3.3) cannot hold for any (if 0
x
n
then
x 0 and (
x)T AT =
xT AT tends to infinity as tends to
infinity). Therefore we must have that xT AT 0 for all x 0n , and
this holds if and only if AT 0n , which means that (II) is feasible.
0n , then
On the other hand, if (I) has a feasible solution, say x
T T
then x
A
= bT
> 0.
A
x = b, so if there is a solution to (II), say ,
T
n
n
> 0 (since x
0 ), a contradiction. Hence (II) is inBut then A
feasible.

3.3

Convex functions

Definition 3.31 (convex function) Suppose that S Rn . A function

S if
f : Rn R {+} is convex at x

xS

(0, 1)
= f (
x + (1 )x) f (
x) + (1 )f (x).

x + (1 )x S
S.
The function f is convex on S if it is convex at every x

57

Convex analysis
In other words, a convex function is such that a linear interpolation
never is lower than the function itself.1
From the definition follows that a function f : Rn R {+} is
convex on a convex set S Rn if and only if

x1 , x2 S
= f (x1 + (1 )x2 ) f (x1 ) + (1 )f (x2 ).
(0, 1)
Definition 3.32 (concave function) Suppose that S Rn . A function
S if f is convex at x
.
f : Rn R {+} is concave at x
S.
The function f is concave on S if it is concave at every x
Definition 3.33 (strictly convex/concave function) A function f : Rn
S if
R {+} is strictly convex at x

x S, x 6= x

(0, 1)
= f (
x + (1 )x) < f (
x) + (1 )f (x).

x + (1 )x S

The function f is strictly convex (concave) on S if it is strictly convex

S.
(concave) at every x

In other words, a strictly convex function is such that a linear interpolation is strictly above the function itself.
Figure 3.10 illustrates a strictly convex function.
f (x1 )
f (x1 ) + (1 )f (x2 )

f
f (x2 )

f (x1 + (1 )x2 )

x1

x1 + (1 )x2

x2

Figure 3.10: A strictly convex function.

1 Words like lower and above should be understood in the sense of the comparison between the y-coordinates of the respective function at the same coordinates
in x.

58

Convex functions
Example 3.34 (convex functions) By using the definition of a convex
function, the following can be established:
(a) The function f : Rn R defined by f (x) := kxk is convex on
Rn .
n
T
Pn(b) Let c R , a R. The affine function xn7 f (x) := c x + a =
j=1 cj xj + a is both convex and concave on R . The affine functions
are also the only finite functions that are both convex and concave.
Figure 3.11 illustrates a non-convex function.
f (x1 )

f (x1 + (1 )x2 )
f (x1 ) + (1 )f (x2 )

f (x2 )
f

x1

x1 + (1 )x2

x2

Figure 3.11: A non-convex function.

Proposition 3.35 (sums of convex functions) Suppose that S Rn . Let

fk , k K, with K finite, be a collection of functions fk : Rn R{+}.
S, then
Let k 0, k K. If each function fk , k K, is convexP
at x
so is the function f : Rn R{+} defined by f (x) := kK k fk (x).
In particular, suppose that S is convex and that fk is convex on S
for each k K. Then,
X
k fk is convex on S
k 0, k K =
kK

holds.
Proof. The proof is left as an exercise.
Proposition 3.36 (convexity of composite functions) Suppose that S
Rn and P R. Let further g : S R be a function which is convex on
S, and f : P R be convex and non-decreasing [y x = f (y) f (x)]
on P . Then, the composite function f (g) is convex on the set { x S |
g(x) P }.
59

Convex analysis
Proof. Let x1 , x2 S { x Rn | g(x) P }, and (0, 1). Then,
f (g(x1 + (1 )x2 )) f (g(x1 ) + (1 )g(x2 ))

f (g(x1 )) + (1 )f (g(x2 )),

where the first inequality follows from the convexity of g and the property of f being increasing, and the second inequality from the convexity
of f .
The following example functions are important in the development
of penalty methods in linear and nonlinear optimization; their convexity
is crucial when developing a convergence theory for such algorithms.
Example 3.37 (convex composite functions) Suppose that the function
g : Rn R is convex.
(a) The function x 7 log(g(x)) is convex on the set { x Rn |
g(x) < 0 }. (This function will be of interest in the analysis of interior
point methods; see Section 13.1.)
(b) The function x 7 1/g(x) is convex on the set { x Rn | g(x) <
0 }.
[Note: This function is convex, but the above rule for composite
functions cannot be used. Utilize the definition of a convex function
instead. The domain of the function must here be limited, because
x 7 1/x is convex only for positive x.]
(c) The function x 7 1/ log(g(x)) is convex on the set { x Rn |
g(x) < 1 }.
[Note: This function is convex, but the above rule for composite
functions cannot be used. Utilize the definition of a convex function
instead. The domain of the function must here be limited, because
x 7 1/x is convex only for positive x.]

We next characterize the convexity of a function on Rn by the convexity of its epigraph in Rn+1 .
[Note: the graph of a function f : Rn R is the boundary of epi f ,
which still resides in Rn+1 . See Figure 3.12 for an example, corresponding to the convex function in Figure 3.10.]
Definition 3.38 (epigraph) The epigraph of a function f : Rn R
{+} is the set
epi f := { (x, ) Rn R | f (x) }.

(3.4)

epiS f := { (x, ) S R | f (x) }.

(3.5)

The epigraph of the function f restricted to the set S Rn is

60

Convex functions
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
1111111111111111111111111111111111
0000000000000000000000000000000000
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111
0000000000000000000000000000000000
1111111111111111111111111111111111

epi f

x
Figure 3.12: A convex function and its epigraph.

Theorem 3.39 Suppose that S Rn is a convex set. Then, the function f : Rn R {+} is convex on S if, and only if, its epigraph
restricted to S is a convex set in Rn+1 .
Proof. [=] Suppose that f is convex on S. Let (x1 , 1 ), (x2 , 2 )
epiS f . Let (0, 1). By the convexity of f on S,
f (x1 + (1 )x2 ) f (x1 ) + (1 )f (x2 )
1 + (1 )2 .

Hence, [x1 + (1 )x2 , 1 + (1 )2 ] epiS f , so epiS f is a convex

set in Rn+1 .
[=] Suppose that epiS f is convex. Let x1 , x2 S, whence
(x1 , f (x1 )), (x2 , f (x2 )) epiS f.

Let (0, 1). By the convexity of epiS f , it follows that

[x1 + (1 )x2 , f (x1 ) + (1 )f (x2 )] epiS f,

that is, f (x1 + (1 )x2 ) f (x1 ) + (1 )f (x2 ). Hence, f is convex

on S.
When f is in C 1 (once differentiable, with continuous partial derivatives) or C 2 (twice differentiable, with continuous partial second derivatives), then convexity can be characterized also in terms of these derivatives. The results show how with stronger differentiability properties the
characterizations become more and more useful in practice.
61

Convex analysis
Theorem 3.40 (convexity characterizations in C 1 ) Let f C 1 on an
open convex set S.
(a) f is convex on S f (y) f (x) + f (x)T (y x), for all
x, y S.
(b) f is convex on S [f (x) f (y)]T (x y) 0, for all
x, y S.
The result in (a) states, in words, that every tangent plane to the
function surface in Rn+1 lies on, or below, the epigraph of f , or, that
a first-order approximation is below f .
The result in (b) states that f is monotone on S.
[Note: when n = 1, the result in (b) states that f is convex if and
only if its derivative f is non-decreasing, that is, that it is monotonically
increasing.]
Proof. (a) [=] Take x1 , x2 S and (0, 1). Then,
f (x1 ) + (1 )f (x2 ) f (x1 + (1 )x2 )
[ > 0]

f (x1 ) f (x2 ) (1/)[f (x1 + (1 )x2 ) f (x2 )].

Let 0. Then, the right-hand side of the above inequality tends to the
directional derivative of f at x2 in the direction of (x1 x2 ), so that in
the limit it becomes
f (x1 ) f (x2 ) f (x2 )T (x1 x2 ).
The result follows.
[=] We have that
f (x1 ) f (x1 + (1 )x2 ) + (1 )f (x1 + (1 )x2 )T (x1 x2 ),
f (x2 ) f (x1 + (1 )x2 ) + f (x1 + (1 )x2 )T (x2 x1 ).

Multiply the inequalities by and (1 ), respectively, and add them

together to get the result sought.
(b) [=] Using (a), and the two inequalities
f (y) f (x) + f (x)T (y x),
T

f (x) f (y) + f (y) (x y),

x, y S,

x, y S,

added together, yields that [f (x)f (y)]T (xy) 0, for all x, y S.

[=] The mean-value theorem states that
f (x2 ) f (x1 ) = f (x)T (x2 x1 ),
62

(3.6)

Convex functions
where x = x1 +(1)x2 for some (0, 1). By assumption, [f (x)
f (x1 )]T (x x1 ) 0, so (1 )[f (x) f (x1 )]T (x2 x1 ) 0. From
this follows that f (x)T (x2 x1 ) f (x1 )T (x2 x1 ). By using this
inequality and (3.6), we get that f (x2 ) f (x1 ) + f (x1 )T (x2 x1 ).
We are done.
Figure 3.13 illustrates part (a) of Theorem 3.40.
f

 
x
f (
x)
1
Figure 3.13: A tangent plane to the graph of a convex function.
f (
x) + f (
x)(y x
)

By replacing the inequalities in (a) and (b) in the theorem by strict

inequalities, and adding the requirement that x 6= y holds in the statements, we can establish a characterization also of strictly convex functions. The statement in (a) then says that the tangential hyperplane lies
strictly below the function except at the tangent point, and (b) states
that the gradient mapping is strictly monotone.
Still more can be said in C 2 :
Theorem 3.41 (convexity characterizations in C 2 , I) Let f be in C 2 on
an open, convex set S Rn .
(a) f is convex on S 2 f (x) is positive semidefinite for all x S.
(b) 2 f (x) is positive definite for all x S = f is strictly convex
on S.
[Note: When n = 1 and S is an open interval, the above reduce to the
following familiar results: (a) f is convex on S if and only if f (x) 0
for every x S; (b) f is strictly convex on S if f (x) > 0 for every
x S.]
S. We must show
Proof. (a) [=] Suppose that f is convex and let x
that pT 2 f (
x)p 0 for all p Rn holds.
+ p S whenever || 6= 0
Since S open, for any given p Rn , x
is small enough. We utilize Theorem 3.40(a) as follows: by the twice
63

Convex analysis
differentiability of f ,
f (
x + p) f (
x) + f (
x)T p,

1
x)p + o(2 ).
f (
x + p) = f (
x) + f (
x)T p + 2 pT 2 f (
2

(3.7)
(3.8)

Subtracting (3.8) from (3.7), we get

1 2 T 2
p f (
x)p + o(2 ) 0.
2
Dividing by 2 and letting 0 it follows that pT 2 f (
x)p 0.
[=] Suppose that the Hessian matrix is positive semidefinite at each
point in S. The proof depends on the following second-order mean-value
theorem: for every x, y S, there exists [0, 1] such that
1
f (y) = f (x)+f (x)T (yx)+ (yx)T 2 f [x+(yx)](yx). (3.9)
2

By assumption, the last term in (3.9) is non-negative, whence we obtain

the convexity characterization in Theorem 3.40(a).
(b) [=] By the assumptions, the last term in (3.9) is always positive
when y 6= x, whence we obtain the strict convexity characterization in
C1.
It is important to note that the opposite direction in the result (b) is
false. A simple example that establishes this fact is the function defined
by f (x) := x4 , S := R; f is strictly convex on R (why?), but its second
derivative at zero is f (0) = 0.
The case of quadratic functions is interesting to mention in particular.
For quadratic functions, that is, functions of the form
f (x) := (1/2)xT Qx q T x + a,

(3.10)

for some symmetric matrix Q Rnn , vector q Rn and scalar a R,

it holds that 2 f (x) Q for every x where f is defined, so the value
2 f (x) does not depend on x. In this case, we can state a stronger
result than in Theorem 3.41: the quadratic function f is convex on the
open, convex set S Rn if and only if Q is positive semidefinite; f is
strictly convex on S if and only if Q is positive definite. To prove this
result is simple from the above result for general C 2 functions, and is
left as an exercise.
What happens when S is not full-dimensional (which is often the
case)? Take, for example, f (x) := x21 x22 and S := { x R2 | x1
R; x2 = 0 }. Then, f is convex on S but 2 f (x) is not positive semidefinite anywhere on S. The below result covers this type of case. Its proof
is left as an exercise.
64

Convex functions
Theorem 3.42 (convexity characterizations in C 2 , II) Let S Rn be a
nonempty convex set and f : Rn R be in C 2 on Rn . Let C be the
subspace parallel to the affine hull of S. Then,
f is convex on S

pT 2 f (x)p 0 for every x S and p C.

In particular, when S has a nonempty interior, f is convex if and only

if 2 f (x) is positive semidefinite for every x S.
We have already seen that the convexity of a function is intimately
connected to the convexity of a certain set, namely the epigraph of the
function. The following result shows that a particular type of set, defined
by those vectors that bound a convex function from above, is a convex
set. Later, we will utilize this result to establish the convexity of feasible
sets in some optimization problems.
Definition 3.43 (level set) Let g : Rn R be a function. The level set
of g with respect to the value b R is the set
levg (b) := { x Rn | g(x) b }.

(3.11)

Figure 3.14 illustrates a level set of a convex function.

f
b

levf (b)

Figure 3.14: A level set of a convex function.

Proposition 3.44 (convex level sets from convex functions) Suppose that
the function g : Rn R is convex. Then, for every value of b R, the
level set levg (b) is a convex set. It is moreover closed.
Proof. The result follows immediately from the definitions of a convex
set and a convex function. Let x1 , x2 both satisfy the constraint that
65

Convex analysis
g(x) b holds, and let (0, 1). (If not two such points x1 , x2 can
be found, then the result holds vacuously.) Then, by the convexity of g,
g(x1 + (1 )x2 ) b + (1 )b = b, so the set levg (b) is convex.
The fact that a convex function which is defined on Rn is continuous
establishes that the set levg (b) is always closed.2 (Why?)
Definition 3.45 (convex problem) Suppose that the set X Rn is
closed and convex. Suppose further that f : Rn R is convex and
that the functions gi : Rn R, i I, are convex. Suppose, finally, that
the functions gi : Rn R, i E, are affine. Then, the problem to
minimize
subject to

f (x),
gi (x) 0,

i I,

gi (x) = 0, i E,
x X,

is called a convex problem.

The name is natural, because the objective function is a convex one,
and the feasible set is closed and convex as well. In order to establish the
latter, we refer first to Proposition 3.44 to establish that the inequality
constraints define convex sets [note that in the similar problem (1.1) the
inequalities are given as -constraints, and then we require gi , i I, to
be concave functions in order to have a convex problem], and ask the
reader to prove that a constraint of the form aT
i x = bi defines a convex
set as well. Finally, we refer to Proposition 3.3 to establish that the
intersection of all the convex sets defined by X, I, and E is convex.

3.4

Application: the projection of a vector

onto a convex set

In Figure 3.15 we illustrate the Euclidean projection of some vectors

onto a convex set.
We see that the Euclidean projection of w Rn is the vector in S
which is nearest (in the Euclidean norm) to w: the vector ProjS (w) is
the unique optimum in the problem of finding
minimum kx wk.
x S

2 That

66

convex functions are continuous will be established in Theorem 4.27.

Application: the projection of a vector onto a convex set

1z
0
0
1

ProjS (z)

1
0
0w
1
1
0
0
1
1
0
1
0

ProjS (w)

Figure 3.15: The projection of two vectors onto a convex set.

The vector w ProjS (w) clearly is normal to the set S. The point z
has the Euclidean projection ProjS (z), but there are also several other
vectors with the same projection; the figure shows in a special shading
the set of vectors z which all have that same projection onto S. This set
is a cone, which we refer to as the normal cone to S at x = ProjS (z).
In the case of the point ProjS (w) the normal cone reduces to a ray
which of course is also a cone. (The difference between these two sets
is largely the consequence of the fact that there is only one constraint
active at ProjS (w), while there are two constraints active at ProjS (z);
when developing the KKT conditions in Chapter 5 we shall see how
strongly the active constraints influence the appearance of the optimality
conditions.)
We will also return to this image already in Section 4.6.3, because it
contains the building blocks of the optimality conditions for an optimization problem with an objective function in C 1 over a closed convex set.
For now, we will establish only one property of the projection operation
ProjS , namely that the distance function, distS , defined by
distS (x) := kx ProjS (x)k,

x Rn ,

(3.12)

is a convex function on Rn . In particular, then, this function is continuous. (Later, we will establish also that the projection operation ProjS is
a well-defined operation whenever S is nonempty, closed and convex, and
that the operation has particularly nice continuity properties. Before we
can do so, however, we need to establish some results on the existence
67

Convex analysis
of optimal solutions.)
Let x1 , x2 Rn , and (0, 1). Then,
distS (x1 + (1 )x2 ) = k(x1 + (1 )x2 )

ProjS (x1 + (1 )x2 )k

k(x1 + (1 )x2 )

(ProjS (x1 ) + (1 )ProjS (x2 ))k

kx1 ProjS (x1 )k

+ (1 )kx2 ProjS (x2 )k

= distS (x1 ) + (1 )distS (x2 ),

where the first inequality comes from the fact that ProjS (x1 ) + (1
)ProjS (x2 ) S, but it does not necessarily define ProjS (x1 + (1
)x2 ) (it may have a longer distance), and the second is the triangle
inequality.
The proof is illustrated in Figure 3.16.
0
x1 1
0
1
1
0
0
1
1
0
0
1

ProjS (x1 )

1
0
01
1
0
0
1

2
1
0
0x
1

1
0
0
1

ProjS (x2 )

Figure 3.16: The distance function is convex. From the intermediate

vector x1 + (1 )x2 , the distance to the vector ProjS (x1 ) + (1
)ProjS (x2 ) [the dotted line segment] clearly is longer than the distance
to its projection on S [shown as a solid line].

68

Notes and further reading

3.5

Notes and further reading

The subject of this chapterconvex analysishas a long history, going

back about a century. Much of the early work on convex sets and functions, for example, the theory of separation of convex sets, go back to
the work of Minkowski [Min10, Min11]. Other expositions are found in
[Fen51, Roc70, StW70], which all are classical in the field. More easily
accessible are the modern books [BoL00, BNO03]. Lighter introductions
are also found in [BSS93, HiL93]. The most influential of all of these is
Convex Analysis by R. T. Rockafellar [Roc70].
Caratheodorys Theorem 3.8 is found in [Car07, Car11]. Farkas
Lemma in Theorem 3.30 is found in [Far1902]. Theorem 3.42 is given as
Exercise 1.8 in [BNO03].
The early history of polyhedral convexity is found in [Mot36].

3.6

Exercises

Exercise 3.1 (convexity of polyhedra) Let A Rmn and b Rm . Show that

the polyhedron
P := { x Rn | Ax b },
is a convex set.
Exercise 3.2 (polyhedra) Which of the following sets are polyhedra?
(a) S := { y1 a + y2 b | 1 y1 1; 1 y2 1 }, where a, b Rn are
fixed.
Pn
Pn
2
(b) S := { x Rn | x 0n ; xT 1n = 1;
i=1 xi ai = b1 ;
i=1 xi ai =
b2 }, where ai R for i = 1, . . . , n, and b1 , b2 R are fixed.
y k2 = 1 }.
(c) S := { x Rn | x 0n ; xT y 1 for all y such that kP
(d) S := { x Rn | x 0n ; xT y 1 for all y such that n
i=1 |yi | = 1 }.
(e) S := { x Rn | kx x0 k2 kx x1 k2 }, where x0 , x1 Rn are fixed.
(f) S := { x Rn | kx x0 k2 kx xi k2 , i = 1, . . . , k }, where
0
x , . . . , xk Rn are fixed.
Exercise 3.3 (extreme points) Consider the polyhedron P defined by
x1 + x2 2,

x2 1,

x3 2,

x2 + x3 2.

(a) Is x1 := (1, 1, 0)T an extreme point to P ?

(b) Is x2 := (1, 1, 1)T an extreme point to P ?

69

Convex analysis
Exercise 3.4 (existence of extreme points in LPs) Let A Rmn be such that
rank A = m, and let b Rm . Show that if the polyhedron
P := { x Rn | Ax = b;

x 0n }

has a feasible solution, then it has an extreme point.

Exercise 3.5 (illustration of the Representation Theorem) Let
Q := { x R2 | 2x1 + x2 1;
C := { x R | 2x1 + x2 0;
2

x1 x2 1;
x1 x2 0;

x1 x2 1 },
x1 x2 0 },

and P be the convex hull of the extreme points of Q. Show that the feasible
= (1, 1)T can be written as
point x

x = p + ,
where

p P and C.

Exercise 3.6 (separation) Show that there is only one hyperplane in R3 which
separates the disjoint closed convex sets A and B defined by
A := { (0, x2 , 1)T | x2 R },

B := { x R3 | x 03 ; x1 x2 x23 },

and that this hyperplane meets both A and B.

Exercise 3.7 (separation) Show that each closed convex set A in
intersection of all the closed halfspaces in Rn containing A.

Rn

is the

Exercise 3.8 (application of Farkas Lemma) In a paper submitted for publication in an operations research journal, the author considered the set
 

P :=

x
n+m
y R

Ax + By ; x 0n ; y 0m ,

where A Rmn , B Rmm is positive semidefinite, and Rm . The

author explicitly assumed that the set P is compact in Rn+m . A reviewer
of the paper pointed out that the only compact set of the above form is the
empty set. Prove the reviewers assertion.
Exercise 3.9 (convex sets) Let S1 := { x R2 | x1 + x2 1; x1 0 },
S2 := { x R2 | x1 x2 0; x1 1 }, and S := S1 S2 . Prove that S1 and
S2 are convex sets and that S is not convex. Hence, the union of convex sets
is not necessarily a convex set.
Exercise 3.10 (convex functions) Determine if the function f (x) := 2x21
3x1 x2 + 5x22 2x1 + 6x2 is convex, concave, or neither, on R2 .

70

Exercises
Exercise 3.11 (convex functions) Let a > 0. Consider the following functions
in one variable:
(a) f (x) := ln x, for x > 0;
(b) f (x) := ln x, for x > 0;
(c) f (x) := ln(1 eax ), for x > 0;
(d) f (x) := ln(1 + eax );
(e) f (x) := eax ;
(f) f (x) := x ln x, for x > 0.
Which of these functions are convex (or, strictly convex)?
Exercise 3.12 (convex functions) Consider the following functions:
(a) f (x) := ln(ex1 + ex2 );
P
a j xj
(b) f (x) := ln n
, where aj , j = 1, . . . , n, are constants;
j=1 e
v
uX
n
u
x2j ;
(c) f (x) := t
j=1

(d) f (x) := x21 /x2 , for x2 > 0;

(e) f (x) := x1 x2 , for x1 , x2 > 0;

(f) f (x) :=

n
Y

!1/n

xj

, for xj > 0, j = 1, . . . , n.

j=1

Which of these functions are convex (or, strictly convex)?

Exercise 3.13 (convex functions) Consider the following function:
f (x, y) := 2x2 2xy +

1 2
y + 3x y.
2

(a) Express the function in matrixvector form.

(b) Is the Hessian singular?
(c) Is f a convex function?
Exercise 3.14 (convex sets) Consider the following sets:
(a) { x R2 | x21 + x22 1; x21 + x22 1/4 };
(b) { x Rn | xj 0, j = 1, . . . , n };
(c) { x Rn | x21 + x22 + + x2n = 1 };
(d) { x R2 | x1 + x22 5; x21 x2 10; x1 0; x2 0 };
(e) { x R2 | x1 x22 1; x31 +x22 10; 2x1 +x2 8; x1 1; x2 0 }.
Investigate, in each case, whether the set defined is convex or not. In the
latter case, provide a counter-example.
Exercise 3.15 (convex sets) Is the set defined by
S := { x R2 | 2ex1 +x2 4; x21 + 3x1 x2 3x22 1 }
2

a convex set?

71

Convex analysis
Exercise 3.16 (convex sets) Is the set defined by
S := { x R2 | x1 x22 1; x31 + x22 10; 2x1 + x2 8; x1 1; x2 0 }
a convex set?
Exercise 3.17 (convex problem) Suppose that the function g :
convex on Rn and that d Rn . Is the problem to
maximize
subject to

n
X

x2j ,

j=1

1
0,
ln(g(x))

dT x = 2,
g(x) 2,
x 0n
a convex problem?
Exercise 3.18 (convex problem) Is the problem to
maximize x1 ln x1 ,
subject to x21 + x22 0,

x 02

a convex problem?

72

Rn

is

Part III

Optimality Conditions

An introduction to
optimality conditions

4.1

IV

Local and global optimality

Consider the problem to

minimize f (x),
subject to x S,

(4.1a)
(4.1b)

where S Rn is a nonempty set and f : Rn R {+} is a given

function.
Consider the function given in Figure 4.1.
f (x)

4 5 6 7S

Figure 4.1: A one-dimensional function and its possible extremal points.

For a minimization problem of f in one variable over a closed interval
S, the interesting points are:
(i) boundary points of S;
(ii) stationary points, that is, where f (x) = 0;
(iii) discontinuities in f or f .

An introduction to optimality conditions

In the case of the function in Figure 4.1 the points 1 and 7 are of
category (i); 2, 3, 4, 5, and 6 of category (ii); and none of category (iii).
Definition 4.1 (global minimum) Consider the problem (4.1). Let x
S. We say that x is a global minimum of f over S if f attains its lowest
value over S at x .
In other words, x S is a global minimum of f over S if
f (x ) f (x),

xS

(4.2)

holds.
Let B (x ) := { y Rn | ky x k < } be the open Euclidean ball
with radius centered at x .
Definition 4.2 (local minimum) Consider the problem (4.1). Let x
S.
(a) We say that x is a local minimum of f over S if there exists a
small enough ball intersected with S around x such that it is a globally
optimal solution in that smaller set.
In other words, x S is a local minimum of f over S if
> 0 such that f (x ) f (x),

x S B (x ).

(4.3)

(b) We say that x S is a strict local minimum of f over S if, in

(4.3), the inequality holds strictly for x 6= x .
Note that a global minimum in particular is a local minimum. When
is a local minimum a global one? This question is resolved in the case
of convex problems, as the following fundamental theorem shows.
Theorem 4.3 (Fundamental Theorem of global optimality) Consider the
problem (4.1), where S is a convex set and f is convex on S. Then,
every local minimum of f over S is also a global minimum.
Proof. Suppose that x is a local minimum but not a global one, while
is a global minimum. Then, f (
x
x) < f (x ). Let (0, 1). By the
convexity of S and f ,
x + (1 )x S, and f (
x + (1 )x )

f (
x) + (1 )f (x ) < f (x ). Choosing > 0 small enough then leads
to a contradiction to the local optimality of x .
There is an intuitive image that can be seen from the proof design:
If x is a local minimum, then f cannot go down-hill from x in any
has a lower value, then f has to go down-hill sooner
direction, but if x
or later. No convex function can have this shape.
76

Local and global optimality

The example in Figure 4.2 shows a case where, without convexity, a
vector x may be a local minimum of a function f C 1 with respect to
every line segment that passes through x , and yet it is not even a local
minimum of f over Rn .

0.06

0.05

0.04

0.03

0.02

0.01
0.1
0
0.2

0
0.15

0.1

0.05

0.05

0.1
0.1

0.15

0.2

Figure 4.2: A three-dimensional graph of the function f (x, y) := (y

x2 )(y 4x2 ). The origin is a local minimum with respect to every line
that passes through it, but it is not a local minimum of f .
In fact, this situation may also occur in the convex case when f
/ C 1.
In the simple one-dimensional example in Figure 4.1, finding and
checking the different points of the form (i)(iii) was easy; however there
are, of course, examples even in R which makes this algorithm impossible to use, and when considering the multi-dimensional case (that is,
n > 1) this is a completely absurd method for solving an optimization
problem.
Remark 4.4 (checking for local optimality is hard) It cannot be overemphasized that it is hard to check, for a general constrained problem,
whether a given feasible solution actually is locally or globally optimal.
We refer to one such negative result: Pardalos and Schnitger [PaS88]
consider a constrained quadratic optimization problem, that is, where
the objective function has the form (3.10), see also (4.6) below; citing
from their abstract, the problem of checking local optimality for a feasible point and the problem of checking if a local minimum is strict, are
NP-hard problems. As a consequence the problem of checking whether
a function is locally strictly convex is also NP-hard. Since finding a
globally optimal solution to problems in this class is also NP-hard (cf.
[PaV91]), it follows that checking for local optimality is, in terms of
77

An introduction to optimality conditions

worst-case complexity, just as difficult as that to solve the entire problem.
In the following we will develop necessary and sufficient conditions
for x to be a local or a global optimal solution to the problem (4.1) for
any dimension n 1, and which are useful and possible to check. Before
we do that, however, we will establish when a globally optimal solution
to the problem (4.1) exists.

4.2
4.2.1

Existence of optimal solutions

A classic result

We first pave the way for a classic result from calculus: Weierstrass
Theorem.
Definition 4.5 (weakly coercive, coercive functions) Let S Rn be a
nonempty and closed set, and f : S R be a given function.
(a) We say that f is weakly coercive with respect to the set S if S is
bounded or for every N > 0 there exists an M > 0 such that f (x) N
whenever kxk M .
In other words, f is weakly coercive if either S is bounded or
lim

kx k
x S

f (x) =

holds.
(b) We say that f is coercive with respect to the set S if S is bounded
or for every N > 0 there exists an M > 0 such that f (x)/kxk N
whenever kxk M .
In other words, f is coercive if either S is bounded or
lim

kx k
x S

f (x)/kxk =

holds.
The weak coercivity of f : S R is (for nonempty closed sets S)
equivalent to the property that f has bounded level sets restricted to S
(cf. Definition 3.43). (Why?)
A coercive function grows faster than any linear function. In fact, for
convex functions f , f being coercive is equivalent to x 7 f (x) aT x
78

Existence of optimal solutions

being weakly coercive for every vector a Rn . This property is a very
useful one for certain analyses in the context of Lagrangian duality.1
We next introduce two extended notions of continuity.
Definition 4.6 (semi-continuity) Consider a function f : S R, where
S Rn is nonempty.
S if
(a) The function f is said to be lower semi-continuous at x
, that
the value f (
x) is less than or equal to every limit of f as xk x
.
is, limk xk = x
S if
In other words, f is lower semi-continuous at x

xk x

f (
x) lim inf f (xk ).
k

S if
(b) The function f is said to be upper semi-continuous at x
.
the value f (
x) is greater than or equal to every limit of f as xk x
S if
In other words, f is upper semi-continuous at x

xk x

f (
x) lim sup f (xk ).
k

We say that f is lower semi-continuous on S (respectively, upper

semi-continuous on S) if it is lower semi-continuous (respectively, upper
S.
semi-continuous) at every x
Lower semi-continuous functions in one variable have the appearance
shown in Figure 4.3.
f

x
Figure 4.3: A lower semi-continuous function in one variable.
Establish the following important relations:
1 For example, in Section 6.3.2 we suppose that the ground set X is compact in
order for the Lagrangian dual function q to be finite. It is possible to replace the
boundedness condition on X with a coercivity condition on f .

79

An introduction to optimality conditions

S
(a) The function f mentioned in Definition 4.6 is continuous at x
.
if and only if it is both lower and upper semi-continuous at x
(b) The lower semi-continuity of f is equivalent to the closedness
of all its level sets levf (b), b R (cf. Definition 3.43), as well as the
closedness of its epigraph (cf. Definition 3.38). (Why?)
Next follows the famous existence theorem credited to Karl Weierstrass (see, however, Section 4.7).
Theorem 4.7 (Weierstrass Theorem) Let S Rn be a nonempty and
closed set, and f : S R be a lower semi-continuous function on S.
If f is weakly coercive with respect to S, then there exists a nonempty,
closed and bounded (thus compact) set of globally optimal solutions to
the problem (4.1).
Proof. We first assume that S is bounded, and proceed by choosing a
sequence {xk } in S such that
lim f (xk ) = inf f (x).

x S

(The infimum of f over S is the lowest limit of all sequences of the form
{f (xk )} with {xk } S, so such a sequence of vectors xk is what we
here are choosing.)
Due to the boundedness of S, the sequence {xk } must have limit
be an
points, all of which lie in S because of the closedness of S. Let x
arbitrary limit point of {xk }, corresponding to the subsequence K Z+ .
Then, by the lower semi-continuity of f ,
f (
x) lim inf f (xk ) = lim f (xk ) = inf f (x).
kK

kK

x S

attains the infimum of f over S, x

is a global minimum of f
Since x
over S. This limit point of {xk } was arbitrarily chosen; any other choice
(provided more than one exists) has the same (optimal) objective value.
Suppose next that f is weakly coercive, and consider the same sequence {xk } in S. Then, by the weak coercivity assumption, either {xk }
is bounded or the elements of the sequence {f (xk )} tend to infinity. The
non-emptiness of S implies that inf x S f (x) < holds, and hence we
conclude that {xk } is bounded. We can then utilize the same arguments
as in the previous paragraph and conclude that also in this case there
exists a globally optimal solution. We are done.
Before moving on we take a closer look at the proof of this result,
because it is instrumental in order to understand the importance of some
of the assumptions that we make about the optimization models that we
80

Existence of optimal solutions

pose. We notice that the closedness of S is really crucial; if S is not closed
then a sequence generated in S may converge to a point outside of S,
which means that we would converge to an infeasible and of course also
non-optimal solution. This is the reason why the generic optimization
model (1.1) stated in Chapter 1 does not contain any constraints of the
form
gi (x) < 0,
i SI,

where SI denotes strict inequality. The reason is that such constraints

in general may describe non-closed sets.

4.2.2

Non-standard results

Weierstrass Theorem 4.7 is next improved for certain convex instances

of the problem (4.1). The main purpose of presenting these results is
to show the role of convexity and to illustrate the special properties of
convex quadratic programs and linear programs. The proofs are complex
and are left out; see the references in Section 4.7.
Theorem 4.8 (existence of optimal solutions, convex polynomials) Suppose that f : Rn R is a convex polynomial function. Suppose further
that the set S can be described by inequality constraints of the form
gi (x) 0, i = 1, . . . , m, where each function gi is convex and polynomial. The problem (4.1) then has a nonempty (as well as closed and
convex) set of globally optimal solutions if and only if f is lower bounded
on S.
In the following result, we let S be a nonempty polyhedron, and
suppose that it is possible to describe it as the following finite (cf. Definition 3.15) set of linear constraints:
S := { x Rn | Ax b;

Ex = d },

(4.4)

where A Rmn , E Rn , b Rm , and d R . The recession

cone to S then is the following set, defining the set of directions that are
feasible at every point in S:2
recS := { p Rn | Ap 0m ;

Ep = 0 }.

(4.5)

(For the definition of the set of feasible directions at a given vector x,

see Definition 4.20.)
We also suppose that
f (x) :=
2 Recall

1 T
x Qx + q T x,
2

x Rn ,

(4.6)

the cone C in the Representation Theorem 3.22.

81

An introduction to optimality conditions

where Q Rnn is a symmetric and positive semidefinite matrix and
q Rn . We define the recession cone to any convex function f : Rn R
as follows: the recession cone to f is the recession cone to the level
set of f (cf. Definition 3.43), defined for any value of b for which the
corresponding level set of f is nonempty.3 In the special case of the
convex quadratic function given in (4.6),
recf = { p Rn | Qp = 0n ;

q T p 0 }.

This is the set of directions that nowhere are ascent directions to f .

Corollary 4.9 (FrankWolfe Theorem) Suppose that S is the polyhedron described by (4.4) and f is the convex quadratic function given
by (4.6), so that the problem (4.1) is a convex quadratic programming
problem. Then, the following three statements are equivalent.
(a) The problem (4.1) has a nonempty (as well as a closed and convex)
set of globally optimal solutions.
(b) f is lower bounded on S.
(c) For every vector p in the intersection of the recession cone recS
to S and the null space N (Q) of the matrix Q, it holds that q T p 0.
In other words,
p recS N (Q)

qTp 0

holds.
The statement in (c) shows that the conditions for the existence of
an optimal solution in the case of convex quadratic programs are milder
than in the general convex case. In the latter case, we can state a slight
improvement over the Weierstrass Theorem 4.7 that if, in the problem
(4.1), f is convex on S where the latter is nonempty, closed and convex,
then the problem has a nonempty, convex and compact set of globally
optimal solutions if and only if recS recf = {0n }. The improvements in
the above results for polyhedral, in particular quadratic, programs stem
from the fact that convex polynomial functions cannot be lower bounded
and yet not have a global minimum.
[Note: Consider the special case of the problem (4.1) where f (x) :=
1/x and S := [1, +). It is clear that f is bounded from below on S,
in fact by the value zero which is the infimum of f over S, but it never
attains the value zero on S, and therefore this problem has no optimal
solution. Of course, f is not a polynomial function.]
3 Check that this cone actually is independent of the value of b under this only
requirement. Also confirm that if the level set levf (b) is (nonempty and) bounded
for some b R then it is bounded for every b R, thanks to the convexity of f .

82

Existence of optimal solutions

Corollary 4.10 (a fundamental theorem in linear programming) Suppose,
in the FrankWolfe Theorem, that f is linear, that is, that Q = 0nn .
Then, the problem (4.1) is identical to a linear programming (LP) problem. Then, the following three statements are equivalent.
(a) The problem (4.1) has a nonempty (as well as a closed and convex
polyhedral) set of globally optimal solutions.
(b) f is lower bounded on S.
(c) For every vector p in the recession cone recS to S, it holds that
q T p 0. In other words,
p recS

qT p 0

holds.
Corollary 4.10 will in fact be established later on in Theorem 8.10, by
the use of polyhedral convexity, when we specialize our treatment of nonlinear optimization to that of linear optimization. Since we have already
established the Representation Theorem 3.22, proving Corollary 4.10 for
the case of LP will be easy: since the objective function is linear, every
feasible direction p recS with q T p < 0 leads to an unbounded solution
from any vector x S.

4.2.3

Special optimal solution sets

Under strict convexity, we can establish the following result.

Proposition 4.11 (unique optimal solution under strict convexity) Suppose that in the problem (4.1) f is strictly convex on S and the set S is
convex. Then, there can be at most one globally optimal solution.
Proof. Suppose, by means of contradiction, that x and x are two
different globally optimal solutions. Then, for every (0, 1), we have
that
f (x + (1 )x ) < f (x ) + (1 )f (x ) = f (x ) [ = f (x )].
Since x + (1 )x S, we have found an entire interval of points
which are strictly better than x or x . This is impossible.
We finally characterize a class of optimization problems over polytopes whose optimal solution set, if nonempty, includes an extreme point.
Consider the maximization problem to
maximize

f (x),

subject to

x P,

(4.7)

83

An introduction to optimality conditions

where f : Rn R is a convex function and P Rn is a nonempty,
bounded polyhedron (that is, a polytope). Then, from the Representation Theorem 3.22 it follows below that an optimal solution can be
found among the extreme points of P . Theorem 8.10 establishes a corresponding result for linear programs that does not rely on Weierstrass
Theorem.
Theorem 4.12 (optimal extreme point) An optimal solution to (4.7) can
be found among the extreme points of P .
Proof. The function f is continuous (since it is convex, cf. Theorem 4.27
below); further, P is a nonempty and compact set. Hence, there exists an
to (4.7) by Weierstrass Theorem 4.7. The Represenoptimal solution x
= 1 v 1 + +k v kPfor some extreme
tation Theorem 3.22 implies that x
1
k
points v , . . . , v of P and 1 , . . . , k 0 such that ki=1 i = 1. But
then (from the convexity of f )
f (
x) = f (1 v 1 + + k v k ) 1 f (v 1 ) + + k f (v k )
1 f (
x) + + k f (
x) = f (
x),

which gives that f (

x) = f (v i ) for some i = 1, . . . , k.
Remark 4.13 Every linear function is convex, so Theorem 4.12 implies,
in particular, that every linear program over a nonempty and bounded
polyhedron has an optimal extreme point.

4.3

Optimality in unconstrained optimization

In Theorem 4.3 we have established that locally optimal solutions also

are global in the convex case. What are the necessary and sufficient
conditions for a vector x to be a local optimum? This is an important question, because the algorithms that we will investigate for solving
important classes of optimization problems are always devised based on
those conditions that we would like to fulfill. This is a statement that
seems to be true universally: efficient, locally or globally convergent iterative algorithms for an optimization problem are directly based on its
necessary and/or sufficient local optimality conditions.
We begin by establishing these conditions for the case of unconstrained optimization, where the objective function is in C 1 . Every proof
is based on the Taylor expansion of the objective function up to order
one or two.
84

Optimality in unconstrained optimization

Our problem here is the following:
minimize
f (x),
n

(4.8)

x R

where f is in C 1 on Rn [for short we say: in C 1 or C 1 (Rn )].

Theorem 4.14 (necessary optimality conditions, C 1 case) Suppose that
f : Rn R is in C 1 on Rn . Then,
x is a local minimum of f over Rn = f (x ) = 0n .

Note that
f (x) =

 f (x) n
xj

j=1

(x )
so the requirement thus is that fx
= 0, j = 1, . . . , n.
j
Just as for the case n = 1, we refer to this condition as x being a
stationary point of f .
[Note: For n = 1, Theorem 4.14 reduces to: x R is a local minimum = f (x ) = 0.]

Proof. (By contradiction.) Suppose that x is a local minimum, but

that f (x ) 6= 0n . Let p := f (x ), and study the Taylor expansion
around x = x in the direction of p:
f (x + p) = f (x ) + f (x )T p + o(),
where o : R R is such that o(s)/s 0 when s 0. We get that
f (x + p) = f (x ) kf (x )k2 + o()
< f (x )

for all small enough > 0, since kf (x )k 6= 0. This completes the

proof.
The opposite direction is false: take f (x) = x3 ; then, x = 0 is
stationary, but it is neither a local minimum nor a local maximum.
The proof is instrumental in that it provides a sufficient condition
for a vector p to define a descent direction, that is, a direction such that
a small step along it yields a lower objective value. We first define this
notion properly.
85

An introduction to optimality conditions

Definition 4.15 (descent direction) Let the function f : Rn R
{+} be given. Let x Rn be a vector such that f (x) is finite. Let
p Rn . We say that the vector p Rn is a descent direction with
respect to f at x if
> 0 such that f (x + p) < f (x) for every (0, ]
holds.
Proposition 4.16 (sufficient condition for descent) Suppose that f :
Rn R {+} is in C 1 around a point x for which f (x) < +,
and that p Rn . Then,
f (x)T p < 0 = p is a descent direction with respect to f at x

holds.
Proof. Since f is in C 1 around x, we can construct a Taylor expansion
of f , as above:
f (x + p) = f (x) + f (x)T p + o().
Since f (x)T p < 0, we obtain that f (x + p) < f (x) for all sufficiently
small values of > 0.
Notice that at a point x Rn there may be other descent directions
p Rn beside those satisfying that f (x)T p < 0; in Example 11.2(b) we
show how directions of negative curvature stemming from eigenvectors
corresponding to negative eigenvalues of the Hessian matrix 2 f (x) can
be utilized.
If f in addition is convex then the opposite implication in the above
proposition is true, thus making the descent property equivalent to the
property that the directional derivative is negative. Since this result can
be stated also for non-differentiable functions f (in which case we must
of course replace the expression f (x)T p with the classic expression
for the directional derivative, f (x; p) := lim0+ 1 [f (x + p) f (x)]),
we shall relegate the proof of this equivalence to our presentation of
the subdifferentiability analysis of convex functions in Section 6.3.1, in
particular to Proposition 6.18.
If f has stronger differentiability properties, then we can say even
more what a local optimum must be like.
Theorem 4.17 (necessary optimality conditions, C 2 case) Suppose that
f : Rn R is in C 2 on Rn . Then,

f (x ) = 0n
x is a local minimum of f =
2 f (x ) is positive semidefinite.
86

Optimality in unconstrained optimization

[Note: For n = 1, Theorem 4.17 reduces to: x R is a local minimum of f over R = f (x ) = 0 and f (x ) 0.]
Proof. Consider the Taylor expansion of f up to order two around x
and in the direction of a vector p Rn :
f (x + p) = f (x ) + f (x )T p +

2 T 2
p f (x )p + o(2 ).
2

Suppose that x satisfies f (x ) = 0n , but that there is a vector p 6= 0n

with pT 2 f (x )p < 0, that is, 2 f (x ) is not positive semidefinite.
Then the above yields that f (x + p) < f (x ) for all small enough
> 0, whence x cannot be a local minimum.
Also in this case, the opposite direction is false; the same counterexample as that after Theorem 4.14 applies.
In Example 11.2(b) we provide an example of a descent direction
that has the form provided in the above proof; it is based on p being an
eigenvector corresponding to a negative eigenvalue of 2 f (x ).
The next result shows that under some circumstances, we can establish local optimality of a stationary point.
Theorem 4.18 (sufficient optimality conditions, C 2 case) Suppose that
f : Rn R is in C 2 on Rn . Then,

f (x ) = 0n
= x strict local min of f over Rn .
2 f (x ) is positive definite
[Note: For n = 1, Theorem 4.18 reduces to: f (x ) = 0 and f (x ) >
0 = x R is a strict local minimum of f over R.]
Proof. Suppose that f (x ) = 0n and 2 f (x ) is positive definite.
Take an arbitrary vector p Rn , p 6= 0n . Then,
2
f (x + p) = f (x ) + f (x )T p + p T 2 f (x )p +o(2 )
{z
}
| {z } 2 |
=0

>0

> f (x )

for all small enough > 0. As p was arbitrary, the above implies that
x is a strict local minimum of f over Rn .
We naturally face the following question: When is a stationary point
a global minimum? The answer is given next. (Investigate the connection between this result and the Fundamental Theorem 4.3.)
87

An introduction to optimality conditions

Theorem 4.19 (necessary and sufficient global optimality conditions) Suppose that f : Rn R is convex and in C 1 on Rn . Then,
x is a global minimum of f over Rn

f (x ) = 0n .

Proof. [=] This has already been shown in Theorem 4.14, since a
global minimum is a local minimum.
[=] The convexity of f yields that for every y Rn ,
f (y) f (x ) + f (x )T (y x ) = f (x ),
where the equality stems from the property that f (x ) = 0n .

4.4

Optimality for optimization over convex sets

We consider a quite general optimization problem of the form:

minimize f (x),

(4.9a)

subject to x S,

(4.9b)

where S Rn is nonempty, closed and convex, and f : Rn R {+}

is in C 1 on S.
A noticeable difference to unconstrained optimization is the fact that
whether a vector p Rn can be used as a direction of movement from
a point x S depends on the constraints defining S; if x is an interior
point of S then every p Rn is a feasible direction, otherwise only certain directions will be feasible. That is, it all depends on whether there
are any active constraints of S at x or not. We will define these terms in
detail next, and then develop necessary and sufficient optimality conditions based on them. The optimality conditions are natural extensions
of those for the case of unconstrained optimization, and reduce to them
when S = Rn . Further, we will develop a way of measuring the distance
to an optimal solution in terms of the value of the objective function f ,
which is valid for convex problems. As a result of this development, we
will also be able to finally establish the Separation Theorem 3.24, whose
proof has been postponed until now. (See Section 4.6.2 for the proof.)
Definition 4.20 (feasible direction) Suppose that x S, where S Rn ,
and that p Rn . Then, the vector p defines a feasible direction at x if
a small step in the direction of p does not lead outside of the set S.
88

Optimality for optimization over convex sets

In other words, the vector p defines a feasible direction at x S if
> 0 such that x + p S for all [0, ]
holds.
Recall that in the discussion following Theorem 4.8 we defined the
set of feasible directions of a polyhedral set, that is, the set of directions
that are feasible at every feasible point. For a general set S it would
hence be the set
{ p Rn | x S > 0 such that x + p S for all [0, ] }.
For nonempty, closed and convex sets S, this set is nonempty if and only
if the set S also is unbounded. (Why?)
Definition 4.21 (active constraints) Suppose that the set S Rn is
defined by a finite collection of equality and inequality constraints:
S = { x Rn | gi (x) = 0,

i E;

gi (x) 0,

i I },

where gi : Rn R (i E I) are given functions. Suppose that x S.

The set of active constraints at x is the union of the equality constraints
and the set of inequality constraints that are satisfied with equality at
x, that is, the set E I(x), where I(x) := { i I | gi (x) = 0 }.
Example 4.22 (feasible directions for linear constraints) Suppose, as a
special case, that the constraints are all linear, that is, that for every
n
i E, gi (x) := eT
i x di (ei R ; di R), and for every i I,
T
n
gi (x) := ai x bi (ai R ; bi R). In other words, in matrix notation,
S := { x Rn | Ex = d; Ax b }.
Suppose further that x S. Then, the set of feasible directions at x
is the set
{ p Rn | Ep = 0 ; aT
i p 0, i I(x) }.
Just as S, this is a polyhedron. Moreover, it is a polyhedral cone.
Clearly, the set of feasible directions of the polyhedral set S (or, the
recession cone of S) is
recS := { p Rn | Ep = 0 ;

Ap 0m },

as stated in (4.5). Note moreover that the above set recS represents the
cone C in the Representation Theorem 3.22.4
4 While that theorem was stated for sets defined only by linear inequalities, we can
always rewrite the equalities Ex = d as Ex d, Ex d; the corresponding
feasible directions are then given by Ep 0 , Ep 0 , that is, Ep = 0 .

89

An introduction to optimality conditions

We can now more or less repeat the arguments for the unconstrained
case in order to establish a necessary optimality condition for constrained
optimization problems over convex sets. This condition will later on in
Chapter 5 be given a general statement for the case of explicit constraints
in the form of the famous KarushKuhnTucker conditions in nonlinear
programming.
Proposition 4.23 (necessary optimality conditions, C 1 case) Suppose
that S Rn and that f : Rn R {+} is in C 1 around a point
x S for which f (x) < +.
(a) If x S is a local minimum of f over S then f (x )T p 0
holds for every feasible direction p at x .
(b) Suppose that S is convex and that f is in C 1 on S. If x S is
a local minimum of f over S then
f (x )T (x x ) 0,

xS

(4.10)

holds.
Proof. (a) We again utilize the Taylor expansion of f around x :
f (x + p) = f (x ) + f (x )T p + o().
The proof is by contradiction. As was shown in Proposition 4.16, if there
is a direction p for which it holds that f (x )T p < 0, then f (x +p) <
f (x ) for all sufficiently small values of > 0. It suffices here to state
that p should also be a feasible direction in order to reach a contradiction
to the local optimality of x .
(b) If S is convex then every feasible direction p can be written as a
positive scalar times the vector x x for some vector x S. (Why?)
The expression (4.10) then follows from the statement in (a).
The inequality (4.10) is sometimes referred to as a variational inequality. We will utilize it for several purposes: (i) to derive equivalent
optimality conditions involving a linear optimization problem as well as
the Euclidean projection operation ProjS introduced in Section 3.4; (ii)
to derive descent algorithms for the problem (4.9) in Section 12.2 and
12.4; (iii) to derive a near-optimality condition for convex optimization
problems in Section 4.5; and (iv) we will extend it to non-convex sets in
the form of the KarushKuhnTucker conditions in Chapter 5.
In Theorem 4.14 we established that for unconstrained C 1 optimization the necessary optimality condition is that f (x ) = 0n holds. Notice that that is exactly what becomes of the variational inequality (4.10)
when S = Rn , because the only way in which that inequality can hold
90

Optimality for optimization over convex sets

for every x Rn is that f (x ) = 0n holds. Just as we did in the case
of unconstrained optimization, we will call a vector x S satisfying
(4.10) a stationary point.
We will next provide two statements equivalent to the variational
inequality (4.10). First up, though, we will provide the extension to
Theorem 4.19 to the convex constrained case. Notice the resemblance of
their respective proofs.
Theorem 4.24 (necessary and sufficient global optimality conditions) Suppose that S Rn is nonempty and convex. Let f : Rn R be convex
and in C 1 on S. Then,
x is a global minimum of f over S

(4.10) holds.

Proof. [=] This has already been shown in Proposition 4.23(b), since
a global minimum is a local minimum.
[=] The convexity of f yields [cf. Theorem 3.40(a)] that for every
y S,
f (y) f (x ) + f (x )T (y x ) f (x ),
where the second inequality stems from (4.10).
First, we will provide the connection to the Euclidean projection of
a vector onto a convex set, discussed in Section 3.4. We claim that the
property (4.10) is equivalent to
x = ProjS [x f (x )],

(4.11)

or, more generally, x = ProjS [x f (x )] for every > 0. In

other words, a point is stationary if and only if a step in the direction of
the steepest descent direction followed by a Euclidean projection onto S
means that we have not moved at all. To prove this fact, we will utilize
Proposition 4.23(b) for the optimization problem corresponding to this
projection. We are interested in finding the point x S that minimizes
the distance to the vector z := x f (x ). We can write this as a
strictly convex optimization problem as follows:
minimize h(x) :=
x S

1
kx zk2 .
2

(4.12)

The necessary optimality conditions for this problem, as stated in Proposition 4.23(b), is that
h(x)T (y x) 0,

y S,

(4.13)
91

An introduction to optimality conditions

holds. Here, h(x) = x z = x [x f (x )]. Since h is strictly
convex, by Theorem 4.24 we know that the variational inequality (4.13)
characterizes x as the unique globally optimal solution to the projection
problem. We claimed that x = x is the solution to this problem if and
only if x is stationary in the problem (4.9). But this follows immediately, since the variational inequality (4.13), for the special choice of h
and x = x , becomes
f (x )T (y x ) 0,

y S,

that is, a statement identical to (4.10). The characterization (4.11) is interesting in that it states that if x is not stationary, then the projection
operation defined therein must provide a step away from x ; this step
will in fact yield a reduced value of f under some additional conditions
on the step length , and so it defines a descent algorithm for (4.9); see
Exercise 4.5, and the text in Section 12.4.
So far, we have two equivalent characterizations of a stationary point
of f at x : (4.10) and (4.11). The following one is based on a linear
optimization problem.
Notice that (4.10) states that f (x )T x f (x )T x for every
x S. Since we obtain an equality by setting x = x we see that x in
fact is a globally optimal solution to the problem to
minimize f (x )T x.
x S

In other words, (4.10) is equivalent to the statement

minimum f (x )T (x x ) = 0.
x S

(4.14)

It is quite obvious that if at some point x S,

minimum f (x)T (y x) < 0,
y S

x, with
then every direction of the form p := y
arg minimum f (x)T (y x),
y
y S

is a feasible descent direction with respect to f at x. Again, we have

a building block of a descent algorithm for the problem (4.9). [The
algorithms that immediately spring out from this characterization are
called the FrankWolfe and Simplicial decomposition algorithms, when
S is polyhedral; we notice that in the polyhedral case, the linear minimization problem in (4.14) is an LP problem. Read more about these
92

Optimality for optimization over convex sets

algorithms in Sections 12.2 and 12.3.] Now having got three equivalent
stationarity conditions(4.10), (4.11), and (4.14), we finally provide
a fourth one. This one is intimately associated with the projection operation, and it introduces an important geometric concept into the theory
of optimality, namely the normal cone to a (convex) set S.
We studied a particular choice of z above, but let us consider an
extension of Figure 3.15 which provided an image of the Euclidean projection.
Notice from the above arguments that if we wish to project the vector
z Rn onto S, then the resulting (unique) projection is the vector x for
which the following holds:
[x z]T (y x) 0,

y S.

Changing sign for clarity, this is the same as

[z x]T (y x) 0,

y S.

The interpretation of this inequality is that the angle between the two
vectors z x (the vector that points towards the point being projected)
and the vector y x (the vector that points towards any vector y S)
is 90 . So, the projection operation has the characterization
[z ProjS (z)]T (y ProjS (z)) 0,

y S.

(4.15)

The above is shown in Figure 4.4 for x = x and z = x f (x ).

NS (x )

x f (x )

1
0
0
1

y
S
Figure 4.4: Normal cone characterization of a stationary point.
Here, the point being projected is z = x f (x ), as used in the
characterization of stationarity.
What is left to complete the picture is to define the normal cone,
NS (x ), to S at x , depicted in Figure 4.4 in the lighter shade.
93

An introduction to optimality conditions

Definition 4.25 (normal cone) Suppose that the set S Rn is closed
and convex. Let x Rn . Then, the normal cone to S at x is the set
(
{ v Rn | v T (y x) 0,
y S }, if x S,
NS (x) :=
(4.16)
,
otherwise.
According to the definition, we can now define our fourth characterization of a stationary point at x as follows:
f (x ) NS (x ).

(4.17)

What this condition states geometrically is that the angle between the
negative gradient and any feasible direction is 90 , which, of course,
whenever f (x ) 6= 0n , is the same as stating that at x there exist
no feasible descent directions. The four conditions (4.10), (4.11), (4.14),
and (4.17) are equivalent, and so according to Theorem 4.24 they all are
also both necessary and sufficient for the global optimality of x as soon
as f is convex.
We remark that in the special case when S is an affine subspace (such
as the solution set of a number of linear equations, S := { x Rn | Ex =
d }), the statement (4.17) means that at a stationary point x , f (x )
is parallel to a normal of the subspace.
The normal cone inclusion (4.17) will later be extended to more general sets, where S is described by a finite collection of possibly nonconvex constraints. The extension will lead us to the famous Karush
KuhnTucker conditions in Chapter 5. [It turns out to be much more
convenient to extend (4.17) than the other three characterizations of
stationarity.]
We finish this section by proving a proposition on the behaviour of
the gradient of the objective function f on the solution set S to convex
problems of the form (4.1). The below result shows that f enjoys a
stability property, and it also extends the result from the unconstrained
case where the value of f always is zero on the solution set.
Proposition 4.26 (invariance of f on the solution set of convex programs) Suppose that S Rn is convex and that f : Rn R is convex
and in C 1 on S. Then, the value of f (x) is constant on the optimal
solution set S .
Further, suppose that x S . Then,
S = { x S | f (x )T (x x ) = 0 and f (x) = f (x ) }.

94

Near-optimality in convex optimization

Proof. Let x S . The definition of the convexity of f shows that
f (x) f (x ) f (x )T (x x ),

x Rn .

(4.18)

S . Then, it follows that f (x )T (

Let x
x x ) = 0. By substituting
T
T
f (x ) x with f (x ) x in (4.18) and using that f (x ) = f (
x), we
obtain that
),
f (x) f (
x) f (x )T (x x

x Rn ,

which is equivalent to the statement that f (

x) = f (x ).

4.5

Near-optimality in convex optimization

We will here utilize Theorem 4.24 in order to provide a measure of the

distance to the optimal solution in terms of the value of f at any feasible
point x.
Let x S, and suppose that f is convex on S. Suppose also that
x S is an arbitrary globally optimal solution, which we suppose
exists. From the necessary and sufficient optimality conditions stated in
Theorem 4.24, it is clear that unless x solves (4.9) there exists a y S
such that f (x)T (y x) < 0, and hence p := y x is a feasible descent
direction.
Suppose now that
arg minimum z(y) := f (x)T (y x).
y
yS

(4.19)

Consider the following string of inequalities and equalities:

f (x) + z(
y ) = f (x) + min z(y) f (x) + z(x ) f (x ) f (x).
y S

The equality follows by definition; the first inequality stems from the
solves the linear minimization problem, while the vector x
fact that y
may not; the second inequality follows from the convexity of f on S [cf.
Theorem 3.40(a)]; the final inequality follows from the global optimality
of x and the feasibility of x.
From the above, we obtain a closed interval wherein we know that the
optimal value of the problem (4.9) lies. Let f := minimumx S f (x) =
arg minimumy S z(y),
f (x ). Then, for every x S and y
f [f (x) + z(
y ), f (x)].

(4.20)
95

An introduction to optimality conditions

Clearly, the length of the interval is defined by how far from zero the value
of z(
y) is. Suppose then that z(
y) , for some small value > 0. (In
an algorithm where a sequence {xk } is constructed such that it converges
to an optimal solution, this will eventually happen for every > 0.)
Then, from the above we obtain that f (x ) f (x) + z(
y ) f (x) ;
in short,
f (x ) f (x) , or, f (x) f + .
(4.21)
We refer to a vector x S satisfying the inequality (4.21) as an -optimal
solution. From the above linear minimization problem we hence have a
simple instrument for evaluating the quality of a feasible solution in our
problem. Note, again, that convexity is a crucial property enabling this
possibility.

4.6
4.6.1

Applications
Continuity of convex functions

A remarkable property of any convex function is that without any additional assumptions it can be shown to be continuous relative to any open
convex set in the intersection of its effective domain (that is, where the
function has a finite value) and its affine hull.5 We establish a slightly
weaker special case of this result below, in which relative interior is
replaced by interior for simplicity.
Theorem 4.27 (continuity of convex functions) Suppose that f : Rn
R {+} is a convex function, and consider an open convex subset S
of its effective domain. The function f is continuous on S.
S. To establish continuity of f at x
, we must show
Proof. Let x
k implies that
that given > 0, there exists > 0 such that kx x
|f (x) f (
x)| . We establish this property in two parts, by showing
(cf. Definition 4.6).
that f is both lower and upper semi-continuous at x
[upper semi-continuity] By the openness of S, there exists > 0 with
k , implying x S. Construct the value of the scalar as
kx x
follows:
:= maximum {max {|f (
x + ei ) f (
x)|, |f (
x ei ) f (
x)|}} ,
i{1,2,...,n}

(4.22)
5 In

other words, f is continuous relative to any relatively open convex subset of

its effective domain.

96

Applications
where ei is the ith unit vector in Rn . If = 0 it follows that f is constant
and hence continuous there, so suppose that
in a neighbourhood of x
> 0. Let now



:= minimum
,
.
(4.23)
n n
k . For every i {1, 2, . . . , n},P
Choose an x with kx x
if xi xi
= ni=1 i z i ,
then define z i := ei , otherwise z i := ei . Then, x x
where i 0 for all i. Moreover,
k = kk.
kx x

(4.24)

k , it follows that i 1/n for all i.

From (4.23), and since kx x
Hence, by the convexity of f and since 0 i n 1, we get
!
" n
#
n
X
1X
+
f (x) = f x
i z i = f
(
x + i nz i )
n i=1
i=1
n

1X
f (
x + i nz i )
n i=1
n

1X
x + i n(
x + z i )]
f [(1 i n)
n i=1
n

1X
[(1 i n)f (
x) + i nf (
x + z i )].
n i=1

Pn
Therefore, f (x) f (
x) i=1 i [f (
x + z i ) f (
x)]. From (4.22) it is
obvious that f (
x + z i ) f (
x) for each i; and since i 0, it follows
that
n
X
f (x) f (
x)
i .
(4.25)
i=1

Noting (4.23), (4.24), it follows that i /n, and (4.25) implies that
k
f (x) f (
x) . Hence, we have so far shown that kx x
implies that f (x) f (
x) . By Definition 4.6(b), f hence is upper
.
semi-continuous at x
k .
[lower semi-continuity] Let y := 2
x x, and note that ky x
Therefore, as above,
f (y) f (
x) .
(4.26)

= 21 y + 12 x, and by the convexity of f , f (

But x
x) 21 f (y) + 12 f (x) follows. Combining this inequality with (4.26), it follows that f (
x)f (x)
, whence Definition 4.6(a) applies. We are done.

97

An introduction to optimality conditions

Note that convex functions need not be continuous everywhere; by
the above theorem we know however that points of non-continuity must
occur at the boundary of the effective domain of f . For example, check
the continuity of the following convex function:
(
x2 , for |x| < 1,
f (x) :=
2,
for |x| = 1.

4.6.2

The Separation Theorem

The previously established Weierstrass Theorem 4.7 will now be utilized

together with the variational inequality characterization (4.10) of stationary points in order to finally establish the Separation Theorem 3.24.
For simplicity, we rephrase the theorem.
Theorem 4.28 (Separation Theorem) Suppose that the set S Rn is
closed and convex, and that the point y does not lie in S. Then, there
exist a vector 6= 0n and R such that T y > and T x for
all x S.
Proof. We may assume that S is nonempty, and define a function
f : Rn R through f (x) := kx yk2 /2, x Rn . By Weierstrass The of f over S, which by the first order
orem 4.7 there exists a minimum x
)T (x
necessary condition given in Proposition 4.23(b) satisfies (y x
) 0 for all x S (since f (
). Setting := y x
6= 0n
x
x) = y x
T
) x
gives the result sought.
and := (y x
A slightly different separation theorem will be used in the Lagrangian
duality theory in Chapter 6. We state it without proof.
Theorem 4.29 (separation of convex sets) Each pair of disjoint (AB =
) nonempty convex sets A and B in Rn can be separated by a hyperplane
in Rn , that is, there exist a vector Rn and R such that T x
for all x A and T y for all y B.
Remark 4.30 A main difference between the Separation Theorems 3.24
and 4.29 is that in Theorem 3.24 there exists a hyperplane that in fact
strictly separates the point y and the closed convex set C, that is, there
exists a vector Rn and an R such that T y > while T x <
holds for all x C. In Theorem 4.29, however, this is not true. Consider,
for example, the sets A := { x R2 | x2 0 } and B := { x R2 | x1 >
0; x2 1/x1 }. Then, the line { x R2 | x2 = 0 } separates A and B,
but the sets cannot be strictly separated.
98

Applications

4.6.3

Euclidean projection

We will finish our discussions on the projection operation, which was

defined in Section 3.4, by establishing an interesting continuity property.
Definition 4.31 (non-expansive operator) Suppose that S Rn is closed
and convex. Let f : S S denote a vector-valued operator from S to S.
We say that f is non-expansive if, as a result of applying the mapping
f , the distance between any two vectors x and y in S does not increase.
In other words, the operator f is non-expansive on S if
kf(x) f (y)k kx yk,

x, y S,

(4.27)

holds.
Theorem 4.32 (the projection operation is non-expansive) Let S be a
nonempty, closed and convex set in Rn . For every x Rn , its projection
ProjS (x) is uniquely defined. The operator ProjS : Rn S is nonexpansive on Rn , and therefore in particular continuous.
Proof. The uniqueness of the operation is the result of the fact that
the function x 7 kx zk2 is both coercive and strictly convex on S,
so there exists a unique optimal solution to the projection problem for
every z Rn . (Cf. Weierstrass Theorem 4.7 and Proposition 4.11,
respectively.)
Next, take x1 , x2 Rn . Then, by the characterization (4.15) of the
Euclidean projection,
[x1 ProjS (x1 )]T (ProjS (x2 ) ProjS (x1 )) 0,

[x2 ProjS (x2 )]T (ProjS (x1 ) ProjS (x2 )) 0.

Summing the two inequalities yields
kProjS (x2 ) ProjS (x1 )k2 [ProjS (x2 ) ProjS (x1 )]T (x2 x1 )

kProjS (x2 ) ProjS (x1 )k kx2 x1 k,

where the last inequality is a consequence of Cauchys inequality; we

obtain that kProjS (x2 ) ProjS (x1 )k kx2 x1 k. Since this is true
for every pair x1 , x2 of vectors in Rn , we have shown that the operator
ProjS is non-expansive on Rn . In particular, non-expansive functions
are continuous. (The proof of the latter is left as an exercise.)
The theorem is illustrated in Figure 4.5.
99

An introduction to optimality conditions

x1
0
0
1
y

1
0
0
1
1
0
0
1

ProjS (x)

1
0
0
1

ProjS (y)

S
Figure 4.5: The projection operation is non-expansive.

4.6.4

Fixed point theorems

Fixed point theorems state properties of a problem of the following form:

Suppose the mapping f is defined on a closed, convex set S in Rn and
that f (x) S for every x S. Is there an x S such that f maps x
onto itself (that is, onto x), or, in other words,
x S such that x f (x)?
Such a point is called a fixed point of f over the set S. If the mapping f
is single-valued rather than set-valued then the question boils down to:
x S such that x = f (x)?
Many questions in optimization and analysis can be reduced to the
analysis of a fixed point problem. For example, an optimization problem can in some circumstances be reduced to a fixed point problem, in
which case the question of the existence of solutions to the optimization
problem can be answered by studying the fixed point problem. Further,
the optimality conditions analyzed in Section 4.4 can be written as the
solution to a fixed point problem, cf. (4.11); we can therefore equate the
search for a stationary point with that of finding a fixed point of a particular function f . This type of analysis is quite useful also when analyzing
the convergence of iterative algorithms for optimization problems.
4.6.4.1

Theory

We begin by studying some classic fixed point theorems, and then we

provide examples of the connections between the results in Section 4.4
with fixed point theory.
100

Applications
Definition 4.33 (contractive operator) Let S Rn be a nonempty set
in Rn . Let f be a mapping from S to S. We say that f is contractive
on S if, as a result of applying the mapping f , the distance between any
two distinct vectors x and y in S decreases.
In other words, the operator f is contractive on S if there exists
[0, 1) such that
kf (x) f (y)k kx yk,

x, y S,

(4.28)

holds.
Clearly, a contractive operator is non-expansive.
In the below result we utilize the notion of a geometric convergence
rate; while its definition is in fact given in the result below, we also refer
to Sections 6.4.1 and 11.10 for more detailed discussions on convergence
rates.
Theorem 4.34 (fixed point theorems) Let S be a nonempty and closed
set in Rn .
(a) [Banachs Theorem] Let f : S S be a contraction mapping.
Then, f has a unique fixed point x S. Further, for every initial
vector x0 S, the iteration sequence {xk } defined by the fixed-point
iteration
xk+1 := f (xk ),
k = 0, 1, . . . ,
(4.29)
converges geometrically to the unique fixed point x . In particular,
kxk x k k kx0 x k,

k = 0, 1, . . . .

(b) [Brouwers Theorem] Let S further be convex and bounded, and

assume merely that f is continuous. Then, f has a fixed point.
Proof. (a) For any x0 S, consider the sequence {xk } defined by
(4.29). Then, for any p 1,
kxk+p xk k

p
X

kxk+i xk+i1 k

i=1
p1

+ + 1)kxk+1 xk k [k /(1 )]kx1 x0 k.

Hence, {xk } is a Cauchy sequence and thus converges as k . By

continuity and the contraction property, the limit point is the unique
fixed point. In detail, this last part of the proof is as follows: Suppose
that xk x S. Then, for any iterate xk 6= x it holds that
kf (x ) x k kf (x ) f (xk )k + kf (xk ) x k
kx xk k + kxk+1 x k,

101

An introduction to optimality conditions

and according to our assumptions both of the latter terms converge to
zero. Hence, x is a fixed point. Suppose then that there is also another
fixed point, x , so that x 6= x . Then, kx x k = kf (x )
f (x )k kx x k, which yields a contradiction since < 1. Hence,
x = x , and the fixed point x is unique.
The convergence speed follows from the identification
kxk x k = kf (xk1 ) f (x )k kxk1 x k,

k = 1, 2, . . . .

Applying this relation recursively yields the result.

(b) [Sketch] In short, the proof is to first establish that any C 1 function on the unit ball has a fixed point there. Extending the reasoning to merely continuous operators is possible, because of the Stone
Weierstrass Theorem (which states that for any continuous operator defined on the unit ball there is a sequence of C 1 functions defined on the
unit ball that uniformly converges to it). Each of these functions can
be established to have a fixed point, and because of the compactness of
the unit ball, so does the merely continuous limit function. For our final
argument, we can assume that the set S has a nonempty interior. Then
there exists a homeomorphism6 h : S B, where B is the unit ball.
Since the composite mapping hf h1 is a continuous operator from B
to B it has a fixed point y in B; therefore, h1 (y) is a fixed point of f .
Notice that Banachs Theorem holds without a convexity assumption
on the set S; the contraction property is indeed very strong. Brouwers
Theorem constitutes a major improvement in that the mapping f need
only be continuous; on the other hand, the requirements on the set S
increase dramatically. (This is a clear example of a case where a result
(in this case: the existence of a fixed point) requires a sufficient critical
mass of properties which however can be distributed in different ways.)
An alternative proof of Banachs Theorem is provided in Exercise 4.10.
A special case in one variable of the result in Brouwers Theorem is
illustrated in Figure 4.6.
4.6.4.2

Applications

Particularly the result of Theorem 4.34(b) is quite remarkably strong.

We provide some sample consequences of it below. In each case, we ask
the reader to find the pair (S, f ) defining the corresponding fixed point
problem.
6 The given function h is a homeomorphism if it is a continuous operator which is
ontothat is, its range, h(S), is identical to the set B defining its image setand
has a continuous inverse.

102

Applications
f (x)
1

Figure 4.6: Consider the case S = [0, 1], and a continuous function
f : S S. Brouwers Theorem states that there exists an x S
with f (x ) = x . This is the same as saying that the continuous curve
starting at (0, f (0)) and ending at (1, f (1)) must pass through the line
y = x inside the square.

[Mountaineering] You climb a mountain, following a trail, in six

hours (noon to 6 PM). You camp on top overnight. Then at noon
the next day, you start descending. The descent is easier, and you
make much better time. After an hour, you notice that your compass is missing, and you turn around and ascend a short distance,
where you find your compass. You sit on a rock to admire the
view. Then you descend the rest of the way. The entire descent
takes four hours (noon to 4 PM). Along the trail there must then
be a place where you were at the same place at the same time on
both days.
[Maps] Suppose you have two city maps over Gothenburg, which
are not of the same scale. You crumple one of them up into a
loose ball and place it on top of the other map entirely within the
borders of the Gothenburg region on the flat map. Then, there is
a point on the crumpled map (that represents the same place in
Gothenburg on both maps) that is directly over its twin on the flat
map. (A more simple problem is defined by a non-crumpled map
and the city of Gothenburg itself; lay down the map anywhere in
Gothenburg, and at least one point on the map will lie over that
exact spot in real-life Gothenburg.)
[Raking of gravel] Suppose you wish to rake the gravel in your
garden; if the area is, say, circular, then any continuous raking
will leave at least one tiny stone (which one is a function of time)

103

An introduction to optimality conditions

in the same place.
[Stirring coffee] Stirring the contents of a (convex) coffee cup in a
continuous way, no matter how long you stir, some particle (which
one is a function of time) will stay in the same position as it did
before you began stirring.
[Meteorology] Even as the wind blows across the Earth there will
be one location where the wind is perfectly vertical (or, perfectly
calm). This fact actually implies the existence of cyclones; not to
mention whorls, or crowns, in your hair no matter how you comb
it. (The latter result also bears its own name: The Hairy Ball
Theorem; cf. [BoL00, pp. 186187].)

Applying fixed point theorems to our own development, we take a

look at the variational inequality (4.10). Rephrasing it in a more general
form, the variational inequality problem (VIP) is, for some set S Rn
and mapping f : S Rn to find x S such that
f (x )T (x x ) 0,

x S.

(4.30)

In order to turn it into a fixed point problem, we construct the following composite operator from Rn to S:
F (x) := ProjS (x f (x)),

x Rn ,

and consider finding a fixed point of F on S. Why is this operator a

correct one? Because it is equivalent to the statement that
ProjS (x f (x)) = x;
the special case for f = f is found in (4.11). Applying a fixed point
theorem to the above problem then proves that the variational inequality problem (4.30) has solutions whenever f is continuous on S and S is
nonempty, convex and compact. (Moreover, we have immediately found
an iterative algorithm for the variational inequality problem: if the operator x 7 ProjS (x f (x)) is contractive for some > 0, then it
defines a convergent algorithm.)
At the same time, we saw that the fixed point problem was defined
through the same type of stationarity condition that we derived in Section 4.4 for differentiable optimization problems over convex sets. We
have thereby also illustrated that stationarity in an optimization problem is intimately associated with fixed points of a particular operator.7
7 The

book [Pat98] analyzes a large variety of optimization algorithms by utilizing

this connection.

104

Applications
As an exercise, we consider the problem to find an x R such that
f (x) = 0, where f : R R is twice differentiable near a zero of f . The
classic NewtonRaphson algorithm has an iteration formula of the form
x0 R;

xk+1 = xk

f (xk )
,
f (xk )

k = 0, 1, . . . .

If we assume that x is a zero of f at which f (x ) > 0,8 then by starting

close enough to x we can prove that the above iteration formula defines
a contraction, and hence we can establish local convergence. (Why?)
Further analyses of Newton methods will be performed in Chapter 11.
A similar technique can be used to establish that a system of linear equations with a symmetric matrix is solvable by the classic Jacobi
algorithm in numerical analysis, if the matrix is diagonally dominant;
this condition is equivalent to the Jacobi algorithms algorithm-defining
operator being a contraction. (Similar, but stronger, results can also be
obtained for the GaussSeidel algorithm; cf. [OrR70, Kre78, BeT89].)
An elegant application of fixed point theorems is the analysis of matrix games. The famous Minimax Theorem of von Neumann is associated with the existence of a saddle point of a function of the form
(v, w) 7 L(v, w) := v T Aw. Von Neumanns minimax theorem states
that if V and W both are nonempty, convex and compact, then
minimum maximum v T Aw = maximum minimum v T Aw.
v V

w W

w W

v V

(4.31)

In order to prove this theorem we can use the above existence theorem
for variational inequalities. Let
 


v
AT v
x=
; f (x) =
; S = V W.
w
Aw
It is a reasonably simple exercise to prove that the variational inequality (4.30) with the above identifications is equivalent to the saddle
point conditions, which can also be written as the existence of a pair
(v , w ) V W such that
(v )T Aw (v )T Aw v T Aw ,

(v, w) V W ;

and we are done immediately.

Saddle point results will be returned to in the study of (Lagrangian)
duality in the coming chapters, especially for linear programming (which
was also von Neumanns special interest).
8 The

sign of f (x ) is immaterial, as long as f (x ) 6= 0.

105

An introduction to optimality conditions

4.7

Notes and further reading

Most of the material of this chapter is elementary (as is relies mostly

on the Taylor expansion of differentiable functions), and can be found
in most basic books on nonlinear optimization, such as [Man69, Zan69,
Avr76, BSS93, Ber99].
Complexity analysis, in the form of the analysis of NP-hard problems
and the P = NP? question, is beyond the scope of this book; we refer
to [GaJ79] for an excellent introduction.
Weierstrass Theorem 4.7 is the strongest existence result for optimal
solutions that does not utilize convexity. The result is credited to Karl
Weierstrass, but it was in fact known already by Bernard Bolzano in
1817 (although then only available in manuscript form); it has strong
connections to the theorem of the existence of intermediate values as well
as to that on the existence of limit points of every bounded sequence (now
often referred to as the BolzanoWeierstrass Theorem), and the notion of
Cauchy sequences, often also credited to Weierstrass and Augustin-Louis
Cauchy, respectively.
The FrankWolfe Theorem in Corollary 4.9 is found in [FrW56]. The
stronger result in Theorem 4.8 is found in [Eav71, BlO72]. Proposition 4.26 on the invariance of the gradient on the solution set is found
in [Man88, BuF91].
The result in Theorem 4.34(a) is due to Banach [Ban22]; Theorem 4.34(b) is due to Brouwer [Bro09, Bro12], and Hadamard [Had10].
Fixed point theorems are developed in greater detail in [GrD03]. Noncooperative game theory was developed in work by John von Neumann,
together with Oskar Morgenstern (see [vNe28, vNM43]), and by John
Nash [Nas50, Nas51].
As far as iterative algorithms go, it is quite often the case that for
the problem (4.9) involving a (convex) feasible set the sequences {xk }
of iterates do not necessarily stay inside the feasible set S Rn . The
reason is that even if the constraints are convex inequalities it is difficult
to check when one reaches the boundary of S. We mention however two
cases where feasible algorithms (that is, those for which {xk } S holds)
are viable:
(I) When S is a polyhedral set, then it is only a matter of solving a series of simple linear systems to check for the maximum step length
along a feasible direction. Among the algorithms that actually are
feasible we count the simplex method for linear programming (LP)
problems (see Chapters 9 and 10), the FrankWolfe and Simplicial
decomposition methods (see Sections 12.2 and 12.3) which build
on solving such LP problems, and the projection method (see Sec106

Exercises
tion 12.4) which builds on the property (4.11); see also Exercise 4.5.
More on these algorithms will be said in Chapter 12.
(II) When the set S has an interior point, we may replace the constraints with an interior penalty function which has an asymptote
whenever approaching the boundary, thus automatically ensuring
that iterates stay (strictly) feasible. More on a class of methods
based on this penalty function is said in Chapter 13.

4.8

Exercises

Exercise 4.1 (redundant constraints) Consider the problem to

minimize f (x),

(4.32)

subject to g(x) b,

where f : R R and g : R R are continuous functions, and b R. Suppose that this problem has a globally optimal solution, x , and that g(x ) < b
holds. Consider also the problem to
n

minimize f (x),
subject to

(4.33)

x Rn ,

in which we have removed the constraint. The question is under which circumstances this is vaild.
(a) Show by means of a counter-example that the vector x may not solve
(4.33); in other words, in general we cannot throw away a constraint that is
not active without affecting the optimal solution, even if it is inactive. Hence,
it would be wrong to call such constraints redundant.
(b) Suppose now that f is convex on Rn . Show that x solves (4.33).
Exercise 4.2 (unconstrained optimization, exam 020826) Consider the unconstrained optimization problem to minimize the function
f (x) :=

3 2
(x1 + x22 ) + (1 + a)x1 x2 (x1 + x2 ) + b
2

over R2 , where a and b are real-valued parameters. Find all values of a and b
such that the problem has a unique optimal solution.
Exercise 4.3 (spectral theory and unconstrained optimization) Let A be a symmetric n n matrix. For x Rn , x 6= 0n , consider the function (x) :=
xTTAx , and the related optimization problem to
x x
minimize
(x).
n
x 6=0

(P)

Determine the stationary points as well as the global minima in the problem (P). Interpret the result in terms of linear algebra.

107

An introduction to optimality conditions

Exercise 4.4 (non-convex QP over subspaces) The FrankWolfe Theorem 4.9
can be further improved for some special cases of linear constraints. Suppose
that f (x) := 12 xT Qx qT x, where Q Rnn is a symmetric matrix and
q Rn . Suppose further that the constraints are equalities, that is, that the
constraints define the linear system Ex = d, where E Rn and d R .
Note that the problem may not be convex, as we have not assumed that Q is
positive semidefinite.
For this set-up, establish the following:
(a) Every locally optimal solution is a globally optimal solution.
(b) A locally [hence globally, by (a)] optimal solution exists if and only if
f is lower bounded on S := { x Rn | Ex = d }.
Exercise 4.5 (descent from projection) Consider the problem (4.9), where f
is in C 1 on the closed and convex set S. Let x S. Let > 0, and define

p := ProjS [x f (x)] x.

Notice that

p is a feasible direction at x. Establish that

1
f (x)T p kpk2

holds. Hence, p is zero if and only if x is stationary [according to the characterization in (4.11)], and if p is non-zero then it defines a feasible descent
direction with respect to f at x.
Exercise 4.6 (optimality conditions for a special problem) Suppose that f
C 1 on the set S := { x Rn | xj 0, j = 1, 2, . . . , n }, and consider
the problem of finding a minimum of f (x) over S. Develop the necessary
optimality conditions for this problem in a compact form.
Exercise 4.7 (optimality conditions for a special problem) Consider the problem to
maximize f (x) := xa1 1 xa2 2 xann ,
subject to

n
X

xj = 1,

j=1

xj 0,

j = 1, . . . , n,

where the values of aj (j = 1, . . . , n) are positive. Find a global maximum and

show that it is unique.
Exercise 4.8 (extensions of convexity, exam 040602) We have stressed that
convexity is a crucial property of functions when analyzing optimization models in general and studying optimality conditions in particular. There are,
however, certain properties of convex functions that are shared also by classes
of non-convex functions. The purpose of this exercise is to relate the convex functions to two such classes of non-convex functions by means of some
example properties.

108

Exercises
Suppose that S Rn and that f : Rn R is continuous on S.
(a) Suppose further that f is in C 1 on S. We say that the function f is
pseudo-convex on S if, for every x, y S,
f (x)T (y x) 0

f (y ) f (x).

Establish the following two statements: (1) if f is a convex function on S

then f is pseudo-convex on S (that is, convexity implies pseudo-convexity);
(2) the reverse statement (pseudo-convexity implies convexity) is not true.
[Note: The definition of a pseudo-convex function is due to Mangasarian
[Man65].]
(b) A well-known property of a differentiable convex function is its role
in necessary and sufficient conditions for globally optimal solutions. Suppose
now that S is convex. Establish that the equivalence stated in Theorem 4.24
still holds if the convexity of f on S is replaced by the pseudo-convexity of f
on S.
(c) Let S be convex. We say that the function f is quasi-convex on S if its
level sets restricted to S are convex. In other words, f is quasi-convex on S if
lev S
f (b) := { x S | f (x) b }
is convex for every b R.
Establish the following two statements for a function f which is in C 1
on S: (1) if f is a convex function on S then f is quasi-convex on S (that
is, convexity implies quasi-convexity); (2) the reverse statement (quasiconvexity implies convexity) is not true.
[Note: The definition of a quasi-convex function is due to De Finetti
[DeF49].]
Exercise 4.9 (illustrations of fixed point results) (a) Let S := { x R | x 1 }
and f (x) := x/2 + 1/x. Show that f is a contraction and find the smallest
value of .
(b) In analysis, a usual condition for the convergence of an iteration xk =
g(xk1 ) is that g is continuously differentiable and
|g (x)| < 1,

x S.

(4.34)

Establsh that (4.34) implies convergence, by using Banachs Theorem 4.34(a).

(c) Show that a fixed-point iteration for calculating the square root of a
given positive number c is
x0 > 0;

xk+1

1
= g(xk ) :=
2

c
xk +
xk

k = 0, 1, . . . .

What condition do we get

from (b)? Starting at x0 = 1, calculate the approximations x1 , x2 , x3 , x4 of 2.

109

An introduction to optimality conditions

Exercise 4.10 (Ekelands Variational Principle and Banachs Theorem) Ekelands variational principle states that if a lower semicontinuous function f
attains a value close to its infimum at some point then a nearby point minimizes a slightly perturbed function exactly. We state and prove this result
and then utilize it to prove Banachs Theorem 4.34(a).
Theorem 4.35 (Ekelands variational principle) Suppose the function f : Rn
R {+} is lower semicontinuous and that the point x Rn satisfies f (x)
f + for some > 0. Then for any real > 0 there is a point v Rn such
that
(a) kx vk ;
(b) f (v) f (x); and
(c) v uniquely minimizes the function

x 7 f (x) + kx v k over Rn .

Proof. We can assume that f is proper, and by assumption it is bounded

below. Since the function f () + k xk therefore has compact level sets, its
set of minimizers M Rn is nonempty and compact.
Choose a minimizer v of f over M . Then for points z 6= v in M we know
that

f (v ) f (z ) < f (z ) + kz vk,

while for z not in M we have that

f (v ) + kv xk < f (z ) + kz xk.

Part (c) follows by the triangle inequality. Since v lies in M we have that

f (z ) + kz xk f (v ) + kv xk,
z Rn .

Setting z = x shows the inequalities

f (v ) + f + f (x) f (v ) + kv xk.

Properties (a) and (b) follow.

Using this result, prove Banachs Theorem 4.34(a).
[Hint: Apply the Ekeland variational principle to the function
(

at an arbitrary point

z 7

kz f (z )k,
+,

if z S,
otherwise

x in S, with the choice of reals

:= kx f (x)k

and

:=

,
1

where is the contraction parameter for f .]

[Note: Ekelands variational principle is found in [Eke74], while its use in
the proof of Banachs Theorem can be found, for example, in [BoL00, Theorem 8.1.2].]

110

Optimality conditions

5.1

Relations between optimality conditions

and CQs at a glance

Optimality conditions are introduced as an attempt to construct an easily verifiable criterion that allows us to examine points in a feasible set,
one after another, and classify them into optimal and non-optimal ones.
Unfortunately, this is impossible in practice, and not only due to the
fact that there are far too many feasible points, but also because it is
impossible to construct such a universal criterion. It is usually possible
to construct either practical (that is, computationally verifiable) conditions that admit some mistakes in the characterization, or perfect ones
which are impossible to use in the computations. It is of course the first
group that is of practical value for us, and it may further be classified
into two distinct subgroups based on the type of mistakes allowed in the
decision-making process. Namely, optimality conditions encountered in
practice are divided into two classes, known as necessary and sufficient
conditions.
Necessary conditions must be satisfied at every locally optimal point;
on the other hand, we cannot guarantee that every point satisfying the
necessary optimality conditions is indeed locally optimal. On the contrary, sufficient optimality conditions provide such guarantees; however,
there may be some locally optimal points that violate the optimality
conditions. Arguably, it is much more important to be able to find a few
candidates for local minima that can be further investigated by other
means, than to eliminate some local (or even global) minima from the
beginning. Therefore, this chapter is dedicated to the development of
necessary optimality conditions. However, for convex optimization problems these conditions turn out to be sufficient.

Optimality conditions
Now, we can concentrate on what should be meant by easily verifiable
conditions. A human being can immediately state whether a given point
belongs to a simple set or not, by just glancing at a picture of it; for a numerical algorithm, a clear algebraic description of a set in terms of equalities and inequalities is vital. Therefore, we start our development with
geometric optimality conditions (Section 5.3), to gain an understanding
about the relationships between the gradient of the objective function
and the feasible set that must hold at every local minimum point. Given
a specific description of a feasible set in terms of inequalities, the geometric conditions immediately imply some relationships between the
gradients of the objective function and the constraints that are active
at the point under consideration (see Section 5.4); these conditions are
known as the Fritz John optimality conditions, and are rather weak (i.e.,
they can be satisfied by many points that have nothing in common with
locally optimal points). However, if we assume an additional regularity of
the system of inequalities and equalities that define our feasible set, then
the geometric optimality conditions imply stronger conditions, known as
the KarushKuhnTucker optimality conditions (see Section 5.5). The
additional regularity assumptions are known under the name constraint
qualifications (CQs), and they vary from very abstract and difficult to
check, but enjoyed by many feasible sets (such as, e.g., Abadies CQ,
see Definition 5.23) to more specific, easily verifiable but also somewhat
restrictive in many situations (such as the linear independence CQ, see
Definition 5.41, or the Slater CQ, see Definition 5.38). In Section 5.8
we show that for convex problems the KKT conditions are sufficient for
local, hence global, optimality.
The contents of this chapter are in principle summarized in the flowchart in Figure 5.1. Various optimality conditions and constraint qualifications that are discussed in this chapter constitute the nodes of the
flow-chart. Logical relationships between them are denoted with edges,
and the direction of the arrow shows the direction of the logical implication; each implication is further labeled with the result that establishes
it. We note that the KKT conditions follow from both geometric
conditions and constraint qualifications satisfied at a given point; also,
global optimality holds if both the KKT conditions are verified and the
optimization problem is convex.

5.2

A note of caution

In this chapter we will discuss various necessary optimality conditions for

a given point to be a local minimum to a nonlinear programming model.
If the NLP is a convex program, any point satisfying these necessary
112

A note of caution

x locally optimal

x globally optimal

Geometric OC

F (x ) TS (x ) =

Theorem 5.15

Theorem 5.45

Theorem 5.8

Fritz John OC (5.8)

Theorem 5.33

Abadies CQ
TS (x ) = G(x ) H(x )

KKT OC (5.17)

Convexity
Proposition 5.36
Proposition 5.44

MFCQ (Definition 5.35)

Proposition 5.42

LICQ (Definition 5.41)

Affine constraints
Proposition 5.39

Slater CQ (Definition 5.38)

Figure 5.1: Relations between optimality conditions and CQs at a glance.

optimality conditions is not only a local minimum, but actually a global
minimum (see Section 5.8). Arguably, most NLP models that arise in
real world applications tend to be nonconvex, and for such a problem,
a point satisfying the necessary optimality conditions may not even be
a local minimum. Algorithms for NLP are usually designed to converge
to a point satisfying the necessary optimality conditions, and as mentioned earlier, one should not blindly accept such a point as an optimum
solution to the problem without checking (e.g., using the second order
necessary optimality conditions, see [BSS93, Section 4.4], or by means
of some local search in the vicinity of the point) that it is at least better
than all the other nearby points. Also, the system of necessary optimality conditions may have many solutions. Finding alternate solutions of
this system, and selecting the best among them, usually leads to a good
113

Optimality conditions
point to investigate further.
We will illustrate the importance of this with the story of the US Air
Forces controversial B-2 Stealth bomber program in the Reagan era of
the 1980s. There were many design variables, such as the various dimensions, the distribution of volume between the wing and the fuselage,
flying speed, thrust, fuel consumption, drag, lift, air density, etc., that
could be manipulated for obtaining the best range (i.e., the distance it
can fly starting with full tank, without refueling). The problem of maximizing the range subject to all the constraints was modeled as an NLP
in a secret Air Force study going back to the 1940s. A solution to the
necessary optimality conditions of this problem was found; it specified
values for the design variables that put almost all of the total volume in
the wing, leading to the flying wing design for the B-2 bomber. After
spending billions of dollars, building test planes, etc., it was found that
the design solution implemented works, but that its range was too low in
comparison with other bomber designs being experimented subsequently
in the US and abroad.
A careful review of the model was then carried out. The review indicated that all the formulas used, and the model itself, are perfectly valid.
However, the model was a nonconvex NLP, and the review revealed a
second solution to the system of necessary optimality conditions for it,
besides the one found and implemented as a result of earlier studies. The
second solution makes the wing volume much less than the total volume,
and seems to maximize the range; while the first solution that is implemented for the B-2 bomber seems to actually minimize the range. (The
second solution also looked like an aircraft should, while the flying wing
design was counter-intuitive.) In other words, the design implemented
was the aerodynamically worst possible choice of configuration, leading
to a very costly error. The aircraft does fly, but apparently, then, has
the only advantage that it is a stealth plane.
For an account, see Skeleton Alleged in the Stealth Bombers Closet,
Science, vol. 244, 12 May 1989 issue, pages 650651.

5.3

Geometric optimality conditions

In this section we will discuss the optimality conditions for the following
optimization problem [cf. (4.1)]:
minimize f (x),
subject to x S,

(5.1)

where S Rn is a nonempty closed set and f : Rn R is a given

differentiable function. Since we do not have any particular description
114

Geometric optimality conditions

of the feasible set S in terms of equality or inequality constraints, the
optimality conditions will be based on purely geometrical ideas. Being
very general, the optimality conditions we develop in this section are
almost useless for computations, because they are also not very easy (in
fact they are even impossible) to verify for an optimization algorithm.
Therefore, in the sections that follow we will use an algebraic description
of the set S and geometric optimality conditions to further develop the
classic Fritz John and KarushKuhnTucker optimality conditions in the
form of easily verifiable systems of equations and inequalities.
The basic idea behind the optimality conditions is that if the point
x S is a local minimum of f over S, it should not be possible to
draw a curve starting at the point x inside S, such that f decreases
along it arbitrarily close to x . Linearizing the objective function and
the constraints along such curves, we eventually establish relationships
between their gradients that are necessary to hold at points of local
minima.
We first define the meaning of possible to draw a curve starting at
x inside S. Arguably, he simplest curves are the straight lines; the
following definition gives exactly the set of lines that locally around x
belong to S.
Definition 5.1 (cone of feasible directions) Let S Rn be a nonempty
closed set. The cone of feasible directions for S at x Rn , known also
as the radial cone, is defined as:
RS (x) := { p Rn | e > 0 such that x + p S, 0 e }. (5.2)

Thus, this is nothing else but the cone containing all feasible directions
at x in the sense of Definition 4.20.

This cone is used in some optimization algorithms, but unfortunately

it is too small to develop optimality conditions that are general enough.
Therefore, we consider less intuitive, but bigger and more well-behaving
sets (cf. Proposition 5.3 and the examples that follow).
Definition 5.2 (tangent cone) Let S Rn be a nonempty closed set.
The tangent cone for S at x Rn is defined as
TS (x) := { p Rn | {xk } S, {k } (0, ) :
lim k (xk x) = p }.

lim xk = x;

(5.3)

115

Optimality conditions
Thus, to construct a tangent cone we consider all the sequences {xk }
in S that converge to the given x Rn , and then calculate all the directions p Rn that are tangential to the sequences at x; such tangential
vectors are described as the limits of {k (xk x)} for arbitrary positive
sequences {k }. Note that to generate a nonzero vector p TS (x) the
sequence {k } must converge to +.
While it is possible that cl RS (x) = TS (x), or even that RS (x) =
TS (x), in general we have only the following proposition, and examples
that follow show that the two cones might be very different.
Proposition 5.3 (relationship between the radial and the tangent cones)
The tangent cone is a closed set, and the inclusion cl RS (x) TS (x)
holds for every x Rn .
Proof. Consider a sequence {pk } TS (x), and assume that {pk } p.
Since every pk TS (x), there exist xk S and k > 0, such that
kxk xk < k 1 and kk (xk x) pk k < k 1 . Then, clearly, {xk } x,
and, by the triangle inequality, kk (xk x) pk kk (xk x) pk k +
kpk pk, and the two terms in the right-hand side converge to 0, which
implies that p TS (x) and thus the latter set is closed.
In view of the closedness of the tangent cone, it is enough to show the
inclusion RS (x) TS (x). Let p RS (x). Then, for all large integers k
it holds that x + k 1 p S, and, therefore, setting xk = x + k 1 p and
k = k we see that p TS (x) as defined by Definition 5.2.
Example 5.4 Let S := { x R2 | x1 0; (x1 1)2 + x22 1 }. Then,
RS (02 ) = { p R2 | p1 > 0 }, and TS (02 ) = { p R2 | p1 0 }, i.e.,
TS (02 ) = cl RS (02 ) (see Figure 5.2).
Example 5.5 (complementarity constraint) Let S := { x R2 | x1
0; x2 0; x1 x2 0 }. In this case, S is a (non-convex) cone, and
RS (02 ) = TS (02 ) = S (see Figure 5.3).
Example 5.6 Let S := { x R2 | x31 +x2 0; x51 x2 0; x2 0 }.
Then, RS (02 ) = , and TS (02 ) = { p R2 | p1 0; p2 = 0 } (see
Figure 5.4).
Example 5.7 Let S := { x R2 | x2 0; (x1 1)2 + x22 = 1 }. Then,
RS (02 ) = , TS (02 ) = { p R2 | p1 = 0; p2 0 } (see Figure 5.5).
We already know that f decreases along any descent direction (cf. Definition 4.15), and that for a vector p Rn it is sufficient to verify the
inequality f (x )T p < 0 to be a descent direction for f at x Rn
116

Geometric optimality conditions

S
1

S
TS (02 )

1
(a)

(b)

Figure 5.2: (a) The set S obtained as the intersection of the solution set
of two constraints; (b) the tangent cone TS (02 ) (see Example 5.4).
(see Proposition 4.16). Even though this condition is not necessary, it
is very easy to check in practice and therefore we will use it to develop
optimality conditions. Therefore, it would be convenient to define a cone
of such directions (which is empty if f (x ) happens to be 0n ):

F (x ) := { p Rn | f (x )T p < 0 }.

(5.4)

Now we have the necessary notation in order to state and prove the
main theorem of this section.
Theorem 5.8 (geometric necessary optimality conditions) Consider the
optimization problem (5.1). Then,

x is a local minimum of f over S = F (x ) TS (x ) = ,

where F (x ) is defined by (5.4), and TS (x ) by Definition 5.2.

Proof. Assume that p TS (x ), i.e., {xk } S, and {k } (0, )
such that limk xk = x and limk k (xk x ) = p. Using the first
order Taylor expansion (2.1) we get:
f (xk ) f (x ) = f (x )T (xk x ) + o(kxk x k) 0,
where the last inequality holds for all enough large k by the local opti117

Optimality conditions

Figure 5.3: S = RS (02 ) = TS (02 ) (see Example 5.5).

mality of x . Multiplying by k > 0 and taking limit we get


o(kxk x k)
T

0 lim k f (x ) (xk x ) + kk (xk x )k

k
kxk x k
= f (x )T p + kpk 0,

and thus p 6 F (x ).
Combining Proposition 5.3 and Theorem 5.8 we get that
x is a local minimum of f over S

F (x ) RS (x ) = ;

but this statement is weaker than Theorem 5.8.

Example 5.9 Consider the differentiable (linear) function f : R2 R

defined by f (x) = x1 . Then, f = (1, 0)T , and F (02 ) = { x R2 |

x1 < 0 }. It is easy to see from geometric considerations that x = 02
is a local (in fact, even global) minimum in either problem (5.1) with
S given by Examples 5.45.7, and equally easy it is to check that the

geometric necessary optimality condition F (02 ) TS (02 ) = is satisfied

in all these examples (which is no surprise, in view of Theorem 5.8).

5.4

The Fritz John conditions

Theorem 5.8 gives a very elegant criterion for checking whether a given
point x S is a candidate for a local minimum of the problem (5.1),
118

The Fritz John conditions

(a)

(b)

Figure 5.4: (a) The set S; (b) the tangent cone TS (02 ) (see Example 5.6).

1
(a)

1
(b)

Figure 5.5: (a) The set S; (b) the tangent cone TS (02 ) (see Example 5.7).
but there is a catch: the set TS (x ) is close to impossible to compute
for general sets S! Therefore, in this section we will use an algebraic
characterization of the set S to compute other cones that we hope could
approximate TS (x ) in many practical situations.
Namely, we assume that the set S is defined as the solution set of a
system of differentiable inequality constraints defined by the functions
gi C 1 (Rn ), i = 1, . . . , m, such that
S := { x Rn | gi (x) 0,

i = 1, . . . , m }.

(5.5)

We can always assume this structure, because any equality constraint

h(x) = 0 may be written in the form h(x) 0, h(x) 0. Treating the
equality constraints in this way we obtain the Fritz John conditions, that
however are somewhat too weak to be practical; on the positive side, it
significantly simplifies the notation and does not affect the development
119

Optimality conditions
of the KKT conditions. Therefore, we keep this assumption for some
time, and state the KKT system that specifically distinguishes between
the inequality and equality constraints in Section 5.6. We will use the
symbol I(x) to denote the index set of active inequality constraints at
x Rn (see Definition 4.21), and |I(x)| to denote the cardinality of this
set, i.e., the number of active inequality constraints at x Rn .
In order to compute approximations to the tangent cone TS (x), similarly to Example 4.22 we consider cones associated with the active constraints at a given point:

G(x) := { p Rn | gi (x)T p < 0, i I(x) },

(5.6)

G(x) := { p Rn | gi (x)T p 0, i I(x) }.

(5.7)

and

The following proposition verifies that G(x) is an inner approximation for RS (x) (and, therefore, for TS (x) as well, see Proposition 5.3),
and G(x) is an outer approximation for TS (x).

Lemma 5.10 For every x Rn it holds that G(x) RS (x), and

TS (x) G(x).

Proof. Let p G(x). For every i 6 I(x) the function gi is continuous

and gi (x) < 0; therefore gi (x + p) < 0 for all small > 0. Moreover, by
Proposition 4.16, p is a direction of descent for every gi at x, i I(x),
which means that gi (x + p) < gi (x) = 0 for all such i and all small

> 0. Thus, p RS (x), and, hence, G(x) RS (x).

Now, let p TS (x), i.e., {xk } S, and {k } (0, ) such that
limk xk = x and limk k (xk x) = p. Exactly as in the proof
of Theorem 5.8, we use the first order Taylor expansion (2.1) of the
functions gi , i I(x), to get:
0 gi (xk ) = gi (xk ) gi (x) = gi (x)T (xk x) + o(kxk xk),
where the first inequality is by the feasibility of xk . Multiplying by
k > 0 and taking limit we get, for i I(x),


o(kxk xk)
0 lim k gi (x)T (xk x) + kk (xk x)k
k
kxk xk
= gi (x)T p + kpk 0,

and thus p G(x).

120

The Fritz John conditions

Example 5.11 (Example 5.4 continued) The set S is defined by the two
inequality constraints g1 (x) := x1 0 and g2 (x) := (x1 1)2 +x22 1

0. Let us calculate G(02 ) and G(02 ). Both constraints are satisfied with
equality at the given point, so that I(02 ) = {1, 2}. Then, g1 (02 ) =

(1, 0)T , g2 (02 ) = (2, 0)T , and thus G(02 ) = { x R2 | x1 > 0 } =

RS (02 ), G(02 ) = { x R2 | x1 0 } = TS (02 ) in this case.

Example 5.12 (Example 5.5 continued) The set S is defined by the

three inequality constraints g1 (x) := x1 0, g2 (x) := x2 0,
g3 (x) := x1 x2 0, which are all active at x = 02 ; g1 (02 ) = (1, 0)T ,

g2 (02 ) = (0, 1)T , and g3 (02 ) = (0, 0)T . Therefore, G(02 ) = (

RS (02 ), and G(02 ) = { x R2 | x1 0, x2 0 } ) TS (02 ).

Example 5.13 (Example 5.6 continued) The set S is defined by the three
inequality constraints g1 (x) := x31 + x2 0, g2 (x) := x51 x2 0,
g3 (x) := x2 0, which are all active at x = 02 ; g1 (02 ) = (0, 1)T ,

g2 (02 ) = (0, 1)T , and g3 (02 ) = (0, 1)T . Therefore, G(02 ) = =

RS (02 ), and G(02 ) = { x R2 | x2 = 0 } ) TS (02 ).

Example 5.14 (Example 5.7 continued) The set S is defined by the

inequality constraint g1 (x) := x2 0, and the equality constraint
h1 (x) := (x1 1)2 +x22 1 = 0; we split the latter into two inequality constraints g2 (x) := h1 (x) 0, and g3 (x) := h1 (x) 0. Thus, we end up
with three active inequality constraints at x = 02 ; g1 (02 ) = (0, 1)T ,

g2 (02 ) = (2, 0)T , and g3 (02 ) = (2, 0)T . Therefore, G(02 ) = =

RS (02 ), and G(02 ) = { x R2 | x1 = 0, x2 0 } = TS (02 ).
Now we are ready to establish the Fritz John optimality conditions.

Theorem 5.15 (Fritz John necessary optimality conditions) Let the set
S be defined by (5.5). If x S is a local minimum of f over S then
there exist multipliers 0 R, Rm , such that
0 f (x ) +

m
X
i=1

i gi (x ) = 0n ,

(5.8a)

i gi (x ) = 0,

i = 1, . . . , m,

(5.8b)

0 , i 0,

i = 1, . . . , m,

(5.8c)

T T

(0 , ) 6= 0

m+1

(5.8d)

121

Optimality conditions
In other words,
x local minimum of f over S = (0 , ) R Rm : (5.8) holds.

Proof. Combining the results of Lemma 5.10 with the geometric optimality conditions provided by Theorem 5.8, we conclude that there is
no direction p Rn such that f (x )T p < 0 and gi (x )T p < 0, i
I(x ). Define the matrix A with columns f (x ), gi (x ), i I(x );

then the system AT p < 01+|I(x )| is unsolvable. By Farkas Lemma

(cf. Theorem 3.30)

there exists a nonzero vector R1+|I(x )| such
1+|I(x )|
n
T
that 0
and A = 0 . (Why?) Now, let (0 , T
I(x ) ) := ,

and set i = 0 for i 6 I(x ). It is an easy exercise to verify that so

defined 0 and satisfy the conditions (5.8).
Remark 5.16 (terminology) The solutions (0 , ) to the system (5.8)
are known as Lagrange multipliers (or just multipliers) associated with
a given candidate x Rn for a local minimum. Note that every multiplier (except 0 ) corresponds to some constraint in the algebraic representation of S. The conditions (5.8a) and (5.8c) are known as the dual
feasibility conditions, and (5.8b) as the complementarity conditions, respectively; this terminology will become more clear in Chapter 6. Owing
to the complementarity constraints, the multipliers i corresponding to
inactive inequality constraints i 6 I(x ) must be zero. In general, the
Lagrange multiplier i bears the important information about how sensitive a particular local minimum is with respect to small changes in the
constraint gi .
In the following examples, as before, we assume that f (x) := x1 , so
that f = (1, 0)T and x = 02 is the point of local minimum.
Example 5.17 (Example 5.4 continued) The Fritz John system (5.8) at
the point x = 02 reduces to:
  

1
1 2
0
+
= 02 ,
0
0
0
(0 , T )T 03 ,

where R2 is a vector of Lagrange multipliers for the inequality

constraints We do not write the complementarity constraints (5.8b), because in our case the two constraints are active, and therefore the equation (5.8b) is automatically satisfied for all . The solutions to this
122

The Fritz John conditions

system are the pairs (0 , ), with = (1 , 21 (0 1 ))T , for every
0 > 0, 0 1 0 . There are infinitely many Lagrange multipliers,
that form an unbounded set, but 0 must always be positive.
Example 5.18 (Example 5.5 continued) Similarly to the previous example, the Fritz John system (5.8) at the point x = 02 reduces to:
  

1
1 0 0
0
+
= 02 ,
0
0 1 0
(0 , T )T 04 ,

where R3 is a vector of Lagrange multipliers for the inequality

constraints. The solution to the Fritz John system is every pair (0 , )
with = (0 , 0, 3 )T for every 0 0, 3 0 such that either of them
is strictly bigger than zero. That is, there are infinitely many Lagrange
multipliers, that form an unbounded set, and it is possible for 0 to
assume the value zero.
Example 5.19 (Example 5.6 continued) The Fritz John system (5.8) at
the point x = 02 reduces to:
  

1
0 0
0
0
+
= 02 ,
0
1 1 1
(0 , T )T 04 ,

where R3 is a vector of Lagrange multipliers for the inequality

constraints. Thus, 0 = 0, = (1 , 2 , 1 2 )T for every 1 > 0,
0 2 1 . That is, there are infinitely many Lagrange multipliers,
that form an unbounded set, and 0 must assume the value zero.
Example 5.20 (Example 5.7 continued) The Fritz John system (5.8) at
the point x = 02 reduces to:
  

1
0 2 2
0
+
= 02 ,
0
1 0 0
(0 , T )T 04 ,

where R3 is a vector of Lagrange multipliers for the inequality

constraints. The solution to the Fritz John system is every pair (0 , )
with = (0, 2 , 2 21 0 )T for every 2 > 0, 0 0 22 . That is,
there are infinitely many Lagrange multipliers, that form an unbounded
set, and it is possible for 0 to assume the value zero.
123

Optimality conditions
The fact that 0 may be zero in the system (5.8) essentially means
that the objective function f plays no role in the optimality conditions.
This is of course a rather unexpected and unwanted situation, and the
rest of the chapter is dedicated to describing how one can avoid it.
Since the cone of feasible directions RS (x) may be a bad approxi

mation of the tangent cone TS (x), so may G(x) owing to Lemma 5.10.
Therefore, in the most general case we cannot improve on the conditions (5.8); however, it is possible to improve upon (5.8) if we assume

that the set S is regular in some sense, i.e., that either G(x) or G(x)
is a tight enough approximation of TS (x). Requirements of this type
are called constraint qualifications, and they will be discussed in more
detail in Section 5.7. However, to get a feeling of what can be achieved
with a regular constraint set S, we show that the multiplier 0 in the system (5.8) cannot vanish (i.e., the KKT conditions hold, see Section 5.5) if

the constraint qualification G(x ) 6= holds (which is quite a restrictive

one, in view of Example 5.22; however, see the much weaker assumption
denoted MFCQ in Definition 5.35).
Proposition 5.21 (KKT optimality conditions, preview) Assume the con

ditions of Theorem 5.8, and assume that G(x ) 6= . Then, the multiplier
0 in (5.8) cannot be zero; dividing all equations by 0 we may assume
that it equals one.
Proof. Assume that 0 = 0 in (5.8), and define the matrix A with

columns g
Since A
= 0n , 0|I(x )| , and
i (x ), i I(x ).

6= 0|I(x )| , the system AT p < 0|I(x )| is unsolvable (see Farkas

Lemma, Theorem 3.30), i.e., G(x ) = .

Example 5.22 Out of the four Examples 5.45.7, only the first verifies

the condition G(x ) 6= assumed in Proposition 5.21, while as we will

see later (and as Examples 5.175.20 may suggest), three out of the four
problems admit solutions to the corresponding KKT systems.

5.5

The KarushKuhnTucker conditions

In this section we develop the famous and classic KarushKuhnTucker

optimality conditions for constrained optimization problems with inequality constraints, which are essentially the Fritz John conditions (5.8)
with the additional requirement 0 6= 0 (in fact, 0 = 1). We establish these conditions as before, for inequality constrained problems (5.5)
124

The KarushKuhnTucker conditions

(which we do without any loss of generality or sharpness of the theory),
and then discuss the possible modifications of the conditions if one wants
to specifically distinguish between equality and inequality constraints in
Section 5.6. Abadies constraint qualification (see Definition 5.23) which
we impose is very abstract and extremely general (this is almost the
weakest condition one can require); of course it is impossible to check
it when it comes to practical problems. Therefore, in Section 5.7 we
list some computationally verifiable assumptions that all imply Abadies
constraint qualification.
We start with a formal definition.
Definition 5.23 (Abadies constraint qualification) We say that at the
point x S Abadies constraint qualification holds if TS (x) = G(x),
where TS (x) is defined by Definition 5.2 and G(x) by (5.7).
Example 5.24 Out of the four Examples 5.45.7, the first and the last
satisfy Abadies constraint qualification (see Examples 5.115.14).
Then, we are ready to prove the main theorem in this chapter.
Theorem 5.25 (KarushKuhnTucker optimality conditions) Assume that
at a given point x S Abadies constraint qualification holds. If x S
is a local minimum of f over S then there exists a vector Rm such
that

f (x ) +

m
X
i=1

i gi (x ) = 0n ,
i gi (x ) = 0,
m

0 .

(5.9a)
i = 1, . . . , m,

(5.9b)
(5.9c)

In other words,
x local minimum of f over S
Abadies CQ holds at x

= Rm : (5.9) holds.

The system will be referred to as the KarushKuhnTucker optimality conditions.

Proof. By Theorem 5.8 we have that F (x ) TS (x ) = , which due

to our assumptions implies that F (x ) G(x ) = .

As in the proof of Theorem 5.15, construct a matrix A with columns

gi (x ), i I(x ). Then, the system AT p 0|I(x )| and f (x )T p >

125

Optimality conditions
0 has no solutions. By Farkas Lemma (cf. Theorem 3.30), the system A = f (x ), 0|I(x )| has a solution. Define the vector
I(x ) = , and i = 0, for i 6 I(x ). Then, the so defined verifies
the KKT conditions (5.9).
Remark 5.26 (terminology) Similarly to the case of the Fritz John necessary optimality conditions, the solutions to the system (5.9) are
known as Lagrange multipliers (or just multipliers) associated with a
given candidate x Rn for a local minimum. The conditions (5.9a)
and (5.9c) are known as the dual feasibility conditions, and (5.9b) as the
complementarity conditions, respectively; this terminology will become
more clear in Chapter 6. Owing to the complementarity constraints, the
multipliers i corresponding to inactive inequality constraints i 6 I(x )
must be zero. In general, the Lagrange multiplier i bears the important information about how sensitive a particular local minimum is with
respect to small changes in the constraint gi .
Remark 5.27 (geometric interpretation) The system of equations and
inequalities defining (5.9) can (and should) be interpreted geometrically
as f (x ) NS (x ) (see Figure 5.6), the latter cone being the normal
cone to S at x S (see Definition 4.25); according to the figure, the
normal cone to S at x is furthermore spanned by the gradients of the
active constraints at x .1
Notice the specific roles played by the different parts of the system (5.9) in this respect: the complementarity conditions (5.9b) force
i to be equal to 0 for the inactive constraints, whence the summation
in the left-hand side of the linear system (5.9a) involves the active constraints only. Further, the sign conditions in (5.9c) ensures that each
vector i gi (x ), i I(x ), is an outward normal to S at x .
Remark 5.28 Note that in the unconstrained case the KKT system (5.9)
reduces to the single requirement f (x ) = 0n , which we have already
encountered in Theorem 4.14.
It is possible to further develop the KKT theory (with some technical
complications) for twice differentiable functions as it has been done for
the unconstrained case in Theorem 4.17. We refer the interested reader
to [BSS93, Section 4.4].
1 Compare with the normal cone characterization (4.17) and Figure 4.4 in the case
of convex feasible sets: we could, roughly, say that the role of a constraint qualification
in the more general context of this chapter is to ensure that the normal cone to the
feasible set at the vector x is a finitely generated convex cone, which moreover is
generated by the gradients of the active constraints describing functions gi at x ,
thus extending the normal cone inclusion in (4.17) to more general sets.

126

The KarushKuhnTucker conditions

f (x)

g2(x)

g1(x)

g1= 0

g2= 0
x

g3= 0

Figure 5.6: Geometrical interpretation of the KKT system.

Example 5.29 (Example 5.4 continued) In this example Abadies constraint qualification is fulfilled; the KKT system must be solvable. Indeed, the system
  

1
1 2
+
= 02 ,
0
0
0
02 ,

possesses solutions = (1 , 21 (1 1 ))T for every 0 1 1. Therefore, there are infinitely many multipliers, that all belong to a bounded
set.
Example 5.30 (Example 5.5 continued) This is one of the rare cases
when Abadies constraint qualification is violated, and nevertheless the
KKT system happens to be solvable:
  
1
1 0
+
0
0 1


0
= 02 ,
0
03 ,
127

Optimality conditions
admits solutions = (1, 0, 3 )T for every 3 0. That is, the set of
Lagrange multipliers is unbounded in this case.
Example 5.31 (Example 5.6 continued) Since, for this example, in the
Fritz John system the multiplier 0 is necessarily zero, the KKT system
admits no solutions:
  

1
0 0
0
+
= 02 ,
0
1 1 1
03 ,

is clearly inconsistent. In this example Abadies constraint qualification

is violated.
Example 5.32 (Example 5.7 continued) This example satisfies Abadies
constraint qualification, and therefore, since a global optimum exists,
the KKT system is solvable:
  

1
0 2 2
+
= 02 ,
0
1 0 0
03 ,

admits the solutions = (0, 2 , 2 21 )T , for all 2 21 . The set of

Lagrange multipliers is unbounded in this case, but this is because we
have split the original equality constraint into two inequalities. In Section 5.6 we formulate the KKT system that keeps the original equalityrepresentation of the set, and thus reduce the number of multipliers for
the equality constraint to just one!

5.6

Proper treatment of equality constraints

Now we consider both inequality and equality constraints, that is, we

assume that the feasible set S is given by
S := { x Rn | gi (x) 0,
hj (x) = 0,

i = 1, . . . , m;
j = 1, . . . , },

(5.10)

instead of (5.5), where gi C 1 (Rn ), i = 1, . . . , m, and hj C 1 (Rn ), j =

1, . . . , . As it was done in Section 5.4, we write S using only inequality
constraints, by defining the functions gei C 1 (Rn ), i = 1, . . . , m + 2,
via:

i = 1, . . . , m,
g i ,
gei := him ,
(5.11)
i = m + 1, . . . , m + ,

him ,
i = m + + 1, . . . , m + 2,

128

Proper treatment of equality constraints

so that
S = { x Rn | gei (x) 0,

i = 1, . . . , m + 2 }.

(5.12)

e
Now, let G(x)
be defined by (5.7) for the inequality representation (5.12)
of S. We will use the old notation G(x) for the cone defined only by the
gradients of the functions defining the inequality constraints active at x
in the representation (5.10), and in addition define the null space of the
matrix defined by the gradients of the functions defining the equality
constraints:
H(x) := { p Rn | hi (x)T p = 0,

i = 1, . . . , }.

(5.13)

Since all inequality constraint functions gei , i = m + 1, . . . , m + 2, are

necessarily active at every x S, it holds that
e
G(x)
= G(x) H(x),

(5.14)

TS (x) = G(x) H(x).

(5.15)

and thus Abadies constraint qualification (see Definition 5.23) for the
set (5.10) may be equivalently written as

Assuming that the latter constraint qualification holds we can write the
KKT system (5.9) for x S, corresponding to the inequality representation (5.12) (see Theorem 5.25):
m
X
i=1

i gi (x ) +

m+
X

i=m+1

i him (x )

m+2
X

i=m++1

i him (x )

+f (x ) = 0n ,
i gi (x ) = 0, i = 1, . . . , m,

(5.16a)
(5.16b)

i him (x ) = 0,

i = m + 1, . . . , m + ,
(5.16c)

i him (x ) = 0,

i = m + + 1, . . . , m + 2,
(5.16d)

0m+2 .

(5.16e)

e Rm R as
ej =
Define the pair of vectors (e
, )
ei = i , i = 1, . . . , m;
m+j m++j , j = 1, . . . , . We also note that the equations (5.16c)
and (5.16d) are superfluous, because x S implies that hj (x ) = 0,
e known
j = 1, . . . , . Therefore, we get the following system for (e
, ),
as the KKT necessary optimality conditions for the sets represented by
129

Optimality conditions
differentiable equality and inequality constraints:
f (x ) +

m
X
i=1

ei gi (x ) +

X
j=1

ej hj (x ) = 0n ,

(5.17a)

ei gi (x ) = 0, i = 1, . . . , m, (5.17b)
e 0m .

(5.17c)

Thus, we have established the following theorem.

Theorem 5.33 (KKT optimality conditions for inequality and equality
constraints) Assume that at a given point x S Abadies constraint
qualification (5.15) holds, where S is given by (5.10). If x is a local
minimum of a differentiable function f over S then there exists a pair of
e Rm R such that the system (5.17) is satisfied.
vectors (e
, )
In other words,
x local minimum of f over S
Abadies CQ holds at x

e Rm R : (5.17) holds.
= (e
, )

Example 5.34 (Example 5.32 revisited) Let us write the system of KKT
conditions for the original representation of the set with one inequality
and one equality constraint (see Example 5.14). As has already been
mentioned, Abadies constraint qualification is satisfied, and therefore,
since an optimum exists, the KKT system is necessarily solvable:
 
 
 
1
0
2
+ 1
+ 1
= 02 ,
0
1
0
1 0,

which admits the unique solution 1 = 0, 1 = 1/2.

5.7

Constraint qualifications

In this section we discuss conditions on the functions involved in the

representation (5.10) of a given feasible set S, that all imply Abadies
constraint qualification (5.15).
130

Constraint qualifications

5.7.1

MangasarianFromovitz CQ (MFCQ)

Definition 5.35 (MangasarianFromovitz CQ) We say that at the point

x S, where S is given by (5.10), the MangasarianFromovitz CQ
holds if the gradients hj (x) of the functions hj , j = 1, . . . , , defining
the equality constraints, are linearly independent, and the intersection

G(x) H(x) is nonempty.

We state the following result without a real proof, but we outline
the ideas.
Proposition 5.36 The MFCQ implies Abadies CQ.
Proof. [Sketch] Since the gradients hj (x), j = 1, . . . , , are linearly

independent, it can be shown that cl(G(x) H(x)) TS (x) (in the

absence of equality constraints, it follows directly from Lemma 5.10).
Furthermore, from Lemma 5.10 applied to the inequality represene
tation of S, i.e., to G(x)
defined by (5.14), we know that TS (x)
(G(x) H(x)).

Finally, since G(x) H(x) 6= , it can be shown that cl(G(x)

H(x)) = G(x) H(x).

Example 5.37 Since MFCQ implies Abadies constraint qualification,

Example 5.5 and 5.6 must necessarily violate it. On the other hand, both
Examples 5.4 and 5.7 verify it (since they also satisfy stronger constraint
qualifications, see Example 5.40 and 5.43).

5.7.2

Slater CQ

Definition 5.38 (Slater CQ) We say that the system of constraints describing the feasible set S via (5.10) satisfies the Slater CQ, if the functions gi , i = 1, . . . , m, defining the inequality constraints are convex,
the functions hj , j = 1, . . . , , defining the equality constraints are affine
with linearly independent gradients hj (x), j = 1, . . . , , and, finally,
S such that gi (
that there exists x
x) < 0, for all i {1, . . . , m}.
Proposition 5.39 The Slater CQ implies the MFCQ.
Proof. Suppose the Slater CQ holds at x S. By the convexity of the
inequality constraints we get:
0 > gi (
x) = gi (
x) gi (x) gi (x)T (
x x),
131

Optimality conditions
for all i I(x). Furthermore, since the equality constraints are affine,
we have that
0 = hj (
x) hj (x) = hj (x)T (
x x),

x G(x) H(x).
j = 1, . . . , . Then, x
Example 5.40 Only Example 5.4 verifies the Slater CQ (which in particular explains why it satisfies MFCQ as well, see Example 5.37).

5.7.3

Linear independence CQ (LICQ)

Definition 5.41 (LICQ) We say that at the point x S, where S is

given by (5.10), the linear independence CQ holds if the gradients gi (x)
of the functions gi , i I(x), defining the active inequality constraints,
as well as the gradients hj (x) of the functions hj , j = 1, . . . , , defining
the equality constraints, are linearly independent.
Proposition 5.42 The LICQ implies the MFCQ.

Proof.
[Sketch] Assume that G(x )H(x ) = , i.e., the system GT p <
|I(x )|
0
and H T p = 0 is unsolvable, where G and H are the matrices
having the gradients of the active inequality and equality constraints, respectively, as their columns. Using a separation result similar to Farkas
Lemma (cf. Theorem 3.30) one can show that the system G+H = 0n ,

0|I(x )| has a nonzero solution (T , T )T R|I(x )|+ , which contradicts the linear independence assumption.
In fact, the solution (, ) to the KKT system (5.17), if one exists,
is necessarily unique in this case, and therefore LICQ is a rather strong
assumption in many practical situations.
Example 5.43 Only Example 5.7 in the original description using both
inequality and equality constraints verifies the LICQ (which in particular
explains why it satisfies the MFCQ, see Example 5.37, and why the
Lagrange multipliers are unique in this case, see Example 5.34).

5.7.4

Affine constraints

Assume that both the functions gi , i = 1, . . . , m, defining the inequality constraints and the functions hj , j = 1, . . . , , defining the equality
constraints in the representation (5.10) are affine, that is, the feasible
132

Sufficiency of the KKT conditions under convexity

set S is a polyhedron. Then, the radial cone RS (x) (see Definition 5.1)
is equal to G(x) H(x) (see Example 4.22). Owing to the inclusions
e
RS (x) TS (x) (Proposition 5.3) and TS (x) G(x)
= G(x) H(x)
e
(Lemma 5.10), where G(x) was defined in Section 5.6 (cf. (5.12) and the
discussion thereafter), Abadies CQ (5.15) holds in this case.
Thus, the following claim is established.
Proposition 5.44 If all (inequality and equality) constraints are affine,
then Abadies CQ is satisfied.

5.8

Sufficiency of the KKT conditions under convexity

In general, the KKT necessary conditions do not imply local optimality,

as has been mentioned before (see, e.g., the example right after the proof
of Theorem 4.14). However, if the optimization problem (5.1) is convex,
then the KKT conditions are sufficient for global optimality.
Theorem 5.45 (sufficiency of the KKT conditions for convex problems)
Assume that the problem (5.1) with the feasible set S given by (5.10)
is convex, i.e., the objective function f as well as the functions gi ,
i = 1, . . . , m, are convex, and the functions hj , j = 1, . . . , , are affine.
Assume further that for x S the KKT conditions (5.17) are satisfied.
Then, x is a globally optimal solution of the problem (5.1).
In other words,

the problem (5.1) is convex
= x global minimum in (5.1).
KKT conditions (5.17) hold at x
Proof. Choose an arbitrary x S. Then, by the convexity of the
functions gi , i = 1, . . . , m, it holds that
gi (x )T (x x ) gi (x ) gi (x) = gi (x) 0,

(5.18)

for all i I(x ), and using the affinity of the functions hj , j = 1, . . . , ,

we get that
hj (x )T (x x ) = hj (x ) hj (x) = 0,

(5.19)

for all j = 1, . . . , . Using the convexity of the objective function, equations (5.17a) and (5.17b), non-negativity of the Lagrange multipliers i ,
133

Optimality conditions
i I(x ), and equations (5.18) and (5.19) we obtain the inequality
f (x)f (x ) f (x )T (xx )
=

iI(x )

0.

i gi (x )T (xx )

X
j=1

j hj (x )T (xx )

The point x S was arbitrary, whence x solves (5.1).

Remark 5.46 (alternative proof of Theorem 5.45) An alternative proof
of Theorem 5.45 is available from the sufficient global optimality condition in unconstrained optimization. Suppose that the conditions of
Theorem 5.45 are fulfilled. By Theorem 4.19 this is equivalent to the
Lagrangian function x 7 L(x, , ) = f (x) + ( )T g(x) + ( )T h(x)
having a global minimum over Rn at x . In particular, then,
L(x , , ) L(x, , )
holds for every feasible x. The rest of the proof is a simple matter of
writing out this inequality explicitly and utilizing the remaining parts of
the KKT conditions (5.17):
f (x ) + ( )T g(x ) + ( )T h(x ) f (x) + ( )T g(x) + ( )T h(x)

f (x ) f (x) +

( ) g(x) ( )T g(x ) + ( )T [h(x) h(x )] .

|
{z
} |
{z
}
| {z }
T

0 [(5.17c)+feas.]

=0 [(5.17b)]

=0 [feas.]

(Here, feas. stands for primal feasibility.) We are done.

Theorem 5.45 combined with the necessity of the KKT conditions

under an appropriate CQ leads to the following statement.
Corollary 5.47 Assume that the problem (5.1) is convex and verifies
the Slater CQ (Definition 5.38). Then, for x S to be a globally
optimal solution of (5.1) it is both necessary and sufficient to verify the
KKT system (5.17).
Not surprisingly, without the Slater constraint qualification the KKT
conditions remain only sufficient (i.e., they are unnecessarily strong), as
the following example demonstrates.
134

Applications and examples

Example 5.48 Consider the optimization problem to
minimize x1 ,
subject to x21 + x2 0,
x2 0,
which is convex but has only one feasible point 02 R2 . At this unique
point both the inequality constraints are active, and thus the Slater CQ
is violated, which however does not contradict the global optimality of
02 . It is easy to check that the KKT system
  

1
0 0
+
= 02 ,
0
1 1
02 ,

is unsolvable, and therefore the KKT conditions are not necessary without a CQ even for convex problems.

5.9

Applications and examples

Example 5.49 Consider a symmetric square matrix A Rnn , and

the optimization problem to
minimize xT Ax,

subject to xT x 1.
The only constraint of this problem is convex; furthermore, (0n )T 0n =
0 < 1, and thus Slaters CQ (Definition 5.38) is verified. Therefore, the
KKT conditions are necessary for the local optimality in this problem.
We will find all the possible KKT points, and then choose a globally
optimal point among them.
(xT Ax) = 2Ax (A is symmetric), and (xT x) = 2x. Thus,
the KKT system is as follows: xT x 1 and
2Ax + 2x = 0n ,
0,
(xT x 1) = 0.

From the first two equations we immediately see that either x = 0n ,

or the pair (, x) is, respectively, a nonnegative eigenvalue and a corresponding eigenvector of A (recall that Ax = x holds). In the former
case, from the complementarity condition we deduce that = 0.
Thus, we can characterize the KKT points of the problem into the
following groups:
135

Optimality conditions
1. Let 1 , . . . , k be all the positive eigenvalues of A (if any), and
define Xi := { x Rn | xT x = 1; Ax = i x } to be the set of
corresponding eigenvectors of length 1, i = 1, . . . , k. Then, (x, i )
is a KKT point with the corresponding multiplier for every x Xi ,
i = 1, . . . , k. Moreover, xT Ax = i xT x = i < 0, for every
x Xi , i = 1, . . . , k.
2. Define also X0 := { x Rn | xT x 1; Ax = 0n }. Then, the
pair (x, 0) is a KKT point with the corresponding multiplier for
every x X0 . We note that if the matrix A is nonsingular, then
X0 = {0n }. In any case, xT Ax = 0 for every x X0 .

Therefore, if the matrix A has any positive eigenvalue, then the global
minima of the problem we consider are the eigenvectors of length one,
corresponding to the largest positive eigenvalue; otherwise, every vector
x X0 is globally optimal.
Example 5.50 Similarly to the previous example, consider the following equality-constrained minimization problem associated with a symmetric matrix A Rnn :
minimize xT Ax,

subject to xT x = 1.
The gradient of the only equality constraint equals 2x, and since 0n is
infeasible, LICQ is satisfied at every feasible point (see Definition 5.41),
and the KKT conditions are necessary for local optimality. In this case,
the KKT system is extremely simple: xT x = 1 and
2Ax + 2x = 0n .
Let 1 < 2 < < k denote all distinct eigenvalues of A, and define
as before Xi := { x Rn | xT x = 1; Ax = i x } to be the set of corresponding eigenvectors of length 1, i = 1, . . . , k. Then, (x, i ) is a KKT
point with the corresponding multiplier for every x Xi , i = 1, . . . , k.
Furthermore, since xT Ax = i for every x Xi , i = 1, . . . , k, it
holds that every x Xk , that is, every eigenvector corresponding to the
largest eigenvalue, is globally optimal.
Considering the problem for AT A and using the spectral theorem,
we deduce the well known fact that kAk = max1ik { |i | }.
Example 5.51 Consider the problem of finding the projection of a given
point y onto the polyhedron { x Rn | Ax = b }, where A Rkn ,
b Rk . Thus, we consider the following minimization problem with
136

Notes and further reading

affine constraints (so that the KKT conditions are necessary for the
local optimality, see Section 5.7.4):
1
kx yk2 ,
2
subject to Ax = b.
minimize

The KKT system in this case is written as follows:

Ax = b,
(x y) + AT = 0n ,
for some Rk . Pre-multiplying the last equation with A, and using
the fact that Ax = b we get:
AAT = Ay b.
Substituting an arbitrary solution of this equation into the KKT system,
we calculate x via x := yAT . It can be shown that the vector AT is
the same constant for every Lagrange multiplier , so using this formula
we obtain the globally optimal solution to our minimization problem.
Now assume that the columns of AT are linearly independent, i.e.,
LICQ holds. Then, the matrix AAT is nonsingular, and the multiplier
is therefore unique:
= (AAT )1 (Ay b).
Substituting this into the KKT system, we finally obtain
x = y AT (AAT )1 (Ay b),
the well known formula for calculating the projection.

5.10

Notes and further reading

One cannot overemphasize the importance of the KarushKuhnTucker

optimality conditions for any development in optimization. We essentially follow the ideas presented in [BSS93, Chapters 4 and 5]; see also
[Ber99, Chapter 3]. The original papers by Fritz John [Joh48], and Kuhn
and Tucker [KuT51] might also be interesting. The work of Karush is a
1939 M.Sc. thesis from the University of Chicago.
Various forms of constraint qualifications play an especially important role in sensitivity analyses and studies of parametric optimization
problems (e.g., [Fia83, BoS00]). Original presentations of constraint
137

Optimality conditions
qualifications, some of which we considered in this chapter, may be found
in the works of Arrow, Hurwitz, and Uzawa [AHU61], Abadie [Aba67],
Mangasarian and Fromowitz [MaF67], Guignard [Gui69], Zangwill [Zan69],
and Evans [Eva70].

5.11

Exercises

Exercise 5.1 Consider the following problem:

minimize
subject to

f (x) := 2x21 + 2x1 x2 + x22 10x1 10x2 ,

x21 + x22 5,

3x1 + x2 6.

Check if the point x0 = (1, 2)T is a KKT point for this problem. Is this an
optimal solution? Which CQs are satisfied at the point x0 ?
Exercise 5.2 (optimality conditions, exam 020529) (a) Consider the following
optimization problem:
x2 ,

minimize

sin(x) 1.

subject to

(5.20)

Find every locally and every globally optimal solution. Write down the KKT
conditions. Are they necessary/sufficient for this problem?
(b) Do the locally/globally optimal solutions to the problem (5.20) satisfy
the FJ optimality conditions?
(c) Question the usefulness of the FJ optimality conditions by finding a
point (x, y), which satisfies the FJ conditions for the problem:
minimize

y,

subject to

x2 + y 2 1,

x3 y 4 ,

but, nevertheless, is neither a local nor a global minimum.

Exercise 5.3 Consider the following linear programming problem:

T x,
subject to Ax b.
minimize

State the KKT conditions for this problem. Verify that every KKT point
satisfies T x = bT , where  is a vector of KKT multipliers.

Exercise 5.4 (optimality conditions, exam 020826) (a) Consider the nonlinear
programming problem with equality constraints:
minimize f (x),

subject to hi (x) = 0,

138

i = 1, . . . , m,

(5.21)

Exercises
where f , h1 , . . . , hm are continuously differentiable functions.
Show that the problem (5.21) is equivalent to the following problem with
one inequality constraint:
minimize f (x),
m
X

subject to

hi (x)

2

i=1

(5.22)

0.

Show (by a formal argument or an illustrative example) that the KKT

conditions for the latter problem are not necessary for local optimality.
Can Slaters CQ or LICQ be satisfied for the problem (5.22)?
(b) Consider the unconstrained minimization problem to
minimize maximum{f1 (x), f2 (x)},
where f1 : R R, f2 : Rn R are in C 1 . Show that if x is a local minimum
for this problem, then there exist 1 , 2 R such that
n

1 0, 2 0,

1 f1 (x ) + 2 f2 (x ) = 0n ,


1 + 2 = 1,

and i = 0 if fi (x ) < max f1 (x ), f2 (x ) , i = 1, 2.

Exercise 5.5 Consider the following optimization problem:

1 T
x x,
2
subject to Ax = b.
minimize

Assume that the matrix A has full row rank. Find the globally optimal solution
to this problem.
Exercise 5.6 Consider the following optimization problem:
n
X

minimize

cj xj ,

j=1
n
X

subject to

j=1

(5.23)

x2j 1,

xj 0,

j = 1, . . . , n.

Assume that min {c1 , . . . , cn } < 0, and let us introduce KKT multipliers 0
and j 0, j = 1, . . . , n for the inequality constraints.
(a) Show that the equalities
xj = min{0, cj }/(2 ),
=

1
2

n
X
j=1

[min {0, cj }]2

j = max{0, cj },

j = 1, . . . , n,

!1/2

j = 1, . . . , n,

define a KKT point for (5.23).

(b) Show that there is only one optimal solution to (5.23).

139

Optimality conditions
Exercise 5.7 (optimality conditions, exam 040308) Consider the following optimization problem:

RR

minimize f (x, y) :=
(x,y)

1
1
(x 2)2 + (y 1)2 ,
2
2

x y 0,

subject to

(5.24)

y 0,

y(x y) = 0.

(a) Find all points of global and local minima (you may do this graphically),
as well as all KKT points. Is this a convex problem? Are the KKT optimality
conditions necessary and/or sufficient for local optimality in this problem?
(b) Demonstrate that LICQ is violated at every feasible point of the problem (5.24). Show that instead of solving the problem (5.24) we can solve two
convex optimization problems that furthermore verify some constraint qualification, and then choose the best point out of the two.
(c) Generalize the procedure from the previous part to the more general
optimization problem to
minimize

g(x),

aTi x bi ,

subject to

i = 1, . . . , n,

xi 0,

i = 1, . . . , n,

xi (aT
i x bi ) = 0,

where x = (x1 , . . . , xn )T Rn , ai
is a convex differentiable function.

i = 1, . . . , n,

Rn , bi R, i = 1, . . . , n, and g : Rn R

Exercise 5.8 Determine the values of the parameter c for which the point
(x, y) = (4, 3) is an optimal solution to the following problem:

RR

minimize cx + y,
(x,y)

subject to x2 + y 2 25,
x y 1.

Exercise 5.9 Consider the following optimization problem:

minimize

f (x) :=

n
X
x2j
j=1

subject to

n
X

cj

xj = D,

j=1

xj 0,

j = 1, . . . , n,

where cj > 0, j = 1, . . . , n, and D > 0. Find the unique globally optimal

solution to this problem.

140

Lagrangian duality

VI

This chapter collects some basic results on Lagrangian duality, in particular as it applies to convex programs with a zero duality gap.

6.1

The relaxation theorem

Given the problem to find

f := infimum f (x),
x

subject to x S,

(6.1a)
(6.1b)

where f : Rn R is a given function and S Rn , we define a relaxation

to (6.1) to be a problem of the following form: find
fR := infimum fR (x),
x

subject to x SR ,

(6.2a)
(6.2b)

where fR : Rn R is a function with the property that fR f on S,

and where SR S. For this pair of problems, we have the following
basic result.
Theorem 6.1 (Relaxation Theorem) (a) [relaxation] fR f .
(b) [infeasibility] If (6.2) is infeasible, then so is (6.1).
(c) [optimal relaxation] If the problem (6.2) has an optimal solution,
xR , for which it holds that
xR S

and

fR (xR ) = f (xR ),

then xR is an optimal solution to (6.1) as well.

(6.3)

Lagrangian duality
Proof. The result in (a) is obvious, as every solution feasible in (6.1)
is both feasible in (6.2) and has a lower objective value in the latter
problem. The result in (b) follows for similar reasons. For the result in
(c), we note that
f (xR ) = fR (xR ) fR (x) f (x),

x S,

from which the result follows.

This basic result will be utilized both in this chapter and later on to
motivate why Lagrangian relaxation, objective function linearization and
penalization constitute relaxations, and to derive optimality conditions
and algorithms based on them.

6.2

Lagrangian duality

In this section we formulate the Lagrangian dual problem and establish

its convexity. The Weak Duality Theorem is also established, and we
introduce the terms Lagrangian relaxation, Lagrange multiplier, and
duality gap.

6.2.1

Lagrangian relaxation and the dual problem

Consider the optimization problem to find

f := infimum f (x),

(6.4)

x X,

subject to

gi (x) 0,

i = 1, . . . , m,

where f : Rn R and gi : Rn R (i = 1, 2, . . . , m) are given functions,

and X Rn .
For this problem, we assume that
< f < ,

(6.5)

that is, that f is bounded from below on the feasible set and the problem
has at least one feasible solution.
Definition 6.2 (Lagrange function, relaxation, multiplier) (a) For an arbitrary vector Rm , the Lagrange function is
L(x, ) := f (x) +

m
X
i=1

142

i gi (x) = f (x) + T g(x).

(6.6)

Lagrangian duality
(b) Consider the problem to
minimize L(x, ),

(6.7)

subject to x X.
Whenever is non-negative, the problem (6.7) is referred to as a Lagrangian relaxation.
(c) We call the vector Rm a Lagrange multiplier vector if it is
non-negative and if f = inf x X L(x, ) holds.
Note that the Lagrangian relaxation (6.7) is a relaxation, in terms of
Section 6.1.
Theorem 6.3 (Lagrange multipliers and global optima) Let be a Lagrange multiplier vector. Then, x is an optimal solution to (6.4) if and
only if x is feasible in (6.4) and
x arg min L(x, ),
x X

and i gi (x ) = 0, i = 1, . . . , m.

(6.8)

Proof. If x is an optimal solution to (6.4), then it is in particular

feasible, and
f = f (x ) L(x , ) infimum L(x, ),
x X

where the first inequality stems from the feasibility of x and the definition of a Lagrange multiplier vector. The second part of that definition
implies that f = inf x X L(x, ), so that equality holds throughout in
the above line of inequalities. Hence, (6.8) follows.
Conversely, if x is feasible and (6.8) holds, then by the use of the
definition of a Lagrange multiplier vector,
f (x ) = L(x , ) = minimum L(x, ) = f ,
x X

so x is a global optimum.
Let
q() := infimum L(x, )
x X

(6.9)

be the Lagrangian dual function, defined by the infimum value of the

Lagrange function over X; the Lagrangian dual problem is to
maximize q(),

(6.10)

subject to 0m .
143

Lagrangian duality
For some , q() = is possible; if it is true for all 0m , then
q := supremum q()
0m

equals . (We can then say that the dual problem is infeasible.)
The effective domain of q is
Dq := { Rm | q() > } .
Theorem 6.4 (convex dual problem) The effective domain Dq of q is
convex, and q is concave on Dq .
Rm , and [0, 1]. We have that
Proof. Let x Rn , ,
= L(x, ) + (1 )L(x,
).
L(x, + (1 ))
Take the infimum over x X on both sides; then,
= inf {L(x, ) + (1 )L(x, )}

inf L(x, + (1 ))

x X

x X

)
inf L(x, ) + inf (1 )L(x,
x X

x X

= inf L(x, ) + (1 ) inf L(x, ),

x X

x X

since [0, 1], and the sum of infimum values may be smaller than the
infimum of the sum, since in the former case we have the possibility to
choose different optimal solutions in the two problems. Hence,
q() + (1 )q()

q( + (1 ))
lie in Dq , then
holds. This inequality has two implications: if and
so Dq is convex; also, q is concave on Dq .
so does + (1 ),
That the Lagrangian dual problem always is convex (we indeed maximize a concave function) is good news, because it means that it can
be solved efficiently. What remains is to show how a Lagrangian dual
optimal solution can be used to generate a primal optimal solution.
Next, we establish that every feasible point in the Lagrangian dual
problem always underestimates the objective function value of every feasible point in the primal problem; hence, also their optimal values have
this relationship.
Theorem 6.5 (Weak Duality Theorem) (a) Let x and be feasible in
the problems (6.4) and (6.10), respectively. Then,
q() f (x).
144

Lagrangian duality
In particular,

q f .

(b) If q() = f (x), then the pair (x, ) is optimal in its respective
problem.
Proof. For all 0m and x X with g(x) 0m ,
q() = infimum L(z, ) f (x) + T g(x) f (x),
z X

so

q = supremum q()
0m

infimum

x X:g(x )0m

f (x) = f .

The result follows.

Weak duality is also a consequence of the Relaxation Theorem: For
any 0m , let
S := X { x Rn | g(x) 0m },

SR := X,
fR := L(, ).

(6.11a)
(6.11b)
(6.11c)

Then, the weak duality statement is the result in Theorem 6.1(a).

If our initial feasibility assumption (6.5) is false, then what does weak
duality imply? Suppose that f = . Then, weak duality implies
that q() = for all 0m , that is, the dual problem is infeasible.
Suppose then that X 6= but that X { x Rn | g(x) 0m } is empty.
Then, f = , by convention. The dual function satisfies q() < for
all 0m , but it is possible that q = , < q < , or q =
(see [Ber99, Figure 5.1.8]). For linear programs, < q < implies
< f < ; see below.
If q = f , then we say that the duality gap (as given by :=
f q ) is zero, or that there is no duality gap. If there exists a Lagrange
multiplier vector, then by the weak duality theorem, this implies that
there is no duality gap. The converse is not true in general: there may be
cases where no Lagrange multipliers exist even when there is no duality
gap;1 in that case though, the Lagrangian dual problem cannot have an
optimal solution, as implied by the following result.
Proposition 6.6 (duality gap and the existence of Lagrange multipliers)
(a) If there is no duality gap, then the set of Lagrange multiplier vectors
equals the set of optimal dual solutions (which however may be empty).
(b) If there is a duality gap, then there are no Lagrange multipliers.
1 Take

the example of minimizing f (x) := x subject to g(x) := x2 0; x X := R.

145

Lagrangian duality
Proof. By definition, a vector 0m is a Lagrange multiplier vector
if and only if f = q( ) q , the equality following from the definition
of q( ) and the inequality from the definition of q as the supremum of
q() over Rm
+ . By weak duality, this relation holds if and only if there
is no duality gap and is an optimal dual solution.
Above we have developed properties of the min-max problem for
finding
q := supremum infimum L(x, ).
0m

x X

What then is the max-min problem to find

p := infimum supremum L(x, )?
x X

0m

Fix x X. Then,
p(x) := supremum L(x, ) =
0m

f (x),
+,

if g(x) 0m ,
otherwise.

(We call the function p : Rm

+ R {+} the primal function, in contrast to the dual function q.) Hence, the max-min problem is essentially
equivalent to minimizing f over the set X { x Rn | g(x) 0m },
that is, the original problem (6.4), and p = f holds. Moreover, we
have seen above that in general q f holds, that is, the min-max
problem has an optimal value which is always at least as large as that of
the max-min problem. This is a general statement, and equality holds
precisely when there exists a saddle point of the function L. The above
development extends that of Von Neumanns matrix game; cf. (4.31).
Before moving on, we remark on the statement of the problem (6.4).
There are several ways in which the original set of constraints of the
problem can be placed either within the definition of the ground set X
(which is kept intact), or within the explicit constraints defined by the
functions gi (which are Lagrangian relaxed). How to distinguish between
the two, that is, how to decide whether a constraint should be kept or be
Lagrangian relaxed, depends on several factors. For example, keeping
more constraints within X may result in a smaller duality gap, and
fewer multipliers also result in a simpler Lagrangian dual problem. On
the other hand, the Lagrangian subproblem defining the dual function
simultaneously becomes more complex and difficult to solve. There are
no immediate rules to follow, but experimentation and experience.
146

Lagrangian duality

6.2.2

Global optimality conditions

The following result characterizes every optimal primal and dual solution. It is however applicable only in the presence of Lagrange multipliers; in other words, the below system (6.12) is consistent if and only if
there exists a Lagrange multiplier vector and there is no duality gap.
Theorem 6.7 (global optimality conditions in the absence of a duality
gap) The vector (x , ) is a pair of primal optimal solution and Lagrange multiplier vector if and only if
0m ,
x arg min L(x, ),

(Dual feasibility)
(Lagrangian optimality)

(6.12a)
(6.12b)

x X,

(Primal feasibility)

(6.12c)

i gi (x )

= 0,

x X

g(x ) 0m ,

i = 1, . . . , m.

(Complementary slackness) (6.12d)

Proof. Suppose that the pair (x ) satisfies (6.12). Then, from (6.12a)
we have that the Lagrangian problem to minimize L(x, ) over x X
is a (Lagrangian) relaxation of (6.4). Moreover, according to (6.12b)
x solves this problem, (6.12c) shows that x is feasible in (6.4), and
(6.12d) implies that L(x , ) = f (x ). The Relaxation Theorem 6.1
then yields that x is optimal in (6.4), which in turn implies that is
a Lagrange multiplier vector.
Conversely, if (x , ) is a pair of optimal primal solution and Lagrange multiplier vector, then they are primal and dual feasible, respectively. The relations (6.12b) and (6.12d) follow from Theorem 6.3.
Theorem 6.8 (global optimality and saddle points) The vector (x , )
is a pair of optimal primal solution and Lagrange multiplier vector if
and only if x X, 0m , and (x , ) is a saddle point of the
Lagrangian function on X Rm
+ , that is,
L(x , ) L(x , ) L(x, ),

(x, ) X Rm
+

(6.13)

holds.
Proof. We establish that (6.12) and (6.13) are equivalent; Theorem 6.7
then gives the result. The first inequality in (6.13) is equivalent to
g(x )T ( ) 0,

Rm
+,

(6.14)
147

Lagrangian duality
for the given pair (x , ) X Rm
+ . This variational inequality is
equivalent to stating that2
0m g(x ) 0m ,

(6.15)

where denotes orthogonality: that is, for any vectors a, b Rn , a b

means that aT b = 0. Because of the sign restrictions posed on and
g, that is, the vectors a and b, the relation a b actually means that
not only does it hold that aT b = 0 but in fact ai bi = 0 must hold for all
i = 1, . . . , n. This complementarity system is, for the given Rm
+,
the same as (6.12a), (6.12c) and (6.12d). The second inequality in (6.13)
is equivalent to (6.12b).
The above two theorems also imply that the set of primaldual optimal solutions (x , ) is a Cartesian product set, X U . For example,
given any optimal dual solution U , every optimal primal solution
x X satisfies (6.12). Hence, we can write, for an arbitrary dual
vector U ,
X = { x Rn | x satisfies (6.12) for = }





m
T

= x arg min L(x, )  g(x ) 0 ; ( ) g(x ) = 0 .

x X

We note that structurally similar results to the above two theorems

which are valid for the general problem (6.4) with any size of the duality
gap can be found in [LaP05].3
We finally note a practical connection between the KKT system (5.9)
and the above system (6.12). The practical use of the KKT system is
normally to investigate whether a primal vector xobtained perhaps
from a solver for our problemis a candidate for a locally optimal solution; in other words, we have access to x and generate a vector of
2 We establish the equivalence between (6.14) and (6.15) as follows. (The proof
extends that for line search problems in unconstrained optimization in a footnote in
Section 11.3.1.)
First, suppose that (6.15) is fulfilled. Then, g (x )T (  ) = g (x )T  0,
for all  0m , that is, (6.14) is fulfilled. Conversely, suppose that (6.14) is fulfilled.
Setting  = 0m yields that g(x )T  0. On the other hand, the choice  = 2
yields that g (x )T  0. Hence, g (x )T  = 0 holds. Last, let  =  + ei ,
where ei is the ith unit vector in Rm . Then, g (x )T (  ) = gi (x ) 0. Since
this is true for all i {1, 2, . . . , m} we have obtained that g (x ) 0m , that is,
g(x ) 0m . We are done.
3 The system (6.12) is there appended with two relaxation parameters which measure, respectively, the near-optimality of x in the Lagrangian subproblem [that is,
the -optimality in (6.12b)], and the violation of the complementarity conditions
(6.12d). The saddle point condition (6.13) is similarly perturbed, and at an optimal
solution, the sum of these two parameter values equals the duality gap.

148

Lagrangian duality
Lagrange multipliers in the investigation of the KKT system (5.9). In
contrast, the system (6.12) is normally investigated in the reverse order;
we formulate and solve the Lagrangian dual problem, thereby obtaining
an optimal dual vector . Starting from that vector, we investigate the
global optimality conditions stated in (6.12) to obtain, if possible, an
optimal primal vector x. In the section to follow, we show when this is
possible, and provide strong connections between the systems (5.9) and
(6.12) in the convex and differentiable case.

6.2.3

Strong duality for convex programs

So far the results have been rather non-technical to achieve: the convexity of the Lagrangian dual problem comes with very few assumptions
on the original, primal problem, and the characterization of the primal
dual set of optimal solutions is simple and also quite easily established.
In order to establish strong duality, that is, to establish sufficient conditions under which there is no duality gap, however, takes much more.
In particular, as is the case with the KKT conditions we need regularity
conditions (that is, constraint qualifications), and we also need to utilize
separation theorems such as Theorem 4.29. Most importantly, however,
is that strong duality is deeply associated with the convexity of the original problem, and it is in particular under convexity that the primal and
dual optimal solutions are linked through the global optimality conditions provided in the previous section. We begin by concentrating on the
inequality constrained case, proving this result in detail. We will also
specialize the result to quadratic and linear optimization problems.
Consider the inequality constrained convex program (6.4), where f :
Rn R and gi (i = 1, . . . , m) are convex functions and X Rn is
a convex set. For this problem, we introduce the following regularity
condition, due to Slater (cf. Definition 5.38):
x X with g(x) < 0m .

(6.16)

Theorem 6.9 (Strong Duality, inequality constrained convex programs)

Suppose that the feasibility condition (6.5) and Slaters constraint qualification (6.16) hold for the convex problem (6.4).
(a) There is no duality gap and there exists at least one Lagrange multiplier vector . Moreover, the set of Lagrange multipliers is bounded
and convex.
(b) If the infimum in (6.4) is attained at some x , then the pair

(x , ) satisfies the global optimality conditions (6.12).

(c) If further f and g are differentiable at x , then the condition
149

Lagrangian duality
(6.12b) can equivalently be written as the variational inequality
x L(x , )T (x x ) 0,

x X.

(6.17)

If, in addition, X is open (such as is the case when X = Rn ), then this

reduces to the condition that
x L(x , ) = f (x ) +

m
X
i=1

i gi (x ) = 0n ,

(6.18)

and the global optimality conditions (6.12) reduce to the KarushKuhn

Tucker conditions stated in Theorem 5.25.
Proof. (a) We begin by establishing the existence of a Lagrange multiplier vector (and the presence of a zero duality gap).
First, we consider the following subset of Rm+1 :
A := {(z1 , . . . , zm , w)T | x X with gi (x) zi , i = 1, . . . , m; f (x) w}.
It is elementary to show that A is convex.
Next, we observe that ((0m )T , f )T is not an interior point of A;
otherwise, for some > 0 the point ((0m )T , f )T A holds, which
would contradict the definition of f . Therefore, by the (possibly nonproper) separation result in Theorem 4.29, we can find a hyperplane
passing through ((0m )T , f )T such that A lies in one of the two corresponding half-spaces. In particular, there then exists a vector (T , )T 6=
((0m )T , 0)T such that
f w + T z,

(z T , w)T A.

(6.19)

This implies that

0;

0m ,

(6.20)

since for each (z T , w)T A, (z T , w + )T A and (z1 , . . . , zi1 , zi +

, zi+1 , . . . , zm , w)T A for all > 0 and i = 1, . . . , m.
We claim that > 0 in fact holds. Indeed, if it was not the case, then
= 0 and (6.19) then implies that T z 0 for every pair (z T , w)T
is such P
A. But since (g(
x)T , f (
x))T A [where x
that it satisfies the
m
Slater condition (6.16)], we would obtain that 0 i=1 i gi (
x) which
m
satisfies the
in view of 0 [cf. (6.20)] and the assumption that x
Slater condition (6.16) implies that = 0m . This means, however, that
(T , )T = ((0m )T , 0)T a contradiction. We may therefore claim that
> 0. We further, with no loss of generality, assume that = 1.
Thus, since (g(x)T , f (x))T A for every x X, (6.19) yields that
f f (x) + T g(x),
150

x X.

Lagrangian duality
A
(g(
x)T , f (
x))T

((0m )T , f )T
z

(T , 1)T

Figure 6.1: Illustration of the set F := { (g(x)T , f (x))T | x X } and

the set A used in the proof of Theorem 6.9. The idea of the proof is
to show that A is convex and that ((0m )T , f )T is not an interior point
of A. A hyperplane passing through ((0m )T , f )T and supporting A is
used to construct a Lagrange multiplier.

Taking the infimum over x X and using that 0m we obtain

f infimum {f (x) + T g(x)} = q() supremum q() = q .
x X

0m

From the Weak Duality Theorem 6.5 follows that is a Lagrange multiplier vector, and that there is no duality gap.
X satisfying (6.16) and a Lagrange multiplier
Take any vector x
vector . By the definition of a Lagrange multiplier vector, f
L(
x, ) holds, which implies that
m
X
i=1

[f (
x) f ]
.
mini=1,...,m {gi (
x)}

Since 0m , boundedness follows. As by Proposition 6.6(a) the set of

Lagrange multipliers is the set of optimal solutions to the dual problem
(6.10), convexity follows from the identification of the dual solution set
with the set of vectors Rm
+ for which
q() q
holds. This is the upper level set for q at the level q ; this set is convex,
by the concavity of q (cf. Theorem 6.4 and Proposition 3.44).
151

Lagrangian duality
(b) The result follows from Theorem 6.7.
(c) The first part follows from Theorem 4.24, as the Lagrangian function L(, ) is convex. The second part follows by identification.
Consider next the extension of the inequality constrained convex program (6.4) in which we seek to find
f := infimum f (x),

(6.21)

x X,

subject to

gi (x) 0,

T
j x

dj = 0,

i = 1, . . . , m,
j = 1, . . . , ,

under the same conditions as stated following (6.4), and where j Rn ,

j = 1, . . . , . For this problem, we replace the Slater condition (6.16)
with the following (cf. [BSS93, Theorem 6.2.4]):
x X with g(x) < 0m and 0m int { Ex d | x X },

(6.22)

where E Rn has rows T

j , and d = (dj )j{1,...,} R .
Note that in the statement (6.22), the int can be stricken whenever
X is polyhedral, so that the latter part simply states that Ex = d.
For this problem, the Lagrangian dual problem is to find

q := supremum q(, ),

(6.23)

(,)

subject to 0m ,
where
q(, ) := infimum L(x, , ) := f (x) + T g(x) + T (Ex d),
x

subject to x X.
Theorem 6.10 (Strong Duality, general convex programs) Suppose that
in addition to the feasibility condition (6.5), Slaters constraint qualification (6.22) holds for the problem (6.21).
(a) The duality gap is zero and there exists at least one Lagrange
multiplier vector pair ( , ).
(b) If the infimum in (6.21) is attained at some x , then the triple

(x , , ) satisfies the global optimality conditions

0m , (Dual feasibility)
x arg min L(x, , ), (Lagrangian optimality)

x X

x X, g(x ) 0m , Ex = d, (Primal feasibility)

i gi (x )

152

(6.24a)
(6.24b)
(6.24c)

= 0, i = 1, . . . , m. (Complementary slackness) (6.24d)

Lagrangian duality
(c) If further f and g are differentiable at x , then the condition
(6.24b) can equivalently be written as
x L(x , , )T (x x ) 0,

x X.

(6.25)

If, in addition, X is open (such as is the case when X = Rn ), then this

reduces to the condition that
x L(x , , ) = f (x ) +

m
X
i=1

i gi (x ) +

j j = 0n , (6.26)

j=1

and the global optimality conditions (6.24) reduce to the KarushKuhn

Tucker conditions stated in Theorem 5.33.
Proof. The proof is similar to that of Theorem 6.9.
We finally consider a special case where automatically a regularity
condition holds.
Consider the affinely constrained convex program to find
f := infimum f (x),

(6.27)

subject to

x X,

aT
i x bi
T
j x dj

0,

= 0,

i = 1, . . . , m,
j = 1, . . . , ,

where f : Rn R is convex and X Rn is polyhedral.

Theorem 6.11 (Strong Duality, affine constraints) If the feasibility condition (6.5) holds for the problem (6.27), then there is no duality gap
and there exists at least one Lagrange multiplier vector.
Proof. Again, the proof is similar to that of Theorem 6.9, except that
no additional regularity conditions are needed.4
The existence of a multiplier vector [which by Proposition 6.6 and
the absence of a duality gap implies the existence of an optimal solution
to the dual problem (6.10)] does not imply the existence of an optimal
solution to the primal problem (6.27) without any additional assumptions. However, when f is either weakly coercive, quadratic or linear,
the existence results are stronger; see the primal existence results in
Theorems 4.7, 4.8, and 6.12 below, for example.
4 For

a detailed proof, see [Ber99, Proposition 5.2.1]. (The special case where f is
moreover differentiable is covered in [Ber99, Proposition 3.4.2].)

153

Lagrangian duality
For convex programs where a Slater CQ holds, the Lagrange multipliers defined in this section, and those that appear in the Karush
KuhnTucker conditions, clearly are identical. Next, we specialize the
above to linear and quadratic programs.

6.2.4

Strong duality for linear and quadratic programs

The following result will be established and analyzed in detail in Chapter 10 on linear programming duality (cf. Theorem 10.6), but can in
fact also be established similarly to above. (See [BSS93, Theorem 2.7.3]
or [Ber99, Proposition 5.2.2], for example.) Its proof will however be
relegated to that of Theorem 10.6.
Theorem 6.12 (Strong Duality, linear programs) Assume, in addition to
the conditions of Theorem 6.11, that f is linear, so that (6.27) is a linear
program. Then, the primal and dual problems have optimal solutions
and there is no duality gap.
The above result states a strong duality result for a general linear
program. We next develop an explicit Lagrangian dual problem for a
linear program.
Let A Rmn , c Rn , and b Rm ; consider the linear program
minimize cT x,

(6.28)

subject to Ax = b,
x 0n .
If we let X := Rn+ , then the Lagrangian dual problem is to
maximize
bT ,
m

(6.29)

subject to AT c.
The reason why we can write it in this form is that
n
o
T
T
q() := infimum
c
x
+

(b

Ax)
= bT + infimum
(c AT )T x,
n
n
x 0

x 0

so that

q() =

bT , if AT c,
, otherwise.

(The infimum is attained at zero if and only if these inequalities are

satisfied; otherwise, the inner problem is unbounded below.)
154

Lagrangian duality
Further, why is it that here is not restricted in sign? Suppose we
were to split the system Ax = b into an inequality system of the form
Ax b,

Ax b.
Let ((+ )T , ( )T )T be the corresponding vector of multipliers, and
take the Lagrangian dual for this formulation. Then, we would have a
Lagrange function of the form
(x, + , ) 7 L(x, + , ) := cT x + (+ )T (b Ax),
and since + can take on any value in Rm we can simply replace it
with the unrestricted vector Rm . This motivates why the multiplier
for an equality constraint never is sign restricted; the same was the case,
as we saw in Section 5.6, for the multipliers in the KKT conditions.
As applied to this problem, Theorem 6.12 states that if both the
primal or dual problems have feasible solutions, then they both have
optimal solutions, satisfying strong duality (cT x = bT ). On the
other hand, if any of the two problems has an unbounded solution, then
the other problem is infeasible.
Consider next the quadratic programming problem to


1 T
T
minimize
x Qx + c x ,
(6.30)
x
2
subject to Ax b,
where Q Rnn , c Rn , A Rmn , and b Rm . We develop an
explicit dual problem under the assumption that Q is positive definite.
By Lagrangian relaxing the inequality constraints, we obtain that the
inner problem in x is solved by letting
x = Q1 (c + AT ).

(6.31)

Substituting this expression into the Lagrangian function yields the Lagrangian dual problem to


1
1
maximize T AQ1 AT (b+AQ1 c)T cT Q1 c , (6.32)

2
2
subject to 0m ,

Strong duality follows for this convex primaldual pair of quadratic

programs, in much the same way as for linear programs.
155

Lagrangian duality
Theorem 6.13 (Strong Duality, quadratic programs) For the primal
dual pair of convex quadratic programs (6.30), (6.32), the following
holds:
(a) If both problems have feasible solutions, then both problems also
have optimal solutions, and the primal problem (6.30) also has a unique
optimal solution, given by (6.31) for any optimal Lagrange multiplier
vector, and in the two problems the optimal values are equal.
(b) If either of the two problems has an unbounded solution, then
the other one is infeasible.
(c) Suppose that Q is positive semidefinite, and that the feasibility
condition (6.5) holds. Then, both the problem (6.30) and its Lagrangian
dual have nonempty, closed and convex sets of optimal solutions, and
their optimal values are equal.
In the result (a) it is important to note that the Lagrangian dual
problem (6.32) is not necessarily strictly convex; the matrix AQ1 AT
need not be positive definite, especially so when A does not have full
rank. The result (c) extends the strong duality result from linear programming, since Q in (c) can be the zero matrix. In the case of (c) we of
course cannot write the Lagrangian dual problem in the form of (6.32)
because Q is not necessarily invertible.

6.2.5

Two illustrative examples

Example 6.14 (an explicit, differentiable dual problem) Consider the problem to
minimize f (x) := x21 + x22 ,
x

subject to x1 + x2 4,

xj 0,

j = 1, 2.

We consider the first constraint to be the complicated one, and hence

define g(x) := x1 x2 + 4 and let X := { (x1 , x2 )T | xj 0, j = 1, 2 }.
Then, the Lagrangian dual function is
q() = minimum L(x, ) := f (x) (x1 + x2 4)
x X

= 4 + minimum {x21 + x22 x1 x2 }

x X

= 4 + minimum {x21 x1 } + minimum {x22 x2 },

x1 0

x2 0

0.

For a fixed 0, the minimum is attained at x1 () = 2 , x2 () = 2 .

Substituting this expression yields q() = f (x())(x1 ()+x2 ()
2
4) = 4 2 .
156

Lagrangian duality
Note that q is strictly concave, and it is differentiable everywhere (due
to the fact that f, g are differentiable and x() is unique), by Danskins
Theorem 6.17(d).
We have that q () = 4 = 0 = 4. As = 4 0, it is the
optimum in the dual problem: = 4; x = (x1 ( ), x2 ( ))T = (2, 2)T .
Also, f (x ) = q( ) = 8.
This is an example where the dual function is differentiable, and
therefore we can utilize Proposition 6.29(c). In this case, the optimum
x is also unique, so it is automatically given as x = x().
Example 6.15 (an implicit, non-differentiable dual problem) Consider
the linear programming problem to
minimize f (x) := x1 x2 ,
x

subject to 2x1 + 4x2 3,

0 x1 2,
0 x2 1.

The optimal solution is x = (3/2, 0)T , f (x ) = 3/2.

Consider Lagrangian relaxing the first constraint, obtaining
L(x, ) = x1 x2 + (2x1 + 4x2 3);

q() = 3 + minimum {(1 + 2)x1 } + minimum {(1 + 4)x2 }

0x1 2
0x2 1

3 + 5, 0 1/4,
2 + , 1/4 1/2,
=

3, 1/2 .

Check that = 1/2, and hence that q( ) = 3/2. For linear

programs, we have strong duality, but how do we obtain the optimal
primal solution from ? It is clear that q is non-differentiable at .
Let us utilize the characterization given in the system (6.12).
First, at , it is clear that X( ) is the set { (2, 0)T | 0 1 }.
Among the subproblem solutions, we next have to find one that is primal
feasible as well as complementary.
Primal feasibility means that 2 2 + 4 0 3 3/4.
Further, complementarity means that (2x1 +4x2 3) = 0 =
3/4, since 6= 0. We conclude that the only primal vector that satisfies
the system (6.12) together with the dual optimal solution = 1/2 is
x = (3/2, 0)T .
In the first example, the Lagrangian dual function is differentiable
since x() is unique. The second one shows that otherwise, there may
157

Lagrangian duality
be kinks in the function q where there are alternative solutions x(); as a
result, to obtain a primal optimal solution becomes more complex. The
DantzigWolfe algorithm, for example, represents a means by which to
automatize the process that we have just shown; the algorithm generates
extreme points of X() algorithmically, and constructs the best feasible
convex combination thereof, obtaining a primaldual optimal solution in
a finite number of iterations for linear programs.
The above examples motivate a deeper study of the differentiability properties of convex (or, concave) functions in general, and the Lagrangian dual objective function in particular.

6.3

Differentiability properties of the dual

function

We have established that the Lagrangian dual problem (6.10) is a convex

one, and further that under some circumstances the primal and dual
optimal values are the same. We now turn to study the Lagrangian dual
problem in detail, and in particular how it can be solved efficiently. First,
we will establish when the dual function q is differentiable. We will see
that differentiability holds only in some special cases, in which we can
recognize the workings of the Lagrange multiplier method; this classic
method was illustrated in Example 6.14. Most often, the function q will
however be non-differentiable, and then this method will fail. This means
that we must devise a more general numerical method which is not based
on gradients but rather subgradients. This type of algorithm is the topic
of the next section; we begin by studying the topic of subgradients of
convex functions in general.

6.3.1

Subdifferentiability of convex functions

Throughout this section we suppose that f : Rn R is a convex function, and study its subdifferentiability properties. We will later on apply
our findings to the Lagrangian dual function q, or, rather, its negative
q. We first remark that a finite convex function is automatically continuous (cf. Theorem 4.27).
Definition 6.16 (subgradient) Let f : Rn R be a convex function.
We say that a vector g Rn is a subgradient of f at x Rn if
f (y) f (x) + g T (y x),

y Rn .

(6.33)

The set of such vectors g defines the subdifferential of f at x, and is

denoted f (x).
158

Differentiability properties of the dual function

For concave functions, the reverse inequality of course holds; for simplicity we will refer also to such vectors g as subgradients.
Notice the connection to the characterization of a convex function in
C 1 in Theorem 3.40(a). The difference between them is that g is not
unique at a non-differentiable point. (Just as the gradient has a role in
supporting hyperplanes to the graph of a convex function in C 1 , the role
of a subgradient is the same; at a non-differentiable point there are more
then one supporting hyperplane to the graph of f .)
We illustrate this in Figure 6.2.
f

Figure 6.2: Three possible slopes of the convex function f at x.

Notice that a minimum x of f over Rn is characterized by the inclusion 0n f (x ); recognize, again, the similarity to the C 1 case.
We list some additional basic results for convex functions next. Proofs
will not be given here; we refer instead to the convex analysis text by
Rockafellar [Roc70].
Proposition 6.17 (properties of a convex function) Let f : Rn R be
a convex function.
(a) [boundedness of f (x)] For every x Rn , f (x) is a nonempty,
convex, and compact set. If X is bounded then x X f (x) is bounded.
(b) [closedness of f ] The subdifferential mapping x 7
7 f (x) is
closed; in other words, if {xk } is a sequence of vectors in Rn converging
to x, and g k f (xk ) holds for every k, then the sequence {g k } of
subgradients is bounded and every limit point thereof belongs to f (x).
(c) [directional derivative and differentiability] For every x Rn , the
directional derivative of f at x in the direction of p Rn satisfies
f (x; p) = maximum g T p.
g f (x )

(6.34)

In particular, f is differentiable at x with gradient f (x) if and only

if it has f (x) as its unique subgradient at x; in that case, f (x; p) =
f (x)T p.
159

Lagrangian duality
(d) [Danskins Theoremdirectional derivatives of a convex max function] Let Z be a compact subset of Rm , and let : Rn Z R be
continuous and such that (, z) : Rn R is convex for each z Z. Let
the function f : Rn R be given by
f (x) := maximum (x, z),
z Z

x Rn .

(6.35)

The function f then is convex on Rn and has a directional derivative at

x in the direction of p equal to
f (x; p) = maximum (x, z; p),

(6.36)

z Z(x )

where (x, z; p) is the directional derivative of (, z) at x in the direction of p, and Z(x) := { z Rm | (x, z) = f (x) }.
In particular, if Z(x) contains a single point z and (, z ) is differen ), where
tiable at x, then f is differentiable at x, and f (x) = x (x, z
(x ,
z)
) is the vector with components xi , i = 1, . . . , n.
x (x, z
If further (, z) is differentiable for all z Z and x (x, ) is continuous on Z for each x, then
f (x) = conv { x (x, z) | z Z(x) },

x Rn .

Proof. (a) This is a special case of [Roc70, Theorem 24.7].

(b) This is [Roc70, Theorem 24.5].
(c) This is [Roc70, Theorem 23.4 and 25.1].
(d) This is [Ber99, Proposition B.25].
Figure 6.3 illustrates the subdifferential of a convex function.
We apply parts of the above results in order to characterize a minimum of a convex function on Rn .
Proposition 6.18 (optimality of a convex function over Rn ) Let f :
Rn R be a convex function. The following three statements are equivalent:
1. f is globally minimized at x Rn ;

2. 0n f (x );

3. f (x ; p) 0 for all p Rn .

160

Differentiability properties of the dual function

f (x)
x
5
4
3
2
1

Figure 6.3: The subdifferential of a convex function f at x.

Proof. We establish the result thus: 1 = 2 = 3 = 1.

[1 = 2]: By the statement 1., we have that f (y) f (x ) for
every y Rn . This implies that for g = 0n , we satisfy the subgradient
inequality (6.33). This establishes the statement 2.
[2 = 3]: We can equivalently write
f (x) = { g Rn | g T p f (x; p),

p Rn }.

With g = 0n this definition immediately yields the statement 3.

[3 = 1]: By the compactness of the subdifferential [cf. Proposition 6.17(a)] and Weierstrass Theorem 4.7 the maximum in the expression (6.34) is attained at some g f (x ). It follows that, in the
subgradient inequality (6.33), we get that
f (x + p) f (x ) + g T p f (x ),

p Rn ,

which is equivalent to the statement 1.

This result implies that a direction p Rn is a descent direction with
respect to f at x if and only if f (x; p) < 0 holds. This result cannot
be extended to non-convex functions, even when the function f is in C 1
or even C 2 . [Take f (x) := x3 ; x = 0; p = 1; see also the discussions
following Proposition 4.16 and on saddle points in Example 11.2(b).]
(A related result for possibly non-convex but differentiable functions is
found in Proposition 4.16.)
161

Lagrangian duality

6.3.2

Differentiability of the Lagrangian dual function

We consider the inequality constrained problem (6.4), where we make

the following standing assumption:
f, gi (i = 1, . . . , m) C 0 , X is nonempty and compact.

(6.37)

Under this assumption, the set of solutions to the Lagrangian subproblem,

X() := arg minimum L(x, ),
Rm ,
(6.38)
x X

is nonempty and compact for any choice of dual vector by Weierstrass

Theorem 4.7. We first develop the subdifferentiability properties of the
associated dual function q, stated in (6.9). The first result strengthens
Theorem 6.4 under these additional assumptions.
Proposition 6.19 (subdifferentiability of the dual function) Suppose that,
in the problem (6.4), the compactness condition (6.37) holds.
(a) The dual function (6.9) is finite, continuous and concave on Rm . If
its supremum over Rm
+ is attained, then the optimal solution set therefore
is closed and convex.
is the singleton
(b) The mapping 7
7 X() is closed on Rm . If X()
m

set {
x} for some R , and for some sequence {k } Rm with
, xk X(k ) for all k, then xk x
.
k
(c) Let Rm . If x X(), then g(x) is a subgradient to q at ,
that is, g(x) q().
(d) Let Rm . Then,
q() = conv { g(x) | x X() }.
The set q() is convex and compact. Moreover, if U is a bounded
set, then U q() is also bounded.
(e) The directional derivative of q at Rm in the direction of
p Rm is
q (; p) = minimum g T p.
g q()

Proof. (a) Theorem 6.4 stated the concavity of q on its effective domain.
Weierstrass Theorem 4.7 states that q is finite on Rm , which is then
also its effective domain. The continuity of q follows from that of any
finite concave function, as we have already seen in Theorem 4.27. The
closedness property of the solution set is a direct consequence of the
continuity of q (the upper level set then automatically is closed), and
complements the result of Theorem 6.9(a).
162

Differentiability properties of the dual function

, and
(b) Let {k } be a sequence of vectors in Rm converging to
let xk X(k ) be arbitrary. Let x be arbitrary in X, and let further
X be an arbitrary limit point of {xk } (at least one exists by the
x
compactness of X). From the property that for all k,
L(xk , k ) L(x, k ),
follows, by the continuity of L, that, in the limit of k in the subsequence
,
in which xk converges to x
L(x,
),
L(
x, )
X(),
as desired. The special case of a singleton set X()

so that x
follows.
Rm be arbitrary and let x
X().
We have that
(c) Let
= infimum L(y,
) = f (x) +
T g(x)
q()
y X

)T g(x) q() + (
)T g(x),
= f (x) + T g(x) + (
which implies that g(x) q().
(d) The inclusion q() conv { g(x) | x X() } follows from (c)
and the convexity of q(). The opposite inclusion follows by applying
the Separation Theorem 3.24.5
(e) See Proposition 6.17(c).
The result in (c) is an independent proof of the concavity of q on Rm .
The result (d) is particularly interesting, because by Caratheodorys
Theorem 3.8 every subgradient of q at any point is the convex combination of a finite number (in fact, at most m + 1) of vectors of the form
g(xs ) with xs X(). Computationally, this has been utilized to devise
efficient (proximal) bundle methods for the Lagrangian dual problem as
well as to devise methods to recover primal optimal solutions.
Next, we establish the differentiability of the dual function under
additional assumptions.
Proposition 6.20 (differentiability of the dual function) Suppose that,
in the problem (6.4), the compactness condition (6.37) holds.
(a) Let Rm . The dual function q is differentiable at if and
only if { g(x) | x X() } is a singleton set, that is, if the value of the
vector of constraint functions is invariant over the set of solutions X()
to the Lagrangian subproblem. Then, we have that
q() = g(x),
5 See

[BSS93, Theorem 6.3.7] for a detailed proof.

163

Lagrangian duality
for every x X().
(b) The result in (a) holds in particular if the Lagrangian subproblem
has a unique solution, that is, X() is a singleton set. In particular, this
property is satisfied for 0m if further X is a convex set, f is strictly
convex on X, and gi (i = 1, . . . , m) are convex, in which case q C 1 .
Proof. (a) The concave function q is differentiable at the point (where
it is finite) if and only if its subdifferential q() there is a singleton, cf.
Proposition 6.17(c).
(b) Under either one of the assumptions stated, X() is a singleton,
whence the result follows from (a). Uniqueness follows from the convexity of the feasible set and strict convexity of the objective function,
according to Proposition 4.11. That q C 1 follows from the continuity
of g and Proposition 6.19(b).

Proposition 6.21 (twice differentiability of the dual objective function)

Suppose that, in the problem (6.4), X = Rn , and f and gi (i = 1, . . . , m)
are convex functions in C 2 . Suppose that, at Rm , the solution x to
the Lagrangian subproblem not only is unique, but also that the partial
Hessian of the Lagrangian is positive definite at the pair (x, ), that is,
x2x L(x, ) is positive definite.
Then, the dual function q is twice differentiable at , with
2 q() = g(x)T [x2x L(x, )]1 g(x).
Proof. The result follows from the Implicit Function Theorem, which
is stated in Chapter 2, applied to the Lagrangian subproblem.6

6.4

Subgradient optimization methods

We begin by establishing the convergence of classic subgradient optimization methods as applied to a general convex optimization problem.

6.4.1

Convex problems

Consider the convex optimization problem to

minimize f (x),

(6.39a)

subject to x X,

(6.39b)

6 See

164

[Ber99, Pages 596598] for a detailed analysis.

Subgradient optimization methods

where f : Rn R is convex and the set X Rn is nonempty, closed

and convex.
The subgradient projection algorithm is as follows: select x0 X,
and for k = 0, 1, . . . generate
g k f (xk ),

(6.40a)

xk+1 = ProjX (xk k g k ),

(6.40b)

where the sequence {k } is generated from one of the following three

rules:
The first rule is termed the divergent series step length rule, and
requires that
k > 0,

k = 0, 1, . . . ;

lim k = 0;

k = +.

(6.41)

k=0

The second rule adds to the requirements in (6.41) the square-summable

restriction

X
(6.42)
2k < +.
k=0

The conditions in (6.41) allow for convergence to any point from any
starting point, since the total step is infinite, but convergence is therefore
also quite slow; the additional condition in (6.42) means fast sequences
are selected. An instance of the step length formulas which satisfies both
(6.41) and (6.42) is the following:
k = + /(k + 1),

k = 0, 1, . . . ,

where > 0, 0.
The third step length rule is
k = k

f (xk ) f
,
kg k k2

0 < 1 k 2 2 < 2,

(6.43)

where f is the optimal value of (6.39). We refer to this step length formula as the Polyak step, after the Russian mathematician Boris Polyak
who invented the subgradient method in the 1960s together with Ermolev and Shor.
How is convergence established for subgradient optimization methods? As shall be demonstrated in Chapters 11 and 12 convergence of
algorithms for problems with a differentiable objective function is typically based on generating descent directions, and step length rules that
result in the sequence {xk } of iterates being strictly descending in the
165

Lagrangian duality
value of f . For the non-differentiable problem at hand, generating descent directions is a difficult task, since it is not true that the negative of
an arbitrarily chosen subgradient of f at a non-optimal vector x defines
a descent direction.
In bundle methods one gathers information from more than one subgradient (hence the term bundle) around a current iteration point so
that a descent direction can be generated, followed by an inexact line
search. We concentrate here on the simpler methodology of subgradient
optimization methods, in which we apply the formula (6.40) where the
step length k is chosen based on very simple rules.
We establish below that if the step length is small enough, an iteration of the subgradient projection method leads to a vector that is closer
to the set of optimal solutions. This technical result also motivates the
construction of the Polyak step length rule, and hence shows that the
convergence of subgradient methods is based on the reduction of the Euclidean distance to the optimal solutions rather than on the reduction of
the value of the objective function f .
Proposition 6.22 (decreasing distance to the optimal solution set) Suppose that xk X is not optimal in (6.39), and that xk+1 is given by
(6.40) for some step length k > 0.
Then, for every optimal solution x in (6.39),
kxk+1 x k < kxk x k
holds for every step length k in the interval
k (0, 2[f (xk ) f ]/kg k k2 ).

(6.44)

Proof. We have that

kxk+1 x k2 = kProjX (xk k g k ) x k

= kProjX (xk k g k ) ProjX (x )k2

kxk k g k x k2

= kxk x k2 2k (xk x )T g k + 2k kg k k2

kxk x k2 2k [f (xk ) f ] + 2k kg k k2
< kxk x k2 ,

where we have utilized the property that the Euclidean projection is nonexpansive (Theorem 4.32), the subgradient inequality (6.33) for convex
166

Subgradient optimization methods

functions, and the bounds on k given by (6.44).

Our first convergence result is based on the divergent series step
length formula (6.41), and establishes convergence to the optimal solution set X under an assumption on its boundedness. With the other
two step length formulas, this condition will be possible to remove.
Recall the definition (3.12) of the minimum distance from a vector to
a closed and convex set; our interest is in the distance from an arbitrary
vector x Rn to the solution set X :
distX (x) := minimum
ky xk.

y X

Theorem 6.23 (convergence of subgradient optimization methods, I) Let

{xk } be generated by the method (6.40), (6.41). If X is bounded and
the sequence {g k } is bounded, then f (xk ) f and distX (xk ) 0
holds.
Proof. We show that the iterates will eventually belong to an arbitrarily
small neighbourhood of the set of optimal solutions to (6.39).
Let > 0 and B := { x Rn | kxk }. Since f is convex,
X is nonempty, closed and convex, and X is bounded, it follows from
[Roc70, Theorem 27.2], applied to the lower semi-continuous, proper7
and convex function f + X 8 that there exists an = () > 0 such that
the level set { x X | f (x) f + } X + B /2 ; this level set is
denoted by X . Moreover, since for all k, kg k k sups {kgs k} < , and
k 0, there exists an N () such that k kgk k2 and k kg k k /2
for all k N ().
The sequel of the proof is based on induction and is organized as
follows. In the first part, we show that there exists a finite k() N ()
such that xk() X + B . In the second part, we establish that if xk
belongs to X + B for some k N () then so does xk+1 , by showing
that either distX (xk+1 ) < distX (xk ) holds, or xk X so that xk+1
X + B since the step taken is not longer than /2.
Let x X be arbitrary. In every iteration k we then have
2

kx xk+1 k = kx ProjX (xk k g k )k

(6.45a)

kx xk + k g k k
(6.45b)


2
2

= kx xk k + k 2g T
, (6.45c)
k (x xk ) + k kg k k

7 A proper function is a function which is finite at least at some vector and nowhere
attains the value . See also Section 1.4.
8 For any set S Rn the function is the indicator function of the set S, that
S
is, S (x) = 0 if x S; and S (x) = + if x 6 S. See also Section 13.1.

167

Lagrangian duality
where the inequality follows from the projection property. Now, suppose
2

2 gT
s (x xs ) + s kg s k <

(6.46)

for all s N (). Then, using (6.45) repeatedly, we obtain that for any
k N (),
k
X

2
kx xk+1 k2 < x xN ()
s ,
s=N ()

and from (6.40) it follows that the right-hand side of this inequality tends
to minus infinity as k , which clearly is impossible. Therefore,
2

2 gT
k (x xk ) + k kg k k

(6.47)

for at least one k N (), say k = k(). From the definition of N (), it

follows that g T
k() (x xk() ) . From the definition of a subgradient

(cf. Definition 6.16) we have that f (x ) f (xk() ) g T

k() (x xk() ),
since x , xk() X. Hence, f (xk() ) f + , that is, xk() X
X + B /2 X + B .
Now, suppose that xk X + B for some k N (). If (6.46) holds
for s = k, then, by using (6.45), we have that kx xk+1 k < kx xk k
for any x X . Hence,
distX (xk+1 ) kProjX (xk ) xk+1 k < kProjX (xk ) xk k
= distX (xk ) .

Thus, xk+1 X +B . Otherwise, (6.47) must hold and, using the same
arguments as above, we obtain that f (xk ) f + , i.e., xk X
x + B /2 . As
kxk+1 xk k = kProjX (xk k g k ) xk k kxk k g k xk k
= k kg k k /2
whenever k N (), it follows that xk+1 X + B /2 + B /2 = X + B .
By induction with respect to k k(), it follows that xk X + B
for all k k(). Since this holds for arbitrarily small values of > 0
and f is continuous, the theorem follows.
We next introduce the additional requirement (6.42); the resulting
algorithms convergence behaviour is now much more favourable, and
the proof is at the same time less technical.
168

Subgradient optimization methods

Theorem 6.24 (convergence of subgradient optimization methods, II) Let

{xk } be generated by the method (6.40), (6.41), (6.42). If X is nonempty
and the sequence {g k } is bounded, then f (xk ) f and xk x X
holds.
Proof. Let x X and k 1. Repeated application of (6.45) yields
2

kx xk k kx x0 k + 2

k1
X
s=0

s g T
s (x xs ) +

k1
X
s=0

2s kg s k .(6.48)

Since x X and g s f (xs ) for all s 0 we obtain that

f (xs ) f f (xs ) + g T
s (x xs ) ,

s 0,

(6.49)

and hence that g T

sup {kg k}
s (x xs ) 0 for all s 0. Define c :=
P
Pk1 k 2 k
2
and p = k=0 k , so that kgs k c for any s 0 and s=0 s < p.
From (6.48) we then conclude that kx xk k2 < kx x0 k2 + pc2 for
any k 1, and thus that the sequence {xk } is bounded.

Assume now that there is no subsequence {xki } of {xk } with g T

ki (x
T

xki ) 0. Then there must exist an > 0 with g s (x xs ) for all

sufficiently large values of s. From (6.48) and the conditions on the step
lengths it follows that kx xs k , which clearly is impossible.
The sequence {xk } must therefore contain a subsequence {xki } such

that g T
ki (x xki ) 0. From (6.49) it follows that f (xki ) f .
The boundedness of {xk } implies the existence of an limit point of the
subsequence {xki }, say x . From the continuity of f it follows that
x X .
To show that x is the only limit point of {xkP
}, let > 0 and choose

an M () such that kx xM() k2 /2 and s=M() 2s /(2c2 ).

Consider any k > M (). Analogously to the derivation of (6.48), and
using (6.49), we then obtain that
k1
X

2

2s kg s k2 < + 2 c2 = .
kx xk k2 x xM() +
2 2c
s=M()

Since this holds for arbitrarily small values of > 0, we are done.

Note that the boundedness condition on {g k } is fulfilled whenever

we know before-hand that the sequence {xk } is bounded, such as in the
case when X itself is bounded; cf. Proposition 6.17(a).
We finally present the even stronger convergence properties of the
subgradient projection method using the Polyak step.
169

Lagrangian duality
Theorem 6.25 (convergence of subgradient optimization methods, III)
Let {xk } be generated by the method (6.40), (6.43). If X is nonempty
then f (xk ) f and xk x X holds.
Proof. From Proposition 6.22 follows that the sequence {kxk x k}
is strictly decreasing for every x X , and therefore has a limit. By
construction of the step length, in which the step lengths are bounded
away from zero and 2[f (xk ) f ]/kg k k2 , it follows from the proof of
Proposition 6.22 that [f (xk ) f ]2 /kg k k2 0 must hold. Since {g k }
must be bounded due to the boundedness of {xk } [Proposition 6.17(a)],
we have that f (xk ) f . Further, xk is bounded, and due to the
continuity property of f every limit point must then belong to X .
It remains to show that there can be only one limit point. This
property follows from the monotone decrease of the distance kxk x k.
In detail, the proof is as follows. Suppose two subsequences of {xk }
exist, such that they converge to two different vectors in X :
xmi x1 ;

xli x2 ;

x1 6= x2 .

We must then have kxli x1 k > 0. Since x1 X and the

distance to X is decreasing, kxk x1 k holds, and in particular
kxmi x1 k , which is a contradiction.
Contrary to the slow convergence of the subgradient projection algorithms that rely on the divergent series step length rule, under additional conditions on the function f a subgradient algorithm based on the
Polyak step length (6.43) is geometrically convergent, in the sense that
there exist c > 0 and (0, 1) with
kxk x k c k ,

k = 0, 1, . . . .

See Section 6.8 for references to other subgradient algorithms than those
presented here.

6.4.2

Application to the Lagrangian dual problem

We remind ourselves that the Lagrangian dual problem is a concave

maximization problem, and that the appearance of the dual function is
similar to that of the following example:
Let h(x) := minimum {h1 (x), h2 (x)}, where h1 (x) := 4 |x| and
h2 (x) := 4 (x 2)2 . Then,
(
4 x,
if 1 x 4;
h(x) =
2
4 (x 2) if x 1, x 4;
170

Subgradient optimization methods

Figure 6.4: A convex min-function with three pieces.

cf. Figure 6.4.

The function h is non-differentiable at x = 1 and x = 4, since its
graph has non-unique supporting hyperplanes there:

{4 2x}, if x < 1;

if x = 1;
[1, 2] ,
h(x) = {1},
if 1 < x < 4;

[4, 1] , if x = 4;

{4 2x}, if x > 4.

the subdifferential is here either a singleton (at differentiable points) or

a closed interval (at non-differentiable points).
Note the monotonically decreasing nature of the relation x 7 h(x).
Note also that 0 h(1), whence x = 1 defines a maximum over R.
and let U be the set of optimal solutions to
Now, let g q(),
(6.10). Then,
) 0 }.
U { Rm | g T (
In other words, any subgradient defines a half-space that contains the
set of optimal solutions; cf. Figure 6.5. We therefore know that a small
enough step in the direction of a subgradient gets us closer to the set of
optimal solutions; cf. Proposition 6.22. But again consider Figure 6.5:
an arbitrary subgradient, like the one depicted, may not define an ascent
direction! As we saw in the previous section, convergence must be based
on other arguments, like the decreasing distance to U alluded to above
and in the previous section. In the next subsection we discuss in brief
the generation of ascent directions.
We consider the Lagrangian dual problem (6.10). We suppose, as
in the previous section, that X is compact so that the infimum in (6.9)
171

Lagrangian duality

1
2
3

q()

4
5

q
Figure 6.5: The half-space defined by the subgradient g of q at . Note
that the subgradient is not an ascent direction.

is attained for every 0m (which is the set over which we wish to

maximize q) and q is real-valued over Rm
+.
In the case of our special concave maximization problem, the iteration
has the form
k+1 = ProjRm
[k + k g k ] = [k + k g k ]+
+
= (maximum {0, (k )i + k (g k )i })m
i=1 ,

(6.50)

where g k q(k ) is arbitrarily chosen; we would typically use g k =

g(xk ), where xk arg minimumx X L(x, k ). The projection operation
onto the first orthant is, as we can see, very simple.
Replacing the Polyak step (6.43) with the corresponding dual form
k = k

q q(k )
,
kg k k2

0 < 1 k 2 2 < 2,

(6.51)

convergence will now be a simple consequence of the above theorems.

The compactness condition (6.37) and the fact that the feasible set of
(6.4) is nonempty ensure that the problem (6.4) has an optimal solution;
in particular, the feasibility condition (6.5) then holds. Further, if we
introduce the Slater condition (6.16), we are ensured that there is no
duality gap, and that the dual problem (6.10) has a compact set U
of optimal solutions. Under these assumptions, we have the following
results for subgradient optimization methods.
Theorem 6.26 (convergence of subgradient optimization methods) Sup172

Subgradient optimization methods

pose that the problem (6.4) is feasible, and that the compactness condition (6.37) and the Slater condition (6.16) hold.
(a) Let {k } be generated by the method (6.50), (6.41). Then,
q(k ) q , and distU (k ) 0.
(b) Let {k } be generated by the method (6.50), (6.41), (6.42). Then,
{k } converges to an optimal solution to (6.10).
(c) Let {k } be generated by the method (6.50), (6.51). Then, {k }
converges to an optimal solution to (6.10).
Proof. The results follow from Theorems 6.23, 6.24, and 6.25, respectively. Note that in the first two cases, boundedness conditions were
assumed for X and the sequence of subgradients. The corresponding
conditions for the Lagrangian dual problem are fulfilled under the CQs
imposed, since they imply that the search for an optimal solution is done
over a compact set; cf. Theorem 6.9(a) and its proof.

6.4.3

The generation of ascent directions

Proposition 6.18 shows that the existence of a descent direction with

Rn hinges on
respect to the convex function f : Rn R at some x
n

R such that f (
) < 0. According
the existence of some vector p
x; p
to the definition of the directional derivative and the compactness of
< 0 for every
f (
x), this is equivalent to the statement that g T p
g f (
x). In the context of Lagrangian duality we show below how we
can generate an ascent direction for q at some Rm .
Definition 6.27 (steepest ascent direction) Suppose that the problem
(6.4) is feasible, and that the compactness condition (6.37) holds. Consider the Lagrangian dual problem (6.10), and let Rm . A vector
Rm with k
p
pk 1 is a steepest ascent direction if
) = maximum q (; p)
q (; p
kpk1

holds.
Proposition 6.28 (the shortest subgradient yields the steepest ascent direction) Suppose that the problem (6.4) is feasible, and that the compactness condition (6.37) holds. Consider the Lagrangian dual problem
of steepest ascent with respect to q at is given
(6.10). The direction p
q() is the shortest subgradient in q() with respect
below, where g
to the Euclidean norm:
(
= 0m ,
0m , if g
=
p
g
6= 0m .
k , if g
kg
173

Lagrangian duality

Proof. By Definition 6.27 and Proposition 6.19(e), the following string

of equalities and inequalities can easily be verified:
maximum q (; p) = maximum infimum g T p
kpk1
kpk1
g q()

infimum maximum g T p
g q()
kpk1
= infimum kgk
g q()
= k
g k.

(6.52)

such that q (; p
) = k
If we can construct a direction p
g k then by (6.52)
is the steepest ascent direction. If g
= 0m then for p
= 0m we
p

) = k
obviously have that q (; p
g k. Suppose then that g 6= 0m , and let
:= g
/k
p
gk. Note that
g T g
) = infimum g T p
= infimum
q (; p
gk
g q()
g q() k


1
=
infimum k
g k2 + g T (g g )
k
gk g q()
1
T (g g
).
= k
gk +
infimum g
k
g k g q()

(6.53)

is the shortest vector in q(), then, by the variational inequality

Since g
characterization of the projection of 0m onto q() established in The ) 0 for every g q(). Hence,
orem 4.24, we obtain that g T (g g
infimumg q() gT (g g ) = 0 is achieved at g . From (6.53) it then fol ) = k
lows that q (; p
g k. We are done.

6.5

Obtaining a primal solution

It remains for us to show how an optimal dual solution can be translated into an optimal primal solution x . Obviously, convexity and
strong duality will be needed in general, if we are to be able to utilize
the primaldual optimality characterization in Theorem 6.7. It turns
out that the generation of a primal optimum is automatic if q is differentiable at , which is also the condition under which the famous
Lagrange multiplier method works. Unfortunately, in many cases, such
as for most non-strictly convex optimization problems (like linear programming), this will not be the case, and then the translation work
becomes more complex.
We start with the ideal case.
174

6.5.1

Obtaining a primal solution

Differentiability at the optimal solution

The following results summarize the optimality conditions for the Lagrangian dual problem (6.10), and their consequences for the availability
of a primal optimal solution in the absence of a duality gap.
Proposition 6.29 (optimality conditions for the dual problem) Suppose
that the compactness condition (6.37) holds in the problem (6.4). Suppose further that the vector is solves the Lagrangian dual problem.
(a) The dual optimal solution is characterized by the inclusion
0m q( ) + NRm
( ).
+

(6.54)

In other words, there then exists q( )an optimality-characterizing subgradient of q at such that
0m 0m .

(6.55)

There exists a finite set of solutions xi X( ), i = 1, . . . , k, where

k m + 1 such that
=

k
X

i g(xi );

i=1

k
X
i=1

i = 1;

i 0, i = 1, . . . , k.

(6.56)

Hence, we have that

k
X

i i gi (xi ) = 0.

(6.57)

i=1

(b) If there is a duality gap, then q is non-differentiable at .

(c) If q is differentiable at , then there is no duality gap. Further,
any vector in X( ) then solves the primal problem (6.4).
Proof. (a) The first result is a direct statement of the optimality conditions of the convex and subdifferentiable program (6.10); the complementarity conditions in (6.55) are an equivalent statement of the inclusion
in (6.54).
The second result is an application of Caratheodorys Theorem 3.8
to the compact and convex set q( ).
(b) The result is established once (c) is.
be any vector in X( ) for which q( ) = g(
(c) Let x
x) holds, cf.
Proposition 6.20(a). We obtain from (6.55) that
0m g(
x) 0m .
) fulfills all the conditions stated in (6.12), so that,
Hence, the pair ( , x

by Theorem 6.7, x is an optimal solution to (6.4).

175

Lagrangian duality
Remark 6.30 (the non-coordinability phenomenon and decomposition algorithms) Many interesting problems do not comply with the conditions
in (c); for example, linear programming is one where the Lagrangian
dual problem often is non-differentiable at every dual optimal solution.9 This is sometimes called the non-coordinability phenomenon (cf.
[Las70, DiJ79]). It was in order to cope with this phenomenon that
DantzigWolfe decomposition ([DaW60, Las70]) and other column generation algorithms, Benders decomposition ([Ben62, Las70]) and generalized linear programming were developed; noticing that the convex combination of a finite number of candidate primal solutions are sufficient to
verify an optimal primaldual solution [cf. (6.57)], methodologies were
developed to generate those vectors algorithmically. See also [LPS99]
for overviews on the subject of generating primal optimal solutions from
dual optimal ones, and [BSS93, Theorem 6.5.2] for an LP procedure that
provides primal feasible solutions for convex programs.
Note that the equation (6.57) in (a) reduces to the complementar :=
ity condition that i gi (
x) = 0 holds, for the averaged solution, x
Pk
i
i=1 i x , whenever all the functions gi are affine.

6.5.2

Everetts Theorem

The next result shows that the solution to the Lagrangian subproblem
solves a perturbed version of the original problem. We state the result
for the general problem to find
f := infimum f (x),

(6.58)

subject to

x X,

gi (x) 0,
hj (x) = 0,

i = 1, . . . , m,
j = 1, . . . , ,

where f : Rn R, gi : Rn R, i = 1, 2, . . . , m, and hj : Rn R,
j = 1, 2, . . . , , are given functions, and X Rn .

Theorem 6.31 (Everetts Theorem) Let (, ) Rm

+ R . Consider
the Lagrangian subproblem to
n
o
minimize f (x) + T g(x) + T h(x) .
(6.59)
x X

is an optimal solution to this problem, and let I()

Suppose that x
{1, . . . , m} denote the set of indices i for which i > 0.
9 In

other words, even if a Lagrange multiplier vector is known, the Lagrangian

subproblem may not identify a primal optimal solution.

176

Sensitivity analysis

is an optimal solution to the perturbed primal problem to

(a) x
minimize f (x),

(6.60)

subject to

x X,

gi (x) gi (
x),
hj (x) = hj (
x),

i I(
x),
j = 1, . . . , .

is feasible in (6.58) and T g(

solves
(b) If x
x) = 0 holds, then x
(6.58), and the pair (, ) then solves the Lagrangian dual problem.
Proof. (a) Let x satisfy the constraints of (6.60). Since we have that
in (6.59) yields
h(x) = h(
x) and T g(x) T g(
x), the optimality of x
f (x) + T g(
x) + T h(
x) f (x) + T g(x) + T h(x)

f (
x) + T g(
x) + T h(
x),

which shows that f (x) f (

x). We are done.
solves
(b) T g(
x) = 0 implies that gi (
x) = 0 for i I(); from (a) x
the problem to
minimize f (x),

(6.61)

subject to

x X,

gi (x) 0,
hj (x) = 0,

i I(
x),
j = 1, . . . , .

In particular, then, since the feasible set of (6.58) is contained in that

is feasible in the former, x
must also solve (6.58). That
of (6.61) and x
the pair (, ) solves the dual problem follows by the equality between
the primal and dual objective functions at (
x, , ), and weak duality.
One important consequence of the result is that if the right-hand side
being
perturbations gi (
x) and hi (
x) all are close to zero, the vector x
near-feasible might mean that it is in fact acceptable as an approximate
solution to the original problem. (This interpretation hinges on the
dualized constraints being soft constraints, in the sense that a small
violation is acceptable. See Section 1.8 for an introduction to the topic
of soft constraints.)

6.6
6.6.1

Sensitivity analysis

Analysis for convex problems

Consider the inequality constrained convex program (6.4), where f :

Rn R and gi , i = 1, . . . , m, are convex functions and X Rn is
177

Lagrangian duality
a convex set. Suppose that the problem (6.4) is feasible, and that the
compactness condition (6.37) and Slater condition (6.16) hold. This is
the classic case where there exist multiplier vectors , according to
Theorem 6.9, and strong duality holds.
For certain types of problems where the duality gap is zero and where
there exist primaldual optimal solutions, we have access to a beautiful
theory of sensitivity analysis. The classic meaning of the term is the
answer to the following question: what is the rate of change in f when
a constraint right-hand side changes? This question answers important
practical questions, like the following in manufacturing: If we buy one
unit of additional resource at a given price, or if the demand of a product
that we sell increases by a certain amount, then how much additional
profit do we make?
We will here provide a basic result which states when this sensitivity
analysis of the optimal objective value can be performed for the problem
(6.4), and establish that the answer is determined precisely by the value
of the Lagrange multiplier vector , provided that it is unique.
Definition 6.32 (perturbation function) Consider the function p : Rm
R {} defined by
p(u) := infimum f (x),

(6.62)

subject to

x X,
gi (x) ui ,

i = 1, . . . , m,

u Rm ;

it is called the perturbation function, or primal function, associated with

the problem (6.4). Its effective domain is the set P := { u Rm | p(u) <
+ }.
Under the above convexity conditions, we can establish that p is a
convex function. Indeed, it holds that for any value of the Lagrange
multiplier vector for the problem (6.4) that
q( ) = infimum {f (x) + ( )T g(x)}
x X

=
=

infimum

{f (x) + ( )T g(x)}

infimum

{f (x) + ( )T u}

{ (u,x )P X|g(x )u }
{ (u,x )P X|g(x )u }

= infimum
u P

infimum

{x X|g(x )u }

{f (x) + ( )T u}.

Since is assumed to be a Lagrange multiplier vector, we have that

q( ) = f = p(0m ). By the definition of infimum, then, we have that
p(0m ) p(u) + ( )T u,
178

u Rm ,

Sensitivity analysis

that is, (notice the sign!) is a subgradient of p at u = 0m (see

Definition 6.16). Moreover, by the result in Proposition 6.17(c), p is
differentiable at 0m if and only if p is finite in a neighbourhood of 0m
and is a unique Lagrange multiplier vector, that is, the Lagrangian
dual problem (6.10) has a unique optimal solution. We have therefore
proved the following result:
Proposition 6.33 (a sensitivity analysis result) Suppose that in the inequality constrained problem (6.4), f : Rn R and gi : Rn R,
i = 1, . . . , m, are convex functions and X Rn is a convex set. Suppose that the problem (6.4) is feasible, and that the compactness assumption (6.37) and Slater condition (6.16) hold. Suppose further that
the perturbed problem defined in (6.62) has an optimal solution in a
neighbourhood of u = 0m , and that on the set of primaldual optimal
solutions to (6.4), (6.10), the dual optimal solution is unique. Then,
the perturbation function p is differentiable at u = 0m , and
p(0m ) =
holds.
It is intuitive that the sign of p(0m ) should be non-positive; if a
right-hand side of the (less-than) inequality constraints in (6.4) increases,
then the feasible set becomes larger. [This means that we might be able
to find feasible vectors x in the new problem with f (x) < f , where
f = p(0m ) is the optimal value of the minimization problem (6.4).]
The result specializes immediately to linear programming problems,
which is the problem type where this type of analysis is most often
utilized. The proof of differentiability of the perturbation function at
zero for that special case can however be done much more simply. (See
Section 10.3.1.)

6.6.2

Analysis for differentiable problems

There exist local versions of the analysis valid also for non-convex problems, where we are interested in the effect of a problem perturbation
on a KKT point. A special such analysis was recently performed by
Bertsekas [Ber04], in which he shows that even when the problem is
non-convex and the Lagrange multipliers are not unique, a sensitivity
analysis is available as long as the functions defining the problem are
differentiable. Suppose then that in the problem (6.4) the functions f
and gi , i = 1, . . . , m are in C 1 and that X is nonempty. We generalize
the concept of a Lagrange multiplier vector to here mean that it is a
179

Lagrangian duality
vector associated with a local minimum x such that

f (x ) +

m
X
i=1

!T

i gi (x )

p 0,

p TX (x ),

(6.63a)

i 0,
i = 0,

i = 1, . . . , m,
i 6 I(x ),

(6.63b)
(6.63c)

where TX (x ) is the tangent cone to X at x (cf. Definition 5.2). Note

that under an appropriate CQ this is equivalent to the KKT conditions,
in which case we are simply requiring here that x is a local minimum.
In the below result we utilize the notation
gi+ (x) := maximum {0, gi (x)},

i = 1, . . . , m,

and let g + (x) be the m-vector of elements gi+ (x), i = 1, . . . , m.

Theorem 6.34 (sensitivity from the minimum norm multiplier vector) Suppose that x is a local minimum in the problem (6.4), and that the set
of Lagrange multipliers is nonempty. Let denote the Lagrange multiplier vector of minimum Euclidean norm. Then, for every sequence
{xk } X of infeasible vectors such that xk x we have that
f (x ) f (xk ) k k kg + (xk )k + o(kxk x k).

(6.64)

Furthermore, if 6= 0m and TX (x ) is convex, the above inequality

is sharp in the sense that there exists a sequence of infeasible vectors
{xk } X such that
f (x ) f (xk )
= k k,
k
kg+ (xk )k
lim

and for this sequence

i
gi+ (xk )
=
,
+
k kg (xk )k
k k
lim

i = 1, . . . , m,

holds.
Theorem 6.34 establishes the optimal rate of cost improvement with
respect to infeasible constraint perturbations (in effect, those that imply
an enlargement of the feasible set).
We finally remark that under stronger conditions still, the operator
u 7 x (u) assigning the (unique) optimal solution x to each perturbation vector u Rm is differentiable at u = 0m . Such a result is
180

Applications
reminiscent to the Implicit Function Theorem, which however only covers equality systems. If we are to study the sensitivity of x to changes
in the right-hand sides of inequality constraints as well, then the analysis
becomes complicated due to the fact that we must be able to predict if
some active constraints may become inactive in the process. In some
circumstances, different directions of change in the right-hand sides may
cause different subsets of the active constraints I(x ) at x to become
inactive, and this would most probably then be a non-differentiable point
of the operator u 7 x (u). A sufficient condition (but not necessary, at
least in the case of linear constraints) for this to not happen is when x
is strictly complementary, that is, when there exists a multiplier vector
with i > 0 for every i I(x ).

6.7

Applications

We provide two example applications of Lagrangian duality. The first

describes the primaldual relationship between currents and voltages in
an electrical network of devices (voltage sources, diodes, and resistors);
this application illustrates that Lagrange multipliers often have direct
interpretations. The second application concerns a classic combinatorial
optimization problem: the traveling salesman problem. We show how
to approximately solve this problem through Lagrangian relaxation and
subgradient optimization.

6.7.1

Electrical networks

An electrical network (or, circuit) is an interconnection of analog electrical elements such as resistors, inductors, capacitors, diodes, and transistors. Its size varies from the smallest integrated circuit to an entire
electricity distribution network. A circuit is a network that has at least
one closed loop. A network is a connection of 2 or more simple circuit
elements, and may not be a circuit. The goal when designing electrical
networks for signal processing is to apply a predefined operation on potential differences (measured in volts) or currents (measured in amperes).
Typical functions for these electrical networks are amplification, oscillation and analog linear algorithmic operations such as addition, subtraction, multiplication, and division. In the case of power distribution
networks, engineers design the circuit to transport energy as efficiently
as possible while at the same time taking into account economic factors,
network safety and redundancy. These networks use components such
as power lines, cables, circuit breakers, switches and transformers.
To design any electrical circuits, electrical engineers need to be able
181

Lagrangian duality
to predict the voltages and currents in the circuit. Linear circuits (that
is, an electrical network where all elements have a linear currentvoltage
relation) can be quite easily analyzed through the use of complex numbers and systems of linear equations,10 while nonlinear elements require
a more sophisticated analysis. The classic electrical laws describing
the equilibrium state of an electrical network are due to G. Kirchhoff
[Kir1847]; referred to as Kirchhoffs circuit laws they express in a mathematical form the conservation of charge and energy.11
Formally, we let an electrical circuit be described by branches (or,
links) connecting nodes. We present a simple example where the only
devices are voltage sources, resistors, and diodes. The resulting equilibrium conditions will be shown to be represented as the solution to a
strictly convex quadratic program. In general, devices such as resistors
can be non-linear, but linearity is assumed throughout this section.
A voltage source maintains a constant branch voltage vs irrespective of the branch current cs . The power absorbed by the device
is vs cs .
A diode permits the branch current cd to flow in one direction only,
but consumes no power regardless of the current or voltage on the
branch. Denoting the branch voltage by vd , the direction condition
can be stated as a complementarity condition:

cd 0;

vd 0;

vd cd = 0.

(6.65)

A resistor consumes power in relation with its resistance, denoted

by Rr . We recognize the following law describing the relationship
between the branch current and voltage in a linear resistor:

vr = Rr cr .

(6.66)

The power consumed is given by

vr cr =

vr2
= Rr c2r ,
Rr

(6.67)

where we have utilized (6.66) to derive two alternative relations.

We must be careful about the direction of flow of currents and voltages, and thus define, for each type of device, a nodebranch incidence
10 For

such networks already Maxwell [Max1865] had stated equilibrium conditions.

laws can be derived from Maxwells equations, but Kirchhoff preceded
Maxwell and derived his equations from work done by G. Ohm.
11 These

182

Applications
matrix of the form

1, if branch j has node i as its origin,

nij := 1,
if branch j ends in node i,

0,
otherwise.

The interpretation of a current flow variable is that the direction is

from the negative to the positive terminal of the device, that is, from
the origin to the ending node of the branch; a negative variable value
will therefore correspond to a flow in the opposite direction. Note that
for the diodes, the latter is not allowed, as seen in (6.65).
For the three types of devices we hence yield incidence matrices
denoted by NS , NR , and ND , creating a partitioned matrix N =
T
T T
T
T
T T
[NS ND NR ]. Similarly, we let c = (cT
S , cD , cR ) and v = (v S , v D , v R )
represent the vectors of branch currents and voltages. We also let p =
T
T T
(pT
S , pD , pR ) denote the vector of node potentials. Before stating the
optimization problem whose minimum describes the equilibrium of the
system, we recall the two fundamental equilibrium laws:
Kirchhoffs current law: The sum of all currents entering a node is equal
to the sum of all currents leaving the node. In other words, N c = 0:12
NS cS + ND cD + NR cR = 0.

(6.68)

Kirchhoffs voltage law: The difference between the node potentials at

the ends of each branch is equal to the branch voltage. In other words,
N T p = v:13
NST p = v S ,

(6.69a)

T
ND
p
T
NR p

= vD ,

(6.69b)

= vR .

(6.69c)

We summarize the equations representing the characteristics of the

electrical devices as follows: For the diodes, (6.65) yields
v D 0;

cD 0;

vT
D cD = 0.

(6.70)

For the resistors, (6.66) yields

v R = RcR ,

(6.71)

12 This law is also referred to as the first law, the point rule, the junction rule, and
the node law.
13 This law is a corollary to Ohms law, and is also referred to as the loop law.

183

Lagrangian duality
R being the diagonal matrix with elements equal to the values Rr .
Hence, (6.68)(6.71) represent the equilibrium conditions of the circuit. We will now describe the optimization problem whose optimality
conditions are, precisely, (6.68)(6.71) [note that v S is fixed]:
1 T
c RcR v T
S cS ,
2 R
subject to NS cS + ND cD + NR cR = 0,
cD 0.
minimize

(6.72)

In the problem (6.72) we wish to determine branch currents cS , cD ,

and cR so as to minimize the sum of half the energy absorbed in the resistors and the energy loss of the voltage source. Note the sign condition
on the diode currents.
Note that this is a convex program with linear constraints, and thus
the KKT conditions are both necessary and sufficient for the global optimality of the currents. It is instrumental to check that the KKT conditions for (6.72) are given by (6.68)(6.71), where the Lagrange multipliers
T
are given by (pT , v T
D) .
In the discussion terminating in the Strong Duality Theorem 6.13,
we showed that the Lagrangian dual of a strictly convex quadratic optimization problem is yet another convex quadratic optimization problem.
In our case, following that development, we can derive the following dual
optimization problem in terms of the node potentials p (notice, again,
that v S is fixed):
1
maximize v T
R1 v R ,
2 R
subject to
NST p = v S ,

(6.73)

T
ND
p v D = 0,

NRT p v R = 0,
v D 0.

In the dual problem (6.73) the matrix R1 is the diagonal matrix of

conductances. The objective function is equivalent to the minimization
of the power absorbed by the resistors, and we wish to determine the
branch voltages v D and v R , and the potential vector p.
Verify that the KKT conditions for this problem, again, reduce to
the equilibrium conditions (6.68)(6.71). In other words, the Lagrange
multipliers for the dual problem (6.73) are the (primal) branch currents.
184

Applications
Finally, let us note that by Theorem 6.13(a) the two problems (6.72)
and (6.73) have the same objective value at optimality. That is,
1
1 T
c RcR + v T
R1 v R v T
S cS = 0.
2 R
2 R
By (6.70)(6.71), the above equation reduces to
T
T
vT
S cS + v D cD + v R cR = 0,

which is precisely the principle of energy conservation.

6.7.2

A Lagrangian relaxation of the traveling salesman problem

Lagrangian relaxation has shown to be remarkably efficient for some

combinatorial optimization problems. This is surprising when taking
into account that such problems are integer or mixed-integer problems,
which suffer from non-zero duality gaps in general. What then lies behind
their popularity?
One can show that Lagrangian relaxation of an integer program is
always at least as good as that of a continuous relaxation14 (in the
sense that the value of fR is higher for Lagrangian relaxation than
for a continuous relaxation);
Together with heuristics for finding primal feasible solution, good
feasible solutions are often found;
The Lagrangian relaxed problems can be made computationally
much simpler than the original problem, while still keeping a lot
of the structure of the original problem.

6.7.2.1

The traveling salesman problem

Let the graph G = (N , L) be defined by a number of cities (or, nodes)

i N and undirected links in between subsets of pairs of them: (i, j)
L N N . Notice that the links (i, j) and (j, i) are identical, and are
in L represented by one non-directed link only.
Let cij denote the distance between the cities i and j, {i, j} N .
We introduce the following binary variables:
(
1, if link (i, j) is part of the TSP tour,
xij :=
(i, j) L.
0, otherwise,
14 The

continuous relaxation amounts to removing the integrality conditions, replacing, for example, xj {0, 1} by xj [0, 1].

185

Lagrangian duality
With these definitions, the undirected traveling salesman problem
(TSP) is to
minimize
x

subject to

cij xij ,

(6.74a)

(i,j)L

(i,j)L:{i,j}S

(i,j)L
X

xij |S| 1,

S N,

xij = n,
xij = 2,

iN :(i,j)L

xij {0, 1},

(6.74b)
(6.74c)

j N,
(i, j) L.

(6.74d)
(6.74e)

The constraints have the following interpretation: (6.74b) implies

that there can be no sub-tours, that is, a tour where fewer than n cities
are visited (if S N then there can be at most |S| 1 links between
nodes in the set S, where |S| is the cardinalitynumber of membersof
the set S); (6.74c) implies that in total n cities must be visited; and
(6.74d) implies that each city is connected to two others, such that we
make sure to arrive from one city and leave for the next.
This problem is NP-hard, which implies that there is no known
polynomial algorithm for solving it. We resort therefore to the use
of relaxation techniques, in particular Lagrangian relaxation. We have
more than one alternative relaxation to perform: If we Lagrangian relax the tree constraints (6.74b) and (6.74c) the remaining problem is a
2-matching problem; it can be solved in polynomial time. If we instead
Lagrangian relax the degree constraints (6.74d) for every node except for
one node the remaining problem is a 1-MST problem, that is, a special
type of minimum spanning tree (MST) problem.
The following definition is classic: a Hamiltonian path (respectively,
cycle) is a path (respectively, cycle) which passes every node in the graph
exactly once. Every Hamiltonian cycle is a Hamiltonian path from a node
s to another node, t, followed by a link (t, s); a subgraph which consists
of a spanning tree plus an extra link such that all nodes have degree two.
This is then a feasible solution to the TSP.
A 1-MST problem is the problem to find an MST in the graph that
excludes node s, followed by the addition of the two least expensive links
from node s to that tree. If all nodes happen to get degree two, then
the 1-MST solution is a traveling salesman tour (that is, a Hamiltonian
cycle). The idea behind solving the Lagrangian dual problem is then
to find proper multiplier values such that the Lagrangian relaxation will
produce feasible solutions.
186

Applications
6.7.2.2

Lagrangian relaxation of the traveling salesman problem

Suppose that we Lagrangian relax the degree constraints (6.74d), except

for that for node 1. We assume that the starting node for the trip,
node s N , and all the links in L connected to it, have been removed
temporarily (in the 1-MST, this data is re-introduced later), but without
changing the notation to reflect this.
The subproblem is the following: a 1-MST defined by

X
X
X
q() = minimum
cij xij +
j 2
xij
x

=2

jN

jN

(i,j)L

j + minimum
x

iN :(i,j)L

(cij i j )xij .

(i,j)L

We see immediately the role of the Lagrange multipliers: a high (low)

value of the multiplier j makes node j attractive (unattractive) in the
above 1-MST problem, and will therefore lead to more (less) links being
attached to it.
When solving the Lagrangian dual problem, we will use the class
of subgradient optimization methods, an overview of which is found in
Section 6.4.
What is the updating step in the subgradient method, and what is its
interpretation? It is as usual an update in the direction of a subgradient,
that is, the direction of
X
xij (),
i N,
hi (x()) := 2
iN :(i,j)L

where the value of xij {0, 1} is the solution to the 1-MST solution
with link costs cij i j . We see from the direction formula that

X
new
xij () ,
j N,
:= j + 2
j
iN :(i,j)L

where > 0 is a step length. It is interesting to investigate what the

update means:

> 2 = j (link cost )

= 2 = j (link cost constant)
current degree at node j :

< 2 = j (link cost )

In other words, the updating formula in a subgradient method is

such that the link cost in the 1-MST subproblem is shifted upwards
187

Lagrangian duality
(downwards) if there are too many (too few) links connected to node j
in the 1-MST. We are hence adjusting the node prices of the nodes in
such a way as to try to influence the 1-MST problem to always choose 2
links per node to connect to.
6.7.2.3

A feasibility heuristic

A feasibility heuristic takes the optimal solution from the Lagrangian

minimization problem over x and adjusts it such that a feasible solution
to the original problem is constructed. Since one cannot predict if, or
when, a primal feasible solution will be found directly from the subproblem, the heuristic will provide a solution that can be used in place of an
optimal one, if one is not found. Moreover, as we know from Lagrangian
duality theory, we then have access to both lower and upper bounds on
the optimal value f of the original problem, and so we have a quality
measure of the feasible solutions found.
A feasibility heuristic which can be used together with our Lagrangian
relaxation is as follows.
Identify a path in the 1-MST with many links. Then form a subgraph
with the remaining nodes and find a path that passes all of them. Put the
two paths together in the best way. The resulting path is a Hamiltonian
cycle, that is, a feasible solution.
6.7.2.4

The Philips example

In 19871988 an M.Sc. project was performed at the department of

mathematics at Link
oping University, in cooperation with the company
Philips, Norrk
oping. The project was initiated with the goal to improve
the current practice of solving a production planning problem.
The problem was as follows: Philips produce circuit boards, perhaps
several hundreds or thousands of the same type. There is a new batch of
patterns (holes) to be drilled every day, and perhaps even several such
batches per day.
To speed up the production process the drilling machine is connected
to a microcomputer that selects the ordering of the holes to be drilled,
given their coordinates. The algorithm for performing the sorting used to
be a simple sorting operation that found, for every fixed x-coordinate,
the corresponding y-coordinates and sorted them in increasing order.
The movement of the drill was therefore from left to right, and for each
fixed x-coordinate the movement was vertical. The time it took to drill
the holes on one circuit board was, however, far too long, simply because
the drill traveled around a lot without performing any tasks, following
a path that was too long. (On the other hand, the actual ordering was
188

Notes and further reading

very fast to produce!) All in all, the complete batch production took too
long because of the poorly planned drill movement.
It was observed that the production planning problem is a traveling salesman problem, where the cities are the holes to be drilled, and
the distances between them correspond to the Euclidean distances between them. Therefore, an efficient TSP heuristic was devised and implemented, for use in conjunction with the microcomputer. In fact, it
was based on precisely the above Lagrangian relaxation, a subgradient
optimization method, and a graph-search type heuristic of the form discussed above.
A typical run with the algorithm took a few minutes, and was always
stopped after a fixed number of subgradient iterations; the generation of
feasible solutions with the above graph search technique was performed
at every K th iteration, where K > 1 is an integer. (Moreover, feasible
solutions were not generated during the first iterations of the dual procedure, because of the poor quality of k for low values of k; often the
traveling salesman tour resulting from the heuristic is better when the
multipliers are near-optimal in the Lagrangian dual problem.)
In one of the examples implemented it was found that the optimal
path length was in the order to 2 meters, and that the upper and lower
bounds on f produced lead to the conclusion that the relative error of
the path length of the best feasible solution found was less than 7 %, a
quite good result, also showing that the duality gap for the problem at
hand (together with the Lagrangian relaxation chosen) is quite small.
After implementing the new procedure, Philips could report an increase in production by some 70 %. Hence, the slightly longer time it
took to provide a better production plan, that is, the traveling salesman
tour for the drill to follow, was more than well compensated by the fact
that the drilling could be done much faster.
This is a case where Lagrangian relaxation helped to solve a largescale, complex and difficult problem by utilizing problem structure.

6.8

Notes and further reading

Lagrangian duality has been developed in many sources, including early

developments by Arrow, Hurwicz, and Uzawa [AHU58], Everett [Eve63],
and Falk [Fal67], and later on by Rockafellar [Roc70]. Our development
follows to a large extent that of portions of the text books by Bertsekas [Ber99], Bazaraa et al. [BSS93], and Rockafellar [Roc70].
The Relaxation Theorem 6.1 can almost be considered to be folklore,
and can be found in a slightly different form in [Wol98, Proposition 2.3].
189

Lagrangian duality
The differentiability properties of convex functions were developed
largely by Rockafellar [Roc70], whose text we mostly follow.
Subgradient methods were developed in the Soviet Union in the
1960s, predominantly by Ermolev, Polyak, and Shor. Text book treatments of subgradient methods are found, for example, in [Sho85, HiL93,
Ber99]. Theorem 6.23 is essentially due to Ermolev [Erm66]; the proof
stems from [LPS96]. Theorem 6.24 is due to Shepilov [She76]; finally,
Theorem 6.25 is due to Polyak [Pol69].
Everetts Theorem 6.31 is due to Everett [Eve63].
Theorem 6.34 stems from [Ber04, Proposition 1.1].
That the equilibrium conditions of an electrical or hydraulic network
are attained as the minimum of the total energy loss were known more
than a century ago. Mathematical programming models for the electrical network equilibrium problems described in Section 6.7.1 date at least
as far back as to Duffin [Duf46, Duf47] and dAuriac [dAu47]. Duffin
constructs his objective function as a sum of integrals of resistance functions. The possibility of viewing the equilibrium problem in at least two
related, dual, ways as that of either finding the optimal flows of currents
or the optimal potentials was also known early in the analysis of electrical networks; these two principles are written out in [Cro36] in work on
pipe networks, and explicitly stated as a pair of primaldual quadratic
programming problems in [Den59]; we followed his development, as represented in [BSS93, Section 1.2.D].
The traveling salesman problem is an essential model problem in
combinatorial optimization. Excellent introductions to the field can be
found in [Law76, PaS82, NeW88, Wol98, Sch03]. It was the work in
[HWC74, Geo74, Fis81, Fis85], among others, in the 1970s and 1980s on
the traveling salesman problem and its relatives that made Lagrangian
relaxation and subgradient optimization popular, and it remains most
popular within the combinatorial optimization field.

6.9

Exercises

Exercise 6.1 (numerical example of Lagrangian relaxation) Consider the convex problem to
1
4
+
,
x1
x2
subject to x1 + x2 4,
minimize

x1 , x2 0.

(a) Lagrangian relax the first constraint, and write down the resulting
implicit dual objective function and the dual problem. Motivate why the

190

Exercises
relaxed problem always has a unique optimum, whence the dual objective
function is everywhere differentiable.
(b) Solve the implicit Lagrangian dual problem by utilizing that the gradient to a differentiable dual objective function can be expressed by using the
functions that are involved in the relaxed constraints and the unique solution
to the relaxed problem.
(c) Give an explicit dual problem (a dual problem only in terms of the
Lagrange multipliers). Solve it to confirm the results in (b).
(d) Find the original problems optimal solution.
(e) Show that strong duality holds.
Exercise 6.2 (global optimality conditions) Consider the problem to
minimize f (x) := x1 + 2x22 + 3x33 ,
subject to x1 + 2x2 + x3 3,
2x21 + x2 2,
2x1 + x3 = 2,

xj 0,

j = 1, 2, 3.

(a) Formulate the Lagrangian dual problem that results from Lagrangian
relaxing all but the sign constraints.
(b) State the global primaldual optimality conditions.
Exercise 6.3 (Lagrangian relaxation) Consider the problem to
minimize f (x) := x21 + 2x22 ,
subject to x1 + x2 2,
x21 + x22 5.

Find an optimal solution through Lagrangian duality.

Exercise 6.4 (Lagrangian relaxation) In many circumstances it is of interest
to calculate the Euclidean projection of a vector onto a subspace. Especially,
consider the problem to find the Euclidean projection of the vector y Rn
onto the null space of the matrix A Rmn , that is, to find an x Rn that
solves the problem to
1
ky xk2 ,
2
subject to Ax = 0m ,
minimize f (x) :=

where A is such that rank A = m.

The solution to this problem is classic: the projection is given by

x = y AT (AAT )1 Ay .

If we let P := I n AT (AAT )1 A, where I n Rnn is the unit matrix, be

the projection matrix, the formula is simply x = P y .

191

Lagrangian duality
Derive this formula by utilizing Lagrangian duality. Motivate every step
by showing that the necessary properties are fulfilled.
[Note: This exercise is similar to that in Example 5.51, but utilizes Lagrangian duality rather than the KKT conditions to derive the projection
formula.]
Exercise 6.5 (Lagrangian relaxation, exam 040823) Consider the following linear optimization problem:
minimize f (x, y) := x 0.5y,

subject to

x + y 1,

2x + y 2,
(x, y)T R2+ .

(a) Show that the problem satisfies Slaters constraint qualification. Derive
the Lagrangian dual problem corresponding to the Lagrangian relaxation of
the two linear inequality constraints, and show that its set of optimal solutions
is convex and bounded.
(b) Calculate the set of subgradients of the Lagrangian dual function at
the dual points (1/4, 1/3)T and (1, 0)T .
Exercise 6.6 (Lagrangian relaxation) Provide an explicit form of the Lagrangian
dual problem for the problem to
minimize
subject to

n
m X
X

xij ln xij

i=1 j=1
m
X
i=1
n
X

xij = bj ,

j = 1, . . . , n,

xij = ai ,

i = 1, . . . , m,

xij 0,

i = 1, . . . , m,

j=1

j = 1, . . . , n,

where ai > 0, bj > 0 for all i, j, and where the linear equalities are Lagrangian
relaxed.
Exercise 6.7 (Lagrangian relaxation) Given is the problem to
minimize
x

subject to

f (x) := 2x21 + x22 + x1 3x2 ,

x21

+ x2 8,
x1 [1, 3],
x2 [2, 5].

(6.75a)
(6.75b)
(6.75c)
(6.75d)

Lagrangian relax the constraint (6.75b) with a multiplier . Formulate

the Lagrangian dual problem and calculate the dual functions value at = 1,
= 2, and = 3. Within which interval lies the optimal value f ? Also, draw
the dual function.

192

Exercises
Exercise 6.8 (Lagrangian duality for integer problems) Consider the primal
problem to
minimize f (x),
subject to

g(x) 0m ,
x X,

where X Rn , f : Rn R, and g : Rn Rm . If the restrictions g(x) 0m

are complicating side constraints which are Lagrangian relaxed, we obtain the
Lagrangian dual problem to
maximize
q(),
m

0

where

q() := minimum {f (x) + T g (x)},

x X

 Rm .

(a) Suppose that the set X is finite (for example, consisting of a finite
number of integer vectors). Denote the elements of X by xp , p = 1, . . . , P .
Show that the dual objective function is piece-wise linear. How many linear
segments can it have, at most? Why is it not always built up by that many
segments?
[Note: This property holds regardless of any properties of f and g.]
(b) Illustrate the result in (a) on the linear 0/1 problem to find
z = maximum z = 5x1 + 8x2 + 7x3 + 9x4 ,
subject to
3x1 + 2x2 + 2x3 + 4x4 5,
2x1 + x2 + 2x3 + x4 = 3,
x1 , x2 , x3 , x4 {0, 1},
where the first constraint is considered complicating.
(c) Suppose that the function f and all components of g are linear, and
that the set X is a polytope (that is, a bounded polyhedron). Show that the
dual objective function is also in this case piece-wise linear. How many linear
pieces can it be built from, at most?
Exercise 6.9 (Lagrangian relaxation) Consider the problem to
minimize z = 2x1 + x2 ,
subject to
x1 + x2 5,
x1
4,
x2 4,
x1 , x2 0, integer.
Lagrangian relax the first constraint. Describe the Lagrangian function and
the dual problem. Calculate the Lagrangian dual function at these four points:
= 0, 1, 2, 3. Give the best lower and upper bounds on the optimal value of
the original problem that you have found.

193

Lagrangian duality
Exercise 6.10 (surrogate relaxation) Consider an optimization problem of the
form
minimize f (x),

subject to gi (x) 0,

i = 1, . . . , m,

(P )

x X,

where the functions f, gi : Rn R are continuous and the set X Rn is

closed and bounded. The problem is assumed to have an optimal solution, x .
Introduce parameters i 0, i = 1, . . . , m, and define
s() := minimum f (x),
subject to

T g (x) 0,
x X.

(S)

This problem therefore has exactly one explicit constraint.

(a) [weak duality] Show that x is a feasible solution to the problem (S) and
that s() f therefore always holds, that is, the problem (S) is a relaxation
of the original one. Motivate also why maximum0m s() f must hold.
Explain the potential usefulness of this result!
(b) [example] Consider the linear 0/1 problem
z = maximum z = 5x1 + 8x2 + 7x3 + 9x4 ,
subject to
3x1 + 2x2 + 3x3 + 3x4 6,
2x1 + 3x2 + 3x3 + 4x4 5,
2x1 + x2 + 2x3 + x4 = 3,
x1 , x2 , x3 , x4 {0, 1}.

(1)
(2)

Surrogate relax the constraints (1) and (2) with multipliers 1 , 2 0 and
= (1, 2)T . Calculate s(
).
formulate the problem (S). Let 
Consider again the original problem and Lagrangian relax the constraints
(1) and (2) with multipliers 1 , 2 0. Calculate the Lagrangian dual objec.
tive value at  = 
Compare the two results!
(c) [comparison with Lagrangian duality] Let  0m and
q() := minimum {f (x) + T g(x)}.
x X

Show that q() s(), and that

maximum
q() maximum
s() f
m
m

0

holds.

194

0

Part IV

Linear Programming

Linear programming:
An introduction

VII

Linear programming (LP) models, that is, the collection of optimization

models with linear objective functions and polyhedral feasible regions,
are very useful in practice. The reason for this is that many real world
problems can be described by LP models (even if several approximations
must be made first) and, perhaps as importantly, there exist efficient
algorithms for solving linear programs; the most famous of them is the
simplex method, which will be presented in Chapter 9. Often, LP models
deal with situations where a number of resources (materials, machines,
people, land, etc.) are available and are to be combined to yield several
products.
To introduce the concept of linear programming we use a simplified
manufacturing problem. In Section 7.1 we describe the problem. From
the problem description we develop an LP model in Section 7.2. It turns
out that the LP model only contains two variables. Hence it is possible to
solve the problem graphically, which is done in Section 7.3. In Section
7.4 we discuss what happens if the data of the problem is modified.
Namely, we see how the optimal solution changes if the supply of raw
material or the prices of the products are modified. Finally, in Section
7.5 we develop what we call the linear programming dual problem to the
manufacturing problem.

7.1

The manufacturing problem

A manufacturer produces two pieces of furniture: tables and chairs. The

production of the furniture requires the use of two different pieces of
raw material: large and small pieces. A table is assembled by putting
together two pieces of each, while a chair is assembled from one of the
larger pieces and two of the smaller pieces (see Figure 7.1).

Linear programming: An introduction

Small piece

Large piece

Chair, x2

Table, x1

Figure 7.1: Illustration of the manufacturing problem.

When determining the optimal production plan, the manufacturer

must take into account that only 6 large and 8 small pieces are available.
A table is sold for 1600 SEK, while a chair sells for 1000 SEK. Under
the assumption that all items produced can be sold, and that the raw
material has already been paid for, the problem is to determine the
production plan that maximizes the total income, within the limited
resources.

7.2

A linear programming model

In order to develop a linear programming model for the manufacturing

problem we introduce the following variables:
x1 = number of tables manufactured and sold,
x2 = number of chairs manufactured and sold,
z = total income.
The variable z is, strictly speaking, not a variable, but will be defined
by the variables x1 and x2 .
The income from each product is given by the price of the product
multiplied by the number of products sold. Hence, the total income is
z = 1600x1 + 1000x2 .

(7.1)

Given that we produce x1 tables and x2 chairs the required number

of large pieces is 2x1 + x2 and the required number of small peaces is
198

Graphical solution
2x1 + 2x2 . But only 6 large pieces and 8 small pieces are available, so
we must have that
2x1 + x2 6,
2x1 + 2x2 8.

(7.2)
(7.3)

Further, it is impossible to produce a negative number of chairs or tables,

which gives that
x1 , x2 0.

(7.4)

(Also, the number of chairs and tables produced must be integers, but
we will not take that into account here.)
Now the objective is to maximize the total income, so if we combine
the income function (7.1) and the constraints (7.2)(7.4) we get the
following linear programming model:
maximize
subject to

z = 1600x1 +1000x2 ,
2x1
2x1
x1 ,

7.3

(7.5)

+x2 6,
+2x2 8,
x2 0.

Graphical solution

The feasible region of the linear programming formulation (7.5) is shown

in Figure 7.2. The figure also includes lines corresponding to different
values of the cost function. For example, the line z = 0 = 1600x1 +
1000x2 passes through the origin, and the line z = 2600 = 1600x1 +
1000x2 passes through the point (1, 1)T . We see that the value of the
cost function increases as these lines move upward, and it follows that
the optimal solution is x = (2, 2)T and z = 5200. Observe that the
optimal solution is an extreme point, which is in accordance with Theorem 4.12. This fact will be very important in the development of the
simplex method in Chapter 9.

7.4

Sensitivity analysis

In this section we investigate how the optimal solution changes if the

data of the problem is changed. We consider three different changes
(made independently of each other), namely
1. an increase in the number of large pieces available;
199

Linear programming: An introduction

x2

x = (2, 2)T

x1

z = 5200
z = 2600
z=0
Figure 7.2: Graphical solution of the manufacturing problem.

2. an increase in the number of small pieces available; and

3. a decrease in the price of the tables.

7.4.1

An increase in the number of large pieces available

Assume that the number of large pieces available increases from 6 to 7.

Then the linear program becomes
maximize
subject to

z = 1600x1 +1000x2 ,
2x1
2x1
x1 ,

200

+x2 7,
+2x2 8,
x2 0.

Sensitivity analysis
The feasible region is shown in Figure 7.3.

x2
2x1 + 2x2 = 12
2x1 + 2x2 = 10
2x1 + x2 = 7
2x1 + x2 = 8

x1

z=0
Figure 7.3: An increase in the number of large and small pieces available.
We see that the optimal solution becomes (3, 1)T and z = 5800,
which means that an additional large piece increases the income by
5800 5200 = 600. Hence the shadow price of the large pieces is 600.
The figure also illustrates what happens if the number of large pieces is
8. Then the optimal solution becomes (4, 0)T and z = 6400. But what
happens if we increase the number of large pieces further? From the
figure it follows that the optimal solution will not change (since x2 0
must apply), so an increase larger than 2 in the number of large pieces
gives no further income. This illustrates that the validity of the shadow
price depends on the actual increment; exactly when the shadow price
is valid is investigated in Theorem 10.8 and Remark 10.9.

7.4.2

An increase in the number of small pieces available

Starting from the original setup, in the same manner as for the large
pieces it follows from Figure 7.3 that two additional small pieces give
the new optimal solution x = (1, 4)T and z = 5600, so the income per
201

Linear programming: An introduction

additional small piece is (5600 5200)/2 = 200. Hence the shadow price
of the small pieces is 200. However, no more than 4 small pieces are
worth this price, since x1 0 must apply.

7.4.3

A decrease in the price of the tables

Now assume that the price of tables is decreased from 1600 to 800. The
new linear program becomes
maximize z = 800x1 +1000x2 ,
subject to

2x1
2x1
x1 ,

+x2 6,
+2x2 8,

x2 0.

This new situation is illustrated in Figure 7.4, from which we see that
x2

x1

1600x1 + 1000x2 = 0

800x1 + 1000x2 = 0

Figure 7.4: A decrease in the price of the tables.

the optimal solution is (0, 4)T , that is, we will not produce any tables.
This is natural, since it takes the same number of small pieces to produce
202

The dual of the manufacturing problem

a table and a chair but the table requires one more large piece, and in
addition the price of a table is now lower than that of a chair.

7.5
7.5.1

The dual of the manufacturing problem

A competitor

Suppose that another manufacturer (let us call them Billy) produce book
shelves whose raw material is identical to those used for the tables and
chairs, that is, the small and large pieces. Billy wish to expand their
production, and are interested in acquiring the resources that our factory sits on. Let us ask ourselves two questions, which (as we shall see)
have identical answers: (1) what is the lowest bid (price) for the total
capacity at which we are willing to sell?; (2) what is the highest bid
(price) that Billy are prepared to offer for the resources? The answer to
those two questions is a measure of the wealth of the company in terms
of their resources.

7.5.2

A dual problem

To study the problem, we introduce the variables

y1 = the price which Billy offers for each large piece,
y2 = the price which Billy offers for each small piece,
w = the total bid which Billy offers.
In order to accept to sell our resources, it is reasonable to require
that the price offered is at least as high as the value that the resources
represent in our optimal production plan, as otherwise we would earn
more by using the resources ourselves. Consider, for example, the net
income on a table sold. It is 1600 SEK, and for that we use two large
and two small pieces. The bid would therefore clearly be too low unless
2y1 + 2y2 1600. The corresponding requirement for the chairs is that
y1 + 2y2 1000.
Billy is interested in minimizing the total bid, under the condition
that the offer is accepted. Observing that y1 and y2 are prices and
therefore non-negative, we have the following mathematical model for
Billys problem:
minimize
subject to

w = 6y1 +8y2 ,

(7.6)

2y1 +2y2 1600,

y1 +2y2 1000,
y1 ,

y2 0.

203

Linear programming: An introduction

This is usually called the dual problem of our production planning
problem (which would then be the primal problem).
The optimal solution to this problem is y = (600, 200)T. (Check
this!) The total offer is w = 5200.
Remark 7.1 (the linear programming dual) Observe that the dual problem (7.6) is in accordance with the Lagrangian duality theory of Section
6.2.4. The linear programming dual will be discussed further in Chapter
10.

7.5.3

Interpretations of the dual optimal solution

From the above we see that the dual optimal solution is identical to the
shadow prices for the resource (capacity) constraints. (This is indeed
a general conclusion in linear programming.) To motivate that this is
reasonable in our setting, we may consider Billy as a fictitious competitor
only, which we use together with the dual problem to measure the value
of our resources. This (fictitious) measure can be used to create internal
prices in a company in order to utilize limited resources as efficiently
as possible, especially if the resource is common to several independent
sub-units. The price that the dual optimal solution provides will then
be a price directive for the sub-units, that will make them utilize the
scarce resources in a manner which is optimal for the overall goal.
We note that the optimal value z = 5200 of the production agrees
with the total value w = 5200 of the resources in our company. (This
is also a general result in linear programming; see the Strong Duality
Theorem 10.6.) Billy will of course not pay more than what the resource
is worth, but can at the same time not offer less than the profit that our
company can make ourselves, since we would then not agree to sell. It
follows immediately that for each feasible production plan x and price
y, it holds that z w, since
z = 1600x1 + 1000x2 (2y1 + 2y2 )x1 + (y1 + 2y2 )x2
= y1 (2x1 + x2 ) + y2 (2x1 + 2x2 ) 6y1 + 8y2 = w,

where in the inequalities we utilize all the constraints of the primal and
dual problems. (Also this fact is general in linear programming; see the
Weak Duality Theorem 10.4.) So, each offer accepted (from our point
of view) must necessarily be an upper bound on our own possible profit,
and this upper bound is what Billy wish to minimize in the dual problem.

204

Linear programming
models

VIII

We begin this chapter with a presentation of the axioms underlying the

use of linear programming models and discuss the modelling process.
Then, in Section 8.2, the geometry of linear programming is studied. It
is shown that every linear program can be transformed into the standard
form which is the form that the simplex method requires. Further, we
introduce the concept of basic feasible solution and discuss its connection
to extreme points. A version of the Representation Theorem adapted to
the standard form is presented, and we show that if there exists an
optimal solution to a linear program in standard form, then there exists
an optimal solution among the basic feasible solutions. Finally, we define
the term adjacent extreme point and give an algebraic characterization
of adjacency which actually proves that the simplex method at each
iteration step moves from one extreme point to an adjacent one.

8.1

Linear programming modelling

Many real world situations can be modelled as linear programs. However, the applicability of a linear program requires certain axioms to be
fulfilled. Hence, often approximations of the real world problem must be
made prior to the formulation of a linear program. The axioms underlying the use of linear programming models are:
proportionality (linearity: no economiesofscale; no fixed costs);
additivity (no substitutetimeeffects);
divisibility (continuity); and
determinism (no randomness).

Linear programming models

George Dantzig presented the linear programming model and the
simplex method for solving it at an econometrics conference in Wisconsin
in the late 40s. The economist Hotelling stood up, devastatingly smiling,
and stated that But we all know the world is nonlinear. The young
graduate student George Dantzig could not respond, but was defended
by John von Neumann, who stood up and concluded that The speaker
titled his talk linear programming and carefully stated his axioms. If
you have an application that satisfies the axioms, well use it. If it does
not, then dont; he sat down, and Hotelling was silenced. (See Dantzigs
account of the early history of linear programming in [LRS91, pp. 19
31].)
Now if the problem considered (perhaps after approximations) fulfills the axioms above, then it can be formulated as a linear program.
However, in practical modelling situations we usually do not talk about
the axioms; they naturally appear when a linear program is formulated.
To formulate a real world problem as a linear program is an art in
itself, and unfortunately there is little theory to help in formulating the
problem in this way. The general approach can however be described by
two steps:
1. Prepare a list of all the decision variables in the problem. This list
must be complete in the sense that if an optimal solution providing
the values of each of the variables is obtained, then the decision
maker should be able to translate it into an optimum policy that
can be implemented.
2. Use the variables from step 1 to formulate all the constraints and
the objective function of the problem.
We illustrate the two-step modelling process by an example.
Example 8.1 (the transportation problem) In the transportation problem we have a set of nodes or locations called sources, which have a
commodity available for shipment, and another set of locations called
demand centers, or sinks, which require this commodity. The amount
of commodity available at each source and the amount required at each
demand center are specified, as well as the cost per unit of transporting
the commodity from each source to each demand center. The problem
is to determine the quantity to be transported from each source to each
demand center, so as to meet all the requirements at minimum total
shipping cost.
Consider the problem where the commodity is iron ore, the sources
are found at mines 1 and 2, where the ore is produced, and the demand
206

Linear programming modelling

Table 8.1: Unit cost of shipping ore from mine to steel plant (KSEK per
Mton).
Plant
Mine 1
Mine 2

1
9
14

2
16
29

3
28
19

centers are three steel plants. The unit costs of shipping ore from each
mine to each steel plant are given in Table 8.1.
Further, the amount of ore available at the mines and the Mtons of
ore required at each steel plant are given in the Tables 8.2 and 8.3.
Table 8.2: Amount of ore available at the mines (Mtons).
Mine 1
Mine 2

103
197

Table 8.3: Ore requirements at the steel plants (Mtons).

Plant 1
Plant 2
Plant 3

71
133
96

We use the two-step modelling process to formulate a linear programming model.

Step 1: The activities in the transportation model are to ship ore from
mine i to steel plant j for i = 1, 2 and j = 1, 2, 3. It is convenient
to represent the variables corresponding to the levels at which these
activities are carried out by double subscripted symbols. Hence, for
i = 1, 2 and j = 1, 2, 3, we introduce the following variables:
xij = amount of ore (in Mtons) shipped from mine i to steel plant j.
We also introduce a variable corresponding to the total cost of the
shipping:
z = total shipping cost.
207

Linear programming models

Step 2: The transportation problem considered is illustrated in Figure
8.1.
Plant 1
x11
Mine 1

x21

x12
x13

x22

Plant 2

Mine 2
x23
Plant 3
Figure 8.1: Illustration of the transportation problem.

The items in this problem are the ore at various locations. Consider
the ore at mine 1. According to Table 8.2 there are only 103 Mtons
of it available, and the amount of ore shipped out of mine 1, which
is x11 + x12 + x13 , cannot exceed the amount available, leading to the
constraint
x11 + x12 + x13 103.
Likewise, if we consider ore at mine 2 we get the constraint
x21 + x22 + x23 197.
Further, at steel plant 1 according to Table 8.3 there are at least 71
Mtons of ore required, so the amount of ore shipped to steel plant 1 has
to be greater than or equal to this amount, leading to the constraint
x11 + x21 71.
In the same manner, for the steel plants 2 and 3 we get
x12 + x22 133,
x13 + x23 96.

208

Linear programming modelling

Of course it is impossible to ship a negative amount of ore, yielding the
constraints
xij 0,

i = 1, 2,

j = 1, 2, 3.

From Table 8.1 follows that the total cost (in KSEK) of shipping is
z = 9x11 + 16x12 + 28x13 + 14x21 + 29x22 + 19x23 .
Finally, since the objective is to minimize the total cost we get the
following linear programming model:
minimize
subject to

z = 9x11 +16x12 +28x13 +14x21 +29x22 +19x23 ,

x11

+x12

+x13
x21

x11

+x22

71,
133,

+x21
x12

+x22
x13

x11 ,

+x23

x12 ,

x13 ,

x21 ,

x22 ,

103,
197,

+x23 96,
x23

0.

The transportation problem may be given in a compact general formulation. Assume that we have N sources and M demand centers. For
i = 1, . . . , N and j = 1, . . . , M , introduce the variables
xij = amount of commodity shipped from source i to demand center j,
and let
z = total shipping cost.
Further for i = 1, . . . , N and j = 1, . . . , M introduce the shipping costs
cij = unit cost of shipping commodity from source i to demand center j.
Also, let
si = amount of commodity available at source i, i = 1, . . . , N,
dj = amount of commodity required at demand center j, j = 1, . . . , M.
Consider source i. The amount of commodity available is given by
si , which gives the constraint
M
X
j=1

xij si .

209

Linear programming models

Similarly, the amount of commodity required at demand center j is given
by dj , leading to the constraint
N
X
i=1

xij dj .

It is impossible to ship a negative amount of commodity, which gives

xij 0,

i = 1, . . . , N,

j = 1, . . . , M.

Finally, the total cost for shipping is

z=

N X
M
X

cij xij ,

i=1 j=1

leading to the following linear program:

minimize

z=

N X
M
X

cij xij ,

i=1 j=1

subject to

M
X
j=1

N
X
i=1

xij si ,

i = 1, . . . , N,

xij dj ,

j = 1, . . . , M,

xij 0,

i = 1, . . . , N,

j = 1, . . . , M.

If, for some reason, it is impossible to transport any commodities from

a source i to a sink j, then we may either remove this variable altogether
from the model, or, more simply, give it the unit price cij = +.
Note, finally, that there
solution to the transportation
PN exists aPfeasible
M
problem if and only if i=1 si j=1 dj .

8.2

The geometry of linear programming

In Section 3.2 we studied the class of feasible sets in linear programming,

namely the sets of polyhedra; they are sets of the form
P := { x Rn | Ax b },
where A Rmn and b Rm . In particular, we proved the Representation Theorem 3.22 and promised that it would be important in the
210

The geometry of linear programming

development of the simplex method. In this section we consider polyhedra of the form
P := { x Rn | Ax = b;

x 0n },

(8.1)

where A Rmn , and b Rm is such that b 0m . The advantage

of this form is that the constraints (except for the non-negativity constraints) are equalities, which admits pivot operations to be carried out.
The simplex method uses pivot operations at each iteration step and
hence it is necessary that the polyhedron (that is, the feasible region)
is represented in the form (8.1). This is, however, not a restriction, as
we will see in Section 8.2.1, since every polyhedron can be transformed
into this form! We will use the term standard form when a polyhedron
is represented in the form (8.1). In Section 8.2.2 we introduce the concept of basic feasible solution and show that each basic feasible solution
corresponds to an extreme point. We also restate the Representation
Theorem 3.22 for polyhedra in standard form and prove that if there
exists an optimal solution to a linear program, then there exists an optimal solution among the extreme points. Finally, in Section 8.2.3, a
strong connection between basic feasible solutions and adjacent extreme
points is discussed. This connection shows that the simplex method at
each iteration step moves from an extreme point to an adjacent extreme
point.

8.2.1

Standard form

A linear programming problem in standard form is a problem of the form

minimize

z = cT x,

subject to

Ax = b,
x 0n ,

where A Rmn and b 0m . The purpose of this section is to show

that every linear program can be transformed into the standard form.
In order to do that we must
express the objective function in minimization form;
transform all the constraints into equality constraints with nonnegative right-hand sides; and
transform any unrestricted and non-positive variables into nonnegative ones.

211

Linear programming models

Objective function
Constant terms in the objective function will not change the set of optimal solutions and can therefore be eliminated. If the objective is to
maximize z = cT x,
then change the objective function so that the objective becomes
minimize z := z = cT x.
This change does not affect the set of feasible solutions to the problem
and the equation
[maximum value of z] = [minimum value of z]
can be used to get the maximum value of the original objective function.
Inequality constraints and negative right-hand sides
Consider the inequality constraint
aT x b,
where a Rn and b R. By introducing a non-negative slack variable
s this constraint can be written as
aT x + s = b,
s 0,

(8.2a)
(8.2b)

which has the desired form of an equation. Suppose that b < 0. By

multiplying both sides of (8.2a) by 1 the negativity in the right-hand
side is eliminated and we are done. Similarly, a constraint of the form
aT x b,
can be written as
aT x s = b,
s 0.
We call such variables s surplus variables.
Remark 8.2 (on the role of slack and surplus variables) Slack and surplus variables may appear to be only help variables, but they often have
a clear interpretation as decision variables. Consider, for example, the
212

The geometry of linear programming

model (7.5) of a furniture production problem. The two inequality constraints are associated with the capacity of production stemming from
the availability of raw material. Suppose then that we introduce slack
variables in these constraints, which leads to the equivalent problem to
maximize z = 1600x1 +1000x2,
subject to
2x1
+x2 +s1
= 6,
2x1
+2x2
+s2 = 8,
x1 ,
x2 , s1 , s2 0.
The new variables s1 and s2 have the following interpretation: the
value of si (i = 1, 2) is the level of inventory (or, remaining capacity of
raw material of type i) that will be left unused when the production plan
(x1 , x2 ) has been implemented. This interpretation makes it clear that
the values of s1 and s2 are clear consequences of our decision-making.
Surplus variables have a corresponding interpretation. In the case of
the
PNtransportation model of the previous section, a demand constraint
( i=1 xij dj , j = 1, . . . , M ) may be fulfilled with equality (in which
case the customer gets an amount exactly according to the demand) or
it is fulfilled with strict inequality (in which case the customer gets a
surplus of the product asked for).
Unrestricted and non-positive variables
Consider the linear program
minimize

z = cT x,

subject to

Ax = b,
x2 0,
xj 0,

j = 3, . . . , n,

which is assumed to be in standard form except for the unrestricted

variable x1 and the non-positive variable x2 . The x2 -variable can be
replaced by the non-negative variable x
2 := x2 . The x1 -variable can be
transformed into the difference of two non-negative variables. Namely,

introduce the variables x+

1 0 and x1 0 and let x1 = x1 x1 .
+

Substituting x1 with x1 x1 wherever it occurs transforms the problem

into standard form. The drawback of this method to handle unrestricted
variables is that often the resulting problem is numerically unstable.
However, there are other techniques to handle unrestricted variables that
overcome this problem; one of them is discussed in Exercise 8.5.
213

Linear programming models

Example 8.3 (standard form) Consider the linear program
maximize z = 9x1 7x2 +3y1 ,
subject to
3x1 +x2 y1 1,

(8.4)

4x1 x2 +2y1 3,
x1 , x2
0.

In order to transform the objective function into the minimization form,

let
z := z = 9x1 + 7x2 3y1 .
Further, by introducing the slack variable s1 and the surplus variable s2
the constraints can be transformed into an equality form by
3x1 +x2 y1 +s1

= 1,

4x1 x2 +2y1
s2 = 3,
x1 , x2 ,
s1 , s2 0.
Finally, by introducing the variables y1+ and y1 we can handle the unrestricted variable y1 by substituting it by y1+ y1 wherever it occurs.
We arrive at the standard form to
minimize
subject to

z = 9x1 +7x2 3y1+ +3y1 ,

3x1

4x1
x1 ,

+x2 y1+ +y1 +s1

x2 +2y1+ 2y1
s2
+

x2 , y1 , y1 , s1 , s2

(8.5)
= 1,
= 3,
0.

Clearly, an optimal solution (x1 , x2 , y1+ , y1 , s1 , s2 ) to (8.5) can be

transformed into an optimal solution (x1 , x2 , y1 ) to (8.4) by using the
substitution y1 = y1+ y1 .

8.2.2

Basic feasible solutions and the Representation

Theorem

In this section we introduce the concept of basic feasible solution and

show the equivalence between extreme point and basic feasible solution.
From this we can draw the conclusion that if there exists an optimal
solution then there exists an optimal solution among the basic feasible
solutions. This fact is crucial in the simplex method which searches for
an optimal solution among the basic feasible solutions.
214

The geometry of linear programming

Consider a linear program in standard form,
minimize

z = cT x,

subject to

Ax = b,
x 0n ,

(8.6)

where A Rmn with rank A = rank(A, b) = m (otherwise, we can

is a basic solution
always delete rows), n > m, and b 0m . A point x
of (8.6) if
, that is, A
1. the equality constraints are satisfied at x
x = b; and

2. the columns of A corresponding to the non-zero components of x

are linearly independent.
A basic solution that also satisfies the non-negativity constraints is
called a basic feasible solution, or, in short, a BFS.
Since rank A = m, we can solve the system Ax = b by selecting
m variables of x corresponding to m linearly independent columns of
A. Hence, we partition the columns of A into two parts: one with
n m columns of A corresponding to components of x that are set to 0;
these are called the non-basic variables and are denoted by the subvector
xN Rnm . The others represent the basic variables, and are denoted
by xB Rm . According to this partition,
x=

xB
xN

A = (B, N ),

which yields that

Ax = BxB + N xN = b.
Since xN = 0nm by construction, we get the basic solution
x=

xB
xN

 1 
B b
.
0nm

Further, if B 1 b 0m then x is a basic feasible solution.

Remark 8.4 (degenerate basic solution) If more than n m variables
are zero at a basic solution x, then the corresponding partition is not
unique. Such a basic solution is called degenerate.
215

Linear programming models

Example 8.5 (partitioning) Consider the linear program
minimize z = 4x1 +3x2 +7x3 2x4 ,
subject to
x1
x3
x1 x2
2x1
x1 ,

x2 ,

= 3,

2x4
= 1,
+x4 +x5 = 7,

x3 ,

x4 , x5 0.

The constraint matrix and the right-hand side vector are given by

1 0 1 0 0
3
A = 1 1 0 2 0 , b = 1 .
2 0
0
1 1
7
(a) The partition xB = (x2 , x3 , x4 )T , xN = (x1 , x5 )T ,

0 1 0
1 0
B = 1 0 2 , N = 1 0 ,
0
0
1
2 1

corresponds to the basic solution

x2
15
  x3  1  3

xB
B b

x=
=
=
2
x4 =
7 .
xN
0
x1
0
x5
0

This is, however, not a basic feasible solution (since x2 and x3 are negative).
(b) The partition, xB = (x1 , x2 , x5 )T , xN = (x3 , x4 )T ,

1 0 0
1 0
B = 1 1 0 , N = 0 2 ,
2 0 1
0
1
corresponds to the basic solution


x1
3
  x2  1  2


xB
B b

x=
=
x5
=
=
2

1 .
xN
0
x3
0
x4
0

This is clearly a basic feasible solution.

216

The geometry of linear programming

(c) The partition xB = (x2 , x4 , x5 )T , xN = (x1 , x3 )T ,

0
0 0
1 1
B = 1 2 0 , N = 1 0 ,
0
1 1
2 0

does not correspond to a basic solution since the system BxB = b is

infeasible.
(d) Finally, the partition xB = (x1 , x4 , x5 )T , xN = (x2 , x3 )T ,

1 0 0
0 1
B = 1 2 0 , N = 1 0 ,
2 1 1
0
0
corresponds to the basic feasible solution


x1
3
  x4  1  1


xB
B b

=
=
x=
2
x5 =
0 ,
xN
0
0
x2
x3
0

which is degenerate (since a basic variable, x5 , has value zero).

Remark 8.6 (partitioning) The above partitioning technique will be
used frequently in what follows and from now on when we say that
A = (B, N ) is a partition of A we will always mean that the columns
of A and the variables of x have been rearranged so that B corresponds
to the basic variables xB and N to the non-basic variables xN .
We are now ready to prove the equivalence between extreme point
and basic feasible solution.
Theorem 8.7 (equivalence between extreme point and BFS) Suppose
A Rmn with rank A = m, and b Rm . Then, a vector x Rn
is an extreme point of the set { x Rn | Ax = b; x 0n } 6= if and
only if it is a basic feasible solution.
Proof. Let x be a basic feasible solution with the corresponding partition A = (B, N ), where rank B = m (such a partition exists since
rank A = m). Then the equality subsystem (see Section 3.2.3) of
Ax = b,
x 0n
217

Linear programming models

is given by
BxB + N xN = b,
xN = 0nm
(if some of the basic variables equal zero we get additional rows of the
form xi = 0 but these will not affect the proof). Since rank B = m it
follows that


B
N
rank
= n.
0(nm)m I nm
The result then follows from Theorem 3.17.
Remark 8.8 (degenerate extreme point) An extreme point that corresponds to more than one BFS is said to be degenerate. This typically
occurs when we have redundant constraints.
We present a reformulation of the Representation Theorem 3.22 that
is adapted to the standard form. Consider the polyhedral cone C :=
{ x Rn | Ax = 0m ; x 0n }. From Theorem 3.28 it follows that C is
finitely generated, that is, there exist vectors d1 , . . . , dr Rn such that

(
)
r

X
1
r
i
n 
C = cone {d , . . . , d } := x R  x =
i d ; 1 , . . . , r 0 .

i=1

There are, of course, infinitely many ways to generate a certain polyhedral cone C. Assume that C = cone {d1 , . . . , dr }. If there exists a
vector di {d1 , . . . , dr } such that


di cone {d1 , . . . , dr } \ {di } ,

then di is not necessary in the description of C. If we similarly continue

to remove vectors from {d1 , . . . , dr }, one at a time, we end up with a set
generating C such that none of the vectors of the set can be written as
a non-negative linear combination of the others. Such a set is naturally
called the set of extreme directions of C (cf. Definition 3.11 of extreme
point).
Theorem 8.9 (Representation Theorem) Let P := { x Rn | Ax =
b; x 0n } and V := {v 1 , . . . , v k } be the set of extreme points of P .
Further, let C := { x Rn | Ax = 0m ; x 0n } and D := {d1 , . . . , dr }
be the set of extreme directions of C.
218

The geometry of linear programming

(a) P is nonempty if and only if V is nonempty (and finite).
(b) P is unbounded if and only if D is nonempty (and finite).
(c) Every x P can be represented as the sum of a convex combination of the points in V and a non-negative linear combination of the
points in D, that is,
x=

k
X

i v i +

i=1

r
X

j d j ,

j=1

for some 1 , . . . , k 0 such that

Pk

i=1

i = 1, and 1 , . . . , r 0.

We have arrived at the important result that if there exists an optimal solution to a linear program in standard form then there exists an
optimal solution among the basic feasible solutions.
Theorem 8.10 (existence and properties of optimal solutions) Let the
sets P , V , and D be defined as in Theorem 8.9 and consider the linear
program
minimize
subject to

z = cT x,
x P.

(8.7)

(a) This problem has a finite optimal solution if and only if P is

nonempty and z is lower bounded on P , that is, if P is nonempty and
cT dj 0 for all dj D.
(b) If the problem has a finite optimal solution, then there exists an
optimal solution among the extreme points.
Proof. (a) Let x P . Then it follows from Theorem 8.9 that
x=

k
X

i v i +

i=1

for some 1 , . . . , k 0 such that

Hence
cT x =

k
X
i=1

r
X

j d j ,

(8.8)

j=1

Pk

i=1

i cT v i +

i = 1, and 1 , . . . , r 0.

r
X

j cT dj .

(8.9)

j=1

As x varies over P , the value of z clearly corresponds to variations of

the weights i and j . The first term in the right-hand side of (8.9) is
Pk
bounded as i=1 i = 1. The second term is lower bounded as x varies
219

Linear programming models

over P if and only if cT dj 0 holds for all dj D, since otherwise we
could let j + for an index j with cT dj < 0, and get that z .
If cT dj 0 for all dj D, then it is clearly optimal to choose j = 0 for
j = 1, . . . , r. It remains to search for the optimal solution in the convex
hull of V .
(b) Assume that x P is an optimal solution and let x be represented
as in (8.8). From the above we have that we can choose 1 = = r =
0, so we can assume that
x=

k
X

i v i .

i=1

Further, let
a arg minimum cT v i .
i{1,...,k}

Then,
cT v a = cT v a

k
X

i =

i=1

k
X
i=1

i cT v a

k
X

i cT v i = cT x,

i=1

that is, the extreme point v is a global minimum.

Note that part (b) of the theorem implies that if there exists an
optimal solution to (8.7), then there exists an optimal solution with no
more than m positive variables. This interesting fact does not hold for
a general optimization problem.
Remark 8.11 The bounded case of Theorem 8.10 was already given in
Theorem 4.12.

8.2.3

Adjacent extreme points

Consider the polytope in Figure 8.2. Clearly, every point on the line
segment joining the extreme points x and u cannot be written as a convex combination of any pair of points that are not on this line segment.
However, this is not true for the points on the line segment between the
extreme points x and w. The extreme points x and u are said to be
adjacent (while x and w are not adjacent).
Definition 8.12 (adjacent extreme points) Two extreme points x and
u of a polyhedron P are adjacent if each point y on the line segment
between x and u has the property that if
y = v + (1 )w,
220

The geometry of linear programming

w (not adjacent to x)

u (adjacent to x)

x
Figure 8.2: Illustration of adjacent extreme points.

where (0, 1) and v, w P , then both v and w must be on the line

segment between x and u.
Now, consider the polyhedron in standard form,
P := { x Rn | Ax = b;

x 0n }.

Let u P be a basic feasible solution (and hence an extreme point of

P ) corresponding to the partition A = (B 1 , N 1 ), where rank B 1 = m,
that is,
  1 1 

uB 1
(B ) b
u=
=
.
uN 1
0nm
Further, let B 1 = (b1 , . . . , bm ) and N 1 = (n1 , . . . , nnm ) (that is, bi
Rm , i = 1, . . . , m, and nj Rm , j = 1, . . . , n m, are columns of A).
Construct a new partition (B 2 , N 2 ) by replacing one column of B 1 , say
b1 , with one column of N 1 , say n1 , that is,
B 2 = (n1 , b2 , . . . , bm ),
N 2 = (b1 , n2 , . . . , nnm ).
Assume that the partition (B 2 , N 2 ) corresponds to a basic feasible solution v (i.e., v is an extreme point), and that the two extreme points
u and v corresponding to (B 1 , N 1 ) and (B 2 , N 2 ), respectively, are not
equal. Then we have the following elegant result.
Proposition 8.13 (algebraic characterization of adjacency) Let u and v
be the extreme points that correspond to the partitions (B 1 , N 1 ) and
(B 2 , N 2 ) described above. Then u and v are adjacent BFSs.
221

Linear programming models

Proof. If the variables of v are ordered in the same way as the variables
of u, then the vectors must be of the form
u = (u1 , . . . , um , 0, 0, . . . , 0)T ,
v = (0, v2 , . . . , vm+1 , 0, . . . , 0)T .
Take a point x on the line segment between u and v, that is,
x = u + (1 )v
for some (0, 1). In order to prove the theorem we must show that
if x can be written as a convex combination of two feasible points, then
these points must be on the line segment between u and v. So assume
that
x = y 1 + (1 )y 2
for some feasible points y 1 and y 2 , and (0, 1). Then it follows that
y 1 and y 2 must be solutions to the system
y1 b1 + + ym bm + ym+1 n1 = b,
ym+2 = . . . = yn = 0,
y 0n ,
or, equivalently [by multiplying both sides of the first row by (B 1 )1 ],

 1 1 
ym+1 (B 1 )1 n1
(B ) b
,
y=
+
ym+1
0nm
0nm1
y 0n .

But this is in fact the line segment between u and v (if ym+1 = 0 then
y = u and if ym+1 = vm+1 then y = v). In other words, y 1 and y 2 are
on the line segment between u and v, and we are done.
Since the simplex method at each iteration performs exactly the
above replacement action the proposition actually shows that the simplex method at each non-degenerate iteration moves from one extreme
point to an adjacent.
Remark 8.14 Actually a converse of the implication in Proposition
8.13 also holds. Namely, if two extreme points u and v are adjacent,
then there exists a partition (B 1 , N 1 ) corresponding to u and a partition (B 2 , N 2 ) corresponding to v such that the columns of B 1 and B 2
are the same except for one. The proof is similar to that of Proposition
8.13.
222

Notes and further reading

8.3

Notes and further reading

The material in this chapter can be found in most books on linear programming, such as [Dan63, Chv83, Mur83, Sch86, DaT97, Pad99, Van01,
DaT03, DaM05].

8.4

Exercises

Exercise 8.1 (LP modelling) Let A Rmn and b Rm . Formulate the

following problems as P
linear programming problems.
(a) minimizex n Pm
i=1 |(Ax b)i | subject to maxi=1,...,n |xi | 1.
(b) minimizex n m
i=1 |(Ax b)i | + maxi=1,...,n |xi |.

R
R

Exercise 8.2 (LP modelling) Consider the sets V := {v1 , . . . , vk } Rn and

W := {w1 , . . . , wl } Rn . Formulate the following problems as linear programming problems.
(a) Construct, if possible, a hyperplane that separates the sets V and W ,
that is, find a Rn , with a 6= 0n , and b R such that

aT v b, for all v V ,
aT w b, for all w W .

(b) Construct, if possible, a sphere that separates the sets V and W , that
is, find a center xc Rn and a radius R 0 such that
kv xc k2 R,

v V,
kw x k2 R, for all w W .
for all

Exercise 8.3 (linear-fractional programming) Consider the linear-fractional program

minimize
subject to

f (x) := ( T x + )/(dT x + ),

(8.10)

Ax b,

where , d R , A R
, and b Rm . Further, assume that the polyhen
dron P := {x R | Ax b} is bounded and that dT x + > 0 for all x P .
Show that (8.10) can be solved by solving the linear program
n

mn

minimize

g(y, z) := T y + z,

subject to

Ay z b 0 ,
dT y + z = 1,

(8.11)

z 0.

[Hint: Suppose that y together with z are a solution to (8.11), and show
that z > 0 and that y /z is a solution to (8.10).]

223

Linear programming models

Exercise 8.4 (standard form) Transform the linear program
minimize

x1 5x2 7x3 ,

z=

5x1 2x2 +6x3 5,

subject to

3x1 +4x2 9x3 = 3,

7x1 +3x2 +5x3 9,

2,

x1

into standard form.

Exercise 8.5 (standard form) Consider the linear program
minimize
subject to

z = 5x1 +3x2 7x3 ,

2x1 +4x2 +6x3

= 11,

3x1 5x2 +3x3 +x4 = 11,

x1 ,

x2 ,

x4 0.

(a) Show how to transform this problem into standard form by eliminating
one constraint and the unrestricted variable x3 .
(b) Why cannot this technique be used to eliminate variables with nonnegativity restrictions?
Exercise 8.6 (basic feasible solutions) Suppose that a linear program includes
a free variable xj . When transforming this problem into standard form, xj is
replaced by

xj = x+
j xj ,

x+
j , xj 0.

Show that no basic feasible solution can include both x+

j and xj as non-zero
basic variables.

Exercise 8.7 (equivalent systems) Consider the system of equations

n
X

aij xj = bi ,

i = 1, . . . , m.

(8.12)

j=1

Show that this system is equivalent to the system

n
X
j=1
m X
n
X
i=1 j=1

224

aij xj bi ,
aij xj

m
X
i=1

i = 1, . . . , m,
bi .

(8.13a)
(8.13b)

The simplex method

IX

This chapter presents the simplex method for solving linear programs.
In Section 9.1 the algorithm is presented. First, we assume that a basic feasible solution is known at the start of the algorithm, and then
we describe what to do when a BFS is not known from the beginning.
In Section 9.2 we discuss termination characteristics of the algorithm.
It turns out that if all the BFSs of the problem are non-degenerate,
then the algorithm terminates. However, if there exist degenerate BFSs
then there is a possibility that the algorithm cycles between degenerate
BFSs and hence never terminates. Fortunately, the simple Blands rule
eliminates cycling, and which we describe. We close the chapter by discussing the computational complexity of the simplex algorithm. In the
worst case, the algorithm visits all the extreme points of the problem,
and since the number of extreme points may be exponential in the dimension of the problem, the simplex algorithm does not belong to the
desirable polynomial complexity class. The simplex method is therefore
not theoretically satisfactory, but in practice it works very well and thus
it frequently appears in commercial linear programming codes.

9.1

The algorithm

Assume that we have a linear program in standard form:

minimize

z = cT x,

subject to

Ax = b,
x 0n ,

where A Rmn , n > m, rank A = m, b 0m , and c Rn . (This

is not a restriction, as was shown in Section 8.2.1.) At each iteration

The simplex method

the simplex algorithm starts at the current basic feasible solution (BFS)
and moves to an adjacent BFS such that the objective function value
decreases. It terminates with an optimal BFS (if there exists a finite
optimal solution), or a direction of unboundedness, that is, a point in
C := { p Rn | Ap = 0m ; p 0n } along which the objective function
diverges to . (Observe that if p C is a direction of unboundedness
is a feasible solution, then every solution y() of the form
and x
+ p,
y() := x

0,

is feasible. Hence if cT p < 0 then z = cT y() as +.)

9.1.1

A BFS is known

T T
Assume that a basic feasible solution x = (xT
B , xN ) corresponding to
the partition A = (B, N ) is known. Then we have that
 
xB
= BxB + N xN = b,
Ax = (B, N )
xN

or, equivalently,
xB = B 1 b B 1 N xN .

(9.1)

T T
Further, rearrange the components of c such that c = (cT
B , cN ) has the
T
T T
same ordering as x = (xB , xN ) . Then from (9.1) it follows that
T
cT x = cT
B xB + cN xN
1
= cT
b B 1 N xN ) + cT
B (B
N xN

1
T 1
= cT
b + (cT
N )xN .
BB
N cB B

(9.2)

The principle of the simplex algorithm is now easy to describe. At

the start of an iteration we are located at the BFS given by
   1 
xB
B b
x=
=
,
xN
0nm
which is an extreme point according to Theorem 8.7. Proposition 8.13
implies that if we construct a new partition by replacing one column of B
by one column of N such that the new partition corresponds to a basic
, not equal to x, then x
is adjacent to x. The principle
feasible solution, x
of the simplex algorithm is to move to an adjacent extreme point such
that the objective function value decreases. From (9.2) follows that if
226

The algorithm
we increase the j th component of the non-basic vector xN from 0 to 1,
then the change in the objective function value becomes
T 1
(
cN )j := (cT
N )j ,
N cB B

that is, the change in the objective function value resulting from a unit
increase of the non-basic variable (xN )j from zero is given by the j th
T
T 1
T
component of the vector c
N.
N := cN cB B
We call (
cN )j the reduced cost of the non-basic variable (xN )j for j =
T
= (
T
1, . . . , n m. Actually, we can define the reduced cost, c
cT
B, c
N) ,
of all the variables at the given BFS by
1
T 1
T
T := cT cT
A = (cT
(B, N )
c
B , cN ) cB B
BB
T 1
= ((0m )T , cT
N );
N cB B

B = 0m .
note that the reduced costs of the basic variables are c
If (
cN )j 0 for all j = 1, . . . , n m, then there exists no adjacent
extreme point such that the objective function value decreases and we
stop; x is then an optimal solution.
Proposition 9.1 (optimality in the simplex method) Let x be the basic feasible solution that corresponds to the partition A = (B, N ). If
(
cN )j 0 for all j = 1, . . . , n m, then x is an optimal solution.

1
Proof. Since cT
b is constant, it follows from (9.2) that the original
BB
linear program is equivalent to

minimize

T
c
N xN

z=

subject to

xB +B 1 N xN = B 1 b,
xB
0m ,
xN 0nm ,

or, equivalently [by reducing the xB variables through (9.1)],

minimize
subject to

cT
N xN

z=
B

(9.3)

N xN B

b,

nm

xN 0

Since x is a BFS it follows that xN := 0nm is feasible in (9.3). But

N 0nm so xN = 0nm is in fact optimal in (9.3). (Why?) Hence
c
 1 
B b
x =
0nm
is an optimal solution to the original problem.

227

The simplex method

Remark 9.2 (optimality condition) Proposition 9.1 states that if (
cN )j
0 for all j = 1, . . . , n m, then x is an optimal extreme point. But
is it true that if x is an optimal extreme point, then (
cN )j 0 for all
j = 1, . . . , n m? The answer to this question is no: if the optimal basic
feasible solution x is degenerate, then there may exist basis representations of x such that (
cN )j < 0 for some j. However, it holds that if x
is an optimal extreme point, then there exists at least one basis representation of x such that (
cN )j 0 for all j = 1, . . . , n m. Proposition
9.1 can hence be strengthened: x is an optimal extreme point if and
only if there exists a basis representation of it such that cN 0nm .
If some of the reduced costs are negative, then we choose a non-basic
variable with the most negative reduced cost to enter the basis. We must
also choose one variable from xB to leave the basis. Suppose that the
variable (xN )j has been chosen to enter the basis. Then, according to
(9.1), when the value of (xN )j is increased from zero we will move along
the half-line (or, ray)
 1 


B b
B 1 N j
+
,
0,
l() :=
0nm
ej
where ej is the j th unit vector. In order to maintain feasibility we must
have that l() 0n . If l() 0n for all 0, then z as
+, that is,


B 1 N j
p=
ej
is a direction of unboundedness and z along the half-line l(),
0. Observe that this occurs if and only if
B 1 N j 0m .
Otherwise, the maximal value of in order to maintain feasibility is
=

minimum
1

i{ k | (B

N j )k >0 }

(B 1 b)i
.
(B 1 N j )i

If > 0 it follows that l( ) is an extreme point adjacent to x. Actually

we move to l( ) by choosing the outgoing variable (xB )i , where
i arg

minimum

i{ k | (B 1 N j )k >0 }

(B 1 b)i
,
(B 1 N j )i

to leave the basis.

We are now ready to state the simplex algorithm.
228

The algorithm
The Simplex Algorithm:
T T
Step 0 (initialization: BFS). Let x = (xT
B , xN ) be a BFS corresponding to the partition A = (B, N ).

Step 1 (descent direction or termination: entering variable, pricing). Calculate the reduced costs of the non-basic variables:
T 1
(
cN )j := (cT
N )j ,
N cB B

j = 1, . . . , n m.

If (
cN )j 0 for all j = 1, . . . , n m then stop; x is then optimal.
Otherwise choose (xN )j , where
j arg minimum (
cN )j ,
j{1,...,nm}

to enter the basis.

Step 2 (line search or termination: leaving variable). If
B 1 N j 0m ,
T
then the problem is unbounded, stop; p := ((B 1 N j )T , eT
j ) is
then a direction of unboundedness. Otherwise choose (xB )i , where

i arg

minimum

i{ k | (B 1 N j )k >0 }

(B 1 b)i
,
(B 1 N j )i

to leave the basis.

Step 3 (update: change basis). Construct a new partition by swapping
(xB )i with (xN )j . Go to Step 1.
Remark 9.3 (the simplex algorithm as a feasible descent method) In the
above description, we have chosen to use terms similar to those that
will be used for several descent methods in nonlinear optimization that
are described in Part V; see, for example, the algorithm description in
Section 11.1 for unconstrained nonlinear optimization problems. The
simplex method is a very special type of descent algorithm: in order
to remain feasible we generate feasible descent directions p (Step 1)
that follow the boundary of the polyhedron; because of the fact that
the objective function is linear, a line search would yield an infinite
step unless a new boundary makes such a step infeasible; this is the
role of Step 2. Finally, termination at an optimal solution (Step 1) is
based on a special property of linear programming which allows us to
decide on global optimality based only on local information (that is, the
current BFSs reduced costs). (The convexity of LP is a crucial property
for this principle to be valid.) More on the characterization of this
optimality criterion, and its relationships to the optimality principles in
the Chapters 46 will be discussed in the next chapter.
229

The simplex method

Remark 9.4 (calculating the reduced costs) When calculating the reduced costs of the non-basic variables at the pricing Step 1 of the simplex
algorithm, it is appropriate to first calculate
1
y T := cT
BB

through the system

B T y = cB ,

and then calculate the reduced costs by

T
T
T
c
N = cN y N .

By this procedure we avoid the matrixmatrix multiplication B 1 N .

Remark 9.5 (alternative pricing rules) If n is very large, it can be costly
to compute the reduced costs at the pricing Step 1 of the simplex algorithm. A methodology which saves computations is partial pricing, in
which only a subset of the elements (
cN )j is calculated.
Another problem with the standard pricing rule is that the use of the
criterion to minimizej{1,...,nm} {(
cN )j } does not take into account the
actual improvement that is made. In particular, a different scaling of
the variables might mean that a unit change is a dramatic move in one
variable, and a very small move in another. The steepest-edge rule eliminates this scaling problem somewhat: With (xN )j being the entering
variable we have that
 new
 


xB
xB
B 1 N j
:=
+ (xN )j pj ,
pj =
.
xN
xN
ej
Choose j in
arg minimum

j{1,...,nm}

c T pj
,
kpj k

1
that is, the usual pricing rule based on cT pj = cT
N j ) + (cN )j =
B (B
(
cN )j is replaced by a rule wherein the reduced costs are scaled by the
length of the candidate search directions pj . (Other scaling factors can
of course be used.)

Remark 9.6 (initial basic feasible solution) Consider the linear program
minimize
subject to

z = cT x,
Ax b,

x 0n ,

230

(9.4)

The algorithm
where A Rmn , b 0m , and c Rn . By introducing slack variables
s Rm we get
minimize

z = cT x,

(9.5)
m

subject to

Ax +I s = b,
x
0n ,

s 0m .

Since b 0m it then follows that the partition (I m , A) corresponds to a

basic feasible solution to (9.5), that is, the slack variables s are the basic
variables. (This corresponds to the origin in the problem (9.4), which is
clearly feasible when b 0m .)
Similarly, if we can identify an identity matrix among the columns of
the constraint matrix, then (if the right-hand side is non-negative, which
is the case if the problem is in standard form) we obtain a BFS by taking
the variables that correspond to these columns as basic variables.
Example 9.7 (the simplex method) Consider the linear program
minimize z =
subject to

2x2 4x3 +4x4 ,

x2 +2x3
+x4 4,
2x1
+x2
+x3 4x4 5,
x1
x2
+2x4 3,
x1 ,
x2 ,
x3 ,
x4 0.
x1

By introducing the slack variables x5 , x6 and x7 we get the problem to

minimize z =
subject to

2x2 4x3 +4x4 ,

x2 +2x3
+x4 +x5
= 4,
2x1
+x2
+x3 4x4
+x6
= 5,
x1
x2
+2x4
+x7 = 3,
x1 ,
x2 ,
x3 ,
x4 , x5 ,
x6 ,
x7 0.
x1

According to Remark 9.6 we can take xB = (x5 , x6 , x7 )T and xN =

(x1 , x2 , x3 , x4 )T as the initial basic and non-basic vector, respectively.
The reduced costs of the non-basic variables then become
T 1
cT
N = (1, 2, 4, 4),
N cB B

and hence we choose x3 as the entering variable. Further, from

B 1 b = (4, 5, 3)T ,
B 1 N 3 = (2, 1, 0)T ,
231

The simplex method

arg

minimum
1

i{ k | (B

N 3 )k >0 }

(B 1 b)i
= {1},
(B 1 N 3 )i

so we choose x5 to leave the basis. The new basic and non-basic vectors
are xB = (x3 , x6 , x7 )T and xN = (x1 , x2 , x5 , x4 )T , and the reduced costs
of the non-basic variables become
T 1
cT
N = (1, 4, 2, 6),
N cB B

so x2 is the entering variable, and from

B 1 b = (2, 3, 3)T ,
B 1 N 2 = (1/2, 3/2, 1)T,
arg

minimum
1

i{ k | (B

N 2 )k >0 }

(B 1 b)i
= {2},
(B 1 N 2 )i

and hence x6 is the leaving variable. The new basic and non-basic vectors
become xB = (x3 , x2 , x7 )T and xN = (x1 , x6 , x5 , x4 )T , and the reduced
costs of the non-basic variables are
T 1
cT
N = (13/3, 8/3, 2/3, 6),
N cB B

so x4 is the entering variable and

B 1 b = (3, 2, 5)T ,
B 1 N 4 = (1, 3, 1)T .
But since B 1 N 4 03 it follows that the objective function diverges to
along the half-line given by
l() = (x1 , x2 , x3 , x4 )T = (0, 2, 3, 0)T + (0, 3, 1, 1)T,

0.

We conclude that the problem is unbounded.

9.1.2

A BFS is not known: phase I & II

Often a basic feasible solution is not known initially. (In fact, only if
the origin is feasible in (9.4) we know a BFS immediately.) However, an
initial basic feasible solution can be found by solving a linear program
that is a pure feasibility problem. We call this the phase I problem.
Consider the following linear program in standard form:
minimize
subject to

z = cT x,
Ax = b,
x 0n .

232

(9.6)

The algorithm
In order to find a basic feasible solution we introduce the artificial variables a Rm and consider the phase I problem to
minimize w =
(1m )T a,
subject to
Ax +I m a = b,
x

(9.7)

0n ,
a 0m .

In other words, we introduce an additional (artificial) variable ai for

every linear constraint i = 1, . . . , m, and thus construct the unit matrix
in Rmm sought.
We obtain a BFS to the phase I problem (9.7) by taking the artificial
variables a as the basic variables. (Remember that b 0m ; the simplicity of finding an initial BFS for the phase I problem is in fact the reason
why we require this to hold!) Then the phase I problem (9.7) can be
solved by the simplex method stated in the previous section. Note that
the phase I problem is bounded from below [(1m )T a 0] which means
that an optimal solution to (9.7) always exists by Theorem 8.10.
Assume that the optimal objective function value is w . We observe
that if and only if the part x of an optimal solution ((x )T , (a )T )T
to the problem (9.7) is a feasible solution to the original problem (9.6),
then ((x )T , (0m )T )T is an optimal solution to the phase I problem and
w = 0. Hence, if w > 0, then the original linear program is infeasible.
We have the following cases:
1. If w > 0, then the original problem is infeasible.
2. If w = 0, then if the optimal basic feasible solution is (xT , aT )T
we must have that a = 0m , and x corresponds to a basic feasible
solution to the original problem.1
Therefore, if there exists a feasible solution to the original problem
(9.6), then a BFS is found by solving the phase I problem (9.7). This
BFS can then be used as the starting BFS when solving the original
problem, which is called the phase II problem, with the simplex method.
Remark 9.8 (artificial variables) The purpose of introducing artificial
variables is to get an identity matrix among the columns of the constraint
matrix. If some of the columns of the constraint matrix of the original
problem consists of only zeros except for one positive entry, then it is not
1 Notice that if the final BFS in the phase I problem is degenerate then one or
several artificial variables ai may remain in the basis with value zero; in order to
remove them from the basis a number of degenerate pivots may have to be performed;
this is naturally always possible.

233

The simplex method

necessary to introduce an artificial variable in the corresponding row. An
example of a linear constraint for which an original variable naturally
serves as a basic variable is a -constraint with a positive right-hand
side, in which case we can use the corresponding slack variable.
Example 9.9 (phase I & II) Consider the following linear program:
minimize
subject to

z = 2x1 ,
x1
x3
= 3,
x1 x2
2x4 = 1,
2x1
+x4 7,
x1 , x2 , x3 ,
x4 0.

By introducing a slack variable x5 we get the equivalent linear program in standard form:
minimize
subject to

z = 2x1 ,
x1
x3
x1 x2
2x4

(9.8)
= 3,
= 1,

2x1
+x4 +x5 = 7,
x1 , x2 , x3 , x4 , x5 0.
We cannot identify the identity matrix among the columns of the
constraint matrix of the problem (9.8), but the third unit vector e3 is
found in the column corresponding to the x5 -variable. Therefore, we
leave the problem (9.8) for a while, and instead introduce two artificial
variables a1 and a2 and consider the phase I problem to
minimize
subject to

w=
x1
x3
x1 x2
2x4
2x1
+x4 +x5
x1 , x2 ,
x3 ,
x4 ,
x5 ,

a1
+a1

+a2

= 3,
+a2 = 1,
= 7,
a1 ,
a2 0.

Let xB = (a1 , a2 , x5 )T and xN = (x1 , x2 , x3 , x4 )T be the initial basic

and non-basic vector, respectively. The reduced costs of the non-basic
variables then become
T 1
cT
N = (2, 1, 1, 2),
N cB B

and hence we choose x1 as the entering variable. Further, from

B 1 b = (3, 1, 7)T ,
B 1 N 1 = (1, 1, 2)T ,
234

The algorithm
(B 1 b)i
= {2},
1
i{ k | (B N 1 )k >0 } (B
N 1 )i
so we choose a2 as the leaving variable. The new basic and non-basic vectors are xB = (a1 , x1 , x5 )T and xN = (a2 , x2 , x3 , x4 )T , and the reduced
costs of the non-basic variables become
arg

minimum
1

T 1
cT
N = (2, 1, 1, 2),
N cB B

so x4 is the entering variable, and from

B 1 b = (2, 1, 5)T ,
B 1 N 4 = (2, 2, 5)T ,
(B 1 b)i
= {1, 3},
1
i{ k | (B 1 N 4 )k >0 } (B
N 4 )i
and we choose a1 to leave the basis. The new basic and non-basic vectors
become xB = (x4 , x1 , x5 )T and xN = (a2 , x2 , x3 , a1 )T , and the reduced
costs of the non-basic variables are
arg

minimum

T 1
cT
N = (1, 0, 0, 1),
N cB B

so by choosing the basic variables as xB = (x4 , x1 , x5 )T we get an optimal

basic feasible solution of the phase I problem, and w = 0. This means
that by choosing the basic variables as xB = (x4 , x1 , x5 )T we get a basic
feasible solution of the phase II problem (9.8).
We return to the problem (9.8). By letting xB = (x4 , x1 , x5 )T and
xN = (x2 , x3 )T the reduced costs are
T 1
T

N = (0, 2),
cT
N = cN cB B

which means that

x4
1
  x1  1  3


xB
B b

x5
=
=
=
x=
2

0
xN
0
x2
0
x3
0

is an optimal basic feasible solution to the original problem. (Observe

that the BFS found when solving the phase I problem typically is not
an optimal solution to the phase II problem!) But since the reduced
cost of x2 is zero there is a possibility that there are alternative optimal
solutions. Let x2 enter the basic vector. From
B 1 b = (1, 3, 0)T ,
B 1 N 1 = (0.5, 0, 0.5)T ,
235

The simplex method

arg

minimum
1

i{ k | (B

N 1 )k >0 }

(B 1 b)i
= {1},
(B 1 N 1 )i

so x4 is the leaving variable. We get xB = (x2 , x1 , x5 )T and xN =

(x4 , x3 )T , and since the reduced costs become
T
T 1

cT
N = (0, 2),
N = cN cB B

2
x2
  x1  1  3


xB
B b

x5
=
=
x=
=
2

1
xN
0
x4
0
x3
0
is an alternative optimal basic feasible solution.

9.1.3

Alternative optimal solutions

As we saw in Example 9.9 there can be alternative optimal solutions to

a linear program. However, this can only happen if some of the reduced
costs of the non-basic variables of an optimal solution is zero.
Proposition 9.10 (unique optimal solutions in linear programming) Consider the linear program in standard form
minimize

z = cT x,

subject to

Ax = b,
x 0n .

T T
Let x = (xT
B , xN ) be an optimal basic feasible solution that corresponds
to the partition A = (B, N ). If the reduced costs of the non-basic variables xN are all strictly positive, then x is the unique optimal solution.

Proof. As in the proof of Proposition 9.1 we have that the original

linear program is equivalent to
minimize
subject to

T
c
N xN

z=

xB +B 1 N xN = B 1 b,
xB

0m ,

xN 0nm .
236

Termination
Now if the reduced costs of the non-basic variables are all strictly positive, that is,
cN > 0nm , it follows that a solution for which (xN )j > 0
for some j = 1, . . . , n m cannot be optimal. Hence
   1 
xB
B b
x=
=
xN
0nm
is the unique optimal solution.

9.2

Termination

So far we have not discussed whether the simplex algorithm terminates

in a finite number of iterations. Unfortunately, if there exist degenerate
BFSs it can happen that the simplex algorithm cycles between degenerate solutions and hence never terminates. However, if all of the BFSs
are non-degenerate this kind of cycling never occurs.
Theorem 9.11 (finiteness of the simplex algorithm) If all of the basic
feasible solutions are non-degenerate, then the simplex algorithm terminates after a finite number of iterations.
Proof. If a basic feasible solution is non-degenerate then it follows that
it has exactly m positive components, and hence has a unique associated
basis. In this case, in the minimum ratio test,
=

minimum
1

i{ k | (B

N j )k >0 }

(B 1 b)i
> 0.
(B 1 N j )i

Therefore, at each iteration the objective value decreases, and hence a

basic feasible solution that has appeared once can never reappear. Further, from Corollary 3.18 follows that the number of extreme points,
hence the number of basic feasible solutions, is finite. We are done.
Cycling resulting from degeneracy does not seem to occur often among
the numerous degenerate linear programs encountered in practical applications. However, the fact that it can occur is not theoretically satisfactory. Therefore, methods have been developed that avoid cycling.
One of them is Blands rule.
Theorem 9.12 (Blands rule) Fix an ordering of the variables. (This
ordering can be arbitrary, but once it has been selected it cannot be
changed.) If at each iteration step the entering and leaving variables are
237

The simplex method

chosen as the first variables that are eligible2 in the ordering, then the
simplex algorithm terminates after a finite number of iteration steps.

9.3

Computational complexity

The simplex algorithm is very efficient in practice. Although the total

number of basic feasible solutions can be as many as
 
n!
n
=
m
(n m)! m!
(the number of different ways m objects can be chosen from n objects),
which grows exponentially, it is rare that more than 3m iterations are
needed, and in practice the expected number is in the order of 3m/2.
Since each iteration costs no more than a polynomial (O(m3 ) for factorizations and O(mn) for the pricing) the algorithm is polynomial in
practice. Its worst-case behaviour is however bad, in fact exponential.
The bad worst-case behaviour of the simplex method led to a huge
amount of work being laid down to find polynomial algorithms for solving linear programs. Such a polynomial time competitor to the simplex
method nowadays is the class of interior point algorithms. Its main feature is that the optimal extreme points are not approached by following
the edges, but by moving within the interior of the polyhedron. The
famous Karmarkar algorithm is one, which however has been improved
much in recent years. An analysis of interior point methods for linear programs is made in Chapter 13, as they are in fact to be seen as
instances of the interior penalty algorithm in nonlinear programming.

9.4

Notes and further reading

The simplex method was developed by George Dantzig [Dan51]. The version of the simplex method presented is usually called the revised simplex
method, and was first described by Dantzig [Dan53] and Orchard-Hays
[Orc54]. The first book describing the simplex method was [Dan63].
In the (revised) simplex algorithm several computations are performed using B 1 . The major drawback in this approach is that roundoff
2 By eligible entering variables we mean the variables (x ) for which (
N )j < 0,
N j
and when we have chosen the entering variable j, the eligible leaving variables are
the variables (xB )i such that

i arg

238

minimum

i{ k | (B 1 N j )k >0 }

(B 1 b)i
.
(B 1 N j )i

Exercises
errors accumulate as the algorithm proceeds. This drawback can however be alleviated by using stable forms of LU decomposition or Cholesky
factorization. Most of the software packages for linear programming use
LU decomposition. Early references on numerically stable forms of the
simplex method are [BaG69, Bar71, Sau72, GiM73]. Books that discuss
the subject are [Mur83, NaS96].
The first example of cycling of the simplex algorithm was constructed
by Hoffman [Hof53]. Several methods have been developed for avoiding
cycling, such as the perturbation method of Charnes [Cha52], the lexicographic method of Dantzig, Orden and Wolfe [DOW55], and Blands rule
[Bla77]. In practice, however, cycling is rarely encountered. Instead, the
problem is stalling, which means that the value of the objective function
does not change (or changes very little) for a very large number of iterations3 before it eventually starts to make substantial progress again. So
in practice, we are interested in methods that primarily prevent stalling,
and only secondarily avoid cycling (see, e.g., [GMSW89]).
In 1972, Klee and Minty [KlM72] showed that there exist problems
of arbitrary size that cause the simplex method to examine every possible basis when the standard (steepest-descent) pricing rule is used, and
hence showed that the simplex method is an exponential algorithm in
the worst case. It is still an open question, however, whether there exists
a rule for choosing entering and leaving basic variables that makes the
simplex method polynomial. The first polynomial-time method for linear
programming was given by Khachiyan [Kha79, Kha80], by adapting the
ellipsoid method for nonlinear programming of Shor [Sho77] and Yudin
and Nemirovskii [YuN77]. Karmarkar [Kar84a, Kar84b] showed that interior point methods can be used in order to solve linear programming
problems in polynomial time.
General text books that discuss the simplex method are [Dan63,
Chv83, Mur83, Sch86, DaT97, Pad99, Van01, DaT03, DaM05].

9.5

Exercises

Exercise 9.1 (checking feasibility: phase I) Consider the system

3x1 +2x2 x3 3,

x1 x2 +2x3 1,
x1 ,

x2 ,

x3 0.

Show that this system is infeasible.

3 Very

large normally refers to a number of iterations which is an exponential

function of the number of variables of the LP problem.

239

The simplex method

Exercise 9.2 (the simplex algorithm: phase I & II) Consider the linear program
minimize

z = 3x1 +2x2 +x3 ,

subject to

+x3 3,

2x1

2x1 +2x2 +x3 = 5,

x1 ,

x2 , x3 0.

(a) Solve this problem by using the simplex algorithm with phase I & II.
(b) Is the optimal solution obtained unique?
Exercise 9.3 (the simplex algorithm) Consider the linear program
z = T x,

minimize

Ax = b,
x 0n .

subject to

Suppose that at a given step of the simplex algorithm, there is only one
possible entering variable, (xN )j . Also assume that the current BFS is nondegenerate. Show that (xN )j > 0 in any optimal solution.
Exercise 9.4 (cycling of the simplex algorithm) Consider the linear program
minimize

2
x5
5
3
+ x5
5
1
+ x5
5
2
+ x5
5

z=

subject to

x1
x2
x3
x4

x1 , x2 , x3 , x4 ,

x5 ,

2
x6
5
32
x6
5
9
x6
5
8
x6
5
+x6

9
+ x7 ,
5
24
+ x7 = 0,
5
3
+ x7 = 0,
5
1
+ x7 = 0,
5
= 1,

x6 ,

x7 0.

In solving this problem by the simplex algorithm starting at the BFS xB :=

(x1 , x2 , x3 , x4 )T , in each iteration step select the entering variable as xs , where
j < 0 },
s := minimum { j |
and select the pivot row as the r th row, where
r := minimum { i | row i is eligible }.
Show that cycling occurs.

240

Linear programming
duality and sensitivity
analysis

10.1

Introduction

Consider the linear program

minimize
subject to

z = cT x,
Ax = b,

(10.1)

x 0n ,
where A Rmn , b Rm , and c Rn , and assume that this problem
T T
has been solved by the simplex algorithm. Let x = (xT
be
B , xN )
an optimal basic feasible solution corresponding to the partition A =
(B, N ). Introduce the vector y Rm through
1
(y )T := cT
.
BB

Since x is an optimal solution it follows that the reduced costs of

the non-basic variables are greater than or equal to zero, that is,
T 1
cT
N (0nm )T
N cB B

T
nm T
cT
) .
N (y ) N (0

T
T
T 1
Further, cT
B = (0m )T , so we have that
B (y ) B = cB cB B

cT (y )T A (0n )T ,
or equivalently,
AT y c.

LP duality and sensitivity analysis

Now, for every y Rm such that AT y c and every feasible solution
x to (10.1) it holds that
cT x y T Ax = y T b = bT y.
But
1
T
bT y = bT (B 1 )T cB = cT
b = cT
BB
B xB = c x ,

so in fact we have that y is an optimal solution to the linear program

maximize

bT y,

subject to

(10.2)

A y c,

y free.

Observe that the linear program (10.2) is exactly the Lagrangian

dual problem to (10.1) (see Section 6.2.4). Also, note that the linear
programs (10.1) and (10.2) have the same optimal objective function
values, which is in accordance with the Strong Duality Theorem 6.12
(see also Theorem 10.6 below for an independent proof).
The linear program (10.2) is called the linear programming dual to
the linear program (10.1) (which is called the primal linear program). In
this chapter we will study linear programming duality. In Section 10.2 we
discuss how to construct the linear programming dual to a general linear
program. Section 10.3 presents duality theory, such as that of weak and
strong duality and complementary slackness. This theory specializes
that of Lagrangian duality presented in Chapter 6. The dual simplex
method is developed in Section 10.4. Finally, in Section 10.5 we discuss
how the optimal solutions of a linear program change if the right-hand
side b or the objective function coefficients c are modified.

10.2

The linear programming dual

For every linear program it is possible to construct the Lagrangian dual

problem through the Lagrangian relaxation of the affine constraints. We
will refer to this problem as the dual linear program. It is quite tedious
to construct the Lagrangian dual problem for every special case of a
linear program, but fortunately the dual of a general linear program
can be constructed just by following some simple rules. These rules are
presented in this section. (It is, however, a good exercise to show the
validity of these rules by constructing the Lagrangian dual in each case.)
242

The linear programming dual

10.2.1

Canonical form

When presenting the rules for constructing the linear programming dual
we will utilize the notation of canonical form. The canonical form is
connected with the directions of the inequalities of the problem and with
the objective. If the objective is to maximize the objective function, then
every inequality of type is said to be of canonical form. Similarly, if
the objective is to minimize the objective function, then every inequality
of type is said to be of canonical form. Further, we consider nonnegative variables to be variables in canonical form.
Remark 10.1 (mnemonic rule for canonical form) Consider the LP
minimize
subject to

z = x1
x1 1.

This problem is unbounded from below and hence an optimal solution

does not exist. However, if the problem is to
minimize

z = x1

subject to

x1 1,

then an optimal solution exists, namely x1 = 1. Hence it seems natural to

consider inequalities of type as canonical to minimization problems.
Similarly, it is natural that inequalities of type are canonical to
maximization problems.

10.2.2

Constructing the dual

From the notation of canonical form introduced in Section 10.2.1 we can

now construct the dual, (D), to a general linear program, (P), according
to the following rules.
Dual variables
To each constraint of (P) a dual variable, yi , is introduced. If the ith
constraint of (P) is an inequality of canonical form, then yi is a nonnegative variable, that is, yi 0. Similarly, if the ith constraint of (P) is
an inequality that is not of canonical form, then yi 0. Finally, if the
ith constraint of (P) is an equality, then the variable yi is unrestricted.
Dual objective function
If (P) is a minimization (respectively, a maximization) problem, then (D)
is a maximization (respectively, a minimization) problem. The objective
243

LP duality and sensitivity analysis

function coefficient for the variable yi in the dual problem equals the
right-hand side constant bi of the ith constraint of (P).
Constraints of the dual problem
If A is the constraint matrix of (P), then AT is the constraint matrix of
(D). The j th right-hand side constant of (D) equals the j th coefficient cj
in the objective function of (P).
If the j th variable of (P) is non-negative, then the j th constraint of
(D) is an inequality of canonical form. If the j th variable of (P) is nonpositive, then the j th constraint of (D) is an inequality of non-canonical
form. Finally, if the j th variable of (P) is unrestricted, then the j th
constraint of (D) is an equality.
Summary
The above rules can be summarized as follows:
primal/dual constraint

dual/primal variable

canonical inequality 0
non-canonical inequality 0

unrestricted

equality

Consider the following general linear program:

minimize z =

n
X

cj xj ,

j=1

subject to

n
X

j=1
n
X

j=1
n
X

aij xj bi ,

i C,

aij xj bi ,

i N C,

aij xj = bi ,

i E,

xj 0,
xj 0,

j P,
j N,

j=1

xj free,

j F,

where C stands for canonical, N C for non-canonical, E for equality, P for positive, N for negative, and F for free. Note that
244

The linear programming dual

P N F = {1, . . . , n} and C N C E = {1, . . . , m}. If we apply the
rules above we get the following dual linear program:
maximize w =

m
X

bi yi ,

i=1

subject to

m
X

i=1
m
X

i=1
m
X

aij yi cj ,

j P,

aij yi cj ,

j N,

aij yi = cj ,

j F,

i=1

yi 0,

yi 0,
yi free,

i C,

i N C,
i E.

From this it is easily established that if we construct the dual of the dual
linear program, then we return to the original (primal) linear program.

Examples
In order to illustrate how to construct the dual linear program we present
two examples. The first example considers a linear program with matrix
block structure. This is a usual form of linear programs and it is particularly easy to construct the dual linear program. The other example deals
with the transportation problem presented in Section 8.1. The purpose
of constructing the dual to this problem is to show how to handle double
subscripted variables and indexed constraints.
Example 10.2 (the dual to a linear program of matrix block form) Consider the linear program
maximize
subject to

cT x+dT y,
Ax +By b,
x

Dy = e,
0n1 ,
y 0n2 ,

245

LP duality and sensitivity analysis

where A Rm1 n1 , B Rm1 n2 , D Rm2 n2 , b Rm1 , e Rm2 ,
c Rn1 , and d Rn2 . The dual of this linear program is
bT u +eT v,

minimize

AT u

subject to

c,

B u+D v d,
u
0 m1 ,
v free.

Observe that the constraint matrix of the primal problem is



A
B
,
0m2 n1 D
and if we transpose this matrix we get
 T

A
0n1 m2
.
BT
DT
Also note that the vector of objective function coefficients of the primal
problem, (cT , dT )T , is the right-hand side of the dual problem, and
the right-hand side of the primal problem, (bT , eT )T , is the vector of
objective function coefficients of the dual problem.
Example 10.3 (the dual of the transportation problem) Consider the
transportation problem (see Example 8.1) to
minimize

z=

N X
M
X

cij xij ,

i=1 j=1

subject to

M
X
j=1

N
X
i=1

xij si ,

i = 1, . . . , N,

xij dj ,

j = 1, . . . , M,

xij 0,

i = 1, . . . , N,

j = 1, . . . , M.

The dual linear program is given by

maximize

w=

N
X

si u i +

i=1

subject to

ui + vj cij ,
ui 0,
vj 0,

246

M
X

dj vj ,

j=1

i = 1, . . . , N,
i = 1, . . . , N,
j = 1, . . . , M.

j = 1, . . . , M,

Linear programming duality theory

Observe that there are N + M constraints in the primal problem and
hence N + M dual variables. Also, there are N M variables of the primal
problem, hence N M constraints in the dual problem. The form of the
constraints in the dual problem arises from the fact that xij appears twice
in the column of the constraint matrix corresponding to this variable:
once in the constraints over i = 1, . . . , N and once in the constraints over
j = 1, . . . , M . Also note that all coefficients of the constraint matrix in
the primal problem equal +1, and since we have one dual constraint for
each column, we finally get the dual constraint ui + vj cij .

10.3

Linear programming duality theory

In this section we present some of the most fundamental duality theorems. Throughout the section we will consider the primal linear program
minimize
subject to

z = cT x,
Ax = b,

(P)

x 0n ,
where A Rmn , b Rm , and c Rn , and its dual linear program
maximize
subject to

w = bT y,

(D)

A y c,
y free.

We note that theorems similar to those presented below can be given

also for other primaldual pairs of linear programs.

10.3.1

Weak and strong duality

We begin by proving the Weak Duality Theorem.

Theorem 10.4 (Weak Duality Theorem) If x is a feasible solution to
(P) and y a feasible solution to (D), then cT x bT y.
Proof. We have that
cT x (AT y)T x

= y T Ax = y T b

[c AT y,
[Ax = b]

x 0n ]

= b y,
and we are done.

247

LP duality and sensitivity analysis

Corollary 10.5 If x is a feasible solution to (P), y is a feasible solution
to (D), and cT x = bT y, then x is an optimal solution to (P) and y is
an optimal solution to (D).
Next we show that the duality gap is zero, that is, strong duality
holds. Note that this can also be established by the use of the Lagrangian
duality theory in Chapter 6.
Theorem 10.6 (Strong Duality Theorem) If the primal problem (P) and
the dual problem (D) have feasible solutions, then there exist optimal
solutions to (P) and (D), and their optimal objective function values are
equal.
Proof. Since the dual (D) is feasible it follows from the Weak Duality Theorem 10.4 that the objective function value of (P) is bounded
from below. Hence Theorem 8.10 implies that there exists an optimal
T T
BFS, x = (xT
B , xN ) , to (P). We construct an optimal solution to (D).
(Actually we have already done this in detail in Section 10.1.) Set
1
.
(y )T := cT
BB

(10.3)

Since x is an optimal basic feasible solution the reduced costs of the

non-basic variables are non-negative, which gives that (for details see
Section 10.1)
AT y c.
Hence, y is feasible to (D). Further, we have that
1
T
bT y = bT (B 1 )T cB = cT
b = cT
BB
B xB = c x ,

so by Corollary 10.5 it follows that y is an optimal solution to (D).

See Exercise 10.15 for another formulation of the Strong Duality Theorem.
Remark 10.7 (dual solution from the primal solution) Note that the
proof of Theorem 10.6 is constructive. We construct an optimal dual
solution from an optimal basic feasible solution through (10.3).
When a linear program is solved by the simplex method we obtain an
optimal basic feasible solution (if the LP is not unbounded or infeasible).
Hence from (10.3) we then alsowithout any additional effortobtain
an optimal dual solution from the last pricing step of the simplex algoN 0nm .
rithm when we conclude that c
248

Linear programming duality theory

Interpretation of the optimal dual solution
We have from (10.3) that
1
bT y = cT
b,
BB

for any optimal basic feasible solution to (P). If xB > 0m , then a small
change in b does not change the basis, and so the optimal value of (D)
(and (P)), namely
v(b) := bT y
is linear at, and locally around, the value b. If, however, some (xB )i = 0,
then in this degenerate case it could be that the basis changes in a nondifferentiable manner with b. We summarize:
Theorem 10.8 (shadow price) If, for a given vector b Rm , the optimal
solution to (P) corresponds to a non-degenerate basic feasible solution,
then its optimal value is differentiable at b, with
v(b)
= yi ,
bi

i = 1, . . . , m,

that is, v(b) = y .

Remark 10.9 (shadow price) The optimal dual solution is indeed the
shadow price for the constraints. If a unit change in one right-hand side
bi does not change the optimal basis, then the above states that the
optimal value will change exactly with the amount yi .
It is also clear that non-degeneracy at x in (P) implies that the
optimal solution in (D) must be unique. Namely, we can show that
the function v is convex on its effective domain (why?) and the nondegeneracy property clearly implies that v is also finite in a neighbourhood of b. Then, its differentiability at b is equivalent to the uniqueness
of its subgradients at b; cf. Proposition 6.17(c).
Farkas Lemma
In Section 3.2 we proved Farkas Lemma 3.30 by using the Separation
Theorem 3.24. Having access to LP duality, Farkas Lemma can easily
be proved by using the Strong Duality Theorem 10.6.
Theorem 10.10 (Farkas Lemma) Let A Rmn and b Rm . Then,
exactly one of the systems
Ax = b,

(I)

x0 ,
249

LP duality and sensitivity analysis

and
AT y 0n ,

(II)

b y > 0,

has a feasible solution, and the other system is inconsistent.

Proof. If (I) has a solution x, then
bT y = xT AT y > 0.
But x 0n , so AT y 0n cannot hold, which means that (II) is infeasible.
Assume that (II) is infeasible. Consider the linear program
maximize

bT y,

subject to

(10.4)
n

A y0 ,
y free,

and its dual program

minimize
subject to

(0n )T x,
Ax = b,

(10.5)

x 0n .
Since (II) is infeasible, y = 0m is an optimal solution to (10.4). Hence
the Strong Duality Theorem 10.6 implies that there exists an optimal
solution to (10.5). This solution is feasible in (I).
What we have proved above is the equivalence
(I) (II).
Logically, this is equivalent to the statement that
(I) (II).
We have hence established that precisely one of the two systems (I) and
(II) has a solution.

10.3.2

Complementary slackness

A further relationship between (P) and (D) at an optimal solution is

given by the Complementary Slackness Theorem.
250

Linear programming duality theory

Theorem 10.11 (Complementary Slackness Theorem) Let x be a feasible solution to (P) and y a feasible solution to (D). Then

x optimal to (P)
xj (cj AT
j = 1, . . . , n, (10.6)
j y) = 0,
y optimal to (D)
where Aj is the j th column of A.
Proof. If x and y are feasible we get
cT x (AT y)T x = y T Ax = bT y.

(10.7)

Further, by the Strong Duality Theorem 10.6 and the Weak Duality
Theorem 10.4, x and y are optimal if and only if cT x = bT y, so in fact
(10.7) holds with equality, that is,
cT x = (AT y)T x

xT (c AT y) = 0.

Since x 0n and AT y c, xT (c AT y) = 0 is equivalent to each term

in the sum being zero, that is, that (10.6) holds.
Often the Complementary Slackness Theorem is stated for the primal
dual pair given by
maximize

cT x

subject to

Ax b,
x 0n ,

(10.8)

and
minimize
subject to

bT y

(10.9)

A y c,

y 0m .

The Complementary Slackness Theorem then becomes as follows. (Its

proof is similar to that of Theorem 10.11.)
Theorem 10.12 (Complementary Slackness Theorem) Let x be a feasible solution to (10.8) and y a feasible solution to (10.9). Then x is
optimal to (10.8) and y optimal to (10.9) if and only if
xj (cj y T Aj ) = 0,
yi (Ai x bi ) = 0,

j = 1, . . . , n,

(10.10a)

i = 1 . . . , m,

(10.10b)

where Aj is the j th column of A and Ai the ith row of A.

251

LP duality and sensitivity analysis

Remark 10.13 (interpretation of the Complementary Slackness Theorem)
From the Complementary Slackness Theorem 10.12 follows that, for a
primaldual pair of optimal solutions, if there is slack in one constraint,
then the respective variable in the other problem is zero. Further, if a
variable is positive, then there is no slack in the respective constraint in
the other problem.
Example 10.14 (the transportation problem) Consider again the transportation problem (cf. Examples 8.1 and 10.3). The following complementarity conditions are particularly simple and intuitive:
xij (ui + vj cij ) = 0,

i = 1, . . . , N,

j = 1, . . . , M.

Hence, given an optimal dual solution (u , v ), transportation can occur

on link (i, j), that is, xij > 0 may hold, only if cij = ui + vj holds. The
dual variables may be viewed as node prices that determine whether the
price cij of transportation is sufficiently low. See Section 10.6 for further
reading on duality in linear network flow optimization.
The consequence of the Complementary Slackness Theorem is the
following characterization of an optimal solution to a linear program.
We state it for the primaldual pair given by (10.8) and (10.9), but it
holds as well for each primaldual pair of linear programs.
Theorem 10.15 (necessary and sufficient conditions for global optimality)
For x Rn to be an optimal solution to the linear program (10.8), it is
both necessary and sufficient that
(a) x is a feasible solution to (10.8);
(b) corresponding to x there is a dual feasible solution y Rm to
(10.9); and
(c) the primaldual pair (x, y) satisfies the complementarity conditions (10.10).
The simplex method is very well adapted to these conditions. After
phase I, (a) holds. Every basic solution (feasible or not) satisfies (c),
since if xj is in the basis, then cj = cj y T Aj = 0, and if cj 6= 0,
then xj = 0. So, the only condition that the simplex method does not
satisfy for every basic feasible solution is (b). The proof of the Strong
Duality Theorem 10.6 shows that it is satisfied exactly at an optimal
basic feasible solution. The entering criterion is based on trying to better
satisfy it. Indeed, by choosing as an entering variable xj such that
j arg minimum cj ,
j{1,...,n}

252

Linear programming duality theory

we actually identify a dual constraint
m
X
i=1

aij yi cj ,

which is among the most violated at the complementary solution y T =

1
cT
given by the current BFS. After the basis change we will have
BB
equality in this dual constraint, and hence the basis change corresponds
to making a currently most violated dual constraint feasible!
Example 10.16 (illustration of complementary slackness) Consider the
primaldual pair given by
maximize
subject to

z = 3x1 +2x2 ,
x1 +x2 80,

(10.11)

2x1 +x2 100,

x1
40,
x1 ,

x2 0,

and
minimize

w = 80y1 +100y2 +40y3 ,

subject to

y1
y1
y1 ,

+2y2
+y2
y2 ,

(10.12)

+y3 3,
2,
y3 0.

We use Theorem 10.15 to show that x = (20, 60)T is an optimal

solution to (10.11).
(a) (primal feasibility) Obviously x is a feasible solution to (10.11).
(c) (complementarity) The complementarity conditions imply that
y1 (x1 + x2 80)

y2 (2x1 + x2 100)
y3 (x1 40)
x1 (y1 + 2y2 + y3 3)
x2 (y1 + y2 2)

=0
=0
=0

y3 = 0

=0
=0

=
=

y1 + 2y2 = 3 [x1 > 0]

y1 + y2 = 2 [x2 > 0]

[x1 = 20 6= 40]

which gives that y1 = 1, y2 = 1 and y3 = 0.

(b) (dual feasibility) Clearly y = (1, 1, 0)T is feasible in (10.12).
From Theorem 10.15 it then follows that x = (20, 60)T is an optimal
solution to (10.11) and y = (1, 1, 0)T an optimal solution to (10.12).

253

LP duality and sensitivity analysis

10.4

The dual simplex method

The simplex method presented in Chapter 9, which we here refer to

as the primal simplex method, starts with a basic feasible solution to
the primal linear program and then iterates until the primal optimality
conditions are fulfilled, that is, until a basic feasible solution is found
such that the reduced costs satisfy
T
T 1

cT
N (0nm )T .
N := cN cB B

This is equivalent to the dual feasibility condition

AT y c,
where y := (B 1 )T cB . We call a basis such that all of the reduced
costs are greater than or equal to zero a dual feasible basis; otherwise we
call it a dual infeasible basis. Hence, the primal simplex method starts
with a primal feasible basis and then moves through a sequence of dual
infeasible (but primal feasible) bases until a dual feasible basis is found.
The dual simplex method is a variant of the primal simplex method
that works in a dual manner, in the sense that it starts with a dual
feasible basis and then moves through a sequence of primal infeasible
(but dual feasible) bases until a primal (and dual) feasible basis is found.
In order to derive the dual simplex algorithm, let xB be a dual feasible
basis with the corresponding partition (B, N ). If
:= B 1 b 0m ,
b
then xB is primal feasible and since it is also dual feasible all of the
reduced costs are greater than or equal to zero; hence, xB is an optimal
is strictly negative, say b1 ,
BFS. Otherwise some of the components of b
that is,
(xB )1 +

nm
X

(B 1 N )1j (xN )j = b1 < 0,

j=1

so (xB )1 < 0 in the current basis and will be the leaving variable. If
(B 1 N )1j 0,

j = 1, . . . , n m,

(10.13)

then there exists no primal feasible solution to the problem. (Why?)

Hence, if (10.13) is fulfilled, then we say that the primal infeasibility
criterion is satisfied. Otherwise (B 1 N )1j < 0 for some j = 1, . . . , nm.
Assume that (B 1 N )1k < 0 and choose (xN )k to replace (xB )1 in the
254

The dual simplex method

basis. (Note that this yields that (xN )k = b1 /(B 1 N )1k > 0 in the new
basis.) The new reduced costs then become
(
cN )k
,
(B 1 N )1k
(
cB )j := 0,
j = 2, . . . , m,

(
cB )1 :=

(
cN )j := (
cN )j (
cN )k

(B 1 N )1j
,
(B 1 N )1k

j = 1, . . . , n m.

Since we want the new basis to be dual feasible it must hold that all of
the new reduced costs are non-negative, that is,
(
cN )j (
cN )k

(B 1 N )1j
,
(B 1 N )1k

j = 1, . . . , n m,

or, equivalently,
(
cN )k
(
cN )j

,
(B 1 N )1k
(B 1 N )1j

for all j such that (B 1 N )1j < 0.

Therefore, in order to preserve dual feasibility, as entering variable we

must choose (xN )k such that
k arg

maximum

i{ j

| (B 1 N )

1j <0 }

(
cN )j
.
1
(B N )1j

We have now derived an infeasibility criterion and criteria for choosing the leaving and the entering variables, and are ready to state the
dual simplex algorithm.
The Dual Simplex Algorithm:
T T
Step 0 (initialization: DFS) Assume that x = (xT
B , xN ) is a dual feasible basis corresponding to the partition A = (B, N ).

Step 1 (leaving variable or termination) Calculate

:= B 1 b.
b
0m , then stop; the current basis is optimal. Otherwise,
If b
choose an s such that bs < 0, and let (xB )s be the leaving variable.
Step 2 (entering variable or termination) If
(B 1 N )sj 0,

j = 1, . . . , n m,
255

LP duality and sensitivity analysis

then stop; the (primal) problem is infeasible. Otherwise, choose a
k such that
k arg

maximum

i{ j | (B 1 N )sj <0 }

(
cN )j
,
1
(B N )sj

and let (xN )k be the entering variable.

Step 3 (update: change basis) Construct a new partition by swapping
(xB )s with (xN )k . Go to Step 1.
Similarly to the primal simplex algorithm it can be shown that the
dual simplex algorithm terminates in a finite number of steps if cycling
is avoided. Also, there exist rules for choosing the leaving and entering
variables (among the eligible ones) such that cycling is avoided.
If a dual feasible solution is not available from the start, it is possible
to add a constraint to the original problem that makes it possible to
construct a dual feasible basis, and then run the dual simplex algorithm
on this modified problem (see Exercise 10.12).
Remark 10.17 (unboundedness of the primal problem) If the dual problem is known to be feasible, the primal problem cannot be unbounded
by the Weak Duality Theorem 10.4. Hence the dual simplex algorithm
terminates with a basis that satisfies either the optimality criterion or
the primal infeasibility criterion.
Example 10.18 (illustration of the dual simplex algorithm) Consider the
linear program
minimize
subject to

3x1 +4x2 +2x3 +x4 +5x5 ,

x1 2x2 x3 +x4 +x5 3,

x1 x2 x3 +x4 +x5 2,
x1 +x2 2x3 +2x4 3x5 4,
x1 ,

x2 ,

x3 ,

x4 ,

x5 0.

By introducing the slack variables x6 , x7 , x8 , we get the following linear

program:
minimize
subject to

3x1 +4x2 +2x3 +x4 +5x5 ,

x1 2x2 x3 +x4 +x5 +x6

x1 x2 x3 +x4 +x5
x1 +x2 2x3 +2x4 3x5
x1 ,

256

x2 ,

x3 ,

x4 ,

=3,
+x7

=2,
+x8 = 4,

x5 , x6 , x7 , x8 0.

Sensitivity analysis
We see that the basis xB := (x6 , x7 , x8 )T is dual feasible, but primal
infeasible. Hence we use the dual simplex algorithm to solve the problem.
We have that
:= B 1 b = (3, 2, 4)T ,
b
so we choose (xB )1 = x6 to leave the basis. Further we have that the
reduced costs of xN := (x1 , x2 , x3 , x4 , x5 )T are
cT
N = (3, 4, 2, 1, 5),
and
(B 1 N )1 = (1, 2, 1, 1, 1),

[the 1st row of B 1 N ]

so we choose x2 as the entering variable. The new basis becomes xB :=

(x2 , x7 , x8 )T , xN := (x1 , x6 , x3 , x4 , x5 )T . We get that
:= B 1 b = (1.5, 0.5, 2.5)T.
b
Hence, we choose (xB )2 = x7 as the leaving variable. Further,

cT
N = (5, 2, 0, 3, 7),
(B 1 N )2 = (1.5, 0.5, 0.5, 0.5, 0.5)T,
which gives that x3 is the entering variable. The new basis becomes
xB := (x2 , x3 , x8 )T . We get that

b := B 1 b = (1, 1, 5)T ,
which means that the optimality criterion (primal feasibility) is satisfied,
and an optimal solution to the original problem is given by
x = (x1 , x2 , x3 , x4 , x5 )T = (0, 1, 1, 0, 0)T.
Check that this is indeed true, for example by using Theorem 10.12.

10.5

Sensitivity analysis

In this section we study two kinds of perturbations of a linear program

in standard form,
minimize

z = cT x,

subject to

Ax = b,
x 0n ,

(10.14)

namely
257

LP duality and sensitivity analysis

1. perturbations in the objective function coefficients cj ; and
2. perturbations in the right-hand side coefficients bi .
1 T
T T
We assume that x = (xT
b) , (0nm )T )T is an optimal
B , xN ) = ((B
basic feasible solution to (10.14) with the corresponding partition A =
(B, N ).

10.5.1

Perturbations in the objective function

Assume that the objective function coefficients of the linear program

(10.14) are perturbed by the vector p Rn , that is, we consider the
perturbed problem to
minimize z = (c + p)T x,
subject to
Ax = b,

(10.15)

x 0n .
The optimal solution x to the unperturbed problem (10.14) is obviously
a feasible solution to (10.15), but is it still optimal? To answer this question, we note that a basic feasible solution is optimal if the reduced costs
of the non-basic variables are greater than or equal to zero. The reduced
costs for the non-basic variables of the perturbed problem (10.15) are
T T
given by [let p = (pT
B , pN ) ]
T
T 1
T
c
N.
N = (cN + pN ) (cB + pB ) B

N 0nm is sufficient for x to be an optimal solution to

Hence, c
the perturbed problem (10.15). (Observe, however, that this is not a
necessary condition unless x is non-degenerate.)
Perturbations of a non-basic cost coefficient
If only one component of cN is perturbed, that is,
   m
pB
0
p=
=
,
pN
ej
for some R and j {1, . . . , n m}, then we have that x is an
optimal solution to the perturbed problem if
1
(cN )j + cT
Nj 0
BB

+ (
cN )j 0,

so in this case we only have to check that the perturbation is not less
than (
cN )j in order to guarantee that x is an optimal solution to the
perturbed problem.
258

Sensitivity analysis
Perturbations of a basic cost coefficient
If only one component of cB is perturbed, that is,
  

pB
ej
p=
=
,
pN
0nm
for some R and j {1, . . . , m}, then we have that x is an optimal
solution to the perturbed problem if
1
T
N (0nm )T
(cN )T (cT
B + ej )B

1
nm T
eT
N + cT
) .
j B
N (0

In this case all of the reduced costs of the non-basic variables may change,
and we must check that the perturbation multiplied by the j th row of
T
B 1 N plus the original reduced costs c
N is a vector whose components
all are greater than or equal to zero.
Perturbations that make x non-optimal
If the perturbation p is such that some of the reduced costs of the perturbed problem becomes strictly negative for the basis xB , then x is
perhaps not an optimal solution anymore. If this happens, let some of
the variables with strictly negative reduced cost enter the basis and continue the simplex algorithm until an optimal solution is found (or until
the unboundedness criterion is satisfied).

10.5.2

Perturbations in the right-hand side coefficients

Now, assume that the right-hand side b of the linear program (10.14)
is perturbed by the vector p Rm , that is, we consider the perturbed
problem to
minimize
z = cT x,
subject to Ax = b + p,

(10.16)

x 0n .
The reduced costs of the unperturbed problem do not change as the
right-hand side is perturbed, so the basic feasible solution given by the
partition A = (B, N ) is optimal to the perturbed problem (10.16) if
259

LP duality and sensitivity analysis

and only if it is feasible, that is,
   1

xB
B (b + p)
=
0n ,
xN
0nm
which means that we have to check that B 1 (b + p) 0m .
Perturbations of one component of the right-hand side
Suppose that only one of the components of the right-hand side is perturbed, that is,
p = ej ,
for some R and j {1, . . . , m}. The basic feasible solution corresponding to the partition A = (B, N ) is then feasible if and only if
B 1 (b + ej ) 0m

B 1 ej + B 1 b 0m ,

so it must hold that multiplied by the j th column of B 1 plus the

vector B 1 b equals a vector whose components all are greater than or
equal to zero.
Perturbations that make x infeasible
If the perturbation p is such that the basis xB becomes infeasible, then
some of the components in the updated right-hand side, B 1 (b + p), is
strictly negative. However, the reduced costs are independent of p, so
the basis xB is still a dual feasible basis. Hence, we can continue with
the dual simplex algorithm until an optimal solution is found (or until
the primal infeasibility criterion is satisfied).

10.6

Notes and further reading

For an account of the early history of LP duality theory, see [LRS91].

Linear programming duality theory was introduced by John von Neumann [vNe47]. His results build upon his earlier work in game theory.
The first published proof of the Strong Duality Theorem is found in Gale,
Kuhn, and Tucker [GKT51]. The Complementary Slackness Theorem is
due to Dantzig and Orden [DaO53].
Text books that discuss LP duality and sensitivity analysis are [Dan63,
Chv83, Mur83, Sch86, DaT97, Pad99, Van01, DaT03, DaM05].
More on the modelling of, and algorithms and duality for, linear
network optimization can be found in [AMO93].
260

Exercises

10.7

Exercises

Exercise 10.1 (constructing the LP dual) Consider the linear program

maximize
subject to

z = 6x1 3x2 2x3 +5x4 ,

4x1 +3x2 8x3 +7x4 = 11,

3x1 +2x2 +7x3 +6x4 23,

7x1 +4x2 +3x3 +2x4 12,

x1 ,

0,

x2

x3
x4

0,

free.

Construct its linear programming dual.

Exercise 10.2 (constructing the LP dual) Consider the linear program
minimize
subject to

z = T x,

Ax = b,
l x u.

(a) Construct its linear programming dual.

(b) Show that the dual problem is always feasible (independently of
l, and u).

A, b,

Exercise 10.3 (application of the Weak and Strong Duality Theorems) Consider the linear program
minimize
subject to

z = T x,

(P)

z = T x,
Ax = b,

(P)

Ax = b,
x 0n ,

and the perturbed problem to

minimize
subject to

x 0n .

Show that if (P) has an optimal solution, then the perturbed problem (P)
).
cannot be unbounded (independently of b
Exercise 10.4 (application of the Weak and Strong Duality Theorems) Consider the linear program
minimize
subject to

z = T x,

(10.17)

Ax b.

Assume that the objective function vector cannot be written as a linear

combination of the rows of A. Show that (10.17) cannot have an optimal
solution.

261

LP duality and sensitivity analysis

Exercise 10.5 (application of the Weak and Strong Duality Theorems) Consider the linear program
z = T x,

minimize

(10.18)

Ax b,
x 0n .

subject to

Construct a polyhedron that equals the set of optimal solutions to (10.18).

Exercise 10.6 (application of the Weak and Strong Duality Theorems) Consider the linear program
minimize

z = T x,

subject to

Ax b,
x 0n .

(10.19)

Let x be optimal in (10.19) with the optimal value z , and let

in the LP dual of (10.19). Show that

y be optimal

z = (y )T Ax .

Exercise 10.7 (linear programming primaldual optimality conditions) Consider

the linear program
maximize

4x2 +3x3 +2x4 8x5 ,

z=

subject to

3x1 +x2 +2x3 +x4

x1 x2
x1 ,

x2 ,

x3 ,

= 3,

+x4 x5 2,
x4 ,

x5 0.

Use the LP primaldual optimality conditions to find an optimal solution.

Exercise 10.8 (linear programming primaldual optimality conditions) Consider
the linear program (the continuous knapsack problem)

where

maximize

z = T x,

subject to

aT x b,
x 1n ,
x 0n ,

> 0n , a > 0n , b > 0, and

c1
c2
cn

.
a1
a2
an
Show that the feasible solution
xj = 1, j = 1, . . . , r 1,
where r is such that

262

Pr1
j=1

x given by

xr =

aj b and

Pr1
j=1

ar

Pr

j=1

aj

xj = 0, j = r + 1, . . . , n,

aj > b, is an optimal solution.

Exercises
Exercise 10.9 Prove Theorem 10.15.
Exercise 10.10 (KKT versus LP primaldual optimality conditions) Consider
the linear program
minimize
subject to

z = T x,

Ax b.

Show that the KKT conditions are equivalent to the LP primaldual optimality
conditions.
Exercise 10.11 (Lagrangian primaldual versus LP primaldual) Consider the
linear program
minimize
subject to

z = T x,

Ax b.

Show that the Lagrangian primaldual optimality conditions are equivalent to

the LP primaldual optimality conditions.
Exercise 10.12 (the dual simplex method) Show that by adding the constraint
x1 + + xn M,
where M is a positive constant, to a linear program in standard form, it is
always possible to construct a dual feasible basis.
Exercise 10.13 (sensitivity analysis: perturbations in the objective function)
Consider the linear program
maximize

z=

subject to

x1 +18x2 +c3 x3 +c4 x4 ,

x1 +2x2 +3x3 +4x4 3,

3x1 +4x2 5x3 6x4 1,

x1 ,

x2 ,

x3 ,

x4 0.

Find the values of c3 and c4 such that the basic solution that corresponds
to the partition xB := (x1 , x2 )T is an optimal basic feasible solution to the
problem.
Exercise 10.14 (sensitivity analysis: perturbations in the right-hand side) Consider the linear program
minimize
subject to

z = x1 +2x2 +x3 ,

2x1 +x2 x3 7,

x1 +2x2 +3x3 3 + ,
x1 ,

x2 ,

x3 0.

263

LP duality and sensitivity analysis

(a) Let = 0. Show that the basic solution that corresponds to the
partition xB := (x1 , x3 )T is an optimal solution to the problem.
(b) Find the values of the perturbation R such that the above BFS is
optimal.
(c) Find an optimal solution when = 7.
Exercise 10.15 (a version of the Strong Duality Theorem) Consider the linear
program
z = T x,

(P)

subject to

Ax = b,
x 0n ,

maximize

w = bT y ,

(D)

subject to

A y ,
y free.

minimize

and its dual linear program

Show that if one of the problems (P) and (D) has a finite optimal solution,
then so does its dual, and their optimal objective function values are equal.
Exercise 10.16 (an LP duality paradox) For a standard primaldual pair of
LPs, consider the following string of inequalities:
maximum { T x | Ax b;

x 0n } minimum { bT y | AT y ; y 0m }
maximum { bT y | AT y ; y 0m }
minimum { T x | Ax b; x 0n }
maximum { T x | Ax b; x 0n }
minimum { bT y | AT y ; y 0m }
maximum { bT y | AT y ; y 0m }
minimum { T x | Ax b; x 0n }
maximum { T x | Ax b; x 0n }.

Since equality must hold throughout, the range of T x is a constant over the
primal polyhedron, and bT y is constant over the dual polyhedron, yet , A,
and b are arbitrary. What is wrong in the above line of arguments?
[Note: This and other paradoxes in optimization are found on Harvey
Greenbergs page http://www.cudenver.edu/~hgreenbe/myths/myths.html.]

264

Part V

Algorithms

Unconstrained
optimization

11.1

XI

Introduction

We consider the unconstrained optimization problem to

minimize
f (x),
n

(11.1)

x R

where f C 0 on Rn (f is continuous). Mostly, we will assume that f

C 1 holds (f is continuously differentiable), in some cases even f C 2 .
The method of choice for this problem depends on many factors:
What is the size of the problem (that is, n)?
Are f (x) and 2 f (x) available, and if so at what cost?
What it is the solution requirement? (Do we need a global minimum or a local minimum or simply a stationary point?)
What are the convexity properties of f ?
Do we have a good estimate of the location of a stationary point
x ? (Can we use locally-only convergent methods?)
We will discuss some basic approaches to the problem (11.1) and refer
to questions such as the ones just mentioned during the development.
Example 11.1 (non-linear least squares data fitting) Suppose that we
have m data points (ti , bi ) which we believe are related through an algebraic expression of the form
x1 + x2 exp(x3 ti ) + x4 exp(x5 ti ) = bi ,

i = 1, . . . , m,

where however the parameters x1 , . . . , x5 are unknown. (Here, exp(x) =

ex .) In order to best describe the above model, we minimize the total
residual error given by the norm of the residual
fi (x) := bi [x1 + x2 exp(x3 ti ) + x4 exp(x5 ti )],

i = 1, . . . , m.

Unconstrained optimization
A minimization will then yield the best fit with respect to the data points
available. The following then is the resulting optimization problem to
be solved:
minimize
f (x) :=
5
x R

m
X
i=1

|fi (x)|2 =

m
X

[fi (x)]2 .

i=1

This type of problem is very often solved within numerical analysis and
mathematical statistics. Note that the 2-norm is not the only measure
of the residual used; sometimes the maximum norm is used.
What is the typical form of an algorithm in unconstrained optimization (in fact, for almost every problem class)? Take a look at Figure 11.1
of the level curves1 of a convex, quadratic function, and the algorithm
description below.

pk+1

5
4
3
2

pk

1
0

xk+1
5

1
2

3
1

xk
2

Figure 11.1: At xk , the descent direction pk is generated. A step k is

taken in this direction, producing xk+1 . At this point, a new descent
direction pk+1 is generated, and so on.

Descent algorithm:
Step 0 (initialization). Determine a starting point x0 Rn . Set k := 0.

level curve (or, iso-curve, or iso-cost line) is a set of the form { x Rn | f (x) =
k } for a fixed value of k R.
1A

268

Descent directions
Step 1 (descent direction). Determine a descent direction pk Rn .

Step 2 (line search). Determine a step length k > 0 such that f (xk +
k pk ) < f (xk ) holds.
Step 3 (update). Let xk+1 := xk + k pk .
Step 4 (termination check). If a termination criterion is fulfilled, then
stop! Otherwise, let k := k + 1 and go to step 1.
This type of algorithm is inherently local, since we cannot in general
use more than the information that can be calculated at the current point
xk , that is, f (xk ), f (xk ), and 2 f (xk ). As far as our local sight
is concerned, we sometimes call this type of method (for maximization
problems) the near-sighted mountain climber, reflecting the situation
in which the mountain climber is in a deep fog and can only check her
barometer for the height and feel the steepness of the slope under her
feet. Notice then that Figure 11.1 was plotted using several thousands of
function evaluations; in realityand definitely in higher dimension than
twowe never have this type of orienteering map.
We begin by analyzing Step 1, the most important step of the abovedescribed algorithm. Based on the result in Proposition 4.16 it makes
good sense to generate pk such that it is a direction of descent.

11.2

Descent directions

11.2.1

Introduction

Recall Definition 4.15 of a direction of descent at a given point. Usually,

we have many possible such choices; see for example Proposition 4.16
for a sufficient criterion for a continuously differentiable function. In
this section we discuss some details on how descent directions can be
generated, depending on a particular situation.
Example 11.2 (example descent directions) (a) Let f C 1 (N ) in some
neighborhood N of xk Rn . If f (xk ) 6= 0n , then p = f (xk ) is a
descent direction for f at xk (this follows directly from Proposition 4.16).
This is the search direction used in the steepest descent method, and it
naturally bears the name of steepest descent direction because it solves
the minimization problem to2
minimize f (xk )T p.

pRn :kpk=1

(11.2)

2 We have that f (x)T p = kf (x)k kpk cos , where is the angle between
the vectors f (x) and p; this expression is clearly minimized by making cos =
1, that is, by letting p have the angle 180 with f (x); in other words, p =
f (x)/kf (x )k.

269

Unconstrained optimization
(b) Let f C 2 (N ) in some neighborhood N of xk . If f (xk ) = 0n
we cannot use the steepest descent direction anymore. However, we can
work with second order information provided by the Hessian to find a
descent direction in this case also, provided that f is non-convex at xk .
Assume that 2 f (xk ) is not positive semidefinite (otherwise, xk is likely
to be a local minimum; see Theorem 4.17). If 2 f (xk ) is indefinite we
call the stationary point xk a saddle point of f . Let p be an eigenvector
corresponding to a negative eigenvalue of 2 f (xk ). Then, we call p a
direction of negative curvature for f at xk , and p is a descent direction
since for all > 0 small enough, f (xk + p) f (xk ) = f (xk )T p +
2
2 T 2
2
2
2
2 p f (xk )p + o( ) = 2 kpk + o( ) < 0.
(c) Assume the conditions of (a), and let Q Rnn be an arbitrary
symmetric, positive definite matrix. Then p = Qf (xk ) is a descent
direction for f at xk : f (xk )T p = f (xk )T Qf (xk ) < 0, due to
the positive definiteness of Q. (This is of course true only if xk is nonstationary, as assumed.)
Pre-multiplying by Q may be interpreted as a scaling of f if we
choose a diagonal matrix Q; the use of more general matrices is of course
possible and leads to exceptionally good computational results for clever
choices of Q. Newton and quasi-Newton methods are based on constructing directions in this way. Note that setting Q = I n (the identity
matrix in Rnn ), we obtain the steepest descent direction.
To find some arbitrary direction of descent is not a very difficult
task as demonstrated by Example 11.2 [in fact, the situation when
f (xk ) = 0n appearing in (b) is quite an exotic one already, so typically one can always use directions constructed in (a), or, more generally
(c), as descent directions]. However, in order to secure the convergence
of numerical algorithms we must provide descent directions that behave well numerically. Typical requirements, additional to the basic
requirement of being a direction of descent, are:
|f (xk )T pk | s1 kf (xk )k2 ,

and kpk k s2 kf (xk )k,

(11.3)

f (xk )T pk
s1 ,
kf (xk )k kpk k

and kpk k s2 kf (xk )k,

(11.4)

or

where s1 , s2 > 0, and xk and pk are, respectively, iterates and search

directions of some iterative algorithm.
The purpose of these condition is to prevent the descent directions
to deteriorate in quality, in terms of providing good enough descent.
270

Descent directions
For example, the first condition in (11.3) states that if the directional
derivative of f tends to zero then it must be that the gradient of f also
tends to zero, while the second condition makes sure that a bad direction
in terms of the directional derivative is not compensated by the search
direction becoming extremely long in norm. The first condition in (11.4)
is equivalent to the requirement that the cosine of the angle between
f (xk ) and pk is positive and bounded away from zero by the value
of s1 , that is, the angle must be acute and not too close to /2; this is
another way of saying that the direction pk must be steep enough. The
purpose of the second condition in (11.4) then is to ensure that if the
search direction vanishes then so does the gradient. Methods satisfying
(11.3), (11.4) are sometimes referred to as gradient related, since they
cannot be based on search directions that are very far from those of the
steepest descent method.
The choice pk = f (xk ) fulfills (11.3), (11.4) with s1 = s2 = 1.
Another example is as follows: set pk = Qk f (xk ), where Qk
Rnn is a symmetric and positive definite matrix such that mksk2
sT Qk s M ksk2 , for all s Rn , holds. [All eigenvalues of Qk lie in the
interval [m, M ] (0, ).] Then, the requirement (11.3) is verified with
s1 = m, s2 = M , and (11.4) holds with s1 = m/M , s2 = m.

11.2.2

Newtons method and extensions

What should a good descent direction accomplish? Roughly speaking, it should provide as large descent as possible, that is, minimize
f (x + p) f (x) over some large enough region of p around the origin.
In principle, this is the idea behind the optimization problem (11.2),
because, according to (2.1), f (x + p) f (x) f (x)T p.
Therefore, more insights into how the scaling matrices Q appearing
in Example 11.2(c) should be constructed and, in particular, reasons why
the steepest descent direction is not a very wise choice, can be gained if
we consider more general approximations than the ones given by (2.1).
Namely, assume that f C 1 near x, and that for some positive definite
matrix Q it holds that
1
f (x + p) f (x) x (p) := f (x)T p + pT Q1 p.
2

(11.5)

For example, if f C 2 , 2 f (x) 0nn , and assuming that o(kpk2 ) 0

[cf. (2.3)] we may use Q1 = 2 f (x).
Using the optimality conditions, we can easily check that the search
direction defined in Example 11.2(c) is a solution to the following optimization problem:
minimize
x (p),
(11.6)
n
pR

271

Unconstrained optimization
where x (p) is defined by (11.5). The closer x (p) approximates f (x +
p) f (x), the better we can expect the quality of the search directions
generated by the method described in Example 11.2(c) to be.
As was already mentioned, setting Q = I n , which absolutely fails
to take into account any information about f (that is, it is a one-sizefits-all approximation), gives us the steepest descent direction. (Cases
can easily be constructed such that the algorithm converges extremely
slowly; convergence can actually be so bad that the authors of the book
[BGLS03] decree that the steepest descent method should be forbidden!)
On the other hand, the best second-order approximation is given by
the Taylor expansion (2.3), and therefore we would like to set Q =
[2 f (x)]1 ; this is exactly the choice made in Newtons method.
Remark 11.3 (a motivation for the descent property in Newtons method)
The search direction in Newtons method is based on the solution of the
following linear system of equations: find p Rn such that
p x (p) := f (x) + 2 f (x)p = 0n .
Consider the case of n = 1. We should then solve
f (x) + f (x)p = 0.

(11.7)

It is obvious that unless f (x) = 0 (whence we are at a stationary point

and p = 0 solves the equation) we cannot solve (11.7) unless f (x) 6= 0.
Then, the solution p := f (x)/f (x) to (11.7) is well-defined. We
distinguish between two cases:
(a) f (x) > 0. The derivative of the second-order approximation
p 7 f (x)p + 21 f (x)p2 then has a positive slope. Hence, if f (x) > 0
then p < 0, and if f (x) < 0 then p > 0 holds. In both cases, therefore,
the directional derivative f (x)
p < 0, that is, p is a descent direction.
(b) f (x) < 0. The derivative of the second-order approximation
p 7 f (x)p + 21 f (x)p2 then has a negative slope. Hence, if f (x) > 0
then p > 0, and if f (x) < 0 then p < 0 holds. In both cases, therefore,
the directional derivative f (x)
p > 0, that is, p is an ascent direction.
From the above it is clear that Newtons method3 provides the same
search direction regardless of whether the optimization problem is a minimization or a maximization problem; the reason is that the search direction is based on the stationarity of the second-order approximation
and not its minimization/maximization. We also see that the Newton
direction p is a descent direction if the function f is of the strictly convex
3 For

n = 1 it is often referred to as the NewtonRaphson method; cf. Section 4.6.4.2.

272

Descent directions
type around x [that is, if f (x) > 0], and an ascent direction if it is of the
strictly concave type around x [that is, if f (x) < 0]. In other words, if
the objective function is (strictly) convex or concave, the Newton equation will give us the right direction, if it gives us a direction at all. In
the case when n > 1, Newtons method acts as a descent method if the
Hessian matrix 2 f (x) is positive definite, and as an ascent method if
it is negative definite, which is appropriate.
An essential problem arises if the above-described is not what we
want; for example, we may be interested in maximizing a function which
is neither convex nor concave, and around a current point the function
is of strictly convex type (that is, the Hessian is positive definite). In
this case the Newton direction will not point in an ascent direction,
but instead the opposite. How to solve a problem with a Newton-type
method in a non-convex world is the main topic of what follows. As
always, we consider minimization to be the direction of interest for f .
So, why might one want to choose a matrix Q different from the
best choice [2 f (x)]1 ? There are several reasons:
Lack of positive definiteness The matrix 2 f (x) may not be positive definite. As a result, the problem (11.6) may even lack solutions
and [2 f (x)]1 f (x) may in any case not be a descent direction.
This problem can be cured by adding to 2 f (x) a diagonal matrix
E, so that 2 f (x) + E is positive definite. For example, E = I n , for
smaller than all the non-positive eigenvalues of 2 f (x), may be used
because such a modification shifts the original eigenvalues of 2 f (x)
by > 0. The value of needed will automatically be found when
solving the Newton equation 2 f (x)p = f (x), since eigenvalues
of 2 f (x) are pivot elements in Gaussian-elimination procedures. This
modification bears the name LevenbergMarquardt.
[Note: as becomes large, p resembles more and more the steepest
descent direction.]
Lack of enough differentiability The function f might not be in
C 2 , or the matrix of second derivatives might be too costly to compute.
Either being the case, in quasi-Newton methods one approximates
the Newton equation by replacing 2 f (xk ) with a matrix B k that is
cheaper to compute, typically by only using values of f at the current
and some previous points.
Using a first-order Taylor expansion (2.1) for f (xk ) we know that
2 f (xk )(xk xk1 ) f (xk ) f (xk1 ),
273

Unconstrained optimization
so the matrix B k is taken to satisfy the similar system
B k (xk xk1 ) = f (xk ) f (xk1 ).
Notice that for n = 1, this corresponds to the secant method, in
which at iteration k we approximate the second derivative as
f (xk )

f (xk ) f (xk1 )
.
xk xk1

The matrix B k has n2 elements and is hence under-determined by

these n equations; additional requirements, such as ones that make sure
that B k is symmetric and positive definite, result in particular quasiNewton methods. Typically, starting from B 0 = I n , B k+1 is calculated
from B k using a rank-one or rank-two matrix update; in particular,
this allows us to update the factorization of B k to efficiently obtain the
factorization of B k+1 using standard algorithms in linear algebra.
There are infinitely many choices that may be used, and the following
(called the BroydenFletcherGoldfarbShanno, or BFGS, method after
the original publications [Bro70, Fle70, Gol70, Sha70]) is considered to
be the most effective one:
B k+1 = B k

(B k sk )(B k sk )T
y yT
+ kT k ,
T
sk B k sk
y k sk

where sk = xk+1 xk , and y k = f (xk+1 ) f (xk ). Interestingly

enough, should f be quadratic, B k will be identical to the Hessian of f
after a finite number of steps (namely, n).
Quasi-Newton methods with various updating rules for B k are very
popular for unconstrained optimization; see Section 11.9.
Computational burden The solution of a linear system B k pk =
f (xk ), or (which is the same if we identify Q1 = B k ) finding the
optimum of (11.6), may be too costly. This is exactly the situation when
one would like to use the steepest descent method, which avoids any such
calculations.
Other possibilities are: (a) In a quasi-Newton method, keep the matrix B k (and, obviously, its factorization) fixed for k0 > 1 subsequent
steps; this way, we need only to perform matrix factorization (the most
computationally consuming part) every k0 steps.
(b) Solve the optimization problem (11.6) only approximately, based
on the following arguments. Assume that xk violates the second-order
necessary optimality conditions for f , and consider the problem (11.6),
where we replace the matrix Q1 with an iteration-dependent, perhaps
274

The line search problem

only positive semidefinite matrix B k . As a first example, suppose we
consider the Newton method, whence we choose B k = 2 f (xk ). Then,
by the assumption that the second order necessary optimality conditions
are violated, p = 0n is not a minimum of x k (p) in the problem (11.6).
e 6= 0n be any vector with x k (e
Let p
p) < x k (0n ) = 0. Then,
1 T
e+ p
e Bk p
e < x k (0n ) = 0,
x k (e
p) = f (xk )T p
2
| {z }
0

e < 0. This means that if the Newton

which implies that f (xk )T p
equations are solved inexactly, a descent direction is still obtained. This
can of course be generalized for quasi-Newton methods as well, since we
only assumed that the matrix B k is positive semidefinite.
We summarize the above development of search directions in Table 11.1. The iterate is xk ; for each algorithm, we describe the linear
system solved in order to generate the search direction pk . In the table,
k 0 and B k Rnn is a symmetric and positive definite matrix.
Table 11.1: Search directions.
Algorithm
Steepest descent
Newtons method
LevenbergMarquardt
Quasi-Newton

Linear system
pk = f (xk )
2 f (xk )pk = f (xk )
2
[ f (xk ) + k I n ]pk = f (xk )
B k pk = f (xk )

11.3

The line search problem

11.3.1

A characterization of the line search problem

Executing Step 2 in the iterative algorithm is naturally done by finding

an approximate solution to the one-dimensional problem to
minimize () := f (xk + pk ).
0

(11.8)

Its optimality conditions are that4

4 These

( ) 0;

( ) = 0;

0,

(11.10)

conditions are the same as those in Proposition 4.23(b). To establish this

fact, lets suppose first that we satisfy (4.10) which here becomes the statement that
( )( ) 0,

0.

(11.9)

275

Unconstrained optimization
that is,
f (xk + pk )T pk 0;

f (xk + pk )T pk = 0;

0,

holds. So, if > 0, then ( ) = 0 must hold, which therefore means

that f (xk + pk )T pk = 0; that is, the search direction pk is orthogonal
to the gradient of f at the point xk + pk .
Figure 11.2 shows an example of the one-dimensional function along
a descent direction with a well-defined minimum.

pk

5
4

()

3
2

xk + pk
5

xk
4

1
0

1
2
3
1

4
2

xk + pk

Figure 11.2: A line search in a descent direction.

In the quest for a stationary point it is of relatively minor importance

to perform a line search accuratelythe stationary point is most probably not situated somewhere along that half-line anyway. Therefore, most
line search strategies used in practice are approximate. It should also be
noted that if the function f is non-convex then so is probably the case
with as well, and globally minimizing a non-convex function is difficult
even in one variable.

11.3.2

Approximate line search strategies

First, we consider the case where f is quadratic; this is the only general
case where an accurate line search is practical.
Let f (x) := (1/2)xT Qx q T x + a, where Q Rnn is symmetric,
q Rn , and a R. Suppose we wish to minimize the function for this
Setting first = 0 in (11.9), then ( ) 0 follows. On the other hand, setting
= 2 in (11.9), then ( ) 0 follows. So, ( ) = 0 must hold. Also,
setting = + 1 in (11.9), we obtain that ( ) 0. This establishes that (11.10)
follows from (4.10). To establish the reverse conclusion and therefore prove that the
two conditions are the same, we note that if we satisfy (11.10), then it follows that
for every 0, ( )( ) = ( ) 0, and we are done.

276

The line search problem

special case. Then, we can solve the equation () = 0 analytically:
() = f (x+p)T p = [Q(x+p)q]T p = pT Qp(qQx)T p = 0

= (q Qx)T p/pT Qp.

Lets check the validity and meaning of this solution. We suppose naturally that p is a descent direction, whence (0) = (q Qx)T p < 0
holds. Therefore, if Q is positive definite, we are guaranteed that the
value of will be positive.
Among the classic approximate line searches we mention very briefly
the following:
Interpolation Take f (xk ), f (xk ), f (xk )T pk to model a quadratic
function approximating f along pk . Minimize it by using the analytic formula above.
Newtons method Repeat the improvements gained from a quadratic
approximation: := ()/ ().
Golden Section The golden section method is a derivative-free method
for minimizing unimodal functions.5 The method reduces an interval wherein the reduction is based only on evaluating . The
portion
left of the length of the previous interval after reduction is

51
0.618.
2

An approximate line search methodology often used is known as the

Armijo step length rule. The idea is to quickly generate a step length
which provides a sufficient decrease in the value of f . Note that
f (xk + pk ) f (xk ) + f (xk )T pk for very small values of > 0.
The requirement of the step length rule is that we get a decrease in
the left-hand side of the above approximate relation which is at least a
fraction of that predicted in the right-hand side.
Let (0, 1) be the fraction of decrease required. Then, the step
lengths accepted by the Armijo step length rule are the positive values
which satisfy the inequality
() (0) (0),

(11.11a)

that is,
f (xk + pk ) f (xk ) f (xk )T pk .

(11.11b)

Figure 11.3 illustrates the Armijo step length rule.

5 is unimodal in an interval [a, b] of R if it has a unique global minimum in
[a, b], and is strictly increasing to the left as well as to the right of the minimum.
This notion is equivalent to that of having a minimum over [a, b] and being strictly
quasi-convex there.

277

Unconstrained optimization

()

R
(0) + (0)

(0) + (0)

Figure 11.3: The interval R accepted by the Armijo step length rule.

The typical choices are the following: the value of is small [

(0.001, 0.01)]; and take := 1. If = 1 does not satisfy the inequality
(11.11), then take := /2, and check the inequality (11.11) again, and
so on. The choice of initial trial step = 1 is especially of interest
in Newton-type methods, where, locally around a stationary point x
where 2 f (x ) is positive definite, local convergence with step length
one is guaranteed. (See also Section 4.6.4.2.)
We can select any starting guess > 0 and any fraction (0, 1) in
place of the choice = 21 made above.
The Armijo condition is satisfied for any sufficiently small step length,
provided that f (xk )T pk < 0 holds (see Exercise 11.1). In itself it
therefore does not guarantee that the next iterate is much better in
terms of the objective value than the current one. Often, therefore, it is
combined with a condition such that
| (k )| | (0)|,
that is,
|f (xk + pk )T pk | |f (xk )T pk |,
holds for some [0, 1). This is called the Wolfe condition. A relaxed
condition, the weak Wolfe condition, of the form
(k ) (0)
is often preferred, since the latter takes less computations to fulfill. The
choices 0 < < < 1 lead to interesting descent algorithms when
the Armijo and weak Wolfe conditions are combined, and it is possible
(why?) to find positive step lengths that satisfy these two conditions
278

Convergent algorithms
provided only that f is bounded from below and pk is a direction of
descent.

11.4

Convergent algorithms

This section presents two basic convergence results for descent methods
under different step length rules.
Theorem 11.4 (convergence of a gradient related algorithm) Suppose
that f C 1 , and that for the initial point x0 the level set levf (f (x0 )) :=
{ x Rn | f (x) f (x0 ) } is bounded. Consider the iterative algorithm
defined by the description in Section 11.1. In this algorithm, we make
the following choices, valid for each iteration k:
pk satisfies the sufficient descent condition (11.4);
kpk k M , where M is some positive constant; and
the Armijo step length rule (11.11) is used.

Then, the sequence {xk } is bounded, the sequence {f (xk )} is descending

and lower bounded and therefore has a limit, and every limit point of
{xk } is stationary.
Proof. That the sequence {xk } is bounded follows since the algorithm,
as stated, is a descent method, and we assumed that the level set of f at
the starting point is bounded; therefore, the sequence of iterates must
remain in that set and is therefore bounded.
is a limit
The rest of the proof is by contradiction. Suppose that x
point of {xk } but that f (
x) 6= 0n . It is clear that by the continuity of f ,
f (xk ) f (
x). Hence, f (xk )f (xk+1 ) 0 must hold. According to the
Armijo rule, then, k f (xk )T pk 0. Here, there are two possibilities.
Suppose that k 0. Then, there must be some iteration k after which
the initial step length is not accepted by the inequality (11.11), and
therefore,
f (xk + (k /)pk ) f (xk ) > (k /)f (xk )T pk ,

k k.

Dividing both sides by k / we obtain in the limit that

(1 )f (
x)T p 0,
for any limit point p of the bounded sequence {pk }. But in the limit
of the inequalities in (11.4) we then clearly reach a contradiction to our
claim. So, in fact, we must have that k 6 0. In this case, then, by the
279

Unconstrained optimization
above we must have that f (xk )T pk 0 holds, so by letting k tend to
infinity we obtain that
f (
x)T p = 0,
which again produces a contradiction to the initial claim because of
(11.4). We conclude that f (
x) = 0n must therefore hold.
The above proof can be repeated almost in verbatim to establish
that any step length rule that provides reduction in the value of f that
is at least as good as that guaranteed by the Armijo rule will inherit its
convergence properties. The main argument is based on the inequality
f (xk+1 xk ) f (xk ) f (
xk+1 ) f (xk )
k f (xk )T pk ,
k+1 and
where x
k are the next iterate and step length resulting from
the use of the Armijo rule, respectively. If we repeat the arguments in the
above proof, replacing k with
k , we obtain the same contradictions
to the condition (11.4). For example, this argument can be used to
establish the convergence of gradient related algorithms using exact line
searches.
is a
We further note that there is no guarantee that the limit point x
local minimum; it may also be a saddle point, that is, a stationary point
where 2 f (
x) is indefinite, if it exists.
Another result is cited below from [BeT00]. It allows the Armijo step
length rule to be replaced by a much simpler type of step length rule
which is also used to minimize a class of non-differentiable functions (cf.
Section 6.4). The proof requires the addition of a technical assumption:
Definition 11.5 (Lipschitz continuity) A C 1 function f : Rn R is said
to have a Lipschitz continuous gradient mapping on Rn if there exists a
scalar L 0 such that
kf (x) f (y)k Lkx yk

(11.12)

holds for every x, y Rn .

Check that the gradient of a C 2 function f is Lipschitz continuous
whenever its Hessian matrix is bounded over Rn .
Theorem 11.6 (on the convergence of gradient related methods) Let f
C 1 . Consider the sequence {xk } generated by the formula xk+1 :=
xk + k pk . Suppose that:
f is Lipschitz continuous on Rn ;
c1 kf (xk )k2 f (xk )T pk ,

280

c1 > 0;

Finite termination criteria

kpk k c2 kf (xk )k,

c2 > 0;

k > 0 satisfies that k 0 and limk

Pk

s=1

s = .

Then, either f (xk ) holds, or f (xk ) f and f (xk ) 0n .

In Theorem 11.4 convergence is only established in terms of that of
subsequences, and the requirements include a level set boundedness condition that can be difficult to check. A strong convergence result is available for the case of convex functions f and the steepest descent method
whenever we know that there exists at least one optimal solution; it follows immediately from Theorem 12.4 on the gradient projection method
for differentiable optimization over convex sets. In fact, we have already
seen such a result in Theorem 6.25 for possibly even non-differentiable
convex functions.
Theorem 11.7 (convergence of the steepest descent method under convexity) Suppose that f : Rn R is convex and in C 1 on Rn . Suppose further that the problem (11.1) has at least one optimal solution.
Consider the steepest descent algorithm, where the step lengths k are
determined by the Armijo step length rule. Then, the sequence {xk }
converges to some optimal solution to (11.1).
We have so far neglected Step 4 in the algorithm description in Section 11.1 in that we assume in the above results that the sequence {xk }
is infinite. A termination criterion must obviously be applied if we are
to obtain a result in a finite amount of time. This is the subject of the
next section.

11.5

Finite termination criteria

As noted above, convergence to a stationary point is only asymptotic.

How does one know when to terminate? A criterion based only on a
small size of kf (xk )k is no good. Why? Because we compare with 0!
The recommendation is the combination of the following:
1. kf (xk )k 1 (1 + |f (xk )|), 1 > 0 small;

2. f (xk1 ) f (xk ) 2 (1 + |f (xk )|), 2 > 0 small; and

3. kxk1 xk k 3 (1 + kxk k), 3 > 0 small.

The right-hand sides are constructed in order to eliminate some of the

possible influences of bad scaling of the variable values, of the objective
function, and of the gradient, and also of the possibility that some values
are zero at the limit point.
281

Unconstrained optimization
Notice that using the criterion 2. only might mean that we terminate
too soon if f is very flat; similarly, using only 3., we terminate prematurely if f is steep around the stationary point we are approaching. The
presence of the constant 1 is to remove the dependency of the criterion
on the absolute values of f and xk , particularly if they are near zero.
We also note that using the k k2 norm may not be good when n is
very large: suppose
that f (
x) = (, , . . . , )T = (1, 1, . . . , 1)T . Then,

kf (
x)k2 = n , which illustrates that the dimension of the problem
may enter the norm. Better then is to use the -norm: kf (
x )k :=
)
f (x
max1jn | xj | = ||, which does not depend on n.
Norms may have other bad effects. From
xk1 = (1.44453, 0.00093, 0.0000079)T,
xk = (1.44441, 0.00012, 0.0000011)T;
kxk1 xk k = k(0.00012, 0.00081, 0.0000068)Tk = 0.00081
follows. Here, the termination test would possibly pass, although the
number of significant digits is very small (the first significant digit is still
changing in two components of x!) Norms emphasize larger elements, so
small ones may have bad relative accuracy. This is a case where scaling
is needed.
Suppose we know that x = (1, 104 , 106 )T . If, by transforming
= (1, 1, 1)T , then the same
the space, we obtain the optimal solution x
relative accuracy would be possible to achieve for all variables. Let then

1 0
0
= Dx, where D := 0 104
0 .
x
0 0 106
Let f (x) := 21 xT Qx q T x, where

8
Q := 3 104
0

3 104
4 108
1010

0
1010
6 1012

11
and q := 8 104 .
7 106

Hence, x = Q1 q = (1, 104 , 106 )T .

= Dx, we get the transformed problem to minimize f(
With x
x) :=
1
1
1 T
1 T

x
(D
QD
)
x

(D
q)
x
,
with
2
D 1 QD1

282

8 3
= 3 4
0 1

0
1 ;
6

11
D1 q = 8 ,
7

A comment on non-differentiability
= (1, 1, 1)T . Notice the change in the condition number of the
and x
matrix!
The steepest descent algorithm takes only f (x) into account, not
2 f (x). Therefore, if the problem is badly scaled, it will suffer from a
poor convergence behaviour. Introducing elements of 2 f (x) into the
search direction helps in this respect. This is the precisely the effect of
using second-order (Newton-type) algorithms.

11.6

A comment on non-differentiability

The subject of non-differentiable optimization will not be taken up in

generality here; it has been analyzed more fully for Lagrangian dual
problems in Chapter 6. The purpose of this discussion is to explain, by
means of an example, that things can go terribly wrong if we apply methods for the minimization of differentiable functions when the function is
non-differentiable.
Rademachers Theorem states that a Lipschitz continuous function
[cf. (11.12) for a statement of the Lipschitz condition for a vector-valued
function] automatically is differentiable almost everywhere. It seems to
imply that we need not worry about differentiability, because it is very
unlikely that a non-differentiable point will be hit by mistake. This is
certainly true if the subject is simply to pick points at random, but the
subject of optimization deals with searching for a particular, extremal
point in the sense of the objective function, and such points tend to
be non-differentiable with a higher probability than zero! Suppose for
example that we consider the convex (why?) function
f (x) := maximum {cT
i x + bi },
i{1,...,m}

x Rn ,

that is, a function defined by the point-wise maximum of affine functions.

It has the appearance shown in Figure 11.4.
Clearly, the minimum of this function is located at a point where it
is non-differentiable.
We next look at a specific problem to which we will apply the method
of steepest descent. Suppose that we are given the following convex
(why?) objective function:6
(
5(9x21 + 16x22 )1/2 , if x1 > |x2 |,
f (x1 , x2 ) :=
9x1 + 16|x2 |,
if x1 |x2 |.
For x1 > |x2 |, f is actually continuously differentiable.
6 This

example is due to Wolfe [Wol75].

283

Unconstrained optimization
f (x)

x
Figure 11.4: A piece-wise affine convex function.

If we start at x0 anywhere in the region x1 > |x2 | > (9/16)2 |x1 |

then we obtain a sequence generated by steepest descent with exact
line searches that defines a polygonal path with successive orthogonal
= (0, 0)T .
segments, converging to x
is not a stationary point! The reason why it went wrong is that
But x
the gradients calculated say very little about the behaviour of f at the
limit point (0, 0)T . In fact, f is non-differentiable there. In this example,
it in fact holds that limx1 f (x1 , 0) = , so steepest descent has
failed miserably.
In order to resolve this problem, we need to take some necessary
measures; the ones below applies to convex functions:
a) At a non-differentiable point, f (x) must be replaced by a welldefined extension. Usually, we would replace it with a subgradient,
that is, one of the vectors that define a supporting hyperplane to
it is the set defined by the convex hull of the
the graph of f . At x
two vectors (9, 16)T and (9, 16)T .

b) The step lengths must be chosen differently; exact line searches are
clearly forbidden, as we have just seen.

From such considerations, we may develop algorithms that find optima to non-differentiable problems. They are referred to as subgradient
algorithms, and are analyzed in Section 6.4.

11.7

Trust region methods

Trust region methods use quadratic models like Newton-type methods

do, but avoiding a line search by instead bounding the length of the
search direction, thereby also influencing its direction.
284

Conjugate gradient methods

Let k (p) := f (xk ) + f (xk )T p + 21 pT 2 f (xk )p. We say that the
model k is trusted only in a neighbourhood of xk : kpk k . The
use of this bound is apparent when 2 f (xk ) is not positive semidefinite.
The problem to minimize k (p) subject to kpk k can be solved
(approximately) quite efficiently. The idea is that when 2 f (xk ) is
badly conditioned, the value of k should be kept lowthus turning
the algorithm more into a steepest descent-like method [recall (11.2)]
while if 2 f (xk ) is well conditioned, k should become large and allow
unit steps to be taken. (Prove that the direction of pk tends to that of
the steepest descent method when k 0!)
The vector pk that solves the trust region problem satisfies [2 f (xk )+
k I n ]pk = f (xk ) for some k 0 such that 2 f (xk )+I n is positive
semidefinite. The bounding enforced hence has a similar effect to that of
the LevenbergMarquardt strategy discussed in Section 11.2.2. Provided
that the value of k is small enough, f (xk + pk ) < f (xk ) holds. Even
if f (xk ) = 0n holds, f (xk + pk ) < f (xk ) if 2 f (xk ) is not positive
semidefinite. So, progress is made also from stationary points if they are
saddle points or local maxima. The robustness and strong convergence
characteristics have made trust region methods quite popular.
The update of the trust region size is based on the following measure
of similarity between the model k and f : Let
k =

f (xk ) f (xk + pk )
actual reduction
=
.
f (xk ) k (pk )
predicted reduction

If k let xk+1 = xk (unsuccessful step), else

xk+1 = xk + pk (successful step).
The value of k is updated in the following manner, depending on
the value of k :
k = k+1 = 12 k ,
< k < = k+1 = k ,
k = k+1 = 2k .
Here, 0 < < < 1, with typical choices being = 41 and = 43 ; is a
bound used for deciding when the model can or cannot be trusted even
within the region given, while is used for deciding when the model is
good enough to be used in a larger neighbourhood.
Figure 11.5 illustrates the trust region subproblem.

11.8

Conjugate gradient methods

When applied to nonlinear unconstrained optimization problems conjugate direction methods are methods intermediate between the steepest
285

Unconstrained optimization

xk

Figure 11.5: Trust region and line search step. The dashed ellipses
are two level curves of the quadratic model constructed at xk , while the
dotted circle is the boundary of the trust region. A step to the minimum
of the quadratic model is here clearly inferior to the step taken within
the trust region.

descent and Newton methods. The motivation behind them is similar to that for quasi-Newton methods: accelerating the steepest descent
method but avoid the evaluation, storage and inversion of the Hessian
matrix. They are analyzed for quadratic problems only; extensions to
non-quadratic problems utilize that close to an optimal solution every
problem is nearly quadratic. Even for non-quadratic problems, the last
few decades of developments have resulted in conjugate direction methods being among the most efficient general methodologies available.

11.8.1

Conjugate directions

Definition 11.8 (conjugate direction) Let Q Rnn be symmetric.

Two vectors p1 and p2 in Rn are Q-orthogonal, or, conjugate with respect to Q, if pT
1 Qp2 = 0.
Note that if Q is the zero matrix then every pair of vectors in Rn are
conjugate; when Q is the unit matrix, conjugacy reduces to orthogonality. The following result is easy to prove (see Exercise 11.14).
286

Conjugate gradient methods

Proposition 11.9 (conjugate vectors are linearly independent) If Q
Rnn is symmetric and positive definite and the collection p1 , p2 , . . . , pk
are mutually conjugate with respect to Q, then they are also linearly
independent.
The usefulness of conjugate directions for the quadratic problem to
minimize
f (x) :=
n
x R

1 T
x Qx q T x,
2

(11.13)

where from now on Q is assumed to be symmetric and positive definite, is

clear from the following identification: if the vectors p0 , p1 , . . . , pn1 are
Q-orthogonal, then Proposition 11.9 implies that there exists a vector
w Rn with
n1
X
x =
wi pi ;
(11.14)
i=0

multiplying the equation by Q and scalar multiplying the result by pi

yields
pT q
pT Qx
= Ti
,
(11.15)
wi = iT
pi Qpi
pi Qpi
so that
x =

n1
X
i=0

pT
i q
pi .
pT
i Qpi

(11.16)

Two ideas are embedded in (11.16): by selecting a proper set of orthogonal vectors pi , and by taking the appropriate scalar product all terms
but i in (11.14) disappear. This could be accomplished by using any n orthogonal vectors, but (11.15) shows that by making them Q-orthogonal
we can express wi without knowing x .

11.8.2

Conjugate direction methods

The corresponding conjugate direction method for (11.13) is given by

xk+1 = xk + k pk ,

k = 0, . . . , n 1,

where x0 Rn is arbitrary and k is obtained from an exact line search

with respect to f in the direction of pk ; cf. (11.8). The principal result
about conjugate direction methods is that successive iterates minimize
f over a progressively expanding linear manifold that after at most n
iterations includes the minimizer of f over Rn . In other words, defining
Mk := { x Rn | x = x0 + subspace spanned by {p0 , p1 , . . . , pk } },
287

Unconstrained optimization
{xk+1 } = arg minimum f (x)
x Mk

(11.17)

holds. To show this, note that by the exact line search rule, for all i,

f (xi + pi ) 
= f (xi+1 )T pi = 0.


=i

and for i = 0, 1, . . . , k 1,

f (xk+1 )T pi = (Qxk+1 q)T pi

T

k
X
= xi+1 +
j pj Qpi q T pi
j=i+1

xT
i+1 Qpi

q T pi

= f (xi+1 )T pi ,

where we used the conjugacy of pi and pj , j = i + 1, . . . , k. Hence,

f (xk+1 )T pi = 0 for every i = 0, 1, . . . , k, which verifies (11.17).
It is easy to get a picture of what is going on if we look at the case
where Q = I n and q = 0n ; since the level curves are circles, minimizing
over the n coordinates one by one gives us x in n steps; in each iteration
we also identify the optimal value of one of the variables. Conjugate
directions in effect does this, although in a transformed space.7
The discussion so far has been based on an arbitrary selection of
conjugate directions. There are many ways in which conjugate directions could be generated. For example, we could let the vectors pi ,
i = 0, . . . , n 1 be defined by the eigenvectors of Q, as they are mutually orthogonal as well as conjugate with respect to Q. (Why?) Such a
procedure would however be too costly in large-scale applications. The
remarkable feature of the conjugate gradient method to be presented below is that the new vector pk can be generated directly from the vector
pk1 ; there is no need to remember any of the vectors p0 , . . . , pk2 , and
yet pk will be conjugate to them all.

11.8.3

Generating conjugate directions

Given a set of linearly independent vectors d0 , d1 , . . . , dk we can generate

a set of mutually Q-orthogonal vectors p0 , p1 , . . . , pk such that they
7 Compare this to Newtons method as applied to the problem (11.13); its convergence in one step corresponds to the convergence in one step of the steepest descent
method when we first have performed a coordinate transformation such that the level
curves become circular.

288

Conjugate gradient methods

span the same subspace, by using the GramSchmidt procedure. We
start the recursion with p0 = d0 . Suppose that for some i < k we have
d0 , d1 , . . . , di such that they span the same subspace as p0 , p1 , . . . , pi .
Then, let pi+1 take the following form:
pi+1 = di+1 +

i
X

ci+1
m pm ,

m=0

ci+1
m

choosing
so that pi+1 is Q-orthogonal to p0 , p1 , . . . , pi . This will
be true if, for each j = 0, 1, . . . , i,
!T
i
X
T
pT
ci+1
Qpj = 0.
i+1 Qpj = di+1 Qpj +
m pm
m=0

Since p0 , p1 , . . . , pi are Q-orthogonal we have pT

m Qpj = 0 if m 6= j, so
ci+1
=
j

dT
i+1 Qpj
,
pT
j Qpj

j = 0, 1, . . . , i.

Some notes are in order regarding the above development:

1. pT
j Qpj 6= 0.

2. pi+1 6= 0n ; otherwise it would contradict the linear independence

of d0 , d1 , . . . , dk .
3. di+1 lies in the subspace spanned by p0 , p1 , . . . , pi+1 , while pi+1
lies in the subspace spanned by d0 , d1 , . . . , di+1 , since these vectors span the same space. Therefore, the subspace identification
above is true for i + 1, and we have shown that the GramSchmidt
procedure has the property asked for.

11.8.4

Conjugate gradient methods

The conjugate gradient method applies the above GramSchmidt procedure to the vectors
d0 = f (x0 ),

d1 = f (x1 ),

...,

dn1 = f (xn1 ).

Thus, the conjugate gradient method is to take xk+1 = xk + k pk ,

where k is determined through an exact line search and pk is obtained through step k of the GramSchmidt procedure to the vector
dk = f (xk ) and the previous vectors p0 , p1 , . . . , pk1 . In particular,
pk = f (xk ) +

k1
X
j=0

f (xk )T Qpj
pj .
pT
j Qpj

(11.18)

289

Unconstrained optimization
It holds that p0 = f (x0 ), and termination occurs at step k if f (xk ) =
0n ; the latter happens exactly when pk = 0n . (Why?)
[Note: the search directions are based on negative gradients of f ,
f (xk ) = q Qxk , which are identical to the residual in the linear
system Qx = q that identifies the optimal solution to (11.13).]
The formula (11.18) can in fact be simplified. The reason is that,
because of the successive optimization over subspaces, f (xk ) is orthogonal to the subspace spanned by p0 , p1 , . . . , pk1 .
Theorem 11.10 (the conjugate gradient method) The directions of the
conjugate gradient method are generated by
p0 = f (x0 );

pk = f (xk ) + k pk1 ,

(11.19a)
k = 1, 2, . . . , n 1,

(11.19b)

where
k =

f (xk )T f (xk )
.
f (xk1 )T f (xk1 )

(11.19c)

Moreover, the method terminates after at most n steps.

Proof. We first use induction to show that the gradients f (xk ) are linearly independent. It is clearly true for k = 0. Suppose that the method
has not terminated after k steps, and that f (x0 ), f (x1 ), . . . , f (xk1 )
are linearly independent. Being a conjugate gradient method we know
that the subspace spanned by these vectors is the same as that spanned
by the vectors p0 , p1 , . . . , pk1 :
span (p0 , p1 , . . . , pk1 ) = span [f (x0 ), f (x1 ), . . . , f (xk1 )].
(11.20)
Now, either f (xk ) = 0n , whence the algorithm terminates at the
optimal solution, or f (xk ) 6= 0n , in which case (by the expanding manifold property) it is orthogonal to p0 , p1 , . . . , pk1 . By (11.20) f (xk ) is
linearly independent of f (x0 ), f (x1 ), . . . , f (xk1 ), completing the
induction. Since we have at most n linearly independent vectors in Rn
the algorithm must stop after at most n steps.
The proof is completed by showing that the simplification in (11.19c)
is possible. For all j with f (xj ) 6= 0n we have that
f (xj+1 ) f (xj ) = Q(xj+1 xj ) = j Qpj ,
290

Conjugate gradient methods

and, since j 6= 0,
1
f (xi )T [f (xj+1 ) f (xj )]
j
(
0,
if j = 0, 1, . . . , i 2,
= 1
T
j f (xi ) f (xi ), if j = i 1,

f (xi )T Qpj =

and also that

pT
j Qpj =

1 T
p [f (xj+1 ) f (xj )].
j j

Substituting these two relations into the GramSchmidt formula, we

obtain that (11.19b) holds, with
k =

f (xk )T f (xk )
.
pT
k1 [f (xk ) f (xk1 )]

From (11.19b) follows that pk1 = f (xk1 ) + k1 pk2 . Using this

equation and the orthogonality of f (xk ) and f (xk1 ) we can write
the denominator in the expression for k as desired. We are done.
We can deduce further properties of the algorithm. If the matrix
Q has the eigenvalues 1 2 n then we have the following
estimate of the distance to the optimal solution after iteration k + 1:
kxk+1

x k2Q

nk 1
nk + 1

2

kx0 x k2Q ,

where kzk2Q = z T Qz, z Rn . What does this estimate tell us about

the behaviour of the conjugate gradient algorithm? Suppose that we
have a situation where the matrix Q has m large eigenvalues, and the
remaining n m eigenvalues all are approximately equal to 1. Then the
above tells us that after m + 1 steps of the conjugate gradient algorithm,
kxm+1 x kQ (nm 1 )kx0 x kQ .
For a small value of nm 1 this implies that the algorithm gives a
good estimate of x already after m + 1 steps. The conjugate gradient
algorithm hence eliminates the effect of the largest eigenvalues first, as
the convergence rate after the first m + 1 steps does not depend on the
m + 1 largest eigenvalues.
The exercises offer additional insight into this convergence theory.
291

Unconstrained optimization
This is in sharp contrast with the convergence rate of the steepest
descent algorithm, which is known to be
kxk+1

x k2Q

n 1
n + 1

2

kxk x k2Q ;

in other words, the rate of convergence worsens as the condition number

of the matrix Q, (Q) := n /1 , increases.8
Nevertheless, the conjugate gradient method often comes with a preconditioning, which means that the system solved is not Qx = q but
M Qx = M q for some invertible square matrix M , constructed such
that the eigenvalues of M Q are better clustered than Q itself. (In other
words, the condition number is reduced.)

11.8.5

Extension to non-quadratic problems

Due to the orthogonality of f (xk ) and f (xk1 ), we could rewrite

(11.19c) as
f (xk )T [f (xk ) f (xk1 )]
k =
.
(11.21)
f (xk1 )T f (xk1 )

The formula (11.19c) is often referred to as the FletcherReeves formula

(after the paper [FlR64]), while the formula (11.21) is referred to as the
PolakRibi`ere formula (after the paper [PoR69]).
For the quadratic programming problem, the two formulas are identical. However, they would not produce the same sequence of iterates
if f were non-quadratic, and the conjugate gradient method has been
extended also to such cases. The normal procedure is then to utilize
the above algorithm for k < n steps, after which a restart is made at
the current iterate using the steepest descent direction; that is, we use
the conjugate gradient algorithm several times in succession, in order to
not lose conjugacy. The algorithm is not any more guaranteed to terminate after n steps, of course, but the algorithm has been observed to be
quite efficient when the objective function and gradient values are cheap
to evaluate; especially, this is true when comparing the algorithm class
to that of quasi-Newton. (See [Lue84, Ber99] for further discussions on
8 This type of bound on the convergence rate of the steepest descent method can
also be extended to non-quadratic problems: suppose x is the unique optimal solution to the problem of minimizing the C 2 function f and that 2 f (x ) is positive
definite. Then, with 0 < 1 n being the eigenvalues of 2 f (x ) we have
that for all k,

f (xk+1 ) f (x )

292

1
n + 1
n

2

[f (xk ) f (x )].

A quasi-Newton method: DFP

such computational issues.) It is also remarked in several sources that
the PolakRibi`ere formula (11.21) is preferable in the non-quadratic case.

11.9

A quasi-Newton method: DFP

As we have already touched upon in Section 11.2.2, most quasi-Newton

methods are based on the idea to try to construct the (inverse) Hessian,
or an approximation of it, through the use of information gathered in the
process of solving the problem; the algorithm then works as a deflected
gradient method where the matrix scaling of the negative of the gradient
vector is the current approximation of the inverse Hessian matrix.
The BFGS updating formula that was given in Section 11.2.2 is a
rank-two update of the Hessian matrix. There are several other versions
of the quasi-Newton method, the most popular being based on rank-two
updates but of the inverse of the Hessian rather than the Hessian matrix
itself. We present one such method below.
The DavidonFletcherPowell algorithm is given in the two papers
[Dav59, FlP63]. The algorithm is of interest to us here especially because
we can show that through a special choice of matrix update, the quasiNewton method implemented with an exact line search works exactly like
a conjugate gradient method! Moreover, since quasi-Newton methods
do not rely on exact line searches for convergence, we learn that quasiNewton methods are, in this sense, more general than conjugate gradient
methods.
The algorithm can be explained like this, as applied to the problem
(11.13), the matrix Q being symmetric and positive definite: start with
a symmetric and positive definite matrix H0 Rnn , a point x0 Rn ,
and with k = 0; then set
dk = Hk f (xk );

(11.22a)

{k } = arg minimum f (xk + dk );

(11.22b)

xk+1 = xk + k dk ;

(11.22c)

pk = k dk ;
q k = f (xk+1 ) f (xk );
Hk+1 = Hk +

pk pT
(Hk q k )(q T
k
k Hk )

;
T
T
pk q k
q k Hk q k

(11.22d)
(11.22e)
(11.22f)

and repeat with k := k + 1.

We note that the matrix update in (11.22f) is a rank two update, since
the two matrices added to Hk both are defined by the outer product of
a given vector with itself.
293

Unconstrained optimization
We first demonstrate that the matrices Hk are positive definite. For
any x Rn we have
xT Hk+1 x = xT Hk x +
1/2

(xT pk )2
(xT Hk q k )2
.

pk q k
qT
k Hk q k

1/2

Defining a = Hk x and b = Hk q k we can write this as

xT Hk+1 x =

(aT a)(bT b) (aT b)2

(xT pk )2
+
.
T
pT
b b
k qk

We also have that

T
T
T
pT
k q k = pk f (xk+1 ) pk f (xk ) = pk f (xk ),

since
pT
k f (xk+1 ) = 0

(11.23)

due to the line search being exact. Therefore, by the definition of pk ,

T
pT
k q k = k f (xk ) Hk f (xk ),

and hence
xT Hk+1 x =

(aT a)(bT b) (aT b)2

(xT pk )2
+
.
k f (xk )T Hk f (xk )
bT b

Both terms in the right-hand side are non-negative, the first because
of the CauchyBunyakowskiSchwarz inequality. We must finally show
that not both can be zero at the same time. The first term disappears
precisely when a and b are parallel. This in turn implies that x and q k
are parallel, say, x = q k for some R. But this would mean that
T
T
pT
k x = pk q k = k f (xk ) Hk f (xk ) 6= 0,

whence xT Hk+1 x > 0 holds.

Notice that the fact that the line search is exact is not actually used;
it is enough that the k chosen yields that pT
k q k > 0.
The following proposition shows that the DavidonFletcherPowell
(DFP) algorithm (11.22) is a conjugate gradient algorithm which provides an optimal solution to (11.13) in at most n steps, when H n = Q1 .
Theorem 11.11 (finite convergence of the DFP algorithm) Consider the
algorithm (11.22) for the problem (11.13). Then,
pT
i Qpj = 0,
Hk+1 Qpi = pi ,
holds.
294

0 i < j k,
0ik

(11.24a)
(11.24b)

A quasi-Newton method: DFP

Proof. We have that
q k = f (xk+1 ) f (xk ) = Qxk+1 Qxk = Qpk ,

(11.25)

and
Hk+1 Qpk = Hk+1 q k = pk ,

(11.26)

the latter from (11.22f).

Proving (11.24) by induction, we see from the above equation that it
is true for k = 0. Assume (11.24) is true for k 1. We have that
f (xk ) = f (xi+1 ) + Q(pi+1 + + pk1 ).
Therefore, from (11.24a) and (11.23),
T
pT
i f (xk ) = pi f (xi+1 ) = 0,

0 i < k.

Hence from (11.24b)

pT
i QHk f (xk ) = 0.

Thus, since pk = k Hk f (xk ) and since k 6= 0, we obtain

pT
i Qpk = 0.

i < k,

(11.27)

which proves (11.24a) for k.

Now, since from (11.24b) for k 1, (11.25), and (11.27)
T
T
qT
k Hk Qpi = q k pi = pk Qpi = 0,

0 i < k,

we have that
Hk+1 Qpi = Hk Qpi = pi ,

0 i < k.

This together with (11.26) proves (11.24b) for k.

Since the pk -vectors are Q-orthogonal and since we minimize f successively over these directions, the DFP algorithm is a conjugate direction method. Especially, if the initial matrix H0 is taken to be the unit
matrix, it becomes the conjugate gradient method. In any case, however,
convergence is obtained after at most n steps.
Finally, we note that (11.24b) shows that the vectors p0 , p1 , . . . , pk
are eigenvectors corresponding to unity eigenvalues of the matrix Hk+1 Q.
These eigenvectors are linearly independent, since they are Q-orthogonal,
and therefore we have that Hn = Q1 . In other words, with any choice
of initial matrix H0 (as long as it is symmetric and positive definite) n
steps of the 2-rank updates in (11.22f) result in the final matrix being
identical to the inverse of the Hessian.
295

Unconstrained optimization

11.10

Convergence rates

The local convergence rate is a statement about the speed in which one
iteration takes the guess closer to the solution.
Definition 11.12 (local convergence rate) Suppose that {xk } Rn and
that xk x . Consider for large k the quotients
qk :=

kxk+1 x k
.
kxk x k

(a) [linear convergence rate] The speed of convergence is linear if

lim sup qk < 1.
k

A linear convergence rate is roughly equivalent to the statement that we

get one new correct digit per iteration.
(b) [superlinear convergence rate] The speed of convergence is superlinear if
lim qk = 0.
k

(c) [quadratic convergence rate] The speed of convergence is quadratic

if
lim sup
k

qk
c,
kxk x k

c 0.

A quadratic convergence rate is roughly equivalent to the statement that

the number of correct digits is doubled in every iteration.
The steepest descent method has, at most, a linear rate of convergence, moreover often with a constant qk near unity. Newton-like algorithms have, however, superlinear convergence if 2 f (x ) is positive
definite, and even quadratic local convergence can be achieved for Newtons method if 2 f is Lipschitz continuous in a neighbourhood of x .

11.11

Implicit functions

Suppose that the value of f (x) is given through a simulation procedure,

according to Figure 11.6.
If the response y = y(x) from the input x is unknown explicitly, then
we cannot differentiate x 7 f (x, y(x)) with respect to x. If, however,
we believe that y() is differentiable, which means that y is very stable
with respect to changes in x, then x y(x), and hence x f (x, y(x)) can
be calculated numerically. The use of the Taylor expansion technique
that follows is only practical if y(x) is cheap to calculate.
296

Notes and further reading

x Rn

Simulation

y Rm

Figure 11.6: A simulation procedure.

Let ei = (0, 0, . . . , 0, 1, 0, . . . , 0)T be the unit vector in Rn (the only

non-zero entry is in position i). Then,
2
T 2
f (x + hei ) = f (x) + heT
i f (x) + (h /2)ei f (x)ei +

= f (x) + hf (x)/xi + (h2 /2) 2 f (x)/x2i + .

So, for small h > 0,

f (x)
f (x + hei ) f (x)

,
(forward difference)
xi
h
f (x)
f (x + hei ) f (x hei )

.
(central difference)
xi
2h
The value of h is typically set to a function of the machine precision; if
chosen too large, we get a bad approximation of the partial derivative,
while a too small value might result in numerical cancellation.
The automatic differentiation technique exploits the inherent structure of most practical functions, that they almost always are evaluated
through a sequence of elementary operations. Automatic differentiation
represents this structure in the form of a computational graph; when
forming partial derivatives this graph is utilized in the design of chain
rules for the automatic derivative calculation. In applications to simulation models, this means that differentiation is performed within the
simulation package, thereby avoiding some of the computational cost and
the potential instability inherent in difference formulas.

11.12

Notes and further reading

The material of this chapter is classic; text books covering similar material in more depth include [OrR70, DeS83, Lue84, Fle87, BSS93, BGLS03].
Line search methods were first developed by Newton [New1687], and the
steepest descent method is due to Cauchy [Cau1847]. The Armijo rule is
due to Armijo [Arm66], and the Wolfe condition is due to Wolfe [Wol69].
The classic book by Brent [Bre73] analyzes algorithms that do not use
derivatives, especially line search methods.
297

Unconstrained optimization
Rademachers Theorem [Rad19] states that a Lipschitz continuous
function is differentiable everywhere except on sets of Lebesgue measure
zero. The Lipschitz condition is due to Lipschitz [Lip1877]. Algorithms
for the minimization of non-differentiable convex functions are given in
[Sho85, HiL93, Ber99, BGLS03].
Trust region methods are given a thorough treatment in the book
[CGT00]. The material on the conjugate gradient and BFGS methods
was collected from [Lue84, Ber99]; another good source is [NoW99].
A popular class of algorithms for problems with an implicit objective
function is the class of pattern search methods. With such algorithms
the search for a good gradient-like direction is replaced by calculations of
the objective function along directions specified by a pattern of possible
points. For an introduction to the field, see [KLT03].
Automatic differentiation is covered in the monograph [Gri00].

11.13

Exercises

Exercise 11.1 (well-posedness of the Armijo rule) Through an argument by

contradiction, establish the following: If f C 1 , xk Rn and pk Rn
satisfies f (xk )T pk < 0, then for every choice of (0, 1) there exists
>0
such that every (0,
] satisfies (11.11). In other words, which ever positive
first trial step length we choose, we will find a step length that satisfies
(11.11) in a finite number of trials.
Exercise 11.2 (descent direction) Investigate whether the direction of p =
(2, 1)T is a direction of descent with respect to the function f (x) := x21 +
x1 x2 4x22 + 10 at x := (1, 1)T .
Exercise 11.3 (Newtons method) Suppose that you wish to solve the unconstrained problem to minimize f (x) over x Rn , where f C 2 (Rn ). You are
naturally interested in using Newtons method (with line searches).
(a) At some iteration you get the error message, Step length is zero.
Which reason(s) can there be for such a message?
(b) At some iteration you get the error message, Search direction does
not exist. Which reason(s) can there be for such a message?
(c) Describe at least one means to modify Newtons method such that
neither of the above two error messages will ever appear.
Exercise 11.4 (steepest descent) Is it possible to reach the (unique) optimal
solution to the problem to
minimize
f (x) := (x1 2)2 + 5(x2 + 6)2
n
x

by the use of the steepest descent algorithm, if we first perform a variable

substitution? If so, perform it and thus find the optimal solution.

298

Exercises
Exercise 11.5 (steepest descent) Consider the problem to
f (x) := (2x21 x2 )2 + 3x21 x2 .
minimize
n
x

(a) Perform one iteration of the steepest descent method using an exact
line search, starting at x0 := (1/2, 5/4)T .
(b) Is the function convex around x1 ?
(c) Will it converge to a global optimum? Why/why not?
Exercise 11.6 (Newtons method with exact line search) Consider the problem
to
minimize
f (x) := (x1 + 2x2 3)2 + (x1 2)2 .
n
x

(a) Start from x0 := (0, 0)T , and perform one iteration of Newtons method
with an exact line search.
(b) Are there any descent directions from x1 ?
(c) Is x1 optimal? Why/why not?
Exercise 11.7 (Newtons method with Armijo line search) Consider the problem to
1
minimize
f (x) := (x1 2x2 )2 + x41 .
x n
2

(a) Start from x0 := (2, 1)T , and perform one iteration of Newtons method
with the Armijo rule, using the fraction requirement = 0.1.
(b) Determine the values of (0, 1) such that the step length = 1 will
be accepted.
Exercise 11.8 (Newtons method for nonlinear equations) Suppose the function f : Rn Rn is continuously differentiable and consider the following
system of nonlinear equations:

f (x) = 0n .
Newtons method for the solution of unconstrained optimization problems has
its correspondence for the above problem.
Given an iterate xk we construct the following linear approximation of the
nonlinear function:

f (xk ) + f (xk )(x xk ) = 0n ,

or, equivalently,
where

f (xk )x = f (xk )xk f (xk ),

0

f1 (x)T
B f2 (x)T C
B
C
f (x) = B
C
..

A
.
T
fn (x)

299

Unconstrained optimization
is the Jacobian of f at x. Assuming that f (x) is non-singular, this linear
system has a unique solution which defines the new iterate, xk+1 , that is,

xk+1 = xk f (xk )1 f (xk ).

(One can show that if f satisfies some additional requirements, this sequence
of iterates will converge to a solution to the original nonlinear system, either
from any starting pointglobal convergenceor from a point sufficiently close
to a solutionlocal convergence.)
(a) Consider the nonlinear system


f (x , x )

f (x1 , x2 ) = f1 (x1 , x2 ) =
2
1
2

2(x1 2)3 + x1 2x2

4x2 2x1

 

Perform one iteration of the above algorithm, starting from

Calculate the value of
kf (x1 , x2 )k =

0
.
0

x0 = (1, 0)T .

f1 (x1 , x2 )2 + f2 (x1 , x2 )2

both at x0 and x1 . (Observe that kf (x)k = 0 if and only if f (x) = 0n , whence

the values of kf (xk )k, k = 1, 2, . . . , can be used as a measure of convergence
of the iterates.)
(b) Explain why the above method generalizes Newtons method for unconstrained optimization to a larger class of problems.
Exercise 11.9 (over-determined linear equations) Consider the problem to

minimize
n
x

1
kAx bk2 ,
2

where A Rnn and b Rm . Assume that m n and that rank A = n.

(a) Write down the necessary optimality conditions for this problem. Are
they also sufficient for global optimality? Why/why not?
(b) Write down the globally optimal solution in closed form.
Exercise 11.10 (sufficient descent conditions) Consider the first sufficient descent condition in (11.4). Why does it have that form, and why is the alternative form
f (xk )T pk s1
not acceptable?
Exercise 11.11 (Newtons method under affine transformations) Suppose we
make the following change of variables: y := Ax + b, where A Rnn
is invertible. Show that Newtons method is invariant to such changes of
variables.

300

Exercises
Exercise 11.12 (LevenbergMarquardt, exam 990308) Consider the unconstrained optimization problem to
minimize f (x) := qT x +
subject to

xR ,

1 T
x Qx,
2

(11.28a)
(11.28b)

where Q Rnn is symmetric and positive semidefinite but not positive definite. We attack the problem through a LevenbergMarquardt strategy, that
is, we utilize a Newton-type method where a multiple > 0 of the unit matrix
is added to the Hessian of f (that is, to the matrix Q) in order to guarantee
that the (modified) Newton equation is uniquely solvable. (See Section 11.2.2.)
This implies that, given an iteration point xk , the search direction pk is determined by solving the linear system
[2 f (xk ) + I n ]p = f (xk )

[Q + I ]p = (Qxk + q).
n

(11.29)

(a) Consider the formula

xk+1 := xk + pk ,

k = 0, 1, . . . ,

(11.30)

that is, the algorithm that is obtained by utilizing the Newton-like search
direction pk from (11.29) and the step length 1 in every iteration. Show that
this iterative step is the same as that to let xk+1 be given by the solution to
the problem to

ky xk k2 ,
2
Rn .

minimize f (y ) +
subject to

(11.31a)
(11.31b)

(b) Suppose that an optimal solution to (11.28) exists. Suppose also that
the sequence {xk } generated by the algorithm (11.30) converges to a point x .
(This can actually be shown to hold.) Show that x is optimal in (11.28).
[Note: This algorithm is in fact a special case of the proximal point algorithm. Suppose that f is a convex function on Rn and the variables are
constrained to a non-empty, closed and convex set S Rn .
We extend the iteration formula (11.31) to the following:
minimize f (y ) +
subject to

y S,

k
ky xk k2 ,
2

(11.32a)
(11.32b)

where {k } (0, 2) is a sequence of positive numbers that is bounded away

from zero, and where xk+1 is taken as the unique vector y solving (11.32). If
an optimal solution exists, it is possible to show that the sequence given by
(11.32) converges to a solution. See [Pat98, Ber99] for overviews of this class
of methods. (It is called proximal point because of the above interpretation:
that the next iterate is close, proximal, to the previous one.)]

301

Unconstrained optimization
Exercise 11.13 (unconstrained optimization algorithms, exam 980819) Consider the unconstrained optimization problem to minimize f (x) over x Rn ,
where f : Rn R is in C 1 . Let {xk } be a sequence of iteration points generated by some algorithm for solving this problem, and suppose that it holds
that f (xk ) 0n , that is, the gradient value tends to zero (which of course
is a favourable behaviour of the algorithm). The question is what this means
in terms of the convergence of the more important sequence {xk }.
Consider therefore the sequence {xk }, and also the sequence {f (xk )} of
function values. Given the assumption that f (xk ) 0n , is it true that
{xk } and/or {f (xk )} converges or are even bounded? Provide every possible
case in terms of the convergence of these two sequences, and give examples,
preferably simple ones for n = 1.
Exercise 11.14 (conjugate directions) Prove Proposition 11.9.
Exercise 11.15 (conjugate gradient method) Apply the conjugate gradient
method to the system Qx = q , where
0

2
Q = 1
0

1
2
1

0
1A
2

0 1

and

q = 1A .
1

Exercise 11.16 (convergence of the conjugate gradient method, I) In the conjugate gradient method, prove that the vector pi can be written as a linear
combination of the set of vectors {q , Qq, Q2 q, . . . , Qi q}. Also prove that xi+1
minimizes the quadratic function Rn x 7 f (x) := 12 xT Qx q T x over all
the linear combinations of these vectors.
Exercise 11.17 (convergence of the conjugate gradient method, II) Use the
result of the previous problem to establish that the conjugate gradient method
converges in a number of iterations equal to the number of distinct eigenvalues
of the matrix Q.

302

Optimization over
convex sets

12.1

XII

Feasible direction methods

Consider the problem to

minimize f (x),

(12.1a)

subject to x X,

(12.1b)

where X Rn is a nonempty, closed and convex set and f : Rn

R {+} is a C 1 function on X.
In almost all cases, algorithms for this problem will not be based on
staying feasible (xk X) but rather to reach feasibility and optimality
at the same time. Why? If X is defined by (convex) inequalities of
the form gi (x) bi , where gi is nonlinear, then checking, for example,
whether p is a feasible direction at x, or what the maximum feasible step
from x in the direction of p is, is very difficult. For example, in the latter
case, for which step length > 0 does it happen that gi (x + p) = bi ?
This is a nonlinear equation!
The notable exception is when X is a polyhedral set, which we from
here on in this chapter will assume is the case. How to characterize a
feasible direction in the polyhedral case has already been analyzed in
Example 4.22.
A general framework of feasible-direction methods for the problem
(12.1) can be extended from the unconstrained world as follows (notice
the difference to the description given in Section 11.1):
Feasible descent algorithm:
Step 0 (initialization). Determine a starting point x0 Rn such that
x0 X. Set k := 0.

Optimization over convex sets

Step 1 (feasible descent direction). Determine a search direction pk
Rn such that pk is a feasible direction.
Step 2 (line search). Determine a step length k > 0 such that f (xk +
k pk ) < f (xk ) and xk + k pk X.
Step 3 (update). Let xk+1 := xk + k pk .
Step 4 (termination check). If a termination criterion is fulfilled, then
stop! Otherwise, let k := k + 1 and go to step 1.
This type of algorithm is local, just as in the unconstrained case, and
there are more difficulties compared to the unconstrained case, associated with generating search directions pk : they must simultaneously be
feasible and provide descent. The linearity of the constraints however ensures that it is possible to solve approximations of the original problem,
such as approximations based on a Taylor expansion of f around the
current iterate xk , or to determine the active constraint set at xk and
generating a feasible search direction in a neighbourhood determined by
them. Moreover, we also need to determine a maximum step length in
the line search, and the termination criteria need to be different from
those in the unconstrained case because the gradient of f need not be
zero at a stationary point.
In the following, we will analyze three natural algorithms for the solution of the problem (12.1). The first two, the FrankWolfe and simplicial decomposition algorithms, are based on generating search directions
by solving linear problems, while the third one, the gradient projection
algorithm, corresponds to solving a more difficult, yet related, convex
quadratic problem. In each of these cases, the algorithms are derived
from the necessary optimality conditions associated with the problem
(12.1), which can be found in Section 4.4.
We will establish convergence for a general case, where all that can
be guaranteed is that any limit point of the sequence {xk } is stationary.
We will however also establish what can be achieved in addition when
the problem is convex. It is then not only the (obvious) case that every limit point then is globally optimal, but we can on occasion prove
something stronger. In the FrankWolfe method, we can utilize simpler
step length rules than the Armijo rule; in the simplicial decomposition
method, we can establish finite convergence even when previous information is discarded; and in the gradient projection method, we can prove
convergence to an optimal solution. The latter has interesting consequences for iterative methods, like the steepest descent algorithm, in the
unconstrained case, which it generalizes to the constrained case.
304

The FrankWolfe algorithm

12.2

The FrankWolfe algorithm

Consider the problem (12.1). We suppose that X Rn is a bounded

polyhedron, for the simplicity of the presentation. (See, however, Exercise 12.1.) We suppose further that f C 1 (Rn ).
The FrankWolfe algorithm works as follows:
FrankWolfe algorithm:
Step 0 (initialization). Generate the starting point x0 X (for example
by letting it be any extreme point in X). Set k := 0.
Step 1 (feasible descent direction). Solve the problem to
minimize zk (y) := f (xk )T (y xk ).
yX

(12.2)

Let y k be a solution to this LP problem, and pk := y k xk be the

search direction.
Step 2 (line search). Approximately solve the one-dimensional problem
to minimize f (xk + pk ) over [0, 1]. Let k be the resulting
step length.
Step 3 (update). Let xk+1 := xk + k pk .
Step 4 (termination check). If, for example, zk (y k ) or k is close to zero,
then terminate! Otherwise, let k := k + 1 and go to Step 1.
It should be clear that pk is a feasible direction: note that we can
write the new point xk+1 as y k + (1 )xk for some [0, 1], that
is, we construct an optimal convex combination of two feasible points.
In Step 2, we can utilize the Armijo step length rule or a more accurate line search such as one based on a quadratic interpolation.
The LP problem (12.2) was introduced already in Section 4.4, cf., for
example, (4.14). It was also there shown that zk (y k ) < 0 holds if and
only if xk is not stationary. Hence, we have shown that the FrankWolfe
algorithm is a descent algorithm.
We have also shown in Section 4.4 that a lower bound on the optimal
value f is available whenever f is convex: the first termination criterion
in Step 4 then states that this lower bound is close enough to f .
Theorem 12.1 (convergence of the FrankWolfe algorithm) Suppose that
X Rn is a nonempty, bounded polyhedron, and that the function
f : Rn R is in C 1 on X. Suppose that in Step 2 of the FrankWolfe
algorithm we use the Armijo step length rule. Then, the sequence {xk }
is bounded, and every limit point (at least one exists) is stationary.
If f is convex on X, then every limit point is globally optimal.
305

Optimization over convex sets

Proof. The boundedness of X ensures that the sequences {xk } and
{y k } are bounded. Let x and y be limit points, corresponding to
some subsequence K. Also the sequence {pk } is bounded; let further
p := y x . We first show that
{f (xk )T pk }kK 0

(12.3)

holds. As {f (xk )} is descending, it must hold that its limit in K is

f (x ). Hence, {f (xk+1 ) f (xk )}kK 0. Further,
f (xk+1 ) f (xk ) k f (xk )T pk < 0,

k = 0, 1, . . . ,

(12.4)

which means that either {f (xk )T pk }kK 0 (whence we are done) or

{k }kK 0 holds. In the latter case, there must be an index such
that for every k in K the initial step length is not accepted by the
Armijo rule, that is,
f (xk + (k /)pk ) f (xk ) > (k /)f (xk )T pk ,

k ,

k K.

Dividing both sides of this inequality by k /, in the limit in K of

the resulting inequality we reach the conclusion that f (x )T p 0,
while in the limit in K of the descent inequality f (xk )T pk < 0, the
reverse inequality is obtained. We conclude that (12.3) follows.
From the above follows that f (x )T (y x ) = 0, whence stationarity follows, since in the limit in K of the characterization
f (xk )T (y k xk ) f (xk )T (y xk ),

y X,

we obtain that
f (x )T (y x ) f (x )T (y x ) = 0,

y X.

Since the limit point was arbitrarily chosen, the first result follows.
The second part of the theorem follows from Theorem 4.24.
Figure 12.1 illustrates the LP problem in Step 1 at a non-stationary
point xk , the resulting extreme point y k , and search direction pk .
We have above established the result for the Armijo rule. By applying
the same technique as that discussed after Theorem 11.4 for gradient
related methods in unconstrained optimization, we can also establish
convergence to stationary points under the use of exact line searches.
Under additional technical assumptions we can establish that the
sequence f (xk )T pk 0; see Exercise 12.2.
Under the assumption that f is convex, several additional techniques
for choosing the step lengths are available; see the notes for references.
We refer to one such choice below.
306

The FrankWolfe algorithm

20

f (xk )
15

xk

10

pk

yk
5
5

10

Figure 12.1: Step 1 of the FrankWolfe algorithm.

Theorem 12.2 (convergence of the FrankWolfe algorithm) Suppose that

X Rn is nonempty, convex, closed, and bounded and that f : Rn R
is convex and in C 1 on X.
(a) In the FrankWolfe algorithm, suppose the step lengths k
(0, 1] satisfy that, for some C > 0 and every k large enough,1
k C/k,
k+1
.
1 k+1 =
k

(12.5a)
(12.5b)

If the sequence {xk } is finite, then the last iterate solves (12.1). Otherwise, f (xk ) f , and the sequence {xk } converges to the set of
solutions to (12.1): distX (xk ) 0. In particular, any limit point of
{xk } solves (12.1).
(b) Suppose that f is Lipschitz continuous on X. In the Frank
Wolfe algorithm, suppose the step lengths k (0, 1] are chosen according to the quadratically convergent divergent step length rule (6.41),
(6.42). Then, the conclusions in (a) hold.
1 According

to this step length rule, k 1/k for large k.

307

Optimization over convex sets

12.3

The simplicial decomposition algorithm

Consider the problem (12.1) under the same conditions as stated in Section 12.2. The simplicial decomposition algorithm builds on the Representation Theorem 3.22.
In the below description we let P denote the set of extreme points
of X. We also denote by Pk a subset of the extreme points which have
been generated prior to iteration k and which are kept in memory; an
element of this set is denoted by y i in order to not mix these extreme
points with the vectors y k solving the LP problem.
The simplicial decomposition algorithm works as follows:
Simplicial decomposition algorithm:
Step 0 (initialization). Generate the starting point x0 X (for example
by letting it be any extreme point in X). Set k := 0. Let Pb0 =
0 = x0 .
P0 := . Let x

Step 1 (feasible descent direction). Let y k be a solution to the LP problem (12.2).

bk {k}, for some subset P
bk of Pk .
Let Pk+1 := P

Step 2 (multidimensional line search). Let k+1 be an approximate solution to the restricted master problem to

X
k +
k ) ,
minimize f x
i (y i x
(12.6a)

subject to

iPk+1

iPk+1

i 1,

i 0,

(12.6b)

i Pk+1 ,

(12.6c)

k Xk := conv ({
where x
xk1 } {y i | i Pk }).
P
k + iPk+1 ( k+1 )i (y i x
k ).
Step 3 (update). Let xk+1 := x

Step 4 (termination check). If, for example, zk (y k ) is close to zero, or

if Pk+1 = Pk , then terminate! Otherwise, let k := k + 1 and go to
Step 1.
The description of the simplicial decomposition algorithm does not
k . We say that we use
completely specify the sets Pk or the points x
column dropping in Step 1 of the algorithm when Pbk Pk . Some classic
column dropping rules are given in Table 12.1.
To begin with, suppose that we use the principle in (a). According
to this principle, we run the algorithm by adding one new extreme point
to the previous set of extreme points known so far, solve the problem to
308

The simplicial decomposition algorithm

Table 12.1: Some column dropping principles for Step 1
bk := Pk , and x
k := xk .
(a) [no column dropping]: For all k, P

(b) [Zero weight column dropping]: For k 1,

bk := { i Pk | ( k )i > 0 } .
P

k := x0 .
For all k, x

(c) [bounded size of Pk ]: Let r be a positive integer. For k 1, let

bk := { i Pk | ( k )i > 0 } .
P

bk | < r, then let x

k := x
k1 . If |Pbk | = r, then let
If |P
bk \ ik ,
Pbk := P

ik arg minimum {( k1 )i },
bk
iP

k := xk .
where ties are broken arbitrarily, and x
b
k := xk .
(d) [FrankWolfe]: For all k, Pk := and x
minimize f over the convex hull of them, and repeat until we either get
close enough to a stationary point or if the last LP did not give us a new
extreme point. (In the latter case we are at a stationary point! Why?)
Suppose instead that we drop every extreme point that got a zero
weight in the last restricted master problem, that is, we work according
to the principle in (b). We then remove all the extreme points that we
believe will not be useful in order to describe the optimal solution as a
convex combination of them.
The algorithm corresponding to the principle in (c) is normally called
the restricted simplicial decomposition algorithm; it allows us to drop
extreme points in order to keep the memory requirements below a certain
threshold. In order to do so, we may need to also throw away an extreme
point that had a positive weight at the optimal solution to the previous
restricted master problem, and we implement this by removing one with
the least weight.
The most extreme case of the principle in (c) is to throw away every
point that was previously generated, and keep only the most recent one.
(It corresponds to letting r = 1.) Then, according to the principle in
(d), we are back at the FrankWolfe algorithm!
309

Optimization over convex sets

The restricted master problem (12.6) does not contain the slack vari k . Introducing
able associated with the convexity weight for the vector x
it as 0, we obtain an equivalent statement of the problem:

X
minimize f
xk +
i y i ,
(12.7a)
(,)

subject to

iPk+1

i = 1,

(12.7b)

iPk+1

, i 0,

i Pk+1 .

(12.7c)

We then recognize the feasible set of the restricted master problem

as the particular explicit convex hull that it actually is.
The advantage of using an inner representation of X and to algorithmically improve it is that it is much simpler to deal with in an optimization algorithm than the linear constraints that define X above.
The disadvantage of the representation is that the set P of extreme
points is both very large for a large-scale problem, and it is not known;
compare with the case of the simplex method, where we cannot simply
enumerate the extreme points in order to then pick the best one. In
this nonlinear case, we may also need a large number of them in order
to span an optimal solution. The trick that makes the algorithm work
well is the column dropping in Step 1, which keeps down the size of the
problems to be solved.
The simplicial decomposition algorithm is quite similar to the Frank
Wolfe algorithm (notice this from (12.7) when |Pk+1 | = 1). The main,
and very important, difference, is that the FrankWolfe algorithm drops
all the previous extreme points visited, and only optimizes over line segments. In simplicial decomposition the information is kept, and thanks
to this extra information the algorithm can make much better progress
in each iteration. The method therefore becomes much more efficient
than the FrankWolfe algorithm in practice.
As far as convergence is concerned, the algorithm clearly does at
least as well as the FrankWolfe algorithm does. In fact, if we use the
principle (a) in Table 12.1 then we always add an extreme point in each
iteration, and hence we must arrive at a stationary point in a finite
number of iterations since P is finite. If we use the principle in (b)
finite convergence is still ensured under mild additional conditions on
f , while for finite convergence under the principle in (c) the value of r
is crucial: it must be at least as large as the number of extreme points
needed to describe the optimal solution. In other words, we must have
memory enough to be able to span x . If the value is too small, then
the algorithm cannot converge finitely, and then the behaviour can be as
310

The gradient projection algorithm

bad as in the FrankWolfe algorithm, which is rather bad indeed. (The
convergence rate cannot even be linear.)
We finally note that in both of the FrankWolfe and simplicial decomposition algorithms, the sequence {f (xk )} is decreasing provided
that the line search is exact enough; on the other hand, the sequence
{zk (y k )} is non-monotone in general, although the limit (almost) always is zero. We refer to Theorem 12.1 for the corresponding result for
the FrankWolfe algorithm. The result for simplicial decomposition is
the same, except for the cases when convergence to a stationary point is
finite, as discussed above.2

12.4

The gradient projection algorithm

As was observed in Exercise 4.5 the result of the operation

y := ProjX [x f (x)]
at x X is that y = x if and only if x is a stationary point and if it
is not then p := y x defines a descent direction with respect to f at
x. This characterization is true also if we introduce an arbitrary scalar
> 0 as follows:
y := ProjX [x f (x)].
(Why?) In other words, supposing that xk is not stationary, we generate
the next iteration point as follows:
xk+1 := ProjX [xk k f (xk )],

k = 1, . . . ,

(12.8)

where {k } R++ . As the vector pk := ProjX [xk k f (xk )] xk

defines a feasible descent direction with respect to f at xk it follows that
f (xk+1 ) < f (xk ) provided that the value of k is sufficiently small. The
gradient projection algorithm is based on this observation. How do we
choose k and what does it mean to perform a line search in k ?
We propose here to utilize the Armijo rule, which was introduced for
unconstrained optimization in (11.11). It is however modified such that
the trial points are not xk f (xk ) as in the steepest descent method,
but the projected, feasible, points ProjX [xk f (xk )]. Starting with
2 The line segment representing the feasible set in the line search problem in Step 2
of the FrankWolfe algorithm satisfies [xk , yk ] Xk+1 , that is, the restricted master
problem in the simplicial decomposition algorithm is always defined over a set that
is at least as large. As a consequence, the latter algorithm always will be able to
achieve an improvement in the value of the objective function that is at least as great
as that of the latter. From this observation it is relatively easy to establish a basic
convergence result along the lines of Theorem 12.1.

311

Optimization over convex sets

a trial step
> 0, we check the Armijo criterion in (11.11) for the
feasible point ProjX [xk
f (xk )], and then replace
by
for some
(0, 1) if it is not satisfied, and so on, until the Armijo criterion is
satisfied. Eventually, then, we will satisfy the following inequality:
f (ProjX [xk + k pk ]) f (xk ) k f (xk )T pk .
The resulting step length then is k =
i for some integer i 0 (zero
if the initial step is accepted, otherwise positive), and the new iteration
point is the last point projected, xk+1 := ProjX [xk (
i )f (xk )].
Consider Figure 12.2. It illustrates a case where we imagine that the
initial step
has to be reduced twice (here, = 21 ) before the step is
accepted. As we can see from the figure the line search is not really
a line search, since the feasible points checked rather follow a piece-wise
linear curve than a line; in this example we trace the boundary of X,
and we sometimes refer to this type of line search as a boundary search
or a search along the projection arc.
xk f (xk )
xk
f (xk )

xk (
/2)f (xk )
xk (
/4)f (xk )
xk

Figure 12.2: Trial steps in the boundary search.

Although the technique looks more complex than the use of the
Armijo rule for the steepest descent method, their convergence behaviour
312

The gradient projection algorithm

are the same. Theorem 11.4 on the convergence of gradient related methods in unconstrained optimization can be extended to the case of the
gradient projection method, to state the following (see Exercise 12.6):

Theorem 12.3 (convergence of a gradient projection algorithm) Suppose

that X Rn is nonempty, closed, and convex, and that the function
f : Rn R is in C 1 on X. Suppose further that for the starting point
x0 the level set levf (f (x0 )) intersected with X is bounded. Consider the
iterative algorithm defined by the iteration (12.8), where the step length
k is determined by the Armijo step length rule along the projection arc.
Then, the sequence {xk } is bounded, the sequence {f (xk )} is descending,
lower bounded and therefore has a limit, and every limit point of {xk }
is stationary.
The following theorem shows that the gradient projection method
has a much stronger convergence property in the convex case.

Theorem 12.4 (convergence of a gradient projection algorithm) Suppose

that X Rn is nonempty, closed, and convex, and that the function
f : Rn R is in C 1 on X. Suppose further that f is convex and
that the problem (12.1) has at least one optimal solution. Consider the
iterative algorithm defined by the iteration (12.8), where the step length
k is determined by the Armijo step length rule along the projection
arc. Then, xk x for some optimal solution x to (12.1).
Proof. If {xk } is finite, then the stopping criterion implies that the last
iterate is optimal. Suppose therefore that the sequence is infinite.
Let x be an arbitrary optimal solution to (12.1). We have that
kxk+1xk k2+kxk x k2kxk+1x k2 = 2(xk+1xk )T (xxk ). (12.9)
Further, from the variational inequality characterization of the projection resulting in xk+1 follows that
0 [xk+1 xk + k f (xk )]T (x xk+1 )
= [xk+1 xk + k f (xk )]T (x xk )

+ [xk+1 xk + k f (xk )]T (xk xk+1 ),

313

Optimization over convex sets

which yields
(xk+1 xk )T (x xk ) k f (xk )T (xk x )

[xk+1 xk +k f (xk )]T (xk xk+1 )

k [f (xk )f (x )]

+[xk+1 xk +k f (xk )]T (xk+1 xk )

kxk+1 xk k2 +k f (xk )T (xk+1 xk ),

(12.10)
where we used the convexity characterization in Theorem 3.40(a) in the
second inequality, and the optimality of x in the third. Combining
(12.9) and (12.10) now yields that
kxk+1 xk k2 +kxk x k2 kxk+1 x k2 2[kxk+1 xk k2

+k f (xk )T (xk+1 xk )].

Rearranging terms yields

kxk+1 x k2 kxk x k2 kxk+1 xk k2 2k f (xk )T (xk+1 xk )
kxk x k2 2k f (xk )T (xk+1 xk )
= kxk x k2 + k ,

(12.11)

where
k := 2k f (xk )T (xk ProjX [xk k f (xk )]),

k = 0, 1, . . . .

Note that by the descent property of the algorithm, k 0 for all k.

In view of the Armijo rule,
f (xk )T (xk ProjX [xk k f (xk )]) f (xk ) f (xk+1 ).
Combining the above two inequalities,
k

2k
[f (xk ) f (xk+1 )]
[f (xk ) f (xk+1 )].

(12.12)

By (12.12),

k=0

[f (x0 ) f (x )] < .

We say that the sequence {k } is summable. The consequence for the

inequality (12.11) will become apparent from the following lemma.
314

The gradient projection algorithm

Lemma 12.5 (quasi-Fejer convergence) Let S Rn be nonempty and
{ak } Rn be a sequence such that for all x S and all k,
kak+1 xk2 kak xk2 + k ,

(12.13)

where {k } is a summable sequence in R+ .

(a) Then, {ak } is bounded; and
of {ak } belongs to S then ak a
.
(b) if a limit point a
Proof. (a) Fix some x S. Applying (12.13) iteratively yields, for
some C R+ ,
kak xk2 ka0 xk2 +

k1
X
j=0

k ka0 xk2 +

X
j=0

k C < .

Hence, {ak } is bounded.

S be a limit point of {ak }. Take > 0. Let {alk } be
(b) Let a
. Since {k } is a summable
a subsequence of {ak } which converges
P to a
sequence, there exists k0 such that
j=k0 j /2, and there exists k1
k2 < /2 for any k k1 . Then, for any
such that lk1 k0 and kalk a
k > lk1 ,
2

k kalk1 a
k +
kak a

k1
X

j=lk1

k +
j kalk1 a

j=lk1

j <

+ = .
2 2

.
We conclude that ak a
By the above lemma, we conclude that {xk } is convergent to a vector
x . This vector must be stationary, by Theorem 12.3, which means, by
convexity, that it is also globally optimal. We are done.
Suppose now that X = Rn . Then the gradient projection algorithm
reduces to the steepest descent method in unconstrained optimization,
and the Armijo step length rule along the projection arc reduces to
the classic Armijo rule. The above result then states that the steepest
descent algorithm converges to an optimal solution whenever f is convex
and there exist optimal solutions (see Theorem 11.7).
Finally, we consider the problem of performing the Euclidean projection. This is a strictly convex quadratic programming problem of the
form (4.12). We will show that we can utilize the phase I procedure of
the simplex method (see Section 9.1.2) in order to solve this problem. We
take a slightly more general viewpoint here, and present the algorithm
for a general strictly convex quadratic program.
315

Optimization over convex sets

Consider the problem to
1 T
x Qx q T x,
2
subject to Ax b,
minimize f (x) :=

(12.14)

x 0n ,

where Q Rnn is symmetric and positive definite, q Rn , A Rmn ,

and b Rm . Assuming that there exists a feasible solution to this
problem, its unique optimal solution x is characterized by its KKT
conditions as follows:
Qx + AT y v = q,
Ax b,
y T (Ax b) = 0,
T

v x = 0,
x, y, v 0,

(12.15a)
(12.15b)
(12.15c)
(12.15d)
(12.15e)

where y and v are the vectors of Lagrange multipliers for the constraints
of (12.14). We introduce a slack variable vector s in (12.15b), and can
therefore write the above system equivalently as
Qx + AT y v = q,

(12.16a)

(12.16b)

(12.16c)

(12.16d)
(12.16e)

Ax + I s = b,
y s = 0,

v x = 0,
x, s, y, v 0.

Disregarding the complementarity conditions (12.16c), (12.16d), this is

a set of linear equations over nonnegative variables, a solution to which
can be found by using the phase I procedure of the simplex method (see
Section 9.1). We propose to take the conditions (12.16c), (12.16d) into
account implicitly, in the following way.
Introducing artificial variables in the system (12.16a), (12.16b), multiplying before-hand any equation with a negative right-hand side qj or
bi by 1, we let the artificial variables define the starting BFS in the
phase I problem of minimizing their sum. When deciding on the incoming variable, we then introduce the following rules which make sure that
the conditions (12.16c), (12.16d) are always satisfied:
(a) If a variable xj (respectively, vj ), j = 1, . . . , n, is already
in the basis, then the variable vj (respectively, xj ) is not
admissible to enter the basis.
316

Application: traffic equilibrium

(b) If a variable si (respectively, yi ), i = 1, . . . , m, is already
in the basis, then the variable yi (respectively, si ) is not admissible to enter the basis.
It is not straightforward to argue why it is possible to reach the
optimal solution when we restrict the incoming rule in this way. The
interested reader is referred to the classic linear programming text by
Dantzig [Dan63, Section 24.4], in which the above method is proven to
yield convergence in a finite number of iterations provided that Q is
positive semidefinite.

12.5

Application: traffic equilibrium

12.5.1

Model analysis

The traffic equilibrium problem is a mathematical model that describes

the steady-state of traffic in a transportation network, wherein each traveler minimizes his/her own travel costs for reaching the desired destination, irrespective of the modes of travel used.3 Traffic equilibrium models
are used frequently as simulation models, aiding in the design or improvement of transportation systems. We develop a classic model of traffic
equilibrium, and show that it is naturally given a variational inequality
formulation; under certain conditions, it is also possible to state and
solve it as a (strictly) convex optimization problem. We illustrate the
performance of the FrankWolfe and simplicial decomposition methods
on a sample problem.
Given is a graph (or, network) G = (N , L) of nodes and directed
links, a set (p, q) C of pairs of origindestination (OD) nodes with
fixed demands dpq of units of desired traffic. Wardrops [War52] user
equilibrium principle states that for every OD pair (p, q) C, the travel
costs of the routes utilized are equal and minimal for each individual
user.4 We denote by hr the volume of traffic on route r Rpq , and by
cr (h), r R, the travel cost on the route as experienced by an individual
user given the volume h. Wardrops condition can be written as follows:
3 While travel costs mostly are composed by travel times, modern traffic equilibrium models also take into account the possible uses of congestion tolls and the
differences in the travelers values of time.
4 The rationale behind this principle is, roughly, that if the same network user every
morning travels between the same OD pair and he/she is not travelling along the best
route, then through a trialand-error procedure he/she will eventually reach the best
one and stick to it; if every traveler behaves in the same fashion, the steady-state
that is reached eventually must be a user equilibrium.

317

Optimization over convex sets

0|R| h (c(h) ) 0|R| ,

T

h = d,

(12.17a)
(12.17b)

where the value of pq is interpreted as the minimal (that is, equilibrium)

route cost in OD pair (p, q), and where we introduced the matrix
R|R||C| to be the routeOD pair incidence matrix (i.e., the element rk
is 1 if route r joins OD pair k := (p, q) C, and 0 otherwise). The first
condition essentially states that more costly routes are not used, and the
second describes the demand condition.
Let {0, 1}|L||R| be the linkroute incidence matrix, whose element lr equals one if route r R utilizes link l L, and zero otherwise.
Route r has an additive route cost cr (h) if it is the sum
P of the costs of using all the links defining it. In other words, cr (h) = lL lr tl (v), where
v R|L| is the total volume of traffic on the links, and tl : R|L| R is a
function measuring the travel cost on link l L given the link volume v.
In matrixvector notation, then, c(h) = T t(v) holds. Also, implicit in
this cost relationship is the assumption that the pair (h, v) is consistent,
in the sense that v equals the sum of the route volumes: v = h.
Consider the following variational inequality: find v F such that
v F ,

t(v )T (v v ) 0,

(12.18)

|R|
where F := { v R|L| | h R+ with T h = d and v = h } is the
set of demand-feasible link volumes.
In the case where t is integrable,5 the model (12.18) defines the firstorder optimality conditions for an optimization problem; assuming, further, that t is separable, that is, that tl is a function only of vl , l L,
the optimization problem has the form
X Z vl
tl (s) ds,
(12.19)
minimize f (v) :=
(h,v )

lL

subject to T h = d,

v = h,
h 0|R| .
This is the classic traffic assignment problem.
Since the feasible set of the problem (12.19) is a bounded polyhedron there exists a nonempty and bounded set of optimal link and route
5 If t is continuously differentiable, then integrability is equivalent to the symmetry
of its Jacobian matrix t(v ) everywhere. Integrability is a more general property
than this symmetry property, since t need not be always be differentiable.

318

Application: traffic equilibrium

volumes, and the optimality conditions given by the respective variational inequality (or by the KKT conditions) are necessary for the local
optimality of a pair of link and route volumes [cf. Proposition 4.23(b)].
Moreover, the optimality conditions are exactly the Wardrop conditions of user equilibrium. To see this, suppose that v is a local minimum
in (12.19) and consider the following problem, a solution to which necessarily then is v [cf. (4.14)]:
minimize t(v )T v,
(h,v )

(12.20)

T h = d,

subject to

v h = 0|L| ,

h 0|R| .

Note that we are simply rephrasing (12.18). Introducing the LP dual

variable vectors R|C| and R|L| , the LP dual to (12.20) is to
maximize dT ,
(,)

(12.21)

subject to T 0|R| ,
= t(v ).
Eliminating through = t(v ) the primaldual optimality conditions
are, precisely, the Wardrop conditions (12.17), together with the consistency condition v = h.
Suppose, in addition, that each link cost function tl is increasing; this
is a natural assumption considering that congestion on a link, that is, the
travel time, increases with its volume. According to Theorem 3.40(b)
this means that the function f is convex, and therefore the problem
(12.19) is a convex one. Therefore also the optimality conditions stated
in the variational inequality (12.18) or the (equivalent) Wardrop conditions (12.17) are both necessary and sufficient for a volume v to be an
equilibrium one.
If further tl is strictly increasing for every l L then the solution v
is unique (cf. Proposition 4.11).

12.5.2

Algorithms and a numerical example

When solving this problem by using the FrankWolfe or the simplicial

decomposition method the search directions in Step 1 correspond to a
very special problem: if the current traffic volume is v k then the solution
y k to the LP problem is found by assigning, for each OD pair (p, q) C,
319

Optimization over convex sets

the demand dpq onto a shortest route between the origin and destination node given the fixed link cost vector t(v k ), and then aggregating
these route volumes through the relation v = h; for each OD pair,
the shortest route is found by using, for example, Dijkstras algorithm.
Doing this, we in fact need not store any route information at all, which
saves computer storage.
A Matlab implementation of the two algorithms was in [Jos03] devised and tested on a classic traffic assignment problem, modelling the
small city of Sioux Falls in South Dakota, USA, whose traffic network
representation has 24 nodes, 76 links, and 528 OD pairs.
In the simplicial decomposition algorithm, we tested three algorithms
for the restricted master problems (RSMPs)a Newton method and two
gradient projection methods. In Figure 12.3 we illustrate the solution
times necessary for reaching a given accuracy; accuracy is here measured
in terms of the relative error stemming from the lower and upper bounds
on the optimal value.
Sioux Falls network

10

SD/Grad. proj. 1
SD/Grad. proj. 2
SD/Newton
FrankWolfe

10

Max relative objective function error

10

10

10

10

10

10

10

10

10

11

10

10

20

30

40

50
CPU time (s)

60

70

80

90

100

Figure 12.3: The performance of SD vs. FW on the Sioux Falls network.

It is clear that the FrankWolfe algorithm is extremely slow in comparison with the simplicial decomposition algorithm, regardless of the
method used for solving the restricted master problems. An analysis of
the algorithms behaviour reveals that very small step lengths are taken
320

Notes and further reading

already quite early on in the solution process. The basis for this behaviour is the congestion effects that imply that several OD pairs need
more than one route to have a positive volume at the solution; this
means that the optimal link volume is not an extreme point, and the
solutions to (12.2) will zig-zag between assigning the total volume onto
these routes.

12.6

Notes and further reading

Algorithms for linearly constrained optimization problems are disappearing from modern text books on optimization. It is perhaps a sign of maturity, as we are now better at solving optimization problem with general constraints, and therefore do no longer have to especially consider
the class of linearly constrained optimization problems. Nevertheless we
feel that it provides a link between linear programming and nonlinear
optimization problems with general constraints, being a subclass of nonlinear optimization problems for which primal feasibility can be retained
throughout the procedure.
The FrankWolfe method was developed for QP problems in [FrW56],
and later for more general problems, including non-polyhedral sets, in
[Gil66] and [PsD78, Section III.3], among others. The latter source includes several convergence results for the method under different step
length rules, assuming that f is Lipschitz continuous, for example a
Newton-type step length rule. The convergence Theorem 12.1 for the
FrankWolfe algorithm was taken from [Pat98, Theorem 5.8]. The convergence result for convex problems given in Theorem 12.2 is due to
Dunn and Harshbarger [DuH78]. The version of the FrankWolfe algorithm produced by the selection k := 1/k is known as the method of
successive averages (MSA).
The simplicial decomposition algorithm was developed in [vHo77].
Restricted simplicial decomposition methods have been developed in
[HLV87, Pat98].
The gradient projection method presented here was first given in
[Gol64, LeP66]; see also the textbook [Ber99]. Theorem 12.4 is due to
[Ius03], while Lemma 12.5 is due to [BGIS95].
The traffic equilibrium models of Section 12.5 are described and analyzed more fully in [She85, Pat94].
Apart from the algorithms developed here, there are other classical
algorithms for linearly constrained problems, including the reduced gradient method, Rosens gradient projection method, active set methods,
and other sub-manifold methods. They are not treated here, as some of
321

Optimization over convex sets

them have fallen out of popularity. Reduced gradient methods still constitute the main building block of some commercial software, however.

12.7

Exercises

Exercise 12.1 (extensions of the FrankWolfe algorithm to unbounded sets)

Develop an extension to the FrankWolfe algorithm applicable to cases where
X is unbounded. Which steps need to be changed? What can go wrong?
Exercise 12.2 (convergence of the sequence {zk (y k )}) We are interested in
the convergence of the sequence {zk (y k )} := {f (xk )T pk } := {f (xk )T (y k
xk )} of optimal values of the linear objective function in Step 1 of the Frank
Wolfe algorithm. In addition to the properties of the problem (12.1) and the
execution of the FrankWolfe algorithm assumed in Theorem 12.1, suppose
that
f is Lipschitz continuous on Rn .
Establish that zk (y k ) 0 holds.
[Hint: Utilize the following technical lemma, which is of independent interest and therefore given a proof:
Lemma 12.6 (descent lemma) Suppose that f is Lipschitz continuous on
Let x, p both lie in Rn . Then,

Rn , with modulus L.

f (x + p) f (x) f (x)T p +

L
kpk2
2

holds.
Proof. Let R and g() := f (x + p). The chain rule yields
pT f (x + p). Then,
f (x + p) f (x) = g(1) g(0) =
Z

dg
() d =
d

pT f (x) d + 
p f (x) d +

pT f (x) + kpk
pT f (x) +

1
0
1

0
1

1
0

dg
()
d

pT f (x + p) d

pT [f (x + p) f (x)] d

kpkkf (x + p) f (x)k d

Lkpk d

L
kpk2 .
2

We are done.
Apply this result to the inequality resulting from applying the Armijo
rule at a given iteration k, with x replaced by xk and p replaced by k pk .

322

Exercises
Summing all these inequalities and utilizing that {kpk k} isP
a bounded sequence
2
thanks to the boundedness of X, conclude that the sum
k=0 [zk (y k )] must
be convergent and therefore zk (y k ) 0 must hold.]
Exercise 12.3 (numerical example of the FrankWolfe algorithm) Consider the
problem to
minimize f (x) := (x1 + 2x2 6)2 + (2x1 x2 2)2 ,

subject to 2x1 + 3x2 9,

x1 3,

x1 , x2 0.
(a) Show that the problem is convex.
(b) Apply one step of the FrankWolfe algorithm, starting at the origin.
Provide an interval where f lies.
Exercise 12.4 (numerical example of the FrankWolfe algorithm) Consider the
problem to
maximize f (x) := x21 4x22 + 16x1 + 24x2 ,

subject to x1 + x2 6,
x1 x2 3,

x1 , x2 0.

(a) Show that the problem is convex.

(b) Solve the problem by using the FrankWolfe algorithm, starting at the
origin.
Exercise 12.5 (numerical example of the FrankWolfe algorithm) Consider the
problem to
minimize f (x) :=
subject to

1
2

x1

1
2

2

1 2
x2 ,
2

x1 1,
x2 1,

x1 , x2 0.
Apply two iterations of the FrankWolfe algorithm, starting at
(1, 1)T . Give upper and lower bounds on the optimal value.

x0 :=

Exercise 12.6 (convergence of a gradient projection algorithm) Establish Theorem 12.3.

Exercise 12.7 (numerical example of the simplicial decomposition algorithm)
Solve the problem in Exercise 12.3 by using the simplicial decomposition algorithm.

323

Optimization over convex sets

Exercise 12.8 (numerical example of the simplicial decomposition algorithm)
Solve the problem in Exercise 12.4 by using the simplicial decomposition algorithm.
Exercise 12.9 (numerical example of the simplicial decomposition algorithm)
On the problem in Exercise 12.5 apply two iterations of the simplicial decomposition algorithm. Is x2 optimal? Why/why not?

324

Constrained
optimization

XIII

In this chapter, we will discuss the conversion of nonlinear programming

problems with inequality and equality constraints into (in some sense)
equivalent unconstrained problems or problems with simple constraints.
In practice, a sequence of such equivalent (or, approximating) problems
is solved because of computational considerations.

13.1

Penalty methods

Let us consider a general optimization problem:

minimize f (x),
subject to x S,

(13.1)

where S Rn is a nonempty, closed set and f : Rn R is a given

differentiable function. The basic idea behind all penalty algorithms is
to replace the problem (13.1) with the equivalent unconstrained one:
minimize f (x) + S (x),
where

(13.2)

(
0,
if x S,
S (x) =
+, otherwise.

The role of S , which in the optimization community is known as the

indicator function of the set S, is to make sure that feasibility is top
priority, and only when achieving feasibility do we concentrate on optimizing the function f . Of course, the so defined S is rather bizarre from
the computational point of view: it is non-differentiable, discontinuous,
and even not finite (though it is convex provided S is). Thus, from the

Constrained optimization
practical point of view we would like to replace the additional term S
with a numerically better behaving function.
There are two alternative approaches achieving this. The first is
called the penalty, or the exterior penalty method, in which we add
a penalty to the objective function for points not lying in the feasible
set and thus violating some of the constraints. This method typically
generates a sequence of infeasible points, approaching optimal solutions
to the original problem from the outside (exterior) of the feasible set,
whence the name of the method. The function S is approximated from
below in these methods.
Alternatively, in the barrier, or interior point methods, we add a continuous barrier term that equals + everywhere except in the interior
of the feasible set and thus ensure that globally optimal solutions to the
approximating unconstrained problems do not escape the feasible set of
the original constrained problem. The method thus generates a sequence
of interior points, whose limit is an optimal solution to the original constrained problem. The function S is approximated from above in
these methods.
Clearly we would like to transfer nice properties of original constrained problems, such as convexity, smoothness, to penalized problems
as well. We easily achieve this by carefully choosing penalty functions;
use Exercises 13.1 and 13.2 to verify that convexity may be easily transferred to penalized problems.

13.1.1

Exterior penalty methods

We assume that the feasible set S of the optimization problem (13.1) is

given by a system of inequality and equality constraints:
S := { x Rn | gi (x) 0,
hj (x) = 0,

i = 1, . . . , m;
j = 1, . . . , },

(13.3)

where gi C(Rn ), i = 1, . . . , m, hj C(Rn ), j = 1, . . . , . In this case,

we can choose a continuous function : R R+ such that (s) = 0
if and only if s = 0 (typical examples of () will be 1 (s) = |s|, or
2 (s) = s2 ), and try the approximation:
S (x)
S (x) :=

X
m
i=1

max{0, gi (x)} +

X
j=1




hj (x) , (13.4)

where the real number > 0 is called the penalty parameter. The
different treatment of inequality and equality constraints in the equation (13.4) stems from the fact that equality constraints are violated at
326

Penalty methods

Figure 13.1: The graph of

S and the feasible set S (black).
x whenever hj (x) 6= 0 for some j = 1, . . . , , while inequality constraints
are violated only when gi (x) > 0 for some i = 1, . . . , m; the latter fact
can be equivalently expressed as max{0, gi (x)} 6= 0.
Example 13.1 We repeat the settings of Example 5.7. Let S := { x
R2 | x2 0; (x1 1)2 + x22 = 1 }. Let (s) = s2 . Then, in this
example,

S (x) := [max{0, x2 }]2 + [(x1 1)2 + x22 1]2 .

The graph of the function S , together with the feasible set S, is shown
in Figure 13.1.
To exclude trivial cases, we assume that the original constrained
problem (13.1), and thus its equivalent reformulation (13.2), has an optimal solution x . Furthermore, we assume that for every > 0 the
approximating optimization problem to
minimize
f (x) +
S (x)
n
x R

(13.5)

has at least one optimal solution x .

Clearly,
S is non-negative and, furthermore, S (x) = 0 if and only
if x S. Therefore, from the Relaxation Theorem (Theorem 6.1) we
know that the inequality f (x1 ) + 1 (x
1 ) f (x2 ) + 2 (x
2 )
327

Constrained optimization
f (x ) + S (x ) = f (x ) holds for every positive 1 2 . In fact, we
can establish an even stronger inequality, which will be used later; see
the following lemma.
Lemma 13.2 (penalization constitutes a relaxation) For every positive
1 2 it holds that f (x1 ) f (x2 ).
Proof. The claim is trivial for 1 = 2 , thus we assume that 1 <
2 . Since x1 minimizes f (x) + 1
S (x), and x2 is feasible in this
(unconstrained) optimization problem, it holds that
f (x1 ) + 1
S (x1 ) f (x2 ) + 1
S (x2 ).
Similarly,

(13.6)

f (x2 ) + 2
S (x2 ) f (x1 ) + 2
S (x1 ).

Adding the two inequalities, we conclude that

(2 1 )[
S (x2 )
S (x1 )] 0,
which, substituted into (13.6), implies the claim, because 2 1 > 0.
Now we are ready to show that every limit point of the sequence
{x }, as converges to infinity, is optimal in the problem (13.1). Thus,
the family of problems (13.5) is indeed an approximation of the original
problem (13.1), and setting to a large enough value we can solve the
problem (13.5) in place of (13.1).
Theorem 13.3 (global convergence of a penalty method) Assume that
the original constrained problem (13.1) possesses optimal solutions. Then,
every limit point of the sequence {x }, +, of globally optimal solutions to (13.5) is globally optimal in the problem (13.1).
In other words,

x globally optimal in (13.5)
= x globally optimal in (13.1).
x x as +
Proof. Let x denote an arbitrary globally optimal solution to (13.1).
From the inequality (cf. the Relaxation Theorem 6.1)
f (x ) +
S (x ) f (x ),

(13.7)

and Lemma 13.2, we obtain uniform bounds on the penalty term

S (x )
for all 1:
0
S (x ) f (x ) f (x1 ).
328

Penalty methods
Thus,
S (x ) converges to zero as converges to +, and, owing to the
continuity of
S , every limit point of the sequence {x } must be feasible
in (13.1).
denote an arbitrary limit point of {x }, that is,
Now, let x
,
lim xk = x

for some sequence {k } converging to infinity. Then, we have the following chain of inequalities:
f (
x) = lim f (xk ) lim {f (xk ) + k
S (xk )} f (x ),
k+

k+

where the last inequality follows from (13.7). However, owing to the
in (13.1) the reverse inequality f (x ) f (
feasibility of x
x) must also
hold. The two inequalities combined imply the required claim.
We emphasize that Theorem 13.3 establishes the convergence of globally optimal solutions only; the result may therefore be of limited practical value for nonconvex nonlinear programs. However, assuming more
regularity of the stationary points, such as LICQ (see Definition 5.41),
and using specific continuously differentiable penalty functions, such as
(s) := s2 , we can show that every limit point of sequences of stationary points of (13.5) also is stationary (i.e., a KKT point) in (13.1).
Furthermore, we easily obtain estimates of the corresponding Lagrange

).
multipliers (,
Theorem 13.4 (convergence of a penalty method) Let the objective
function f : Rn R and the functions gi : Rn R, i = 1, . . . , m,
and hj : Rn R, j = 1, . . . , , defining the inequality and equality
constraints of (13.1) be in C 1 (Rn ). Further assume that the penalty
function : R R+ is in C 1 and that (s) 0 for all s 0.
Consider a sequence {xk } of points that are stationary for the sequence of problems (13.5), for some positive sequence of penalty param , and that
eters {k } converging to +. Assume that limk+ xk = x
. Then, if x
is feasible in (13.1) it must also verify the
LICQ holds at x
KKT conditions.
In other words,

xk stationary in (13.5)

as k +
xk x
stationary in (13.1).
= x

LICQ holds at x

feasible in (13.1)
x
329

Constrained optimization
Proof. [Sketch] Owing to the optimality conditions (4.14) for unconstrained optimization we know that every point xk , k = 1, 2, . . . , necessarily satisfies the equation
[f (xk ) + k
S (xk )] = f (xk )
m
X
k [max{0, gi (xk )}]gi (xk )
+

(13.8a)
(13.8b)

i=1

k [hj (xk )]hj (xk ) = 0n .

(13.8c)

j=1

Let, as before, I(
x) denote the index set of active inequality constraints
at x. If i 6 I(
x) then gi (xk ) < 0 for all large k, and the terms corresponding to this index do not contribute to (13.8).
, we know that the vectors { gi (
Since LICQ holds at x
x), hj (
x) |
i I(
x), j = 1, . . . , } are linearly independent. Therefore, we can easily
show that the sequence {k [max{0, gi (xk )}]} must converge to some
limit
i as k + for all i I(
x). Similarly, limk+ k [hj (xk )] =
j , j = 1, . . . , . At last, since k [max{0, gi (xk )}] 0 for all k =

0|I(x )| .
1, 2, . . . , i I(
x) it follows that
Passing to the limit as k + in (13.8) we deduce that
f (
x) +

iI(
x)

i gi (
x) +

X
j=1

j hj (

x) = 0n ,

is a KKT point for (13.1) with Lagrange multipliers (

i.e., x
, ).
Notice that if the original problem (13.1) is convex and verifies LICQ,
and if every penalized problem is also convex (cf. Exercise 13.1), then
Theorems 13.3 and 13.4 essentially work with the same sequences: under
convexity and LICQ globally optimal solutions are KKT points and vice
versa. Therefore, in this case we automatically get feasibility of limit
points in Theorem 13.3, as well as expressions for estimating Lagrange
multipliers in Theorem 13.4.

13.1.2

Interior penalty methods

While the idea behind exterior penalty functions is to nicely approximate

S on the whole of Rn , interior penalty, or barrier, function methods
construct approximations only inside the feasible set and set a barrier
against leaving it. If a globally optimal solution to (13.1) happens to be
330

Penalty methods
located on the boundary of the feasible region, then the method generates
a sequence of interior points that converges to it.
In this section we assume that the feasible set S of the optimization
problem (13.1) has the following form:
S := { x Rn | gi (x) 0,

i = 1, . . . , m }.

(13.9)

For the method to work, we need to assume that there exists a strictly
Rn , that is, such that gi (
feasible point x
x) < 0, i = 1, . . . , m. Thus, in
contrast with the exterior penalty algorithms, we cannot include equality
constraints into the penalty term. While it is possible to extend the
discussion to allow for equality constraints, we prefer to keep the notation
simple and assume that equality constraints are not present.
To formulate a barrier problem, we consider the following approximation of S :
( P
m
i=1 [gi (x)], if gi (x) < 0, i = 1, . . . , m,
S (x)
S (x) :=
+,
otherwise,
(13.10)
and the function : R R+ is a continuous nonnegative function such that (sk ) + for all negative sequences {sk } converging to zero. Typical examples of are 1 (s) := s1 , and 2 (s) :=
log[min{1, s}]. Note that 2 is not differentiable at the point s = 1.
However, dropping the nonnegativity requirement on , the famous differentiable logarithmic barrier function e2 (s) := log(s) gives rise to
the same convergence theory as we are going to present.
Example 13.5 Consider the simple one-dimensional set S := { x R |
x 0 }. Choosing = 1 = s1 , the graph of the barrier function

S is shown in Figure 13.2 for various values of . Note how

S
converges towards S as 0.

Having chosen the function and a penalty parameter > 0, we are

going to solve the following problem:
minimize
f (x) +
S (x).
n
x R

(13.11)

Similarly to the case of exterior penalty functions discussed in the

previous section, we can prove the convergence to globally optimal solutions (however, in this case we need to assume the regularity assumption
S = cl{ x Rn | gi (x) < 0, i = 1, . . . , m }). Rather, we proceed directly
to establish a convergence result for stationary points, similar to Theorem 13.4. Not only is this result more practical than the one concerning
331

Constrained optimization
3

10

=1
=0.1
=0.01

10

10

10

10

10

10
0.5

0.5

1.5

2.5

Figure 13.2: The graph of

S for various choices of . Note the logarithmic scale.
globally optimal solutions, but also the interior point algorithms are most
often applied to convex optimization problems, and thus stationarity implies global optimality (see Section 5.8). The reason is that interior point
algorithms are especially efficient both practically and theoretically for
convex optimization problems. In fact, one can show that the number of
computational steps an interior point algorithm needs in order to achieve
a prescribed accuracy > 0 is bounded by a polynomial function of the
size of the problem (that is, the number of variables and constraints)
and 1 . For non-convex problems, on the contrary, it is known that the
number of steps necessary can grow exponentially. For other algorithms
that can be applied to convex optimization problems, for example, exterior penalty methods, no well-developed complexity theory exists.
The proof of the general convergence theorem for barrier methods
goes in parallel with the corresponding result for exterior penalty methods. An important difference, though, is that now the constrained problem (13.1) is the relaxation of (13.11) for every > 0, and the convergence is studied as 0.
Theorem 13.6 (convergence of an interior point algorithm) Let the objective function f : Rn R and the functions gi , i = 1, . . . , m, defining
the inequality constraints of (13.9) be in C 1 (Rn ). Further assume that
the barrier function : R R+ is in C 1 and that (s) 0 for all s < 0.
Consider a sequence {xk } of points that are stationary for the sequence of problems (13.11) with = k , for some positive sequence of
,
penalty parameters {k } converging to 0. Assume that limk+ xk = x

and that LICQ holds at x. Then, x is a KKT point of (13.1).

332

Penalty methods
In other words,

xk stationary in (13.11)
as k + = x
xk x
stationary in (13.1).

LICQ holds at x
Proof. [Sketch] Owing to the optimality conditions (4.14) for unconstrained optimization we know that every point xk , k = 1, 2, . . . , necessarily satisfies the equation
[f (xk ) + k
S (xk )] =
m
X
f (xk ) +
k [gi (xk )]gi (xk ) = 0n .

(13.12)

i=1

is
Because every point xk is strictly feasible in (13.1), the limit x
clearly feasible in (13.1). Let I(
x) denote the index set of active in.
equality constraints at x
If i 6 I(
x) then gi (xk ) < 0 for all large k, and k [gi (xk )] 0 as
k 0.
, we know that the vectors { gi (
Since LICQ holds at x
x) | i
I(
x) } are linearly independent. Therefore, we can easily show that the
sequence {k [gi (xk )]} must converge to some limit
i as k + for
all i I(
x). At last, since k [gi (xk )] 0 for all k = 1, 2, . . . , i I(
x),
0|I(x )| .
it follows that
Passing to the limit as k + in (13.12) we deduce that
X

i gi (
x) = 0n ,
f (
x) +
iI(
x)

is a KKT point for (13.1) with Lagrange multiplier vector .

that is, x
For example, if we use (s) := 1 (s) := 1/s, then (s) = 1/s2 in
Theorem 13.6, and the sequence {k /gi2 (xk )} converges to the Lagrange
multiplier
i corresponding to the constraint i (i = 1, . . . , m).

13.1.3

Computational considerations

As the penalty parameter increases in the exterior penalty methods,

or decreases in the interior penalty methods, the approximating problem (13.5) [respectively, (13.11)] becomes more and more ill-conditioned.
Therefore, a typical computational strategy is to start from safe values of the penalty parameter (relatively small for exterior penalties, or
333

Constrained optimization
relatively large for barriers), and then proceed step after step slightly
modifying the penalty parameter (e.g., multiplying it with some number
close to 1).
It is natural to use the optimal solution xk as a starting point for an
iterative algorithm used to solve the approximating problem corresponding to the next value k+1 of the penalty parameter. The idea behind
such a warm start is that, typically, k k+1 implies xk xk+1 .
In fact, in many cases we can perform only few (maybe, only one)
steps of an iterative algorithm starting at xk to obtain a satisfactory approximation xk+1 of an optimal solution corresponding to the
penalty parameter k+1 , and still preserve the convergence xk x ,
as k +, towards optimal solutions of the original constrained problem (13.1). This technique is especially applicable to convex optimization problems, and all the complexity estimates for interior penalty algorithms depend on this fact.

13.1.4

Applications and examples

Example 13.7 (exterior penalty) Consider the problem to

minimize f (x) :=

1 2
(x + x22 ) + 2x2 ,
2 1

(13.13)

subject to x2 = 0.
The problem is convex with affine constraints; therefore, the KKT conditions are both necessary and sufficient for the global optimality. The
KKT system is in this case reduces to: x2 = 0 and


   
x1
0
0
+
=
.
x2 + 2
1
0
The only solution to this system is x = 02 , = 2.
Let us use the exterior penalty method with quadratic penalty (s) :=
s2 to solve this problem. That is, we want to
minimize

1 2
(x + x22 ) + 2x2 + x22 ,
2 1

where > 0 is a penalty parameter. This problem is convex as well, so

that stationarity is both necessary and sufficient for global optimality:

  
x1
0
=
,
(1 + 2)x2 + 2
0
which has the unique solution x = (0, 2/(1 + 2))T for every > 0.
Note that lim+ x = 02 is a globally optimal solution to (13.13),
334

Penalty methods
and that

4
= 2 = ,
1 + 2
where is the Lagrange multiplier corresponding to the equality constraint x2 = 0.
lim [(x )2 ] = lim

Example 13.8 (interior penalty) Consider the problem to

minimize f (x) := x21 + x2 ,
subject to x21 + x22 1 0.

(13.14)

The problem is convex and verifies Slaters CQ (see Definition 5.38);

therefore, the KKT conditions are both necessary and sufficient for global
optimality. The KKT system in this case reduces to: x21 + x22 1 and



  
2x1
2x1
0
+
=
,
1
2x2
0
(x21

x22

0,

1) = 0.

After easy calculations, which the reader is encouraged to perform, we

can see that the only solution to this system is x = (0, 1)T , = 1/2.
Now, let us use the barrier method with the barrier function (s) :=
log(s). That is, we want to
minimize x21 + x2 log(1 x21 x22 ),
where > 0 is a penalty parameter. This problem is convex as well
(verify this!), so that stationarity (restricted to the interior of the feasible
set, { x R2 | x21 + x22 < 1 }) is both necessary and sufficient for global
optimality:



  

2x1
2x1
0
+
=
.
1
0
1 x21 x22 2x2

Again, after some transformations

we can verify thatthis system has

two solutions x = (0, 2 + 1)T and y = (0, + 2 + 1)T , out of

which only x is (strictly) feasible. We can easily see that lim+0 x =
(0, 1)T is a globally optimal solution to (13.14), and that

1 ( 2 + 1)2
1
1
= = ,
= lim
+0 2 2 + 1 2
2

lim [(x )21 + (x )22 1] = lim

+0

+0

where is the Lagrange multiplier corresponding to the inequality constraint x21 + x22 1 0.
335

Constrained optimization
Example 13.9 (linear programming) Consider an LP problem of the following form:
maximize bT y,
(13.15)
subject to AT y c,
where b, y Rm , c Rn , and A Rmn . Using standard linear
programming theory (see Theorem 10.15), we can write the primaldual
optimality conditions for this problem in the form:
AT y c,

Ax = b,
x 0n ,

(13.16)

xT (c AT y) = 0,

where x Rn is the vector of Lagrange multipliers for the inequality

constraints, or just a vector of dual variables as it is customary called in
the linear programming literature.
Introducing a vector s Rn of slack variables for the inequality
constraints into the problem (13.15), it assumes the form
maximize bT y,
subject to AT y + s = c,

(13.17)

s0 ,
and the corresponding system of optimality conditions will be:
AT y + s = c,
Ax = b,
n

(13.18)

x 0 , s 0 , x s = 0.
Now, let us apply the barrier method to the optimization problem (13.17). It has equality constraints, which we do not move into the
penalty function, but rather leave them as they are. Thus, we consider
the following problem with equality constraints only:
minimize bT y
T

n
X

log(sj ),

j=1

(13.19)

subject to A y + s = c,
where we use the logarithmic barrier function, > 0 is a penalty parameter, and we have multiplied the original objective function with
336

Sequential quadratic programming

1 to change the maximization problem into a minimization one. The
problem (13.19) is convex with affine constraints, therefore the KKT
conditions are both necessary and sufficient for the global optimality.
The KKT system in this case is: AT y + s = c, and

b
 m
/s1  
A
0

,
(13.20)
..
+
n x =
I
0n
.

/sn

where x Rn is a vector of Lagrange multipliers for the equality constraints in the problem (13.19). Further, the system (13.20) can be
rewritten in the following more convenient form:
AT y + s = c,

(13.21)

Ax = b,
xj sj = ,

j = 1, . . . , n.

Recalling that due to the presence of the barrier the vector s must be
strictly feasible, that is, s > 0n , and that the penalty parameter
is positive, the last equation in (13.21) does in fact imply the strict
inequality x > 0n .
Therefore, comparing (13.21) and (13.18) we see that for linear programs the introduction of a logarithmic barrier amounts to a small perturbation of the complementarity condition. Namely, instead of the requirement
x 0n , s 0n , xj sj = 0, j = 1, . . . , n,

we get a similar one (for small > 0):

x > 0n , s > 0n , xj sj = ,

j = 1, . . . , n.

For the case n = 1 the difference between the two is shown in Figure 13.3.
Note the smoothing effect on the feasible set introduced by the interior
penalty algorithm. We can use Newtons method to solve the system of
nonlinear equations (13.21), but not (13.18).

13.2

Sequential quadratic programming

13.2.1

Introduction

We begin by studying the equality constrained problem to

minimize f (x),
subject to hj (x) = 0,

(13.22a)
j = 1, . . . , ,

(13.22b)
337

Constrained optimization
= .1
= .01
= .001
=0

0.9
0.8
0.7
0.6

0.5
0.4
0.3
0.2
0.1
0
0.1

0.2

0.4

0.6

0.8

Figure 13.3: The approximation of the complementarity constraint resulting from the use of logarithmic barrier functions in linear programming.

where f : Rn R and hj : Rn R are given functions in C 1 on

Rn . The KarushKuhnTucker conditions for this problem state (cf.
Theorem 5.33) that at a local minimum x of f over the feasible set,
where x satisfies some constraint qualification, there exists a vector
R such that
x L(x , ) := f (x ) +

j hj (x ) = 0n ,

(13.23a)

L(x , ) := h(x ) = 0 .

(13.23b)

j=1

It is an appealing idea to find such a point by directly attacking

this system of nonlinear equations, which has n + unknowns as well
as equations. Newtons method is then the natural choice. Let us
see what the Newton subproblem looks like. We now assume, for the
time being, that f and hj , j = 1, . . . , , are in C 2 on Rn . Suppose
that (xk , k ) Rn R . Then, since Newtons method takes a unit
step in the direction towards the approximate problems solution, we
obtain the following characterization of the next iterate (xk+1 , k+1 ):
(xk+1 , k+1 ) := (xk , k ) + (pk , v k ), where (pk , v k ) Rn R solves
the second-order approximation of the stationary point condition for the
338

Sequential quadratic programming

Lagrange function:
2 L(xk , k )

 
pk
= L(xk , k ),
vk

that is,
 2
  

x x L(xk , k ) h(xk ) pk
x L(xk , k )
=
,
vk
h(xk )
h(xk )T
0

(13.24)

where the matrix h(xk )T is the Jacobian of h at xk , comprised of the

rows hj (xk )T for j = 1, . . . , .
This system of linear equations has a nice interpretation, namely as
the KKT system corresponding to the quadratic programming problem
to
1
minimize pT 2x x L(xk , k )p + x L(xk , k )p,
(13.25a)
p
2
subject to hj (xk ) + hj (xk )T p = 0,
j = 1, . . . , .
(13.25b)
This approximate problem has as its objective a second-order approximation of the Lagrange function with respect to the primal variables
x, and the original constraints have been replaced by their first-order
approximations at xk . The Lagrange multiplier vector v k appearing in
(13.24) is the vector of Lagrange multipliers for the constraints (13.25b).
As for Newton methods in unconstrained optimization, convergence
to a stationary point of the Lagrangian in Rn R requires (unless some
sort of line search is introduced) that we start the algorithm close to such
a point and where also the Hessian of the Lagrangian is invertible so that
the algorithm is well-defined. Under the additional conditions that the
stationary point x is a strict minimum of f over the feasible set, that
it satisfies the linear independence constraint qualification LICQ (see
Definition 5.41), and that it together with the KKT multiplier vector
satisfies a second-order sufficient condition (cf. Theorem 4.17), the
sequence {(xk , k )} converges towards the KKT point (x , ) with a
superlinear rate (cf. Section 11.10).
We remark that the convergence theory presented for the above rudimentary Newton method is far from satisfactory, for several reasons:
Convergence is only local, which means that the algorithm must be
combined with an algorithm that converges to a KKT point from
any starting vector, that is, a global algorithm.
The algorithm requires strong assumptions about the problem,
such as that the functions f and hj are in C 2 and that the Hessian
of the Lagrangian is positive definite, in order for the solution to
(13.25) to be well-defined.

339

Constrained optimization
In the next section, we will therefore develop a modification of the
above algorithm, which is globally convergent to stationary points. Moreover, we will work also with inequality constraints, which is not immediate to incorporate into the above Newton-like framework.

13.2.2

A penalty-function based SQP algorithm

In order to introduce a penalty function into the discussion, let us consider first the following one:
P (x, ) := kx L(x, )k2 + kh(x)k2 .

(13.26)

This is an exact penalty function, because its unconstrained minima are

(or, strongly relate to) optimal solutions and/or Lagrange multipliers of
the constrained problem. The exact penalty function (13.26) has been
used extensively in cases where = n and the problem is to find a
solution to hj (x) = 0 for all j. The function has significant drawbacks,
however: it does not distinguish between local minima and maxima, and
it may have local minima that are not global and even do not correspond
to vectors where the value of P is zero; in other words, it may have local
minima that are even infeasible in the original problem.
The above case of penalty function is differentiable; the more popular
penalty functions are non-differentiable. We present such a one next.
Consider the constrained optimization problem to
minimize f (x),
subject to gi (x) 0,
hj (x) = 0,

(13.27a)
i = 1, . . . , m,
j = 1, . . . , ,

(13.27b)
(13.27c)

where f : Rn R, gi : Rn R, and hj : Rn R are given functions in

C 1 on Rn . We introduce the l1 penalty function [cf. (13.4)]

S (x) :=

m
X
i=1

maximum {0, gi (x)} +

X
j=1

|hj (x)|,

(13.28)

and the associated exact penalty function

Pe (x) := f (x) +
S (x),

(13.29)

where > 0.
Proposition 13.10 (an exact penalty function) Suppose that x satisfies the KKT conditions (5.17) of the problem (13.27), together with
340

Sequential quadratic programming

Lagrange multipliers ( , ). Suppose further that the functions f and
gi , i I(x ), all are convex, and that hj , j = 1, . . . , , are affine. If the
value of is large enough such that
maximum{i , i I(x );

|j |, j = 1, . . . , },

then the vector x is also a global minimum of the function Pe .

Proof. [Sketch] Consider the problem of minimizing Pe over Rn . We
can rewrite this problem as follows:

X
X
minimize f (x) +
yi +
zj ,
(13.30a)
i=1

subject to

j=1

yi gi (x) and yi 0, i = 1, . . . , m,
(13.30b)
zj hj (x) and zj hj (x), j = 1, . . . , . (13.30c)

Analyzing the KKT conditions for this problem, we can construct multipliers for the problem (13.30) from the multiplier vectors ( , ) and
show that x is a globally optimal solution to it (note the convexity assumptions).
There are similar results also for more general, non-convex, problems
that establish that if x is a (strict) local minimum to (13.27) then it is
also a (strict) local minimum of the exact penalty function.
We must note, however, that the implication is in a somewhat unsatisfactory direction: there may exist local minima of Pe that do not
correspond to constrained local minima in the original problem, for any
value of . The theory is much more satisfactory in the convex case.
We develop a penalty SQP algorithm, known as the MSQP method
(as in Merit SQP, merit function being synonymous with objective function), for solving the general problem (13.27). Given an iterate xk Rn

and a vector (k , k ) Rm
+ R , suppose we choose a positive definite,
n
symmetric matrix B k R
; for example, it can be an approximation
of 2x x L(xk , k , k ). We then solve the following subproblem:
1 T
p B k p + f (xk )T p,
2
subject to gi (xk ) + gi (xk )T p 0,
minimize

(13.31a)

hj (xk ) + hj (xk ) p = 0,

i = 1, . . . , m,

(13.31b)

j = 1, . . . , .

(13.31c)

Note that if we were to utilize B k := 2x x L(xk , k , k ) then the problem (13.31) would be the optimization problem associated with a secondorder approximation of the KKT conditions for the original problem
341

Constrained optimization
(13.27); the close connection to quasi-Newton methods in unconstrained
optimization should be obvious.
We also took the liberty to replace the term x L(xk , k , k )T p by
the term f (xk )T p. This is without any loss of generality, as the KKT
conditions for the problem (13.31) imply that they can be interchanged
without any loss of generalitythe only difference in the two objectives
lies in the constant term which plays no role in the optimization.
A quasi-Newton type method based on the subproblem (13.31) followed by a unit step and a proper update of the matrix B k , as in the
BFGS algorithm, is locally convergent with a superlinear speed, just
as in the unconstrained case. But we are still interested in a globally
convergent version, whence we develop the theory of an algorithm that
utilizes the exact penalty function (13.29) in a line search rather than
taking a unit step. Our first result shows when the subproblem solution
provides a descent direction with respect to this function.
Lemma 13.11 (a descent property) Given xk Rn consider the strictly
convex quadratic problem (13.31), where B k Rnn is symmetric and
positive definite. Suppose that pk solves this problem together with the
multipliers and . Assume that pk 6= 0n . If
maximum {1 , . . . , m , |1 |, . . . , | |}
then the vector pk is a direction of descent with respect to the l1 penalty
function (13.29) at (xk , k , k ).
Proof. Using the KKT conditions of the problem (13.31) we obtain that
f (xk )T p = pT B k p
= pT B k p +
pT B k p +

m
X
i=1

m
X

i gi (xk )T p
i gi (xk ) +

i=1

m
X
i=1

X
j=1

j hj (xk )T p

j hj (xk )

j=1

i maximum {0, gi (xk )} +

X
j=1

|j ||hj (xk )|

X
X
pT B k p +
maximum {0, gi (xk )} +
|hj (xk )| .
i=1

j=1

In order to investigate the descent properties of pk with respect to

342

Sequential quadratic programming

Pe at xk , we next note that
Pe (xk )Pe (xk +pk ) = f (xk ) f (xk + pk )
m
X
+
[max {0, gi (xk )}max {0, gi (xk +pk )}]
i=1

X
j=1

[|hj (xk )| |hj (xk + pk )|].

Let Oi (respectively, Oj ) denote functions from R to R each function

Or having the property that lim0 Or () = 0, and specially chosen
such that the below identities follow. Then, for > 0 small enough,
f (xk + pk ) = f (xk ) + f (xk )T pk + O0 ().
Also, gi (xk + pk ) = gi (xk ) + gi (xk )T pk + Oi () gi (xk )
gi (xk ) + Oi () holds by the KKT conditions of the problem (13.31).
Hence,
maximum {0, gi (xk + pk )} (1 ) maximum {0, gi (xk )} + |Oi ()|.
Similarly we obtain that hj (xk + pk ) = hj (xk ) + hj (xk )T pk +
Oj () = (1 )hj (xk ) + Oj (), and hence
|hj (xk + pk )| (1 )|hj (xk )| + |Oj ()|.
Using these three expressions in the expression for Pe (xk ) Pe (xk +
pk ) we obtain that for small enough > 0, Pe (xk ) Pe (xk + pk )
P
P
[f (xk )T pk + m
i=1 maximum {0, gi (xk )}+
j=1 |hj (xk )|+O()].
Hence, we obtain that
Pe (xk ) Pe (xk + pk ) [pT
k B k pk + O()] > 0
for every > 0 small enough, due to the positive definiteness of the
matrix B k . We are done.
The MSQP algorithm then works as follows. At some iteration k,
we have at hand a vector xk . Select a symmetric and positive definite
matrix B k Rnn . Solve the QP problem (13.31) in order to obtain
the vector pk and multipliers (k+1 , k+1 ). If pk = 0n we stop with xk
being a KKT point for the original problem (13.27) together with the
multipliers (k+1 , k+1 ). (Why?) Otherwise, find xk+1 = xk + k pk
where k minimizes Pe (xk + pk ) over 0. Increase k by one and
repeat.
Convergence of this rudimentary algorithm follows below.
343

Constrained optimization
Theorem 13.12 (convergence of the MSQP method) The algorithm MSQP either terminates finitely at a KKT point for the problem (13.27)
or it produces an infinite sequence {xk }. In the latter case, we assume
that {xk } lies in a compact set X Rn and that for every x X
and symmetric and positive definite matrix B k the QP (13.31) has a
unique solution, and also unique multiplier vectors and satisfying
max {1 , . . . , m , |1 |, . . . , | |}, where > 0 is the penalty parameter. Furthermore, assume that the sequence {B k } of matrices is bounded
and that every limit point of this sequence is positive definite (or, the
sequence {B 1
k } of matrices is bounded). Then, every limit point of
{xk } is a KKT point for the problem (13.27).
Proof. [Sketch] Clearly, the algorithm stops precisely at KKT points,
so we concentrate on the case where {xk } is an infinite sequence. We
can consider an iteration as a descent step wherein we first construct a
descent direction pk , followed by a line search in the continuous function
Pe , and followed by an update of the matrix B k . By the properties
stated, each of these steps is well defined.
Since the sequence {xk } is bounded, it has an limit point, say x .
Consider from now on this subsequence. By the assumptions stated, also
the sequence {pk } must be bounded. (Why?) Suppose that p is an
limit point of {pk } within the above-mentioned subsequence. Suppose
that it is non-zero. By assumption the sequence {B k } also has limit
points within this subsequence, all of which are positive definite. Suppose B is one such matrix. Then, by Lemma 13.11 the vector p
must define a descent direction for Pe . This contradicts the assumption
that x is an limit point. (Why?) Therefore, it must be the case that
p = 0n , in which case x is stationary, that is, a KKT point. We are
done.
Note that we here have not described any rules for selecting the value
of . Clearly, this is a difficult task, which must be decided upon from
experiments including the results from the above line searches with respect to the merit function Pe . Further, we have no guarantees that the
QP subproblems (13.31) are feasible; in the above theorem we assumed
that the problem is well-defined. Further still, Pe is only continuous and
directionally differentiable, whence we cannot utilize several of the step
length rules devised in Section 11.3. Local superlinear or quadratic convergence of this algorithm can actually be impaired due to the use of this
merit function, as it is possible to construct examples where a unit step
does not reduce its value even very close to an optimal solution. (This
is known as the Maratos effect, after [Mar78].) The Notes Section 13.4
lead to further reading on these issues.
344

Sequential quadratic programming

13.2.3

A numerical example on the MSQP algorithm

Consider the two-dimensional optimization problem to

minimize f (x) := 2x21 + 2x22 2x1 x2 4x1 6x2 ,

subject to g1 (x) :=

2x21

x2 0,

g2 (x) := x1 + 5x2 5 0,
g3 (x) := x1 0,
g4 (x) := x2 0.

(13.32a)
(13.32b)
(13.32c)
(13.32d)
(13.32e)

Check that the vector ( 37 , 83 )T is an unconstrained globally optimal

solution, which however is infeasible.
Suppose we wish to utilize the MSQP algorithm for solving this
problem. We choose := 10, B k to always be the partial Hessian
2x x L(xk , k ) of the Lagrangian (notice that it always is positive definite due to the convexity properties of the problem), and the starting
point x0 := (0, 1)T , which is feasible. Hence, f (x0 ) = Pe (x0 ) = 4. We
also select 0 := 04 . Setting up the first QP subproblem accordingly,
we obtain the problem to
1
(4p21 + 4p22 4p1 p2 ) 6p1 2p2 ,
2
subject to 1 p2 0,
minimize

p1 + 5p2 0,
p1 0,

1 p2 0.

(13.33a)
(13.33b)
(13.33c)
(13.33d)
(13.33e)

7 T
Solving this problem yields the solution p1 = ( 35
31 , 31 ) and the multiT
plier vector 1 (0, 1.032258, 0, 0) .
We next perform a line search in the exact penalty function:

minimize Pe (x0 + p1 ) = 3.16128972 6.3225804 4

0

+ 10 max {0, 2.54942742 + 0.2258064 1}

+ 10 max {0, 0} + 10 max {0, 1.1290322}
+ 10 max {0, 1 + 0.2258064}.

We obtain that 1 0.5835726. (Note that the unconstrained minimum of 7 f (x0 + p1 ) is = 1, which however leads to an infeasible
point having a too high penalty.)
This produces the next iterate, x1 (0.6588722, 0.8682256)T.
345

Constrained optimization
We ask the reader to confirm that this is a near-optimal solution by
checking the KKT conditions , and to confirm that the next QP problem
verifies this.
We were able to find the optimal solution this quickly, due to the
facts that the problem is quadratic and that the value = 10 is large
enough. (Check the value of the Lagrange multipliers.)

13.2.4

On recent developments in SQP algorithms

We have seen that the SQP algorithm above has an inherent decision
problem, namely to choose the right value of the penalty parameter .
In recent years, there has been a development of algorithms where the
penalty parameter is avoided altogether. We call such methods filterSQP methods.
In such methods we borrow a term from multi-objective optimization,
and say that x1 dominates x2 if (x
1 ) (x
2 ) and f (x1 ) f (x2 )
[where
=
S is our measure of infeasibility], that is, if x1 is at least
as good, both in terms of feasibility and optimality. A filter is a list of
pairs (
i , fi ) such that
i <
j or fi < fj for all j 6= i in the list. By
adding elements to the filter, we build up an efficient frontier, that is,
the Pareto set in the bi-criterion problem of simultaneously finding low
objective values and reduce the infeasibility. The filter is used in place
of the penalty function, when the standard Newton-like step cannot be
computed, for example because the subproblem is infeasible.
This algorithm class is quickly becoming popular, and has already
been found to be among the best general algorithms in nonlinear programming, especially because it does not rely on any parameters that
need to be estimated.

13.3

A summary and comparison

Quite a few algorithms of the penalty and SQP type exist, of which
only a small number could be summarized here. Which are the relative
strengths and weaknesses of these methods?
First, we may contrast the two types of methods with regards to
their ill-conditioning. The barrier methods of Section 13.1.2 solve a sequence of unconstrained optimization problems that become more and
more ill-conditioned. In contrast, exact penalty methods need not be
ill-conditioned and moreover only one approximate problem is, at least
in principle, enough to solve the original problem. However, it is known
at least for linear and quadratic programming problems that the inherent ill-conditioning of barrier methods can be eliminated (we say the
346

Notes and further reading

ill-conditioning is benign), because of the special structure of these problems and their optimality conditions.
Among the features of SQP methods is that they can deal with
very large classes of problems, including those with nonlinear equality constraints, and they do not rely on the existence of second-order
derivativesalthough they can make good use of them. While it is
known from practice that the number of quadratic subproblems can
be rather small before reaching a near-locally optimal solution, these
subproblems can be costly to solve. A major development has been
made of specialized quadratic programming methods for solving and reoptimizing large-scale quadratic SQP subproblems, and the most recent
codes are quite robust. Still, which methods to prefer depend on many
factors.
The solver fmincon in the Matlab Optimization Toolbox is an SQP
method.

13.4

Notes and further reading

Exterior and interior penalty methods were popularized by the book

Nonlinear Programming: Sequential Unconstrained Minimization Technique by Fiacco and McCormick [FiM68], although barrier methods had
been presented already in 1961 and exterior penalty methods were developed by Courant much earlier still (in 1943). Their name for many
years was SUMT, after the title of the book. These methods lost
popularity when the classes of SQP and augmented Lagrangian methods (see Exercise 13.8 below) had begun to mature, but following the
discovery of the polynomial complexity of certain interior point methods for LP, and in particular the discovery that some of them could be
derived as special barrier methods where the barrier parameter is updated in a special way, made them popular again. Most text books on
nonlinear optimization concentrate on these two classes of methods; see
[Lue84, Fle87, BSS93, Ber99, NoW99].
In linear programming, several recent text books offer descriptions of
interior point methods; see, e.g., [Pad99, Van01].
Sequential quadratic programming (SQP) methods were first developed by Wilson [Wil63]. Sequential linear programming (SLP) methods
(cf. Exercise 13.7 below), which are based on first-order approximations
of the KKT conditions, were developed by staff in the chemical (especially oil) industry; one reason why SLP methods are effective in such applications is that some important blending problems are only mildly nonlinear. The MSQP method described here stems from [Han75, Pow78].
A formal proof of Theorem 13.12 is given in [BSS93, Theorem 10.4.2].
347

Constrained optimization
The issue of feasibility of the SQP subproblems is taken up in [Fle87].
The boundedness of the subproblem solution is often ensured by combining SQP with a trust region method (cf. Section 11.7), such that
the QP subproblem is further constrained. The Maratos effect has been
overcome during the last decade of research; cf. [PaT91, Fac95]. An excellent paper which addresses most of the computational issues within an
SQP algorithm and provides a very good compromise in the form of the
SNOPT softare is [GMS05]. Filter-SQP algorithms offer a substantial
development over the standard SQP methods. Good references to this
rapidly developing class of methods are [FLT02, UUV04].
We recommend a visit to the NEOS Server for Optimization at
http://www-neos.mcs.anl.gov/neos/ for a continuously updated list
of optimization solvers, together with an excellent software guide for
several types of classes of optimization models.

13.5

Exercises

Exercise 13.1 (convexity, exterior penalty method) Assume that the problem (13.1) is convex. Show that with the choice (s) := s2 [where enters the
definition of the penalty function via (13.4)], for every > 0 the problem (13.5)
is convex.
Exercise 13.2 (convexity, interior penalty method) Assume that the problem (13.1) is convex. Show that with the choice (s) := log(s) [where
enters the definition of the penalty function via (13.10)], for every > 0 the
problem (13.11) is convex.
Exercise 13.3 (numerical example, exterior penalty method) Consider the problem to
minimize f (x) :=

1 2
x1 + x22 ,
2

subject to x1 = 1.
Apply the exterior penalty method with the standard quadratic penalty function.
Exercise 13.4 (numerical example, logarithmic barrier method) Consider the
problem to
minimize f (x) :=

1 2
x1 + x22 ,
2

subject to x1 1.
Apply the interior penalty method with a logarithmic penalty function on the
constraint.

348

Exercises
Exercise 13.5 (logarithmic barrier, exam 990827) Consider the problem to
1 2
x1 + x22 ,
2
subject to x1 + 2x2 10.
minimize f (x) :=

Attack this problem with a logarithmic barrier method. Describe explicitly the
trajectory the method follows, as a function of the barrier parameter. Confirm
that the limit point of the trajectory solves the problem.
Exercise 13.6 (logarithmic barrier method in linear programming) Consider the
linear programming problem to
minimize f (x) := y1 + y2 ,

subject to

y2 1,

y1 1,

y 02 .

Apply the interior penalty method with a logarithmic penalty function on the
non-negativity restrictions on the slack variables.
Exercise 13.7 (sequential linear programming) Consider the optimization problem to
minimize

f (x),

subject to

gi (x) 0,
hj (x) = 0,

(13.34a)
i = 1, . . . , m,
j = 1, . . . , ,

(13.34b)
(13.34c)

where the functions f , gi , i = 1, . . . , m, and hj , j = 1, . . . , , all are continuously differentiable on Rn .

is feasible in the problem (13.34). Prove the following
Suppose that x
satisfies the KKT conditions
statement by using linear programming duality: x
if and only if the following LP problem has the optimal value zero:
minimize
p

subject to

f (
x)T p,

gi (
x) + gi (x)T p 0,
hj (
x) + hj (x)T p = 0,

i = 1, . . . , m,
j = 1, . . . , .

Describe briefly how this LP problem could be used to devise an iterative

method for the problem (13.34).
[Note: Algorithms in this class of methods are referred to as Sequential
Linear Programming (SLP) methods.]
Exercise 13.8 (augmented Lagrangian) Consider the problem
f := infimum f (x),
subject to

(P)

x X,
349

Constrained optimization
and
l := infimum l(x),
subject to

(R)

x G.

If X G and l(x) f (x) for all x X we say that (R) is a relaxation of (P);
cf. Section 6.1. Conversely, (P) is then a restrification of (R).
Consider the problem of the form
f := infimum f (x),
subject to gi (x) = 0,

i = 1, . . . , m,

where f and gi , i = 1, . . . , m, are continuous functions on Rn . Let i ,

i = 1, . . . , m, be multipliers for the constraints and let P : Rm R+ be
a continuous exterior penalty function, that is, a function such that
(

P (y )

= 0,
> 0,

if
if

y = 0m ,
y 6= 0m .

Consider the penalized problem

:= infimum
(x) := f (x) +
n
x

m
X

i gi (x) + P (g(x)),

i=1

where g (x) is the m-vector of gi (x) and where > 0. Show that this problem
is a relaxation of the original one.
[Note: Algorithms based on the relaxation (R)which linearly combines
the Lagrangian and a penalty functionare known as augmented Lagrangian
methods, and the function is known as the augmented Lagrangian function.
They constitute an alternative to exact penalty methods, in that they also
can be made convergent without having to let the penalty parameter tend
to infinity, in this case because of the Lagrangian term; in augmented Lagrangian algorithms the multiplier  plays a much more active role than in
SQP methods.]

350

Part VI

Appendix

Answers to the
exercises

Chapter 1: Modelling and classification

Exercise 1.1 Variables:
xj = number of units produced in process j, j = 1, 2;
y = number of half hours hiring the model.
Optimization model:
maximize f (x, y) := 50(3x1 + 5x2 ) 3(x1 + 2x2 ) 2(2x1 + 3x2 ) 5000y,

subject to

x1 + 2x2 20, 000,

2x1 + 3x2 35, 000,

3x1 + 5x2 1, 000 + 200y

x1 0,

x2 0,

0 y 6.

Exercise 1.2 Variables:

xj = number of trainees trained during month j, j = 1, . . . , 5;
yj = number of technicians available at the beginning of month j, j = 1, . . . , 5.
Optimization model:

Answers to the exercises

minimize z =

5
X

(15000yj + 7500xj )

j=1

160y1 50x1
6000
160y2 50x2
7000
160y3 50x3
8000
160y4 50x4
9500
160y5 50x5
11, 500
0.95y1 + x1 =
y2
0.95y2 + x2 =
y3
0.95y3 + x3 =
y4
0.95y4 + x4 =
y5
y1 =
50
yj , xj Z+, j = 1, . . . , 5.

subject to

Exercise 1.3 We declare the following indices:

i, i = 1, . . . , 3: Work place,
k, k = 1, . . . , 2: Connection point,
and variables

(xi , yi ): Coordinates for work place i;

ti,k : Indicator variable; its value is defined as 1 if work place i is connected to the connection point k, and as 0 otherwise;
z: The longest distance to the window.
The problem to minimize the maximum distance to the window is that to
minimize

z,

(A.1)

subject to the work spaces being inside the rectangle:

d
d
xi l ,
2
2
d
d
yi b ,
2
2

i = 1, . . . , 3,

(A.2)

i = 1, . . . , 3,

(A.3)

that the work spaces do not overlap:

(xi xj )2 + (yi yj )2 d2 ,

i = 1, . . . , 3, j = 1, . . . , 3, i 6= j,

that the cables are long enough:



t1,k

(A.4)

l
(xi )2 + (yi 0)2 a2i ,
2

i = 1, . . . , 3,

(A.5)

b 2
) a2i ,
2

i = 1, . . . , 3,

(A.6)

t2,k (xi l)2 + (yi

that each work space must be connected to a connection point:

ti,1 + ti,2 = 1,

i = 1, . . . , 3,

ti,k {0, 1},

354

i = 1, . . . , 3, k = 1, 2,

(A.7)
(A.8)

Answers to the exercises

and finally that the value of z is at least as high as the longest distance to the
window:
b yi z,

i = 1, . . . , 3.

(A.9)

The problem hence is to minimize the objective function in (A.1) under

the constraints (A.2)(A.9).
Exercise 1.4 We declare the following indices:

i: Warehouses (i = 1, . . . , 10),
j: Department stores (j = 1, . . . , 30),
and variables:

xij : portion (between 0 and 1) of the total demand at department store

j which is served from warehouse i,
yi : Indicator variable; its value is defined as 1 if warehouse i is built,
and 0 otherwise.
We also need the following constants, describing the department stores
that are within the specified maximum distance from a warehouse:
(

aij :=

if dij D,
otherwise,

1,
0,

i = 1, . . . , 10, j = 1, . . . , 30.

(a) The problem becomes:

minimize

10
X

ci yi ,

i=1

xij aij yi ,

subject to
30
X
j=1

ej xij ki yi ,

10
X

i = 1, . . . , 10,

j = 1, . . . , 30,

i = 1, . . . , 10,

xij = 1,

j = 1, . . . , 30,

xij 0,

j = 1, . . . , 30,

i=1

yi {0, 1},

i = 1, . . . , 10.

The first constraint makes sure that only warehouses that are built and
which lie sufficiently close to a department store can supply any goods to it.
The second constraint describes the capacity of each warehouse, and the
demand at the various department stores.
The third and fourth constraints describe that the total demand at a department store must be a non-negative (in fact, convex) combination of the
contributions from the different warehouses.
(b) Additional constraints: xij {0, 1} for all i and j.

355

Answers to the exercises

Chapter 3: Convexity
Exercise 3.1 Use the definition of convexity (Definition 3.1).
Exercise 3.2 (a) S is a polyhedron. It is the parallelogram with the corners
a1 + a2 , a1 a2 , a1 + a2 , a1 a2 , that is, S = conv {a1 + a2 , a1 a2 , a1 +
a2 , a1 a2 } which is a polytope and hence a polyhedron.
(b) S is a polyhedron.
(c) S is not a polyhedron. Note that although S is defined as an intersection
of halfspaces it is not a polyhedron, since we need infinitely many halfspaces.
(d) S = {x Rn | 1n x 1n }, that is, a polyhedron.
(e) S is a polyhedron. By squaring both sides of the inequality, it follows
that 2(x0 x1 )T x kx1 k22 kx0 k22 , so S is in fact a halfspace.
(f) S is a polyhedron. Similarly as in e) above it follows that S is the
intersection of the halfspaces
2(x0 xi )T x kxi k22 kx0 k22 ,

i = 1, . . . , k.

Exercise 3.3 (a) x1 is not an extreme point.

(b) x2 is an extreme point. This follows by checking the rank of the equality subsystem and then using Theorem 3.17.
Exercise 3.4 Let
0

A
b
D :=  A A , d := bA .
I n
0n
P . Now, if
Then P is defined by Dx d. Further, P is nonempty, so let x
x is not an extreme point of P , then the rank of equality subsystem is lower
than n. By using this it is possible to construct an x P such that the
rank of the equality subsystem of x is at least one larger than the rank of the
. If this argument is used repeatedly we end up with
equality subsystem of x
an extreme point of P .
Exercise 3.5 We have that
 

1
1

= 0.5

 

 

 

0
1
1
+ 0.5
+ 0.5
,
1
0
1

and since (0, 1)T , (1, 0)T Q and (1, 1)T C we are done.
Exercise 3.6 Assume that a1 , a2 , a3 , b R satisfy

x A,
a1 x1 + a2 x2 + a3 x3 b, x B.
a1 x1 + a2 x2 + a3 x3 b,

356

(A.10)
(A.11)

Answers to the exercises

From (A.10) follows that a2 = 0 and that a3 b. Further, since (1/n, n, 1)T
B for all n > 0, from (A.11) we have that a3 b. Hence, it holds that
a3 = b. Since (0, 0, 0)T , (1, n2 , n)T B for all n 0, inequality (A.11)
shows that b 0 and a3 0. Hence a2 = a3 = b = 0, and it follows
that H := { x R3 | x1 = 0 } is the only hyperplane that separates A and B.
Finally, A H and (0, 0, 0)T H B, so H meets both A and B.
Exercise 3.7 Let B be the intersection of all closed halfspaces in Rn containing A. It follows easily that A B. In order to show that B A, show
that Ac B c by using the Separation Theorem 3.24.
Exercise 3.8 Assume that P 6= . Then, by using Farkas Lemma (Theorem 3.30), show that there exists a p 6= 0m such that p 0m and Bp 0m .
From this it follows that P is unbounded and hence not compact.
Exercise 3.9
Exercise 3.10 The function is strictly convex on

R2 .

Exercise 3.11 (a) Not convex; (b)(f) strictly convex.

Exercise 3.12 (a)(f) Strictly convex.
Exercise 3.13 (a)


f (x, y) =

1
4
(x, y)
2
2

 

2
1

 

x
x
+ (3, 1)
.
y
y

(b) Yes. (c) Yes.

Exercise 3.14 (a) Non-convex; (b) convex; (c) non-convex; (d) convex; (e)
convex.
Exercise 3.15 Yes.
Exercise 3.16 Yes.
Exercise 3.17 We will try to apply Definition 3.45. It is clear that the
objective function can be written as the minimization of a (strictly) convex
function. The constraints are analyzed thus: the first and third, taken together and applying also Example 3.37(c), describe a closed and convex set;
the second and fourth constraint describes a (convex) polyhedron. By Proposition 3.3 we therefore are done. The answer is Yes.
Exercise 3.18 The first constraint is redundant; the feasible set hence is a
nonempty polyhedron. Regarding the objective function, it is defined only for
positive x1 ; the objective function is strictly convex on R++ , since its second

357

Answers to the exercises

derivative there equals 1/x1 > 0 [cf. Theorem 3.41(b)]. We may extend the
definition of x1 ln x1 to a continuous (in fact convex) function, on the whole
of R+ by defining 0 ln 0 = 0. With this classic extension, together with the
constraint, we see that it is the problem of maximizing a convex function over
a closed convex set. This is not a convex problem. The answer is No.

Chapter 4: An introduction to optimality

conditions
Exercise 4.1 (a)
(b) Argue by contradiction and utilize the convexity of f . The proof is
similar in form to that of Theorem 4.3, and utilizes that moving slightly from
x still makes the removed constraint feasible, thus falsifying the initial claim
that x is optimal in the problem (4.32).
Exercise 4.2 Investigating the Hessian matrix yields that a (4, 2) and
b R implies that the objective function is strictly convex (in fact, strongly
convex, because it is quadratic).
[Note: It would be a mistake to here perform a classic transformation,
namely to observe that the problem is symmetric in x1 and x2 and utilize
this to eliminate one of the variables through the identification x1 = x2 .
Suppose we do so. We then reduce the problem to that of minimizing the
one-dimensional function x 7 (4 + a)x2 2x + b over R. The condition for this
function to be strictly convex, and therefore have a unique solution (see the
above remark on strong convexity), is that a > 4, which is a milder condition
than the above. However, if the value of a is larger than 2 the original problem
has no solution! Indeed, suppose we look at the direction x R2 in which
x1 = x2 = p. Then, the function f (x) behaves like (2 a)p2 2p + b which
clearly tends to minus infinity whenever |p| tends to infinity, whenever a > 2.
It is important to notice that the transformation works when the problem has
a solution; otherwise, it is not.]
Exercise 4.3 Let (x) :=
2

xT Ax . Stationarity for at x means that

xT x

xT x

(Ax (x) x) = 0n .

If xi 6= 0n is an eigenvector of A, corresponding to the eigenvalue i , then

(xi ) = i holds. From the above two equations follow that for x 6= 0n to be
stationary it is both necessary and sufficient that x is an eigenvector.
The global minimum is therefore an arbitrary nonzero eigenvector, corresponding to the minimal eigenvalue i of A.
is a local
Exercise 4.4 (a) The proof is by contradiction, so suppose that x
optimum, x is a global optimum, and that f (
x) < f (x ) holds. We first note

358

Answers to the exercises

and the affine nature of the constraints, it
that by the local optimality of x
must hold that
f (
x)T p = 0m ,

for all vectors

p with Ap = 0m .

.
We will especially look at the vector p := x x
Next, by assumption, f (
x) < f (x ), which implies that (x x )T Q(x
x ) < 0 holds. We utilize this strict inequality together with the above to last
establish that, for every > 0,

x + (x x )) < f (x),
f (
. We are done.
which contradicts the local optimality of x
(b)
Exercise 4.5 Utilize the variational inequality characterization of the projection operation.
Exercise 4.6 Utilize Proposition 4.23(b) for this special case of feasible
set. We obtain the following necessary conditions for x 0n to be local
minimum:
0 xj

f (x )
0,
xj

j = 1, 2, . . . , n,

where (for real values a and b) a b means the condition that a b = 0 holds.
In other words, if xj > 0 then the partial derivative of f at x with respect
to xj must be zero; conversely, if this partial derivative if non-zero then the
value of xj must be zero. (This is called complementarity.)
Exercise 4.7 By P
a logarithmic transformation, we may instead maximize
the function f (x) = n
j=1 aj log xj . The optimal solution is
aj
xj = Pn
i=1

ai

j = 1, . . . , n.

(Check the optimality conditions for a problem defined over a simplex.)

We confirm that it is a unique optimal solution by checking that the objective function is strictly concave where it is defined.
Exercise 4.8
Exercise 4.9
Exercise 4.10

359

Answers to the exercises

Chapter 5: Optimality conditions

Exercise 5.1 (1, 2)T is a KKT point for this problem with KKT multipliers
(1, 0)T . Since the problem is convex, this is also a globally optimal solution
(cf. Theorem 5.45). Slaters CQ (and, in fact, LICQ as well) is verified.
Exercise 5.2 (a) The feasible set of the problem consists of countably
many isolated points xk = /2 + 2k, k = 1, 2, . . . , each of which is thus a
locally optimal solution. The globally optimal solution is x = /2. KKT
conditions are not satisfied at the points of local minimum and therefore they
are not necessary for optimality in this problem. (The reason is of course that
CQs are not verified.)
(b) It is easy to verify that FJ conditions are satisfied (as they should be,
cf. Theorems 5.8 and 5.15).
(c) The point (x, y)T = (0, 0)T is a FJ point, but it has nothing to do with
points of local minimum.
Exercise 5.3 KKT system:

Ax b,
 0,
T
A  = 0,
T
 (Ax b) = 0.
Combining the last two equations we obtain

T x = bT .

ExerciseP 5.4 (a) Clearly,Pthe two problems are equivalent. On the other
m
2
hand, { m
i=1 [hi (x)] } = 2
i=1 hi (x)hi (x) = 0 at every feasible solution.
Therefore, MFCQ is violated at every feasible point of the problem (5.22)
(even though Slaters CQ, LICQ, or at least MFCQ might hold for the original
problem).
(b) The objective function is non-differentiable. Therefore, we rewrite the
problem as
minimize z,
subject to f1 (x) z 0,
f2 (x) z 0,

The problem verifies MFCQ (e.g., the direction (0T , 1)T G(x, z) for all feasible points (xT , z)T . Therefore, the KKT conditions are necessary for local
optimality; these conditions are exactly what we need.
Exercise 5.5 The problem is convex and a CQ is fulfilled, so we need to
find an arbitrary KKT point. The KKT system is as follows:

x + AT  = 0,
Ax = b.
360

Answers to the exercises

Ax + AAT  = 0 and AAT  = b yields x = AT (AAT )1 b.

Exercise 5.6 (a)
(b) Show that
the KKT multiplier is positive at every optimal solution.
P
2
It means that n
j=1 xj = 1 is satisfied at every optimal solution; use convexity
to conclude that there may be only one optimal solution.
Exercise 5.7 (a) Locally and globally optimal solutions may be found
using geometrical considerations; (x, y) = (2, 0) gives us a local min, (x, y) =
(3/2, 3/2) is a globally optimal solution. KKT system incidentally has two [in
the space (x, y)] solutions, but at every point there are infinitely many KKT
multipliers. Therefore, in this particular problem KKT conditions are both
necessary and sufficient for local optimality.
(b) The gradients of the constraints are linearly dependent at every feasible
point; thus LICQ is violated.
The feasible set is a union of two convex sets F1 := { (x, y)T | y = 0; xy
0 } and F2 := { (x, y)T | y 0; x y = 0 }. Thus we can solve two convex
optimization problems to minimize f over F1 , and to minimize f over F2 ; then
simply choose the best solution.
(c) The feasible set may be split into 2n convex parts FI , I { 1, . . . , n },
where
aTi x = bi , and xi 0, i I,

aTi x bi , and xi = 0, i 6 I.

Thus we (in principle) have reduced the original non-convex problem that violates LICQ to 2n convex problems.
Exercise
c 1.

5.8

Use the KKT conditions (convex problem + Slaters CQ).

Exercise 5.9 Slaters CQ implies that the KKT

conditions are necessary
P
for optimality. Prove that xj > 0; then xj = Dcj / n
k=1 ck , j = 1, . . . , n.

Chapter 6: Lagrangian duality

Exercise 6.1 (a) For > 0, the Lagrangian L(, ) is strictly convex;

x1 () = 1/ and x2 () = 4/ uniquely. For = 0, q() = .

(b) = 9/16.

(c) The dual problem is to maximize0 q() = 6 4.

T
(d) x = (4/3, 8/3) .
(e) f = q = 9/4.
Exercise 6.2

361

Answers to the exercises

Exercise 6.3

x = (4/2, 2/3)T ; = 8/3; f = q = 22/9.

Exercise 6.4 From L(x, ) := 12 kx y k2 + T Ax we get that x L(x, ) =

0n when x = y AT  [uniquely by the strict convexity and coercivity of
L(, )]. Inserted into the Lagrangian we obtain q() = y T AT  12 kAT k2 .
From q() = 0m we obtain that  = (AAT )1 Ay (uniquely), which yields
the formula sought.
Exercise 6.5 (a) The Slater CQ is verified since the problem is convex
(even linear), and there is a strictly feasible point [e.g., (x, y)T = (3, 1)T ].
Introducing Lagrange multipliers 1 and 2 we calculate the Lagrangian
dual function q:
q(1 , 2 ) =

min

R {x 1/2y + (x + y + 1) + (2x + y + 2)}

1

(1 ,2 ) 2
+

= 1 + 22 + min (1 1 22 )x + min (1/2 + 1 + 2 )y

x0

1 + 22 ,
,

y0

if 1 + 22 1 and 1 + 2 1/2,
otherwise.

Therefore, the set of optimal Lagrange multipliers is { (1 , 2 ) | 1

0; 2 0; 1 +22 = 1 : 1 +2 1/2 }, which is clearly convex and bounded
(e.g., you may illustrate this graphically) as it should be in the presence of
Slaters CQ.
(b) At (1 , 2 )T = (1/4, 1/3)T the set of optimal solutions to the Lagrangian relaxed problem is the singleton { (0, 0)T }. Hence, the Lagrangian
function is differentiable at this point and its gradient equals the value of the
vector of constraint functions evaluated at the optimal solution to the relaxed
problem, i.e., (0 + 0 + 1, 2 0 + 0 + 2)T = (1, 2)T . Alternatively, differentiate
q at a given point to obtain the result.
At (1 , 2 )T = (1, 0)T the set of optimal solutions to the Lagrangian relaxed problem is not a singleton: it equals { (x, 0)T | x 0 }. Hence, the dual
function is not differentiable, and the set of subgradients is obtained by evaluating the constraint functions at the optimal solutions to the relaxed problem,
i.e., q(1, 0) = { (x + 1, 2x + 2)T | x 0 }.
Exercise 6.6 Introduce the multipliers j , j = 1, . . . , n, and i , i =
1, . . . , m. We obtain that the minimum of the Lagrangian is obtained at
xij (j , i ) = e(1+j +i ) . Inserted into the Lagrangian yields the dual objective function
q(, ) =

m
n X
X
j=1 i=1

e(1+j +i )

n
X
j=1

bj j

which is to be maximized over (, ) Rn Rm .

Exercise 6.7

362

m
X
i=1

a i i ,

Answers to the exercises

=1
=2
=3
Further,

= x1 = 1, x2 = 2,
= x1 = 1, x2 = 5/2,
= x1 = 3, x2 = 3,
f (3, 3) = 21, so 43/4 f

infeasible,
infeasible,
feasible,
21.

q(1) = 6;
q(2) = 43/4;
q(3) = 9.

Exercise 6.8 (a) The value of the Lagrangian dual function is given by
q() := minimump{1,...,P } {f (xp ) + T g(xp )}, which is the point-wise minimum of P affine functions. Therefore, q is piece-wise linear, with no more than
P pieces; the number is less if for some value(s) of  more than one element
xp attains the minimum value of the Lagrangian.
(b)
(c) q() := minimumi{1,...,I} {f (xi ) + T g (xi ), where xi , i {1, . . . , I},
are the extreme points of the polytope X. The number of pieces of the dual
function is bounded by the number I of extreme points of X.
Exercise 6.9 The dual problem is to maximize q() over R+ , where
q() := 5 + minimum (2 )x1 + minimum (1 )x2 .
x1 {0,1,...,4}

x2 {0,1,...,4}

= 0 yields x() = (0, 0)T ; q(0) = 0;

= 1 yields x() = (0, 0)T (for example); q(1) = 5;
= 2 yields x() = (0, 4)T (for example); q(2) = 6;
= 3 yields x() = (4, 4)T ; q(3) = 3.
Since x() is feasible for = 3, we also gain access to a primal feasible
solution; f ((4, 4)T ) = 12.
We conclude that f [6, 12].
Exercise 6.10 (a) The feasible set of (S) includes that of (P ). The result
then follows from the Relaxation Theorem 6.1.
(b)
(c) The Relaxation Theorem 6.1 applies; the objective function in the Lagrangian minorizes f and the former problems feasible set is larger.

Chapter 8: Linear programming models

Exercise 8.1 (a) Introduce the new variables
is equivalent to the linear program
minimize

m
X

y Rm . Then the problem

yi ,

i=1

subject to

y Ax b y ,
1n x 1n .

363

Answers to the exercises

(b) Introduce the new variables
equivalent to the linear program

y Rm and t R. Then the problem is

m
X

minimize

yi + t,

i=1

y Ax b y ,

subject to

t1n x t1n .

Exercise 8.2 (a) Let

0

(v1 )T

B
..
B
.
B
B
k T
B(v )
:= B
1 T
B (w )
B
..
B

.

(w l )T

1
.. C
. C
C
C
1C
C,
1C
C
.. C
. A
1

x :=

 

a
b

Then from the rank assumption it follows that rank B = n + 1, which means
that x 6= 0n+1 implies that Bx 6= 0k+l . Hence the problem can be solved by
solving the linear program
minimize

(0n+1 )T x,

subject to

Bx 0k+l ,
(1k+l )T Bx = 1.

(b) Let = R2 kxc k22 . Then the problem can be solved by solving the
linear program
minimize

(0n )T xc + 0,

subject to

kv i k22 2(v i )T xc ,
kw

i 2
k2

2(w )

i T

i = 1, . . . , k,

x , i = 1, . . . , l,
c

and compute R as R = + kxc k22 (from the first set of inequalities in the
LP above it follows that + kxc k22 0 so this is well defined).
Exercise 8.3 Since P is bounded there exists no y 6= 0n such that Ay 0m .
Hence there exists no feasible solution to the system

Ay 0m ,
dT y = 1,
which implies that z > 0 in every feasible solution to (8.11).

364

Answers to the exercises

Further, let (y , z ) be a feasible solution to (8.11). Then z > 0 and
x = y /z is feasible to (8.10), and f (x ) = g(y , z ). Conversely, let x
be a feasible solution to (8.10). Then by the hypothesis dT x + > 0. Let
z = 1/(dT x + ) and y = z x . Then (y , z ) is a feasible solution to
(8.11) and g(y , z ) = f (x ). These facts imply the assertion.

Exercise 8.4 The problem can be transformed into the standard form:
minimize
subject to

z =

x1 5x+
2 +5x2 7x3 +7x3 ,

5x1 2x+
2 +2x2 +6x3 6x3 s1

3x1

+4x+
2

7x1

+3x+
2

x1 ,

x+
2,

4x
2

3x
2

x
2 ,

9x+
3
+5x+
3

= 15,

+9x
3

5x
3

x+
3,

= 9,
+s2 = 23,

x
3 , s1 , s2 0,

where x1 = x1 + 2, x2 = x+
2 x2 , x3 = x3 x3 , and z = z 2.

Exercise 8.5 (a) The first equality constraint gives that

1
(11 2x1 4x2 ).
6
Now, by substituting x3 with this expression in the objective function and the
second equality constraint the problem is in standard form and x3 is eliminated.
(b) If x3 0, then we must add the constraint (11 2x1 4x2 )/6 0 to
the problem. But this is an inequality, so in order to transform the problem
into standard form we must add a slack variable.
x3 =

Exercise 8.6 Assume that the column in the constraint matrix corresponding to the variable x+
j is aj . Then the column in the constraint matrix corresponding to the variable x
j is aj . The statement follows from the definition
of a BFS, since aj and aj are linearly dependent.
Exercise 8.7 Let P be the set of feasible solutions to (8.12) and Q be the set
of feasible solutions to (8.13). Obviously P Q. In order to show that Q P
assume that there exists an x Q such that x
/ P and derive a contradiction.

Chapter 9: The simplex method

Exercise 9.1 The phase I problem becomes
minimize
subject to

w = a1 + a2 ,
3x1 2x2 + x3 s1 + a1 = 3,

x1 + x2 2x3 s2 + a2 = 1,

x1 , x2 , x3 , s1 , s2 , a1 , a2 0.

365

Answers to the exercises

From the equality constraints follow that a1 + a2 4 for all x1 , x2 , x3 , s1 , s2
0. In particular, it follows that w 4 for all feasible solutions to the phase I
problem, which means that the original problem is infeasible.
Exercise 9.2 (a) The standard form is given by
minimize

3x1 + 2x2 + x3 ,

subject to

2x1 + x3 s1 = 3,

2x1 + 2x2 + x3 = 5,
x1 , x2 , x3 , s1 0.
By solving the phase I problem with the simplex algorithm we get the feasible
basis xB = (x1 , x2 )T . Then by solving the phase II problem with the simplex
algorithm we get the optimal solution x = (x1 , x2 , x3 )T = (0, 1, 3)T .
(b) No, the set of all optimal solution is given by the set
{ x R3 | (0, 1, 3)T + (1 )(0, 0, 5)T ;

[0, 1] }.

Exercise 9.3 The reduced cost for all the variables except for xj must be
greater than or equal to 0. Hence it follows that the current basis is optimal
to the problem that arises if xj is fixed to zero. The assertion then follows
from the fact that the current basis is non-degenerate.
Exercise 9.4

Chapter 10: LP duality and sensitivity analysis

Exercise 10.1 The linear programming dual is given by
minimize
subject to

11y1 +23y2 +12y3 ,

4y1 +3y2 +7y3 6,

3y1 +2y2 +4y3 3,

8y1 +7y2 +3y3 2,

7y1 +6y2 +2y3 = 5,

y2

0,

y3 0.

366

Answers to the exercises

Exercise 10.2 (a) The linear programming dual is given by

b T y 1 + lT y 2 + u T y 3 ,
subject to AT y 1 +I n y 2 +I n y 3 = ,
y2
0n ,
y 3 0n .
maximize

(b) A feasible solution to the linear programming dual is given by

y 1 = 0m ,
y 2 = (max{0, c1 }, . . . , max{0, cn })T ,
y 3 = (min{0, c1 }, . . . , min{0, cn })T .

Exercise 10.3 Use the Weak and Strong Duality Theorems.

Exercise 10.4 The LP dual is infeasible. Hence, from the Weak and Strong
Duality Theorems it follows that the primal problem is either infeasible or
unbounded.
Exercise 10.5 By using the Strong Duality Theorem we get the following
polyhedron:

Ax b,
AT y ,
T x = bT y ,
x 0n ,
y 0m .

Exercise 10.6 From the Strong Duality Theorem it follows that

bT y . Use this to establish the statement.

T x =

Exercise 10.7 The dual problem only contains two variables and hence can
be solved graphically. We get the optimal solution y = (2, 0)T . The complementary slackness conditions then implies that x1 = x2 = x3 = x5 = 0. Hence,
let xB = (x4 , x6 )T . The optimal solution is x = (x1 , x2 , x3 , x4 , x5 , x6 )T =
(0, 0, 0, 3, 0, 1)T .
Exercise 10.8 From the complementary slackness conditions and the fact

367

Answers to the exercises

that c1 /a1 cn /an it follows that
u=

cr
,
ar

yj = cj
yj = 0,

cr
aj , j = 1, . . . , r 1,
ar
j = r, . . . , n,

is a dual feasible solution which together with the given primal solution fulfil
the LP primaldual optimality conditions.
Exercise 10.9
Exercise 10.10
Exercise 10.11
Exercise 10.12
Exercise 10.13 The basis
c4 8.

xB := (x1 , x2 )T is optimal as long as c3 5 and

Exercise 10.14 b) The basis xB := (x1 , x3 )T is optimal for all 6.5.

c) The basis xB := (x1 , x3 ) is not primal feasible for = 7, but it is dual
feasible, so by using the dual simplex method it follows that xB := (x1 , x5 )T
is an optimal basis.
Exercise 10.15
Exercise 10.16 The problem is that min max is not true if the feasible
set is empty. For example, suppose we take max 2x1 +x2 : x1 +x2 1; x 02 .
The dual is min y : y 2; y 1; y 0. A few lines down, where we require
x 02 , the primal is infeasible; thats where the sequence fails.
The conclusion is however true for b = 0m and = 0n .

Chapter 11: Unconstrained optimization

Exercise 11.1
Exercise 11.2 The directional derivative is 13; the answer is No.
Exercise 11.3 (a) The search direction is not a descent direction, for
example because the Hessian matrix is indefinite or negative definite.
(b) The linear system is unsolvable, for example because the Hessian matrix
is indefinite. [Note: Even for indefinite Hessians, the search direction might
exist for some right-hand sides.]

368

Answers to the exercises

(c) Use the LevenbergMarquardt modification.

Exercise 11.4 Let y1 := x1 2 and y2 := 5(x2 + 6). We then get

f (x) = g(y ) = y12 + y22 . At every y R2 the negative gradient points towards
the optimum!
Exercise 11.5 (a) x1 = (1/2, 1)T .
(b) The Hessian matrix is
2 f ( x1 ) =

10
4

4
.
2

The answer is Yes.

(c) The answer is Yes.
Exercise 11.6 (a) x1 = (2, 1/2)T .
(b) The answer is No. The gradient is zero.
(c) The answer is Yes.
Exercise 11.7 (a)
(b) (0, 0.6).

Exercise 11.8 (a) f (x0 ) = (1, 2)T = kf (x0 )k = 5; x1 = (4/3, 2/3)T

= kf (x1 )k = 16/27.
(b) If f is the gradient of a C 2 function f : Rn 7 R we obtain that
f = 2 f , that is, Newtons method for unconstrained optimization is obtained.
Exercise 11.9 (a) x = (AT A)1 AT b.
(b) The objective function is convex, since the Hessian is AT A (which is
always positive semidefinite; check!). Therefore, the normal solution in (a) is
globally optimal.
Exercise 11.10 The reason is that we do not wish to allow for a bad direction pk to be compensated for by simply allowing it to be longer.
Exercise 11.11
Exercise 11.12 (a) We have that
f (y ) + (y xk ) = 0n Qy + q + (y xk ) = 0n
(Q + I n )y = xk q (Q + I n )(y xk ).

Further,
(Q + I n )(y xk ) = xk q (Q + I n )xk = (Qxk + q).

(b) If {xk } converges to x then {pk } = {xk+1 xk } must converge to

zero. From the updating formula we obtain that pk = (Q + I n )1 f (xk )

369

Answers to the exercises

for every k. The sequence {f (xk )} converges to f (x ), since f C 1 . If
f (x ) 6= 0n then {pk } would converge to (Q + I n )1 f (x ) 6= 0n , since
(Q + I n )1 is positive definite when Q + I n is. This leads to a contradiction.
Hence, f (x ) = 0n . Since f is convex x is a global minimum of f over Rn .
Exercise 11.13 Case I: {f (xk )} 0n ; {xk } and {f (xk )} diverge.
Example: f (x) = log x; {xk } ; {f (xk )} ; {f (xk )} 0.
Case II: {f (xk )} 0n ; {xk } diverges; {f (xk )} converges.
Example: f (x) = 1/x; {xk } ; {f (xk )} 0; {f (xk )} 0.

Case III: {f (xk )} 0n ; {xk } 

is bounded; {f (xk )} is bounded.
1 + 1/k,
k even
1 3
Example: f (x) = 3 x x; xk =
1 1/k k odd
{xk } has two limit points: 1; {f (xk )} has two limit points: 2/3.
Case IV: {f (xk )} 0n ; {xk } is bounded; {f (xk )} converges.
Example: f (x) = x2 1; xk as above; {f (xk )} 0.
Case V: {f (xk )} 0n ; {xk } and {f (xk )} converge.
Example: f as in Case IV; xk = 1 + 1/k.
Exercise 11.14
Exercise 11.15
Exercise 11.16
Exercise 11.17

Chapter 12: Optimization over convex sets

Exercise 12.1 If the LP problem (12.2) has an unbounded solution, we
must replace the search direction towards an extreme point with the extreme
direction identified in the last iteration of the simplex method; that is, we
choose pk to be a direction of S in which the objective value in (12.2) tends
to . The feasible set in the line search is R+ .
Under a compactness assumption on the intersection of S with the level
set of f at x0 the convergence properties in Theorem 12.1 can be reached also
for this more general problem and algorithm, otherwise not necessarily.
Exercise 12.2
Exercise 12.3 (a)
(b) x1 = (12/5, 4/5)T ; UBD = f (x1 ) = 8. The LP problem defined at
gives LBD = 0. Hence, f [0, 8].

370

x0

Answers to the exercises

Exercise 12.4 (a)
(b) x = (4, 2)T ; f = 80.
Exercise 12.5 x0 = (1, 1)T ; f (x0 ) = 5/8; y 0 = (0, 0)T ; z(y 0 ) = 7/8; x1 =
(1/4, 1/4)T ; f (x1 ) = 1/16; y 1 = (1, 0)T ; z(y 1 ) = 3/16; x2 = (13/20, 5/20)T ;
f (x2 ) = 1/80.
f [3/16, 1/80].
Exercise 12.6
Exercise 12.7
Exercise 12.8
Exercise 12.9 The answer is yes. The extreme points visited and stored are
(1, 1)T (if we start at the same place as in Exercise 12.5), (0, 0)T , and (1, 0)T ,
which are the same as in the FrankWolfe algorithm. Using the simplicial
decomposition method, the optimal solution x = (1/2, 0)T is found in the
convex hull of these points.

Chapter 13: Constrained optimization

Exercise 13.1
Exercise 13.2
Exercise 13.3
Exercise 13.4
Exercise 13.5 For a given parameter value > 0 the unconstrained problem
to
minimize f (x) log(x1 + 2x2 10)

x 2

uniquely solvable:
x1

= 0;
x1 + 2x2 10

2x2

2
=0
x1 + 2x2 10

2
yields that x1 = x2 must hold; the resulting
pquadratic equation 3x1 10x1 =
0 has two roots, of which x1 () = 5/3 + 25/9 + /3 is strictly feasible. As
0, x1 () = x2 () tends to 10/3.
One then shows that x = ( 10
, 10
)T is a KKT point. The constraint is
3
3

binding, and = 10/3 0. Since the problem is convex, x is optimal.

371

Answers to the exercises

Exercise 13.6
Exercise 13.7 Let us first rewrite the LP problem into the following equivalent form, and note that hj (
x) = 0 for all j, since x is feasible:
minimize
p

subject to

f (
x)T p,

gi (
x)T p gi (x),
i = 1, . . . , m,
x)T p = 0,
j = 1, . . . , .
hj (

Letting  0m and  R be the dual variable vector for the inequality and
equality constraints, respectively, we obtain the following dual program:
maximize



( , )

subject to

m
X

i gi (
x),

i=1
m
X

i=1

i gi (
x)

X
j=1

j hj (
x) = f (x),
i 0,

i = 1, . . . , m.

LP duality now establishes the result sought: First, suppose that the optimal value of the above primal problem over p is zero. Then, the same is true
for the dual problem. Hence, by the sign conditions i 0 and gi (x) 0, each
term in the sum must be zero. Hence, we established that complementarity
holds. Next, the two constraints in the dual problem are precisely the dual

feasibility conditions, which hence are fulfilled. Finally, primal feasibility of x

was assumed. It follows that this vector indeed is a KKT point.
is a KKT point, then the dual problem above has a feasible
Conversely, if x
solution given by any KKT multiplier vector (, ). The dual objective is
upper bounded by zero, since each term in the sum is non-positive. On the
other hand, there is a feasible solution with the objective value 0, namely
any KKT point! So, each KKT point must constitute an optimal solution to
this dual LP problem! It then follows by duality theory that the dual of this
problem, which is precisely the primal problem in p above, has a finite optimal
solution, whose optimal value must then be zero. We are done.
[Note: The LP problem given in the exercise is essentially the subproblem
in the Sequential Linear Programming (SLP) algorithm. By the above analy is not a KKT point, and it must
sis, the optimal value must be negative if x
therefore also be negative (since a zero value is given by setting p = 0n ). The
optimal value of p, if one exists, is therefore a descent direction with respect
. A convergent SLP method introduces additional box constraints on
to f at x
p in the LP subproblem to make sure that the solution is finite, and the update is made according to a line search with respect to some penalty function.]
Exercise 13.8

372

References

[Aba67]

[AMO93]

[Arm66]

[AHU58]

[AHU61]

[Avr76]

[AvG96]

[Ban22]

[Bar71]
[BaG69]

[BSS93]

J. Abadie, On the KuhnTucker theorem, in Nonlinear Programming (NATO Summer School, Menton, 1964), North-Holland,
Amsterdam, 1967, pp. 1936.
R. K. Ahuja, T. L. Magnanti, and J. B. Orlin, Network
Flows: Theory, Algorithms, and Applications, Prentice-Hall, Englewood Cliffs, NJ, 1993.
L. Armijo, Minimization of functions having Lipschitz continuous
first partial derivatives, Pacific Journal of Mathematics, 16 (1966),
pp. 13.
K. J. Arrow, L. Hurwicz, and H. Uzawa, eds., Studies in
Linear and Non-Linear Programming, Stanford University Press,
Stanford, CA, 1958.
K. J. Arrow, L. Hurwicz, and H. Uzawa, Constraint qualifications in maximization problems, Naval Research Logistics Quarterly, 8 (1961), pp. 175191.
M. Avriel, Nonlinear Programming: Analysis and Methods,
Prentice Hall Series in Automatic Computation, Prentice Hall,
Englewood Cliffs, NJ, 1976.
M. Avriel and B. Golany, eds., Mathematical Programming
for Industrial Engineers, vol. 20 of Industrial Engineering, Marcel
Dekker, New York, NY, 1996.
S. Banach, Sur les operations dans les ensembles abstraits et leur
application aux equations integrales, Fundamenta Mathematicae,
3 (1922), pp. 133181.
R. H. Bartels, A stabilization of the simplex method, Numerische
Mathematik, 16 (1971), pp. 414434.
R. H. Bartels and G. H. Golub, The simplex method of linear programming using LU-decomposition, Communications of the
ACM, 12 (1969), pp. 266268 and 275278.
M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear
Programming: Theory and Algorithms, John Wiley & Sons, New
York, NY, second ed., 1993.

References
[Ben62]

[Ber99]
[Ber04]

[BNO03]
[BeT89]

[BeT00]

[Bla77]
[BlO72]

[BGLS03]

[BoS00]

[BoL00]

[BHM77]
[Bre73]

[Bro09]
[Bro12]
[Bro70]

374

J. F. Benders, Partitioning procedures for solving mixed variables programming problems, Numerische Mathematik, 4 (1962),
pp. 238252.
D. P. Bertsekas, Nonlinear Programming, Athena Scientific,
Bellmont, MA, second ed., 1999.
, Lagrange multipliers with optimal sensitivity properties in
constrained optimization, Report LIDS 2632, Department of Electrical Engineering and Computer Science, Massachusetts Institute
of Technology, Cambridge, MA, 2004.
, and A. E. Ozdaglar, Convex
D. P. Bertsekas, A. Nedic
Analysis and Optimization, Athena Scientific, Belmont, MA, 2003.
D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed
Computation: Numerical Methods, Prentice Hall, London, U.K.,
1989.
D. P. Bertsekas and J. N. Tsitsiklis, Gradient convergence in
gradient methods with errors, SIAM Journal on Optimization, 10
(2000), pp. 627642.
R. G. Bland, New finite pivoting rules for the simplex method,
Mathematics of Operations Research, 2 (1977), pp. 103107.
E. Blum and W. Oettli, Direct proof of the existence theorem in
quadratic programming, Operations Research, 20 (1972), pp. 165
167.
J. F. Bonnans, J. C. Gilbert, C. Lemar
echal, and C. A.
bal, Numerical Optimization: Theoretical and PractiSagastiza
cal Aspects, Universitext, Springer-Verlag, Berlin, 2003. Translated from the original French edition, published by SpringerVerlag 1997.
J. F. Bonnans and A. Shapiro, Perturbation Analysis of Optimization Problems, Springer Series in Operations Research,
Springer-Verlag, New York, NY, 2000.
J. M. Borwein and A. S. Lewis, Convex Analysis and Nonlinear
Optimization: Theory and Examples, CMS Books in Mathematics,
Springer-Verlag, New York, NY, 2000.
S. P. Bradley, A. C. Hax, and T. L. Magnanti, Applied
Mathematical Programming, Addison-Wesley, Reading, MA, 1977.
R. P. Brent, Algorithms for Minimization Without Derivatives,
Prentice Hall Series in Automatic Computation, Prentice Hall,
Englewood Cliffs, NJ, 1973. Reprinted by Dover Publications,
Inc., Mineola, NY, 2002.
L. E. J. Brouwer, On continuous vector distributions on surfaces, Amsterdam Proceedings, 11 (1909).

, Uber
Abbildung von Mannigfaltigkeiten, Mathematische
Annalen, 71 (1912), pp. 97115.
C. G. Broyden, The convergence of single-rank quasi-Newton
methods, Mathematics of Computation, 24 (1970), pp. 365382.

References
[BGIS95]

R. Burachik, L. M. G. Drummond, A. N. Iusem, and B. F.

Svaiter, Full convergence of the steepest descent method with inexact line searches, Optimization, 32 (1995), pp. 137146.
[BuF91]
J. V. Burke and M. C. Ferris, Characterization of solution
sets of convex programs, Operations Research Letters, 10 (1991),
pp. 5760.

[Car07]
C. Carath
eodory, Uber
den Variabilit
atsbereich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen, Mathematische Annalen, 64 (1907), pp. 95115.

, Uber
den Variabilit
atsbereich der Fourierschen Konstan[Car11]
ten von positiven harmonischen Funktionen, Rendiconti del Circolo Matematico di Palermo, 32 (1911), pp. 193217.
[Casetal02] E. Castillo, A. J. Conejo, P. R. G. Pedregal, and N. Alguacil, Building and Solving Mathematical Programming Models
in Engineering and Science, Pure and Applied Mathematics, John
Wiley & Sons, New York, NY, 2002.
[Cau1847] A. Cauchy, Methode generale pour la resolution des syst`emes
dequations simultanees, Comptes Rendus Hebdomadaires des
Seances de lAcademie des Sciences (Paris), Serie A, 25 (1847),
pp. 536538.
[Cha52]
A. Charnes, Optimality and degeneracy in linear programming,
Econometrica, 20 (1952), pp. 160170.
tal, Linear Programming, Freeman, New York, NY,
[Chv83]
V. Chva
1983.
[CGT00]
A. R. Conn, N. I. M. Gould, and Ph. L. Toint, Trust-Region
Methods, vol. 1 of MPS/SIAM Series on Optimization, SIAM and
Mathematical Programming Society, Philadelphia, PA, 2000.
[Cro36]
H. Cross, Analysis of flow in networks of conduits or conductors, Bulletin 286, Engineering Experiment Station, University of
Illinois, Urbana, IL, 1936.
[Dan51]
G. B. Dantzig, Maximization of a linear function of variables
subject to linear inequalities, in Activity Analysis of Production
and Allocation, Tj. C. Koopmans, ed., New York, NY, 1951, John
Wiley & Sons, pp. 339347.
[Dan53]
, Computational algorithm of the revised simplex method, Report RM 1266, The Rand Corporation, Santa Monica, CA, 1953.
[Dan57]
, Concepts, origins, and use of linear programming, in Proceedings of the First International Conference on Operational Research, Oxford, 1957, M. Davies, R. T. Eddison, and T. Page, eds.,
London, U.K., 1957, The English Universities Press, pp. 100108.
, Linear Programming and Extensions, Princeton University
[Dan63]
Press, Princeton, NJ, 1963.
[DaM05]
G. B. Dantzig and N. T. Mukund, Linear programming 3: Implementation, Springer Series in Operations Research, SpringerVerlag, New York, NY, 2005.

375

References
[DaO53]

G. B. Dantzig and A. Orden, Notes on linear programming:

Part 2, duality theorems, technical report RM-1265, The Rand
Corporation, Santa Monica, CA, 1953.

[DOW55]

G. B. Dantzig, A. Orden, and P. Wolfe, The generalized simplex method for minimizing a linear form under linear inequality
restraints, Pacific Journal of Mathematics, 5 (1955), pp. 183195.

[DaT97]

G. B. Dantzig and M. N. Thapa, Linear programming 1: Introduction, Springer Series in Operations Research, Springer-Verlag,
New York, NY, 1997.

[DaT03]

, Linear programming 2: Theory and Extensions, Springer

Series in Operations Research, Springer-Verlag, New York, NY,
2003.

[DaW60]

G. B. Dantzig and P. Wolfe, Decomposition principle for linear programs, Operations Research, 8 (1960), pp. 101111.

[dAu47]

A. dAuriac, A propos de lunicite de solution dans les probl`emes

de reseaux mailles, La Houille Blanche, 2 (1947), pp. 209211.

[Dav59]

W. C. Davidon, Variable metric method for minimization, Report ANL-5990 Rev, Argonne National Laboratories, Argonne,
IL, 1959. Also published in SIAM Journal on Optimization, 1
(1991), pp. 117.

[DeF49]

B. De Finetti, Sulla stratificazioni convesse, Annali di Matematica Pura ed Applicata, 30 (1949), pp. 173183.

[Den59]

J. B. Dennis, Mathematical Programming and Electrical Networks, John Wiley & Sons, New York, NY, 1959.

[DeS83]

J. E. Dennis and R. E. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall,
Englewood Cliffs, NJ, 1983.

[DiJ79]

Y. M. I. Dirickx and L. P. Jennergren, Systems Analysis by

Multilevel Methods: With Applications to Economics and Management, vol. 6 of International Series on Applied Systems Analysis,
John Wiley & Sons, Chichester, U.K., 1979.

[Duf46]

R. J. Duffin, Nonlinear networks, I, Bulletin of the American

Mathematical Society, 52 (1946), pp. 833838.

[Duf47]

, Nonlinear networks, IIa, Bulletin of the American Mathematical Society, 53 (1947), pp. 963971.

[DuH78]

J. C. Dunn and S. Harshbarger, Conditional gradient algorithms with open loop step size rules, Journal of Mathematical
Analysis and Applications, 62 (1978), pp. 432444.

[Eav71]

B. C. Eaves, On quadratic programming, Management Science,

17 (1971), pp. 698711.

[EHL01]

T. F. Edgar, D. M. Himmelblau, and L. S. Lasdon, Optimization of Chemical Processes, McGraw-Hill, New York, NY,
second ed., 2001.

376

References
[Eke74]

I. Ekeland, On the variational principle, Journal of Mathematical Analysis and Applications, 47 (1974), pp. 324353.

[Erm66]

Yu. M. Ermolev, Methods for solving nonlinear extremal problems, Kibernetika, 2 (1966), pp. 117. In Russian, translated into
English in Cybernetics, 2 (1966), pp. 114.

[Eva70]

J. P. Evans, On constraint qualifications in nonlinear programming, Naval Research Logistics Quarterly, 17 (1970), pp. 281286.

[Eve63]

H. Everett, III, Generalized Lagrange multiplier method for

solving problems of optimum allocation of resources, Operations
Research, 11 (1963), pp. 399417.

[Fac95]

F. Facchinei, Minimization of SC 1 functions and the Maratos

effect, Operations Research Letters, 17 (1995), pp. 131137.

[Fal67]

J. E. Falk, Lagrange multipliers and nonlinear programming,

Journal of Mathematical Analysis and Applications, 19 (1967),
pp. 141159.

J. Farkas, Uber
die Theorie der einfachen Ungleichungen, Journal f
ur die Reine und Angewandte Mathematik, 124 (1902), pp. 1
24.

[Far1902]

[Fen51]

W. Fenchel, Convex cones, sets and functions, mimeographed

lecture notes, Princeton University, Princeton, NY, 1951.

[Fia83]

A. V. Fiacco, Introduction to sensitivity and stability analysis in

nonlinear programming, vol. 165 of Mathematics in Science and
Engineering, Academic Press Inc., Orlando, FL, 1983.

[FiM68]

A. V. Fiacco and G. P. McCormick, Nonlinear Programming:

Sequential Unconstrained Minimization Techniques, John Wiley
& Sons, New York, NY, 1968. Also published as volume 4 in the
Classics in Applied Mathematics Series, SIAM, Philadelphia, PA,
1990.

[Fis81]

M. L. Fisher, The Lagrangian relaxation method for solving integer programming problems, Management Science, 27 (1981), pp. 1
18.

[Fis85]

, An applications oriented guide to Lagrangian relaxation,

Interfaces, 15 (1985), pp. 1021.

[Fle70]

R. Fletcher, A new approach to variable metric algorithms,

Computer Journal, 13 (1970), pp. 317322.

[Fle87]

, Practical Methods of Optimization, John Wiley & Sons,

Chichester, U.K., second ed., 1987.

[FLT02]

R. Fletcher, S. Leyffer, and Ph. L. Toint, On the global

convergence of a filter-SQP algorithm, SIAM Journal on Optimization, 13 (2002), pp. 4459.

[FlP63]

R. Fletcher and M. J. D. Powell, A rapidly convergent

descent method for minimization, Computer Journal, 6 (1963),
pp. 163168.

377

References
[FlR64]

R. Fletcher and C. M. Reeves, Function minimization by conjugate gradients, Computer Journal, 7 (1964), pp. 149154.
[FrW56]
M. Frank and P. Wolfe, An algorithm for quadratic programming, Naval Research Logistics Quarterly, 3 (1956), pp. 95110.
[GKT51]
D. H. Gale, H. W. Kuhn, and A. W. Tucker, Linear programming and the theory of games, in Activity Analysis of Production
and Allocation, Tj. C. Koopmans, ed., New York, NY, 1951, Wiley, pp. 317329.
[GaJ79]
M. R. Garey and D. S. Johnson, Computers and Intractability:
A Guide to the Theory of NP-Completeness, Freeman, New York,
NY, 1979.
[GaA05]
S. I. Gass and A. A. Assad, An Annotated Timeline of Operations Research. An Informal History, vol. 75 of International Series
in Operations Research & Management Science, Kluwer Academic
Publishers, New York, NY, 2005.
[Geo74]
A. M. Geoffrion, Lagrangean relaxation for integer programming: Approaches to integer programming, Mathematical Programming Study, 2 (1974), pp. 82114.
[Gil66]
E. G. Gilbert, An iterative procedure for computing the minimum of a quadratic form on a convex set, SIAM Journal on Control, 4 (1966), pp. 6180.
[GiM73]
P. E. Gill and W. Murray, A numerically stable form of the
simplex algorithm, Linear Algebra and Its Applications, 7 (1973),
pp. 99138.
[GMS05]
P. E. Gill, W. Murray, and M. A. Saunders, SNOPT: An
SQP algorithm for large-scale constrained optimization, SIAM Review, 47 (2005), pp. 99131.
[GMSW89] P. E. Gill, W. Murray, M. A. Saunders, and M. H. Wright,
A practical anti-cycling procedure for linearly constrained optimization, Mathematical Programming, 45 (1989), pp. 437474.
[Gol70]
D. Goldfarb, A family of variable-metric methods derived by
variational means, Mathematics of Computation, 24 (1970),
pp. 2326.
[Gol64]
A. A. Goldstein, Convex programming in Hilbert space, Bulletin
of the American Mathematical Society, 70 (1964), pp. 709710.
[GrD03]
A. Granas and J. Dugundji, Fixed Point Theory, Springer
Monographs in Mathematics, Springer-Verlag, New York, NY,
1969.
[Gri00]
A. Griewank, Evaluating Derivatives: Principles and Techniques
of Algorithmic Differentiation, vol. 19 of Frontiers in Applied
Mathematics, SIAM, Philadelphia, PA, 2000.
[Gui69]
M. Guignard, Generalized KuhnTucker conditions for mathematical programming problems in a Banach space, SIAM Journal
on Control, 7 (1969), pp. 232241.

378

References
[Had10]

[Han75]

[HaH96]

[HLV87]

[HWC74]

[HiL93]

[Hof53]
[Ius03]

[Joh48]

[JoM74]

[Jos03]

[Kar84a]

[Kar84b]
[Kha79]
[Kha80]

J. Hadamard, Sur quelques applications de lindice de Kronecker,

in Introduction `
a la theorie des fonctions dune variable, J. Tannary, ed., vol. 2, Hermann, Paris, 1910, pp. 875915.
S. P. Han, Penalty Lagrangian methods in a quasi-Newton approach, Report TR 75-252, Computer Science, Cornell University,
Ithaca, NY, 1975.
G. K. Hauer and H. M. Hoganson, Tailoring a decomposition
method to a large forest management scheduling problem in northern Ontario, INFOR, 34 (1996), pp. 209231.
D. W. Hearn, S. Lawphongpanich, and J. A. Ventura, Restricted simplicial decomposition: Computation and extensions,
Mathematical Programming Study, 31 (1987), pp. 99118.
M. Held, P. Wolfe, and H. P. Crowder, Validation of subgradient optimization, Mathematical Programming, 6 (1974), pp. 62
88.
J.-B. Hiriart-Urruty and C. Lemar
echal, Convex Analysis
and Minimization Algorithms, vol. 305 and 306 of Grundlehren der
mathematischen Wissenschaften, Springer-Verlag, Berlin, 1993.
A. Hoffman, Cycling in the simplex algorithm, Report 2974, National Bureau of Standards, Gaithersburg, MD, 1953.
A. N. Iusem, On the convergence properties of the projected gradient method for convex optimization, Computational and Applied
Mathematics, 22 (2003), pp. 3752.
F. John, Extremum problems with inequalities as subsidiary conditions, in Studies and Essays Presented to R. Courant on his
60th Birthday, January 8, 1948, Interscience Publishers, Inc., New
York, NY, 1948, pp. 187204.
L. A. Johnson and D. S. Montgomery, Operations Research
in Production Planning, Scheduling and Inventory Control, John
Wiley & Sons, New York, NY, 1974.
M. Josefsson, Sensitivity analysis of traffic equilibria, Masters
thesis, Department of Mathematics, Chalmers University of Technology, Gothenburg, Sweden, 2003.
N. Karmarkar, A new polynomial-time algorithm for linear programming, in Proceedings of the 16th Annual ACM Symposium on
Theory of Computing, STOC84 (Washington, DC, April 30May
2, 1984), New York, NY, 1984, ACM Press, pp. 302311.
, A new polynomial-time algorithm for linear programming,
Combinatorica, 4 (1984), pp. 373395.
L. G. Khachiyan, A polynomial algorithm in linear programming,
Doklady Akademii Nauk SSSR, 244 (1979), pp. 10931096.
, Polynomial algorithms in linear programming, Akademiya
Nauk SSSR. Zhurnal Vychislitelno Matematiki i Matematichesko Fiziki, 20 (1980), pp. 5168.

379

References
[Kir1847]

[KlM72]

[KLT03]

[Kre78]
[KuT51]

[LaP05]

[LPS96]

[LPS99]

[Las70]
[Law76]
[LRS91]

[LeP66]

[Lip1877]
[Lue84]

380

G. Kirchhoff, Uber
die Ausfl
osung der Gleichungen auf welche
man bei der Untersuchungen der Linearen Vertheilung Galvanisher Str
ome gef
uhrt wird, Pogendorff Annalen Der Physik, 72
(1847), pp. 497508. English translation, IRE Transactions on
Circuit Theory, CT-5 (1958), pp. 48.
V. Klee and G. J. Minty, How good is the simplex algorithm?,
in Inequalities, III. Proceedings of the Third Symposium on Inequalities held at the University of California, Los Angeles, CA,
September 19, 1969; dedicated to the memory of Theodore S.
Motzkin, O. Shisha, ed., New York, NY, 1972, Academic Press,
pp. 159175.
T. G. Kolda, R. M. Lewis, and V. Torczon, Optimization
by direct search: New perspectives on some classical and modern
methods, SIAM Review, 45 (2003), pp. 385482.
E. Kreyszig, Introductory Functional Analysis with Applications,
John Wiley & Sons, New York, NY, 1978.
H. W. Kuhn and A. W. Tucker, Nonlinear programming, in
Proceedings of the Second Berkeley Symposium on Mathematical
Statistics and Probability, 1950, Berkeley and Los Angeles, CA,
1951, University of California Press, pp. 481492.
T. Larsson and M. Patriksson, Global optimality conditions
for discrete and nonconvex optimizationwith applications to Lagrangian heuristics and column generation, technical report, Department of Mathematics, Chalmers University of Technology,
Gothenburg, Sweden, 2005. To appear in Operations Research.
mberg, CondiT. Larsson, M. Patriksson, and A.-B. Stro
tional subgradient optimizationtheory and applications, European Journal of Operational Research, 88 (1996), pp. 382403.
, Ergodic, primal convergence in dual subgradient schemes
for convex programming, Mathematical Programming, 86 (1999),
pp. 283312.
L. S. Lasdon, Optimization Theory for Large Systems, Macmillan, New York, NY, 1970.
E. Lawler, Combinatorial Optimization: Networks and Matroids,
Holt, Rinehart and Winston, New York, NY, 1976.
J. K. Lenstra, A. H. G. Rinnooy Kan, and A. Schrijver,
eds., History of Mathematical Programming. A Collection of Personal Reminiscences, North-Holland, Amsterdam, 1991.
E. S. Levitin and B. T. Polyak, Constrained minimization
methods, USSR Computational Mathematics and Mathematical
Physics, 6 (1966), pp. 150.
R. Lipschitz, Lehrbuch der Analysis, Cohn & Sohn, Leipzig, 1877.
D. G. Luenberger, Linear and Nonlinear Programming, Addison
Wesley, Reading, MA, second ed., 1984. Reprinted by Kluwer
Academic Publishers, Boston, MA, 2003.

References
[Man65]
[Man69]

[Man88]

O. L. Mangasarian, Pseudo-convex functions, SIAM Journal on

Control, 3 (1965), pp. 281290.
, Nonlinear Programming, McGraw-Hill, New York, NY,
1969. Also published as volume 10 in the Classics in Applied
Mathematics Series, SIAM, Philadelphia, PA, 1994.
, A simple characterization of solution sets of convex programs, Operations Research Letters, 7 (1988), pp. 2126.

[MaF67]

O. L. Mangasarian and S. Fromovitz, The Fritz John necessary optimality conditions in the presence of equality and inequality constraints, Journal of Mathematical Analysis and Applications, 17 (1967), pp. 3747.
[Mar78]
N. Maratos, Exact penalty function algorithms for finite dimensional and control optimization problems, PhD thesis, Imperial
College of Science and Technology, University of London, London,
U.K., 1978.
[Max1865] J. C. Maxwell, A dynamical theory of the electromagnetic field,
Philosophical Transactions of the Royal Society of London, 155
(1865), pp. 459512.
[Min10]
[Min11]

H. Minkowski, Geometrie der Zahlen, Teubner, Leipzig, 1910.

, Gesammelte Abhandlungen, vol. II, Teubner, Leipzig, 1911,
ch. Theorie der konvexen K
orper, Insbesondere Begr
undung ihres
Ober fl
achenbegriffs.

[Mot36]

T. Motzkin, Beitr
age zur Theorie del linearen Ungleichungen,
Azriel, Israel, 1936.
K. G. Murty, Linear Programming, John Wiley & Sons, New
York, NY, 1983.

[Mur83]
[Mur95]
[Nas50]

[Nas51]
[NaS96]
[NeW88]

, Operations Research: Deterministic Optimization Models,

Prentice Hall, Englewood Cliffs, NJ, 1995.
J. F. Nash, Jr., Equilibrium points in n-person games, Proceedings of the National Academy of Sciences of the United States of
America, 36 (1950), pp. 4849.
, Non-cooperative games, Annals of Mathematics, 54 (1951),
pp. 286295.
S. G. Nash and A. Sofer, Linear and Nonlinear Programming,
MacGraw-Hill, Singapore, 1996.
G. L. Nemhauser and L. Wolsey, Integer and Combinatorial
Optimization, Wiley-Interscience Series in Discrete Mathematics
and Optimization, John Wiley & Sons, New York, NY, 1988.

[New1687] I. S. Newton, Philosophiae Naturalis Principia Mathematica,

London, U.K., 1687.
[NoW99]
J. Nocedal and S. J. Wright, Numerical Optimization,
Springer Series in Operations Research, Springer-Verlag, New
York, NY, 1999.

381

References
[Orc54]

[OrR70]

[Pad99]

[PaT91]

[PaS82]

[PaS88]

[PaV91]

[Pat94]

[Pat98]

[PoR69]

[Pol69]

[Pow78]

[PsD78]
[Rad19]

382

W. Orchard-Hays, Background, development and extensions of

the revised simplex method, Report RM 1433, The Rand Corporation, Santa Monica, CA, 1954.
J. M. Ortega and W. C. Rheinboldt, Iterative Solution of
Nonlinear Equations in Several Variables, Academic Press, New
York, NY, 1970. Also published as volume 30 in the Classics in
Applied Mathematics Series, SIAM, Philadelphia, PA, 2000.
M. Padberg, Linear Optimization and Extensions, vol. 12 of Algorithms and Combinatorics, Springer-Verlag, Berlin, second ed.,
1999.
E. R. Panier and A. L. Tits, Avoiding the Maratos effect by
means of a nonmonotone line search, I. General constrained problems, SIAM Journal on Numerical Analysis, 28 (1991), pp. 1183
1195.
C. H. Papadimitriou and K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity, Prentice Hall, Englewood
Cliffs, 1982.
P. M. Pardalos and G. Schnitger, Checking local optimality
in constrained quadratic programming is NP-hard, Operations Research Letters, 7 (1988), pp. 3335.
P. M. Pardalos and S. Vavasis, Quadratic programming with
one negative eigenvalue is NP-hard, Journal of Global Optimization, 1 (1991), pp. 1522.
M. Patriksson, The Traffic Assignment ProblemModels and
Methods, Topics in Transportation, VSP BV, Utrecht, The Netherlands, 1994.
, Nonlinear Programming and Variational Inequalities: A
Unified Approach, vol. 23 of Applied Optimization, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1998.
E. Polak and G. Ribi`
ere, Note sur la convergence de
methodes de directions conjuguees, Revue Francaise dInformation
et Recherche Operationnelle, 3 (1969), pp. 3543.
B. T. Polyak, Minimization of unsmooth functionals, USSR
Computational Mathematics and Mathematical Physics, 9 (1969),
pp. 1429.
M. J. D. Powell, A fast algorithm for nonlinearly constrained
optimization calculations, in Numerical Analysis, Proceedings of
the Seventh Biennial Conference held at the University of Dundee,
Dundee, June 28July 1, 1977, G. A. Watson, ed., vol. 630 of Lecture Notes in Mathematics, Berlin, 1978, Springer-Verlag, pp. 144
157.
B. N. Pshenichnyj and Yu. M. Danilin, Numerical Methods in
Extremal Problems, MIR Publishers, Moscow, 1978.

H. Rademacher, Uber
partielle und totale Differenzierbarkeit von

References
Funktionen mehrerer Variabeln under u
ber die Transformation der
Doppelintegrale, Mathematische Annalen, 79 (1919), pp. 340359.
[Rar98]

R. L. Rardin, Optimization in Operations Research, Prentice

Hall, Englewood Cliffs, NJ, 1998.

[Roc70]

R. T. Rockafellar, Convex Analysis, Princeton University

Press, Princeton, NJ, 1970. Reprinted in the series Princeton
Landmarks in Mathematics by Princeton University Press, Princeton, NJ, 1997.

[RoW97]

R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, vol. 317 of Grundlehren der mathematischen Wissenschaften,
Springer-Verlag, Berlin, 1997.

[Sau72]

M. A. Saunders, Large-scale linear programming using the

Cholesky factorization, technical report Stan-cs-72-252, Computer
Sciences Department, Stanford University, Stanford, CA, 1972.

[Sch86]

A. Schrijver, Theory of Linear and Integer Programming, Wiley,

Chichester, 1986.

[Sch03]

, Combinatorial optimization, vol. 24 of Algorithms and

Combinatorics, Springer-Verlag, Berlin, 2003.

[Sha70]

D. F. Shanno, Conditioning of quasi-Newton methods for function minimization, Mathematics of Computation, 24 (1970),
pp. 647656.

[She85]

Y. Sheffi, Urban Transportation Networks: Equilibrium Analysis

with Mathematical Programming Methods, Prentice-Hall, Englewood Cliffs, NJ, 1985.

[She76]

M. A. Shepilov, Method of the generalized gradient for finding the

absolute minimum of a convex function, Cybernetics, 12 (1976),
pp. 547553.

[Sho77]

N. Z. Shor, Cut-off method with space extension in convex programming problems, Cybernetics, 13 (1977), pp. 9496.

[Sho85]

, Minimization Methods for Non-Differentiable Functions,

Springer-Verlag, Berlin, 1985. Translated from the Russian by
K. C. Kiwiel and A. Ruszczy
nski.

[StW70]

J. Stoer and C. Witzgall, Convexity and Optimization in Finite Dimensions I, Springer-Verlag, Berlin, 1970.

[Tah03]

H. A. Taha, Operations Research: An Introduction, Prentice Hall,

Englewood Cliffs, NJ, seventh ed., 2003.

[UUV04]

M. Ulbrich, S. Ulbrich, and L. N. Vicente, A globally convergent primal-dual interior-point filter method for nonlinear programming, Mathematical Programming, 100 (2004), pp. 379410.

[Van01]

R. J. Vanderbei, Linear Programming. Foundations and Extensions, vol. 37 of International Series in Operations Research &
Management Science, Kluwer Academic Publishers, Boston, MA,
second ed., 2001.

383

References
[vHo77]

[vNe28]
[vNe47]
[vNM43]

[Wag75]

[War52]

[Wil99]
[Wil63]

[Wol69]
[Wol75]

[Wol98]

[YuN77]

[Zan69]

384

B. von Hohenbalken, Simplicial decomposition in nonlinear

programming algorithms, Mathematical Programming, 13 (1977),
pp. 4968.
J. von Neumann, Zur Theorie der Gesellschaftsspiele, Mathematische Annalen, 100 (1928), pp. 295320.
, On a maximization problem, unpublished manuscript, Institute for Advanced Study, Princeton, NJ, 1947.
J. von Neumann and O. Morgenstern, Theory of Games and
Economic Behavior, Princeton University Press, Princeton, NJ,
1943.
H. M. Wagner, Principles of Operations Research: With Applications to Managerial Decisions, Prentice Hall, Englewood Cliffs,
NJ, second ed., 1975.
J. G. Wardrop, Some theoretical aspects of road traffic research,
Proceedings of the Institute of Civil Engineers, Part II, (1952),
pp. 325378.
H. P. Williams, Model Building in Mathematical Programming,
John Wiley & Sons, Chichester, UK, fourth ed., 1999.
R. B. Wilson, A simplicial algorithm for concave programming,
PhD thesis, Graduate School of Business Administration, Harvard
University, Cambridge, MA, 1963.
P. Wolfe, Convergence conditions for ascent methods, SIAM Review, 11 (1969), pp. 226235.
, A method of conjugate subgradients for minimizing nondifferentiable functions, Mathematical Programming Study, 3 (1975),
pp. 145173.
L. A. Wolsey, Integer Programming, Wiley-Interscience Series
in Discrete Mathematics and Optimization, John Wiley & Sons,
New York, NY, 1998.
D. B. Yudin and A. S. Nemirovskii, Informational complexity
and efficient methods for the solution of convex extremal problems,
Matekon, 13 (1977), pp. 2545.
W. I. Zangwill, Nonlinear Programming: A Unified Approach,
Prentice Hall, Englewood Cliffs, NJ, 1969.

Index

Abadies CQ, 125

active constraint (I(x)), 89
adjacent extreme point, 220
affine combination, 43
affine function, 11, 59
affine hull, 43
affine independence, 34
affine subspace, 34
affine transformation, 300
algebraic characterization of adjacency, 221
approximate line search, 276
Armijo step, 277, 298, 312
artificial variables, 233
augmented Lagrangian function, 350
augmented Lagrangian method, 350
automatic differentiation, 297
Banachs Theorem, 101
barrier function, 330
barrier problem, 331
basic feasible solution, 215
basic solution, 215
basic variables, 215
basis, 35
BFGS method, 274
Blands rule, 237
boundary, 38
bounded set, 37
Brouwers Theorem, 101
bundle method, 166
calculus rules, 39
canonical form, 243
Caratheodorys Theorem, 45

Cartesian product set, 148

CauchyBunyakowskiSchwarz inequality, 34
central difference formula, 297
classification of optimization models, 11
closed mapping, 159
closed sets, 37
closure, 37
coercive function, 78
column dropping, 308
column generation, 6
combinatorial optimization, 185
complementarity, 148
Complementary Slackness Theorem,
250
composite function, 60, 104
composite operator, 104
concave function, 58
cone, 50
cone of feasible directions, 115
conjugate direction, 286, 294
conjugate gradient, 289
conjugate gradient method, 289
constrained optimization, 12, 88
96, 303350
constraint qualification (CQ), 17,
124, 125, 130, 131
constraints, 4
continuity, 96
continuous function, 38
continuous optimization, 12
continuous relaxation, 185
continuously differentiable function,
39

Index
contractive operator, 101
convergence rate, 296
geometric, 101, 170
linear, 296
quadratic, 296
superlinear, 296
convex analysis, 4172
convex combination, 43
convex function, 57, 96, 159
convex hull, 43
convex programming, 12
convex set, 41
coordinates, 35
CQ, 124
Danskins Theorem, 160
DantzigWolfe algorithm, 158
decision science, 10
decision variable, 6
degenerate basic solution, 215
descent direction, 85, 165
descent lemma, 322
DFP method, 293
Diet problem, 10
differentiability, 163
differentiable function, 38
differentiable optimization, 12
Dijkstras algorithm, 320
diode, 182
direction of unboundedness, 226
directional derivative, 38, 86, 159
distance function, 67
divergent series step length rule, 165,
307
domination, 346
dual feasible basis, 254
dual infeasible basis, 254
dual linear program, 242
dual simplex algorithm, 255
dual simplex method, 254
duality gap, 145
effective domain, 96, 144
efficient frontier, 346
eigenvalue, 36
eigenvector, 36
Ekelands variational principle, 110

386

electrical circuit, 181

electrical network, 181
eligible entering variable, 238
eligible leaving variable, 238
epigraph, 60, 80
-optimal solution, 96
equality constraint, 11
equivalent systems, 224
Euclidean projection, 66
Everetts Theorem, 176
exact penalty function, 340
existence of optimal solution, 219
exterior penalty method, 326330
extreme direction, 218
extreme point, 46
Farkas Lemma, 57, 249
feasibility heuristic, 188
feasible direction, 88, 89
feasible solution, 5, 13
feasible-direction methods, 303
filter, 346
filter-SQP methods, 346
finite termination, 281
finitely generated cone, 55
fixed point, 100
FletcherReeves formula, 292
forward difference formula, 297
FrankWolfe algorithm, 305
FrankWolfe Theorem, 82
Fritz John conditions, 121
GaussSeidel method, 105
geometric convergence rate, 101, 170
global minimum, 76
global optimality conditions, 133,
147
global optimum, 76
necessary and sufficient conditions, 88, 91
sufficient conditions, 133
Golden section, 277
gradient, 38
gradient projection algorithm, 311
gradient related, 271
gradient related method, 279, 280
GramSchmidt procedure, 289

Index
hard constraint, 18
Hessian matrix, 39
I(x), 89
identity matrix I n , 36
ill-conditioning, 346
implicit function, 13, 40, 296
Implicit Function Theorem, 164
indicator function (S ), 167, 325
inequality constraint, 11
infimum, 14
infinite-dimensional optimization, 13
integer programming, 12, 13
integrable function, 318
integrality property, 13
interior, 38
interior penalty function, 107
interior point algorithm, 238, 330
337
interpolation, 277
iso-cost line, 268
iso-curve, 268
Jacobi method, 105
Jacobian, 39, 300, 318, 339
Karmarkars algorithm, 238
KarushKuhnTucker (KKT) conditions, 125, 133
Kirchhoffs laws, 182
Lagrange function, 142
Lagrange multiplier method, 158,
174
Lagrange multiplier vector, 143, 179
Lagrange multipliers, 122
Lagrangian dual function, 143
Lagrangian dual problem, 143
Lagrangian duality, 141194
Lagrangian relaxation, 18, 142, 143,
185
least-squares data fitting, 267
level curve, 268
level set (levg (b)), 65, 66, 78, 80,
151, 167, 279, 281, 313
LevenbergMarquardt, 273, 301
LICQ, 132

limit, 37
limit points, 37
line search, 275
approximate, 276
Armijo step length rule, 277,
298, 312
Golden section, 277
interpolation, 277
Newtons method, 277
linear convergence rate, 296
linear function, 40
linear independence, 34
linear programming, 11, 13, 154,
197264, 336337
linear programming duality, 241
264
linear space, 34
linear-fractional programming, 223
Lipschitz continuity, 280
local convergence, 339
local minimum, 76
local optimum, 76
necessary conditions, 85, 86,
90, 117, 121, 125, 130
sufficient conditions, 87
logarithmic barrier, 331
logical constraint, 5
lower semi-continuity, 79
Maratos effect, 344
mathematical model, 4
mathematical programming, 9
matrix, 35
matrix game, 105
matrix inverse, 36
matrix norm, 35
matrix product, 35
matrix transpose, 35
max function, 160
mean-value theorem, 39
merit function, 341
method of successive averages (MSA),
321
MFCQ, 131
minimax theorem, 105
minimum, 14

387

Index
minimum distance (distS ), 167
multi-objective optimization, 13, 346
near-optimality, 95
negative curvature, 270
neighbourhood, 38
Newtons method, 272, 277, 299
NewtonRaphson method, 105, 272
Nobel laureates, 10
non-basic variables, 215
non-convex programming, 12
non-coordinability, 176
non-differentiable function, 283
non-differentiable optimization, 12
non-expansive operator, 99
nonlinear programming, 11, 13
nonsingular matrix, 36
norm, 34
normal cone (NX ), 94, 126
NP-hard problem, 77, 186
objective function, 4
Ohms law, 183
open ball, 37
open set, 37
operations research, 9
optimal BFS, 226
optimal solution, 5
optimal value, 5
optimality, 9
optimality conditions, 8488, 90
94, 111140, 147149, 175,
227, 228, 252253
optimization under uncertainty, 13
optimize, 3
orthogonality, 34, 148
orthonormal basis, 35
parametric optimization, 137
Pareto set, 346
partial pricing, 230
pattern search methods, 298
penalty, 19
penalty function, 19
penalty parameter, 326
perturbation function (p(u)), 178
Phase I, 315

388

phase I problem, 233

phase II problem, 233
physical constraint, 5
piece-wise linear function, 283
PolakRibi`ere formula, 292
Polyak step, 165
polyhedral cone, 50
polyhedron, 47
polytope, 45
positive (semi)definite matrix, 37
potential, 181
potential difference, 181
pre-conditioning, 292
primal infeasibility criterion, 254
primal simplex method, 254
projection, 66
projection arc, 312
projection operator, 66, 91, 99
projection problem, 315
proof by contradiction, 33
proper function, 15, 167
proximal point algorithm, 301
pseudo-convex function, 109

Q-orthogonal, 286
quadratic convergence rate, 296
quadratic function, 40, 64, 77
quadratic programming, 77, 156,
315
quasi-convex function, 109
quasi-Newton methods, 273, 293,
342
Rademachers Theorem, 283
rank-two update, 293
recession cone, 81, 82
reduced cost, 227
redundant constraint, 107
relaxation, 19, 141, 142, 185, 328
Relaxation Theorem, 141
Representation Theorem, 50, 218,
308
resistor, 182
restricted master problem, 308
restricted simplicial decomposition,
309
restrification, 350

Index
revised simplex method, 238
saddle point, 105, 147, 270
scalar product, 34
secant method, 274
sensitivity analysis, 137, 178, 179
sensitivity analysis for LP, 257
separation of convex sets, 98
Separation Theorem, 52, 98, 163
sequential linear programming (SLP),
349
sequential quadratic programming
(SQP), 337346
set covering problem, 18
shadow price, 249
shortest route, 320
simplex method, 10, 225240
simplicial decomposition algorithm,
308
slack variable, 7
Slater CQ, 131
SLP algorithm, 349
soft constraint, 18, 177
spectral theorem, 136
SQP algorithm, 337350
square matrix, 36
stalling, 239
standard basis, 35
stationary point, 16, 85, 91
steepest descent, 269
steepest-edge rule, 230
stochastic programming, 13
strict inequality, 81
strict local minimum, 76
strictly convex function, 58
strictly quasi-convex function, 277
strong duality, 149
Strong Duality Theorem, 149, 152
154, 156, 248
subdifferentiability, 162
subdifferential, 158
subgradient, 158, 284
subgradient optimization, 166
subgradient projection method, 165
superlinear convergence rate, 296
symmetric matrix, 36

tangent cone, 115

traffic assignment problem, 318
traffic equilibrium, 317
triangle inequality, 36
trust region methods, 284
twice differentiable function, 38
unconstrained optimization, 12, 84
88, 267302
unimodal function, 277
unique optimum, 83
upper level set, 151
upper semi-continuity, 79
user equilibrium, 317
variable, 4
variational inequality, 90, 104
vector, 34
vector-valued functions, 39
voltage source, 182
von Neumanns Minimax Theorem,
105
Wardrops principle, 317
Weak Duality Theorem, 144, 247
weak Wolfe condition, 278
weakly coercive function, 78
Weierstrass Theorem, 80, 98, 162
Wolfe condition, 278

389

Errata and comments list for

An Introduction to Continuous Optimization
Michael Patriksson
17 August, 2005
Page
76
94
98
165
165
165
171
175
276
367

Row

Reads

Should read

2
13
11
17
20
21
Figure 6.4
14
Figure 11.2(b)
Exercise 10.5

has a lower value

than the other
simplicity
means fast
k = + /(k + 1)
where > 0, 0
A convex min-function
k m + 1 such that
xk + pk
y 0m

has a lower function value

than any of the other
the readers convenience
means that fast
k = /(k + 1)
where > 0
A concave min-function
k m + 1, such that

y 0m