Springer Optimization and Its Applications 195

Neculai Andrei

Modern Numerical
Nonlinear
Optimization
Springer Optimization and Its
Applications

Volume 195

Series Editors
Panos M. Pardalos , University of Florida
My T. Thai , University of Florida

Honorary Editor
Ding-Zhu Du, University of Texas at Dallas

Advisory Editors
Roman V. Belavkin, Middlesex University
John R. Birge, University of Chicago
Sergiy Butenko, Texas A&M University
Vipin Kumar, University of Minnesota
Anna Nagurney, University of Massachusetts Amherst
Jun Pei, Hefei University of Technology
Oleg Prokopyev, University of Pittsburgh
Steffen Rebennack, Karlsruhe Institute of Technology
Mauricio Resende, Amazon (United States)
Tamás Terlaky, Lehigh University
Van Vu, Yale University
Michael N. Vrahatis, University of Patras
Guoliang Xue, Arizona State University
Yinyu Ye, Stanford University
Aims and Scope
Optimization has continued to expand in all directions at an astonishing rate.
New algorithmic and theoretical techniques are continually developing and
the diffusion into other disciplines is proceeding at a rapid pace, with a spot
light on machine learning, artificial intelligence, and quantum computing.
Our knowledge of all aspects of the field has grown even more profound. At
the same time, one of the most striking trends in optimization is the con-
stantly increasing emphasis on the interdisciplinary nature of the field.
Optimization has been a basic tool in areas not limited to applied mathe-
matics, engineering, medicine, economics, computer science, operations
research, and other sciences.
The series Springer Optimization and Its Applications (SOIA) aims to
publish state-of-the-art expository works (monographs, contributed volumes,
textbooks, handbooks) that focus on theory, methods, and applications of
optimization. Topics covered include, but are not limited to, nonlinear
optimization, combinatorial optimization, continuous optimization, stochas-
tic optimization, Bayesian optimization, optimal control, discrete optimiza-
tion, multi-objective optimization, and more. New to the series portfolio
include Works at the intersection of optimization and machine learning,
artificial intelligence, and quantum computing.
Volumes from this series are indexed by Web of Science, zbMATH,
Mathematical Reviews, and SCOPUS.
Neculai Andrei

Modern Numerical
Nonlinear Optimization
Neculai Andrei
Academy of Romanian Scientists
Center for Advanced
Modeling and Optimization
Bucharest, Romania

ISSN 1931-6828 ISSN 1931-6836 (electronic)

Springer Optimization and Its Applications
ISBN 978-3-031-08719-6 ISBN 978-3-031-08720-2 (eBook)
https://doi.org/10.1007/978-3-031-08720-2

Mathematics Subject Classification (2010): 90-01, 90-08, 65K05

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Switzerland AG 2022
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher,
whether the whole or part of the material is concerned, specifically the rights of translation,
reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any
other physical way, and transmission or information storage and retrieval, electronic adaptation,
computer software, or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in
this book are believed to be true and accurate at the date of publication. Neither the publisher nor
the authors or the editors give a warranty, expressed or implied, with respect to the material
contained herein or for any errors or omissions that may have been made. The publisher remains
neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface

This book is on nonlinear optimization. Although nowadays excellent books

with great mathematical rigor are available, none of them insists on providing
the performances of nonlinear optimization algorithms for solving real large-
scale applications for a broader audience, ranging from college students to
scientists and professional industry bodies.
The purpose of this book is to present the theoretical details and the
computational performances of the modern optimization algorithms for
solving continuous nonlinear optimization applications from different areas
of activity. The idea is to give a comprehensive description of the theoretical
aspects and details on the computational numerical linear algebra used in
nonlinear optimization algorithms. What distinguishes this book from the
others in the field is a rigorous treatment of the optimization methods,
emphasizing both the convergence properties and the performances of the
algorithms which solve a large class of large-scale unconstrained and
constrained optimization problems and applications. The qualifier modern
in the title refers to those unconstrained and constrained optimization
algorithms that combine and integrate the latest optimization techniques
and advanced computational linear algebra methods.
We have considered only the local solutions of large-scale, complex, and
continuous nonlinear optimization applications. Other types of optimization
problems, such as mixed-integer programming, mixed complementarity,
equilibrium constraints, network optimization, nonlinear integer program-
ming, nonsmooth optimization, and stochastic programming, and the global
solution of nonlinear optimization applications are beyond the goal of
this work.
The book has 20 chapters and is structured in two parts. The first part
(Chaps. 2, 3, 4, 5, 6, 7, 8, and 9) covers the theoretical and practical aspects
of the unconstrained optimization. The second part (Chaps. 12, 13, 14, 15, 16,
17, 18, 19, and 20) is dedicated to detailing the constrained nonlinear
optimization. Chapters 10 and 11 present an overview of the constrained
nonlinear optimization methods and of the optimality conditions for nonlin-
ear optimization, including the duality. Special attention has been given
to the Karush-Kuhn-Tucker (KKT) optimality conditions presented in
Chap. 11, which characterize the local optimal solutions of nonlinear optimi-
zation problems. In general, a nonlinear optimization algorithm is working
under the following strategy: in a current point, a quadratic approximation of
v
vi Preface

the problem is constructed and solved, thus obtaining a new point in which
the original problem is again approximated by a quadratic, the procedure
being repeated in this way.
The unconstrained optimization methods are characterized by two
strategies: the line-search and the trust-region paradigms. The line-search
strategy computes the search direction dk in the current point xk, which must
be a descent direction, and a stepsize αk taken along the search direction to
obtain a new estimation of the minimum point, xkþ1 ¼ xk þ αkdk. On the other
hand, the trust-region methods define a region around the current iterate
within which we trust that the quadratic model is an adequate representation
of the minimizing function and then choose the step to be an approximate
minimizer of the model in this region. Therefore, the trust-region methods
choose the direction and the stepsize simultaneously. Obviously, if a step is
not acceptable, then the size of the region is reduced and a new minimizer is
computed. Both these strategies are operational in nonlinear optimization.
For the unconstrained optimization, the following methods are detailed: the
steepest descent including the relaxed steepest and the accelerated steepest;
the Newton method and its modified variants including the composite New-
ton; the conjugate gradient methods and their variants standard and hybrid;
the memory-less BFGS preconditioned; the quasi-Newton BFGS, DFP, and
SR1; the limited-memory BFGS; the inexact or truncated Newton; the trust-
region methods and the direct searching ones which use only the function
values.
The modern constrained nonlinear optimization methods considered in
this book are based on certain strategies, the most important ones being: the
penalty and the augmented Lagrangian, the sequential quadratic program-
ming based on quadratic programming, the generalized reduced gradient,
the interior-point, the filter methods, the combination of the interior point
with filters, and the direct search methods which do not use the derivative
information.
The book emphasizes and illustrates a number of reliable and robust
packages for solving large-scale unconstrained nonlinear optimization
problems (CG-DESCENT, DESCON, CGOPT, L-BFGS, and TN), as well
as constrained nonlinear optimization problems and applications (SPG,
L-BFGS-B, TNBC, SPENBAR, MINOS, SNOPT, NLPQLP, KNITRO,
CONOPT, filterSQP, and IPOPT). Details on their implementation by
presenting the advanced numerical linear algebra are a priority of our
work. These packages were tested to solve large-scale optimization problems
up to 250,000 variables.
A number of four appendices have been included to facilitate the under-
standing of the theoretical developments and of the numerical performances
of the optimization algorithms. Appendix A reviews the most important
mathematical concepts used throughout the chapters of the book, from
numerical linear algebra to calculus, topology, and convexity. In order to
see the behavior of the unconstrained and constrained optimization
algorithms, the book provides many examples and real applications
described in the next three appendices. Appendix B includes the collection
SMUNO with 16 small-scale continuous unconstrained optimization
Preface vii

applications. The collection LACOP with 18 large-scale continuous

nonlinear optimization applications with constraints is presented in
Appendix C. Appendix D is for six real unconstrained nonlinear optimization
applications from the MINPACK-2 collection of Argonne National Labora-
tory. The vast majority of algorithms have been tested for solving 800 large-
scale optimization problems from the UOP collection (described in Andrei
(2020a)) up to 10,000 variables, as well as optimization applications from
the LACOP and MINPACK-2 collections with 40,000 or 250,000 variables.
The conclusion of these intensive numerical studies is that there is no
algorithm able to solve any nonlinear unconstrained or constrained optimi-
zation problem or application. The most modern, powerful nonlinear optimi-
zation algorithms presented and tested in this book combine and integrate
different optimization methods (modified augmented Lagrangian, sequential
linear or quadratic programming, interior-point methods with filter line-
search or trust-region) and include advanced computational linear algebra
techniques.
The book is an invitation for researchers who work in the nonlinear
optimization area to understand, master, and develop new algorithms with
better properties. It will be of great interest to all those interested in develop-
ing and using new advanced techniques for solving unconstrained and
constrained optimization complex problems. Mathematical programming
researchers, theoreticians and practitioners in operations research,
practitioners in engineering, and industry researchers, as well as graduate
students in mathematics, and doctoral and master’s students in mathematical
programming, can find plenty of information and practical suggestions for
solving large-scale optimization problems and applications.
I am grateful to the Alexander von Humboldt Foundation for its apprecia-
tion and generous fellowship during the 2+ years I spent at different
universities in Germany. My thanks also go to Elizabeth Loew and to all
the staff at Springer for their encouragement and competent assistance with
the preparation of this book. Finally, my deepest thanks go to my wife,
Mihaela, for her constant understanding and support throughout the years.

Bucharest and Tohăniţă / Bran Resort, Romania Neculai Andrei

February 2022
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Mathematical Modeling: Linguistic Models Versus
Mathematical Models . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Mathematical Modeling and Computational Sciences . . . 4
1.3 The Modern Modeling Scheme for Optimization . . . . . . 5
1.4 Classification of Optimization Problems . . . . . . . . . . . . 8
1.5 Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 Collections of Applications for Numerical
Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.7 Comparison of Algorithms . . . . . . . . . . . . . . . . . . . . . . 13
1.8 The Structure of the Book . . . . . . . . . . . . . . . . . . . . . . 14
2 Fundamentals on Unconstrained Optimization. Stepsize
Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Fundamentals on the Convergence of the Line-Search
Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 The General Algorithm for Unconstrained
Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4 Convergence of the Algorithm with Exact
Line-Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5 Inexact Line-Search Methods . . . . . . . . . . . . . . . . . . . . 37
2.6 Convergence of the Algorithm with Inexact
Line-Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.7 Three Fortran Implementations of the Inexact
Line-Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
2.8 Numerical Studies: Stepsize Computation . . . . . . . . . . . 75
3 Steepest Descent Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.1 The Steepest Descent . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.2 The Relaxed Steepest Descent . . . . . . . . . . . . . . . . . . . 92
3.3 The Accelerated Steepest Descent . . . . . . . . . . . . . . . . . 98
3.4 Comments on the Acceleration Scheme . . . . . . . . . . . . 106
4 The Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.1 The Newton Method for Solving Nonlinear Algebraic
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

ix
x Contents

4.2 The Gauss-Newton Method . . . . . . . . . . . . . . . . . . . . . 123

4.3 The Newton Method for Function Minimization . . . . . . 125
4.4 The Newton Method with Line-Search . . . . . . . . . . . . . 129
4.5 Analysis of Complexity . . . . . . . . . . . . . . . . . . . . . . . . 133
4.6 The Modified Newton Method . . . . . . . . . . . . . . . . . . . 140
4.7 The Newton Method with Finite-Differences . . . . . . . . . 145
4.8 Errors in Functions, Gradients, and Hessians . . . . . . . . . 154
4.9 Negative Curvature Direction Methods . . . . . . . . . . . . . 157
4.10 The Composite Newton Method . . . . . . . . . . . . . . . . . . 163
5 Conjugate Gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . 169
5.1 The Concept of Nonlinear Conjugate Gradient . . . . . . . 169
5.2 The Linear Conjugate Gradient Method . . . . . . . . . . . . 172
5.3 General Convergence Results for Nonlinear
Conjugate Gradient Methods . . . . . . . . . . . . . . . . . . . . 188
5.4 Standard Conjugate Gradient Methods . . . . . . . . . . . . . 201
5.5 Hybrid Conjugate Gradient Methods . . . . . . . . . . . . . . . 218
5.6 Conjugate Gradient Methods as Modifications
of the Standard Schemes . . . . . . . . . . . . . . . . . . . . . . . 231
5.7 Conjugate Gradient Methods Memoryless BFGS
Preconditioned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
5.8 Solving Large-Scale Applications . . . . . . . . . . . . . . . . . 257
6 Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
6.1 DFP and BFGS Methods . . . . . . . . . . . . . . . . . . . . . . . 261
6.2 Modifications of the BFGS Method . . . . . . . . . . . . . . . 278
6.3 Quasi-Newton Methods with Diagonal Updating
of the Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
6.4 Limited-Memory Quasi-Newton Methods . . . . . . . . . . . 290
6.5 The SR1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
6.6 Sparse Quasi-Newton Updates . . . . . . . . . . . . . . . . . . . 309
6.7 Quasi-Newton Methods and Separable Functions . . . . . . 310
6.8 Solving Large-Scale Applications . . . . . . . . . . . . . . . . . 312
7 Inexact Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
7.1 The Inexact Newton Method for Nonlinear Algebraic
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
7.2 Inexact Newton Methods for Functions
Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
7.3 The Line Search Newton-CG Method . . . . . . . . . . . . . . 324
7.4 Comparison of TN Versus Conjugate Gradient
Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
7.5 Comparison of TN Versus L-BFGS . . . . . . . . . . . . . . . 327
7.6 Solving Large-Scale Applications . . . . . . . . . . . . . . . . . 329
8 The Trust-Region Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
8.1 The Trust-Region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
8.2 Algorithms Based on the Cauchy Point . . . . . . . . . . . . . 337
8.3 The Trust-Region Newton-CG Method . . . . . . . . . . . . . 340
8.4 The Global Convergence . . . . . . . . . . . . . . . . . . . . . . . 341
Contents xi

8.5 Iterative Solution of the Subproblem . . . . . . . . . . . . . . . 344

8.6 The Scaled Trust-Region . . . . . . . . . . . . . . . . . . . . . . . 351
9 Direct Methods for Unconstrained Optimization . . . . . . . . . . 355
9.1 The NELMED Algorithm . . . . . . . . . . . . . . . . . . . . . . . 355
9.2 The NEWUOA Algorithm . . . . . . . . . . . . . . . . . . . . . . 357
9.3 The DEEPS Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 359
9.4 Numerical Study: NELMED, NEWUOA,
and DEEPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
10 Constrained Nonlinear Optimization Methods:
An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
10.1 Convergence Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
10.2 Infeasible Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
10.3 Approximate Subproblem: Local Models
and Their Solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
10.4 Globalization Strategy: Convergence from Remote
Starting Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
10.5 The Refining the Local Model . . . . . . . . . . . . . . . . . . . 380
11 Optimality Conditions for Nonlinear Optimization . . . . . . . . 383
11.1 General Concepts in Nonlinear Optimization . . . . . . . . . 384
11.2 Optimality Conditions for Unconstrained
Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
11.3 Optimality Conditions for Problems with Inequality
Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
11.4 Optimality Conditions for Problems with Equality
Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
11.5 Optimality Conditions for General Nonlinear
Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . 402
11.6 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
12 Simple Bound Constrained Optimization . . . . . . . . . . . . . . . . 411
12.1 Necessary Conditions for Optimality . . . . . . . . . . . . . . . 412
12.2 Sufficient Conditions for Optimality . . . . . . . . . . . . . . . 414
12.3 Methods for Solving Simple Bound Optimization
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
12.4 The Spectral Projected Gradient Method (SPG) . . . . . . . 417
12.5 L-BFGS with Simple Bounds (L-BFGS-B) . . . . . . . . . . 421
12.6 Truncated Newton with Simple Bounds (TNBC) . . . . . . 428
12.7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
13 Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
13.1 Equality Constrained Quadratic Programming . . . . . . . . 439
13.2 Inequality Constrained Quadratic Programming . . . . . . . 449
13.3 Interior Point Methods . . . . . . . . . . . . . . . . . . . . . . . . . 463
13.4 Methods for Convex QP Problems with Equality
Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467
13.5 Quadratic Programming with Simple Bounds:
The Gradient Projection Method . . . . . . . . . . . . . . . . . . 468
13.6 Elimination of Variables . . . . . . . . . . . . . . . . . . . . . . . 471
xii Contents

14 Penalty and Augmented Lagrangian Methods . . . . . . . . . . . . 475

14.1 The Quadratic Penalty Method . . . . . . . . . . . . . . . . . . . 475
14.2 The Nonsmooth Penalty Method . . . . . . . . . . . . . . . . . . 479
14.3 The Augmented Lagrangian Method . . . . . . . . . . . . . . . 482
14.4 Criticism of the Penalty and Augmented Lagrangian
Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
14.5 A Penalty-Barrier Algorithm (SPENBAR) . . . . . . . . . . . 488
14.6 The Linearly Constrained Augmented Lagrangian
(MINOS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
15 Sequential Quadratic Programming . . . . . . . . . . . . . . . . . . . 521
15.1 A Simple Approach to SQP . . . . . . . . . . . . . . . . . . . . . 526
15.2 Reduced-Hessian Quasi-Newton Approximations . . . . . 531
15.3 Merit Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
15.4 Second-Order Correction (Maratos Effect) . . . . . . . . . . 534
15.5 The Line-Search SQP Algorithm . . . . . . . . . . . . . . . . . 537
15.6 The Trust-Region SQP Algorithm . . . . . . . . . . . . . . . . . 538
15.7 Sequential Linear-Quadratic Programming (SLQP) . . . . 541
15.8 A SQP Algorithm for Large-Scale-Constrained
Optimization (SNOPT) . . . . . . . . . . . . . . . . . . . . . . . . 542
15.9 A SQP Algorithm with Successive Error Restoration
(NLPQLP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
15.10 Active-Set Sequential Linear-Quadratic Programming
(KNITRO/ACTIVE) . . . . . . . . . . . . . . . . . . . . . . . . . . 557
16 Primal Methods: The Generalized Reduced Gradient
with Sequential Linearization . . . . . . . . . . . . . . . . . . . . . . . . 569
16.1 Feasible Direction Methods . . . . . . . . . . . . . . . . . . . . . 569
16.2 Active Set Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 571
16.3 The Gradient Projection Method . . . . . . . . . . . . . . . . . . 573
16.4 The Reduced Gradient Method . . . . . . . . . . . . . . . . . . . 576
16.5 The Convex Simplex Method . . . . . . . . . . . . . . . . . . . . 578
16.6 The Generalized Reduced Gradient Method (GRG) . . . . 579
16.7 GRG with Sequential Linear or Sequential Quadratic
Programming (CONOPT) . . . . . . . . . . . . . . . . . . . . . . . 587
17 Interior-Point Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
17.1 Prototype of the Interior-Point Algorithm . . . . . . . . . . . 601
17.2 Aspects of the Algorithmic Developments . . . . . . . . . . . 603
17.3 Line-Search Interior-Point Algorithm . . . . . . . . . . . . . . 610
17.4 A Variant of the Line-Search Interior-Point
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611
17.5 Trust-Region Interior-Point Algorithm . . . . . . . . . . . . . 627
17.6 Interior-Point Sequential Linear-Quadratic
Programming (KNITRO/INTERIOR) . . . . . . . . . . . . . . 631
18 Filter Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647
18.1 Sequential Linear Programming Filter Algorithm . . . . . 649
18.2 Sequential Quadratic Programming Filter
Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
Contents xiii

19 Interior-Point Filter Line-Search . . . . . . . . . . . . . . . . . . . . . . 661

19.1 Basic Algorithm IPOPT . . . . . . . . . . . . . . . . . . . . . . . . 662
19.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . 669
20 Direct Methods for Constrained Optimization . . . . . . . . . . . . 679
20.1 COBYLA Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 680
20.2 DFL Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
Appendix A: Mathematical Review . . . . . . . . . . . . . . . . . . . . . . . . 691
Appendix B: The SMUNO Collection . . . . . . . . . . . . . . . . . . . . . . 721
Appendix C: The LACOP Collection . . . . . . . . . . . . . . . . . . . . . . 727
Appendix D: The MINPACK-2 Collection . . . . . . . . . . . . . . . . . . 757
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 767
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793
Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 801
List of Algorithms

Algorithm 2.1 General unconstrained optimization algorithm . . . 30

Algorithm 2.2 Hager and Zhang line-search algorithm . . . . . . . . 45
Algorithm 2.3 The update procedure . . . . . . . . . . . . . . . . . . . . 45
Algorithm 2.4 Procedure secant2 . . . . . . . . . . . . . . . . . . . . . . . 45
Algorithm 2.5 Backtracking . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Algorithm 2.6 Backtracking – variant . . . . . . . . . . . . . . . . . . . 47
Algorithm 2.7 Zhang and Hager nonmonotone line-search . . . . . 49
Algorithm 2.8 Nonmonotone line-search Huang, Wan
and Chen . . . . . . . . . . . . . . . . . . . . . . . . . . ... 50
Algorithm 2.9 Ou and Liu nonmonotone line-search . . . . . . ... 51
Algorithm 2.10 Weak Wolfe line-search with simple bisection ... 52
Algorithm 3.1 Steepest descent . . . . . . . . . . . . . . . . . . . . . . . . 82
Algorithm 3.2 Accelerated steepest descent (with Wolfe line
search) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
Algorithm 4.1 Newton for F(x) ¼ 0 . . . . . . . . . . . . . . . . .... 110
Algorithm 4.2 Newton for minf(x) . . . . . . . . . . . . . . . . . .... 126
Algorithm 4.3 Newton with line-search for minf(x) . . . . . .... 130
Algorithm 4.4 Modified Cholesky factorization – Gill
and Murray . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Algorithm 4.5 Newton with finite-difference for F(x) ¼ 0 . . . . . 149
Algorithm 4.6 Negative curvature direction algorithm . . . . . . . 158
Algorithm 4.7 Stable Newton algorithm – Gill and Murray . . . . 159
Algorithm 4.8 Composite Newton of order m . . . . . . . . . . . . . 164
Algorithm 4.9 Composite Newton of order 1 . . . . . . . . . . . . . 164
Algorithm 5.1 Line search . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Algorithm 5.2 Linear Conjugate Gradient . . . . . . . . . . . . . . . . 177
Algorithm 5.3 Preconditioned conjugate gradient . . . . . . . . . . 186
Algorithm 5.4 General nonlinear conjugate gradient . . . . . . . . 202
Algorithm 5.5 General hybrid conjugate gradient algorithm
by using the convex combination of standard
schemes . . . . . . . . . . . . . . . . . . . . . . . . . ... 226
Algorithm 5.6 Guaranteed descent and conjugacy conditions
with a modified Wolfe line-search:
DESCON/DESCONa . . . . . . . . . . . . . . . . ... 240

xv
xvi List of Algorithms

Algorithm 5.7 Conjugate gradient memoryless BFGS

preconditioned: CONMIN . . . . . . . . . . . . . . . . 249
Algorithm 6.1 DFP and BFGS methods . . . . . . . . . . . . . . . . . 265
Algorithm 6.2 L-BFGS two-loop recursion . . . . . . . . . . . . . . . 292
Algorithm 6.3 L-BFGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Algorithm 6.4 Accelerated MM-SR1gen and MM-BFGS . . . . . 304
Algorithm 7.1 Line search Newton-CG (truncated Newton) . . . 325
Algorithm 8.1 Trust-region . . . . . . . . . . . . . . . . . . . . . . . . . . 333
Algorithm 8.2 Cauchy point computation. . . . . . . . . . . . . . . . 337
Algorithm 8.3 Newton conjugate gradient (Steihaug) . . . . . . . . 341
Algorithm 8.4 Trust-region subproblem . . . . . . . . . . . . . . . . . 346
Algorithm 8.5 Generalized Cauchy point . . . . . . . . . . . . . . . . 352
Algorithm 9.1 Nelder-Mead algorithm (NELMED) (one
iteration) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Algorithm 9.2 Powell algorithm (NEWUOA) . . . . . . . . . . . . . 358
Algorithm 9.3 Andrei algorithm (DEEPS) . . . . . . . . . . . . . . . 361
Algorithm 10.1 Prototype for the nonlinear optimization
algorithm . . . . . . . . . . . . . . . . . . . . . ...... 371
Algorithm 10.2 Prototype of the interior point algorithm ...... 376
Algorithm 10.3 Prototype of the line-search method
for the nonlinear optimization . . . . . . . ...... 380
Algorithm 10.4 Prototype of the trust-region method
for the nonlinear optimization . . . . . . . ...... 381
Algorithm 12.1 Gradient projected for simple bounds . . . . . . . . 417
Algorithm 12.2 Spectral projected gradient—SPG . . . . . . . . . . . 418
Algorithm 12.3 Computation of the generalized Cauchy point . . . 426
Algorithm 12.4 L-BFGS-B . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
Algorithm 13.1 Reduced gradient for linear equality
constraints . . . . . . . . . . . . . . . . . . . . . . . . . .. 445
Algorithm 13.2 Preconditioned conjugate gradient for reduced
systems . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 447
Algorithm 13.3 Projected conjugate gradient . . . . . . . . . . . . .. 448
Algorithm 13.4 Active-set method for convex quadratic
programming . . . . . . . . . . . . . . . . . . . . . . . .. 452
Algorithm 13.5 Active-set method with positive definite
Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 454
Algorithm 13.6 Reduced gradient for linear inequality
constraints . . . . . . . . . . . . . . . . . . . . . . . . . .. 454
Algorithm 13.7 Reduced gradient for simple bounds
constraints . . . . . . . . . . . . . . . . . . . . . . . . . .. 455
Algorithm 13.8 Dual algorithm for quadratic programming . . .. 457
Algorithm 13.9 Predictor-Corrector algorithm for quadratic
programming . . . . . . . . . . . . . . . . . . . . . . . .. 466
Algorithm 13.10 Gradient projection for quadratic programming
with box constraints . . . . . . . . . . . . . . . . . . .. 471
List of Algorithms xvii

Algorithm 14.1 Quadratic penalty method . . . . . . . . . . . ..... 476

Algorithm 14.2 l1 penalty method . . . . . . . . . . . . . . . . . ..... 481
Algorithm 14.3 Augmented Lagrangian method: Equality
constraints . . . . . . . . . . . . . . . . . . . . . . ..... 483
Algorithm 14.4 General barrier . . . . . . . . . . . . . . . . . . . ..... 489
Algorithm 14.5 Penalty-Barrier—SPENBAR—Andrei . . . ..... 495
Algorithm 14.6 MINOS–linear constraints–Murtagh
and Saunders . . . . . . . . . . . . . . . . . . . . ..... 508
Algorithm 14.7 MINOS–nonlinear constraints–Murtagh
and Saunders . . . . . . . . . . . . . . . . . . . . ..... 514
Algorithm 15.1 Sequential quadratic programming—equality
constraints . . . . . . . . . . . . . . . . . . . . . . . . . .. 523
Algorithm 15.2 Sequential quadratic programming—inequality
constraints . . . . . . . . . . . . . . . . . . . . . . . . . .. 525
Algorithm 15.3 Sequential quadratic programming—inequality
constraints. Variant . . . . . . . . . . . . . . . . . . . . . 528
Algorithm 15.4 Line-search SQP algorithm . . . . . . . . . . . . . . . 537
Algorithm 15.5 Trust-region SQP algorithm . . . . . . . . . . . . . . . 539
Algorithm 15.6 SNOPT—Gill, Murray and Saunders . . . . . . . . . 551
Algorithm 15.7 Linear search in NLPQLP . . . . . . . . . . . . . . . . 555
Algorithm 15.8 NLPQLP—Schittkowski . . . . . . . . . . . . . . . . . 556
Algorithm 15.9 KNITRO/ACTIVE—Byrd, Gould,
Nocedal, and Waltz . . . . . . . . . . . . . . . . . . .. 560
Algorithm 15.10 Penalty update algorithm . . . . . . . . . . . . . . . .. 561
Algorithm 15.11 Preconditioned projected conjugate
gradient algorithm . . . . . . . . . . . . . . . . . . . .. 562
Algorithm 16.1 Generalized reduced gradient . . . . . . . . . . . . . . 586
Algorithm 16.2 CONOPT—Drud . . . . . . . . . . . . . . . . . . . . . . 589
Algorithm 17.1 Prototype of the interior-point algorithm . . . . . . 602
Algorithm 17.2 Inertia correction and regularization . . . . . . . . . 606
Algorithm 17.3 Line-search interior-point algorithm . . . . . . . . . 610
Algorithm 17.4 Primal-dual interior-point algorithm—PDIP . . . . 620
Algorithm 17.5 Trust-region algorithm for barrier problems . . . . 628
Algorithm 17.6 Trust-region interior-point algorithm . . . . . . . . . 630
Algorithm 17.7 KNITRO/INTERIOR-DIRECT—Byrd, Hribar,
and Waltz . . . . . . . . . . . . . . . . . . . . . . . . . .. 634
Algorithm 17.8 KNITRO/INTERIOR-CG—Byrd, Hribar,
and Waltz . . . . . . . . . . . . . . . . . . . . . . . . . .. 636
Algorithm 17.9 KNITRO crossover algorithm . . . . . . . . . . . .. 639
Algorithm 18.1 Sequential linear programming filter—
filterSD . . . . . . . . . . . . . . . . . . . . . . . . ..... 650
Algorithm 18.2 Basic sequential quadratic programming
filter—filterSQP . . . . . . . . . . . . . . . . . . ..... 654
Algorithm 18.3 Feasibility restoration algorithm . . . . . . . ..... 657
Algorithm 18.4 Filter sequential quadratic programming—
filterSQP . . . . . . . . . . . . . . . . . . . . . . . ..... 658
xviii List of Algorithms

Algorithm 19.1 Line-search filter barrier algorithm—Wächter

and Biegler . . . . . . . . . . . . . . . . . . . . . . . . . . 668
Algorithm 19.2 Inertia correction algorithm . . . . . . . . . . . . . . . 671
Algorithm 19.3 KKT error reduction algorithm . . . . . . . . . . . . . 675
Algorithm 20.1 DFL algorithm . . . . . . . . . . . . . . . . . . . . . . . . 687
Algorithm 20.2 Procedure PEðα, b
α, y, p, γ, αÞ . . . . . . . . . . . . . . 687
List of Applications

Application N1 Solid fuel ignition. Bratu’s problem . . . . . . . . . 120

Application N2 Flow in a drive cavity . . . . . . . . . . . . . . . . . . . 121
Application Q1 Scheduling of three generators to meet the demand
for power over a given period of time . . . . . . . . 462
Application P1 ETA-MACRO simulates a market economy through
a dynamic nonlinear optimization process . . . . . 557
Application S1 Weber (1) Locate a central facility . . . . . . . . . . 721
Application S2 Weber (2) Locate a central facility . . . . . . . . . . 721
Application S3 Weber (3) Locate a central facility . . . . . . . . . . 721
Application S4 Analysis of enzymes reaction . . . . . . . . . . . . . . 721
Application S5 Stationary solution of a chemical reactor . . . . . . 722
Application S6 Robot kinematics problem . . . . . . . . . . . . . . . . 722
Application S7 Solar Spectroscopy . . . . . . . . . . . . . . . . . . . . . 722
Application S8 Estimation of parameters . . . . . . . . . . . . . . . . . 723
Application S9 Propan combustion in air – reduced variant . . . . 723
Application S10 Gear train of minimum inertia . . . . . . . . . . . . . 724
Application S11 Human Heart Dipole . . . . . . . . . . . . . . . . . . . . 724
Application S12 Neurophysiology . . . . . . . . . . . . . . . . . . . . . . 724
Application S13 Combustion application . . . . . . . . . . . . . . . . . . 724
Application S14 Circuit design. . . . . . . . . . . . . . . . . . . . . . . . . 725
Application S15 Thermistor . . . . . . . . . . . . . . . . . . . . . . . . . . . 725
Application S16 Optimal design of a gear train . . . . . . . . . . . . . 726
Application L1 Chemical equilibrium (ELCH) . . . . . . . . . . . . . 727
Application L2 Optimization of an alkilation process (ALKI) . . . 727
Application L3 Optimal design of a reactor as a geometric
programming problem (PREC) . . . . . . . . . . . . . 728
Application L4 Cost minimization of a transformer design
(TRAFO) . . . . . . . . . . . . . . . . . . . . . . . . . . . 729
Application L5 Optimization of a multi-spindle automatic lathe
(LATHE) . . . . . . . . . . . . . . . . . . . . . . . . . . . 730
Application L6 Static power scheduling (PPSE) . . . . . . . . . . . . 731
Application L7 Optimization of a separation process in a membrane
with three stages (MSP3) . . . . . . . . . . . . . . . . . 731
Application L8 Optimization of a separation process in a membrane
with five stages (MSP5) . . . . . . . . . . . . . . . . . . 732

xix
xx List of Applications

Application L9 Blending/pooling with five feeds and two

products (POOL) . . . . . . . . . . . . . . . . . . . . .. 734
Application L10 Distribution of electrons on a sphere (DES) . . .. 735
Application L11 Hanging chain (HANG) . . . . . . . . . . . . . . . . .. 736
Application L12 Determine the optimal mixing policy of two
catalysts along the length of a tubular plug
flow reactor involving several reactions
(CAT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 737
Application L13 Optimal control of a continuous stirred-tank
chemical reactor (CSTC) . . . . . . . . . . . . . . . .. 738
Application L14 Optimal temperature field in a rectangular area
(DIFF) . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 740
Application L15 Stationary flow of an incompressible fluid
in a rectangular area (FLOW/FLOWO) . . . . . .. 744
Application L16 Fed-batch fermenter for penicillin production
(PENICI) . . . . . . . . . . . . . . . . . . . . . . . . . . .. 749
Application L17 A standard linear lumped parameter system
(CONT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754
Application L18 Van der Pol oscillator (POL) . . . . . . . . . . . . . . 755
Application A1 Elastic-Plastic Torsion . . . . . . . . . . . . . . . . . . . 757
Application A2 Pressure Distribution in a Journal Bearing . . . . . 759
Application A3 Optimal Design with Composite Materials . . . . . 759
Application A4 Steady-State Combustion . . . . . . . . . . . . . . . . . 761
Application A5 Minimal Surfaces with Enneper Boundary
Conditions . . . . . . . . . . . . . . . . . . . . . . . . . .. 762
Application A6 Inhomogeneous Superconductors: 1-D
Ginzburg-Landau . . . . . . . . . . . . . . . . . . . . .. 764
List of Figures

Fig. 1.1 The process of modeling and solving (optimizing) a

problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Fig. 1.2 The modern modeling scheme of nonlinear optimization
based on algebraic oriented languages . . . . . . . . . . . . . . . . . . . . . 7
Fig. 2.1 Subroutine back which implements backtracking
(Armijo) . . . . .. . . . .. . . .. . . . .. . . . .. . . . .. . . .. . . . .. . . . .. . . . .. . . .. . . 69
Fig. 2.2 Subroutine LSbis which generate stepsizes satisfying the
weak Wolfe line-search with simple bisection . . . . . . . . . . . . 71
Fig. 2.3 Subroutine wolfeLS which generate safeguarded stepsizes
satisfying the inexact weak Wolfe line-search with cubic
interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Fig. 2.4 Performances of the Hestenes-Stiefel conjugate gradient
algorithm with the inexact weak Wolfe line-search
versus the inexact strong Wolfe line-search . . . . . . . . . . . . . . . 78
Fig. 2.5 Performance profiles of the Hager-Zhang algorithm with
different line-search procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Fig. 3.1 Representation of function f1(x) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
 
Fig. 3.2 (a) Evolution of the error f 1 ðxk Þ  f 1 . (b) Evolution of
kgkk2 . . .. . .. .. . .. . .. .. . .. . .. . .. .. . .. . .. . .. .. . .. . .. . .. .. . .. . .. . .. 83
Fig. 3.3 Evolution of the error |f2(xk)  f2(x)| . n ¼ 500 . . . . . . . . . . 86
Fig. 3.4 (a) Evolution of the error |f2(xk)  f2(x)| . n ¼ 2. (b)
Zigzag evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Fig. 3.5 Steepest descent with weak Wolfe (SDW) versus steepest
descent with backtracking (SDB) . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Fig. 3.6 Evolution of the error |f(xk)  f(x)| for steepest descent
versus relaxed steepest descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Fig. 3.7 Relaxed steepest descent with backtracking (RSDB) ver-
sus steepest descent with backtracking (SDB) . . . . . . . . . . . . 98
Fig. 3.8 Accelerated steepest descent with backtracking. ρ ¼
0.0001. Error |f(xk)  f| and contribution (ak  bk)2/2bk
+ ρak .. . . .. . .. . . .. . . .. . . .. . .. . . .. . . .. . . .. . . .. . .. . . .. . . .. . . .. . .. . 104
Fig. 3.9 Accelerated steepest descent with backtracking. ρ ¼ 0.01.
Error |f(xk)  f| and contribution (ak  bk)2/2bk + ρak . . 104

xxi
xxii List of Figures

Fig. 3.10 Accelerated steepest descent with backtracking (ASDB)

versus steepest descent with backtracking (SDB) . . . .. . .. . 105
Fig. 3.11 Accelerated steepest descent with backtracking (ASDB)
versus relaxed steepest descent with backtracking
(RSDB) and versus steepest descent with backtracking
(SDB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Fig. 4.1 The number of iterations for solving the system from
Example 4.1 when the Newton method is initialized in
points from the domain [1, 3]  [1, 3] . . . . . . . . . . . . . . . . . . . . 115
Fig. 4.2 Evolution of kF(xk)k for applications S5, S6, S9, and S14 118
Fig. 4.3 Solution of Bratu’s problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Fig. 4.4 (a) Flow in a drive cavity (Re ¼ 200). (b) Flow in a drive
cavity (Re ¼ 400). (c) Flow in a drive cavity (Re ¼ 1000) 123
Fig. 4.5 Representation of function f .. . .. .. . .. .. . .. . .. .. . .. .. . .. .. . .. 126
Fig. 4.6 Number of iterations of the Newton method for different
initial points from the domain [1, 1]  [0, 2] . . . . . . . . . . . 127
Fig. 5.1 Performance of the linear conjugate gradient algorithm
for solving the linear system Ax ¼ b, where (a) A ¼ diag
(1, 2, . . ., 1000), (b) the diagonal elements of A are uni-
formly distributed in [0,1), (c) the eigenvalues of A are
distributed in 10 intervals, (d) the eigenvalues of A are
distributed in 5 intervals . . .. . . . . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . 182
Fig. 5.2 Performance of the linear conjugate gradient algorithm
for solving the linear system Ax ¼ b, where the matrix A
has a large eigenvalue separated from the others, which
are uniformly distributed in [0,1) . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Fig. 5.3 Evolution of the error kb  Axkk . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Fig. 5.4 Evolution of the error kb  Axkk of the linear conjugate
gradient algorithm for different numbers (n2) of blocks on
the main diagonal of matrix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Fig. 5.5 Performance profiles of the standard conjugate gradient
methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Fig. 5.6 Performance profiles of the standard conjugate gradient
methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Fig. 5.7 Performance profiles of seven standard conjugate gradi-
ent methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Fig. 5.8 Performance profiles of some hybrid conjugate gradient
methods based on the projection concept . . . . . . . . . . . . . . . . . . 223
Fig. 5.9 Global performance profiles of six hybrid conjugate gra-
dient methods . . .. . . .. . .. . . .. . . .. . .. . . .. . . .. . .. . . .. . . .. . .. . . .. . 224
Fig. 5.10 Performance profiles of NDLSDY versus the standard
conjugate gradient methods LS, DY, PRP, CD, FR, and
HS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Fig. 5.11 Performance profiles of CG-DESCENT versus HS, PRP,
DY, and LS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
List of Figures xxiii

Fig. 5.12 Performance profiles of CG-DESCENTaw

(CG-DESCENT with approximate Wolfe conditions)
versus HS, PRP, DY, and LS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Fig. 5.13 Performance profiles of CG-DESCENT and
CG-DESCENTaw (CG-DESCENT with approximate
Wolfe conditions) versus DL (t ¼ 1) and DLþ (t ¼ 1) . . 237
Fig. 5.14 Performance profile of DESCONa versus HS and versus
PRP . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . 243
Fig. 5.15 Performance profile of DESCONa versus DL (t ¼ 1) and
versus CG-DESCENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Fig. 5.16 Performances of DESCONa versus CG-DESCENTaw . . 244
Fig. 5.17 Performance profiles of CONMIN versus HS, PRP, DY,
and LS .. . .. . . .. . .. . .. . .. . .. . .. . . .. . .. . .. . .. . .. . .. . . .. . .. . .. . .. . 250
Fig. 5.18 Performance profiles of CONMIN versus DL (t ¼ 1),
DLþ (t ¼ 1), CG-DESCENT and DESCONa . . . . . . . . . . . . 251
Fig. 5.19 Performance profiles of DK+w versus CG-DESCENT
and DESCONa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
Fig. 5.20 Performance profiles of DKþaw versus CONMIN and
CG-DESCENTaw .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . 256
Fig. 5.21 Performance profiles of DKþiw versus DKþw and
DKþaw . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
Fig. 6.1 Performance profiles of BFGS versus CONMIN,
DESCON, CG-DESCENT, and CG-DESCENTaw . . . . . . . 268
Fig. 6.2 Performance profiles of BFGSFI versus BFGS, BFGSB,
BFGSC, BFGSD, BFGSE, and BFGSY. CPU time metric 284
Fig. 6.3 Performance profiles of YONS versus QND, WQND,
QCD, and DNRTR. CPU time metric, n ¼ 500 . . . . .. . . . . . 290
Fig. 6.4 Performance profiles of L-BFGS versus CONMIN,
DESCONa, CG-DESCENT, and CG-DESCENTaw . . . . . 293
Fig. 6.5 Solution of application A1. Elastic-plastic torsion. nx ¼
200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
Fig. 6.6 Solution of application A2. Pressure distribution in a
journal bearing. nx ¼ 200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . . 295
Fig. 6.7 Solution of application A3. Optimal design with com-
posite materials. nx ¼ 200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . 295
Fig. 6.8 Solution of application A4. Steady-state combustion. nx
¼ 200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Fig. 6.9 Solution of application A5. Minimal surfaces with
Enneper boundary conditions. nx ¼ 200, ny ¼ 200 . . . . . . 297
Fig. 6.10 Solution of application A6. Inhomogeneous
superconductors: 1-D Ginzburg-Landau. n ¼ 1000 . . . . . . 297
Fig. 6.11 Accelerated MM-SR1gen versus MM-BFGS, range
[1000, 10000] . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. 307
Fig. 6.12 Accelerated MM-SR1gen versus BFGS from CONMIN,
range [100, 1000] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
xxiv List of Figures

Fig. 7.1 Performance of TN (Nash) versus conjugate gradient

algorithms CONMIN, DESCON, CG-DESCENT, and
CG-DESCENTaw .. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . 326
Fig. 7.2 Performance profiles of L-BFGS (m ¼ 5) versus TN
(truncated Newton) based on iterations calls, function
calls, and CPU time, respectively . . . . . . . . . . . . . . . . . . . . . . . . . . 328
Fig. 12.1 SPG: Quadratic interpolation versus cubic interpolation . 420
Fig. 12.2 L-BFGS-B versus SPG with quadratic interpolation
(SPGp) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
Fig. 12.3 L-BFGS-B versus SPG with cubic interpolation (SPGc) 428
Fig. 12.4 TNBC versus L-BFGS-B (m ¼ 5) and versus SPGp . . . . . 429
Fig. 12.5 TNBC versus L-BFGS-B (m ¼ 5) and versus SPGc . . . . . 430
Fig. 12.6 Solution of the application A1 without simple bounds. nx
¼ 200, ny ¼ 200 . . . .. . . .. . . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . . .. . 431
Fig. 12.7 Solution of the application A1 with simple bounds 0  v
 0.01. nx ¼ 200, ny ¼ 200 . .. . . .. . . . .. . . . .. . . . .. . . .. . . . .. . 431
Fig. 12.8 Solution of application A2 without simple bounds.nx ¼
200, ny ¼ 200 . . . .. . . . .. . . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . 432
Fig. 12.9 Solution of application A2 with simple bound 0  v 
1. nx ¼ 200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
Fig. 12.10 Solution of application A3 without simple bounds. nx ¼
200, ny ¼ 200 . . . .. . . . .. . . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . 434
Fig. 12.11 Solution of application A3 with simple bounds 0.02  v
 0. nx ¼ 200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
Fig. 12.12 Solution of application A4 without simple bounds. nx ¼
200, ny ¼ 200 . . . .. . . . .. . . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . 435
Fig. 12.13 Solution of application A4 with simple bounds 0  v 
0.2. nx ¼ 200, ny ¼ 200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
Fig. 12.14 Solution of application A6 without simple bounds. n ¼
1000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
Fig. 12.15 Solution of application A6 with simple bounds 0  v 
0.001. n ¼ 1000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436

Computation of X1 ¼ [x1 y1]T 2 V. The point Xe 2 L

Fig. 16.1 1

corresponds to α ¼ α1, while point X corresponds to

α ¼ α . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
Fig. 16.2 The iterations for solving the system (16.50) starting from
the point Xe lead to the point Xb 2
1 2
= V . . . . . . . . . . . . . . . . . . . . . . . . 585
Fig. 16.3 The iterations for solving the system (16.50) starting from
the point Xe lead to the points Xb , Xb , and then to X1 . . . 585
1 2 3

Fig. L1 Hanging chain of minimal potential energy of length

L ¼ 4 or L ¼ 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737
Fig. L2 Evolution of u(t) given by CONOPT and MINOS . . . . . . . 738
Fig. L3 Evolution of x1(t) and x2(t) given by CONOPT . . . . . . . . . . . 738
Fig. L4 Evolution of x1(t) and x2(t) . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 739
Fig. L5 Evolution of u(t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 739
List of Tables xxv

Fig. L6 Evolution of x1(t) and x2(t) . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 740

Fig. L7 Evolution of u(t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 740
Fig. L8 Solution to the heat transportation problem with two fixed
boundary conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742
Fig. L9 Solution to the heat transportation problem with two fixed
boundary conditions and one heat source on the cell (I5,
J5) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742
Fig. L10 Solution to the heat transportation problem with two fixed
boundary conditions and two heat sources . . . . . . . . . . . . . . . . 742
Fig. L11 Solution of the transient heat transportation problem at 6
time periods .. . .. .. . .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. . .. 743
Fig. L12 Flow domain and its discretization . . . . . . . . . . . . . . . . . . . . . . . . . 746
Fig. L13 Velocity in the direction x .. . .. . .. . .. . .. . .. . .. . .. . .. .. . .. . .. . 746
Fig. L14 Three cross-sections of velocity in the direction x . . . . . . . . 747
Fig. L15 Flow domain with an obstacle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747
Fig. L16 Velocity in the direction x .. . .. . .. . .. . .. . .. . .. . .. . .. .. . .. . .. . 748
Fig. L17 Three cross-sections of velocity in the direction x . . . . . . . . 748
Fig. L18 Three cross-sections of velocity in the direction x with
inertial effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 749
Fig. L19 Velocity in the direction x for two obstacles . . . . . . . . . . . . . . 749
Fig. L20 Three cross-sections of velocity in the direction x . . . . . . . . 749
Fig. L21 Evolution of the control u(t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 752
Fig. L22 State variables y1(t) and y2(t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 752
Fig. L23 State variables y3(t) and y4(t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 752
Fig. L24 Time evolution of variables (backward Euler method) . . . 753
Fig. L25 Time evolution of the state variables . . . . . . . . . . . . . . . . . . . . . . 754
Fig. L26 Time evolution of the control u(t) . . . . . . . . . . . . . . . . . . . . . . . . . . 755
Fig. L27 Evolution of y1(t) and y2(t) . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 755
Fig. L28 Evolution of y3(t) . .. .. . .. .. . .. .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. 756
Fig. L29 Control u(t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756
Fig. A1 Solution of Application A1. nx ¼ 200, ny ¼ 200 . . . . . . . . . 759
Fig. A2 Solution of Application A2. nx ¼ 200, ny ¼ 200 . . . . . . . . 760
Fig. A3 Solution of Application A3. nx ¼ 200, ny ¼ 200 . . . . . . . . 761
Fig. A4 Solution of Application A4. nx ¼ 200, ny ¼ 200 . . . . . . . . 763
Fig. A5 Solution of Application A5. nx ¼ 200, ny ¼ 200 . . . . . . . . 764
Fig. A6 Solution of Application A6. n ¼ 1000 . . . . . . . . . . . . . . . . . . . . . 765
List of Tables

Table 1.1 Algebraic oriented modeling languages (partial list) . . . 6

Table 2.1 Performances of the inexact line-searches:
backtracking, weak Wolfe, and strong Wolfe. Steepest
descent algorithm. Function f1(x) . . . . . . . . . . . . . . . . . . . . . . . . . 76
Table 2.2 Performances of the inexact line-searches:
backtracking, weak Wolfe, and strong Wolfe. Steepest
descent algorithm. Function f2(x) . . . . . . . . . . . . . . . . . . . . . . . . . 76
Table 2.3 Performances of the inexact line-searches:
backtracking, weak Wolfe, and strong Wolfe. Steepest
descent algorithm. Function f3(x) . . . . . . . . . . . . . . . . . . . . . . . . . 76
Table 2.4 Performances of the inexact line-searches:
backtracking, weak Wolfe, and strong Wolfe. Hestenes-
Stiefel conjugate gradient algorithm. Function f1(x) . . . . 76
Table 2.5 Performances of the inexact line-searches:
backtracking, weak Wolfe, and strong Wolfe. Hestenes-
Stiefel conjugate gradient algorithm. Function f2(x) . . . . 77
Table 2.6 Performances of the inexact line-searches:
backtracking, weak Wolfe, and strong Wolfe. Hestenes-
Stiefel conjugate gradient algorithm. Function f3(x) . . . . 77
Table 2.7 Global performances of HZ for solving 800 uncon-
strained optimization problems from the UOP collec-
tion with different line-searches . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Table 3.1 Number of iterations of the steepest descent and of the
relaxed steepest descent for the quadratic problem from
Example 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Table 3.2 Number of iterations and the average stepsize for SDB
and RSDB with backtracking for Example 3.4 . . . . . . . . . 95
Table 4.1 Iterations generated by the Newton method . . . . . . . . . . . . . 111
Table 4.2 Evolution of kx  xkk and kF(xk)k . . . . . . . . . . . . . . . . . . . . . 111
Table 4.3 Stationary solution of a chemical reactor. Initial point,
solution, functions values in these points. Newton
method. 5 iterations. ε ¼ 106 . . . . . . .. . . . . . . . . . . . . . .. . . . . . 117

xxvii
xxviii List of Tables

Table 4.4 Robot kinematics problem. Initial point, solution,

functions values in these points. Newton method. 3
iterations. ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Table 4.5 Propan combustion in air – reduced variant. Initial
point, solution, functions values in these points. Newton
method. 9 iterations. ε ¼ 106 . . . . . . .. . . . . . . . . . . . . . .. . . . . . 119
Table 4.6 Circuit design. Initial point, solution, functions
values in these points. Newton method. 4 iterations.
ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Table 4.7 Initial points, value of function in initial points, number
of iterations. n ¼ 50 , ε ¼ 108, ρ ¼ 0.0001 and
β ¼ 0.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Table 4.8 Initial points, value of function in initial points, number
of iterations. n ¼ 100 , ε ¼ 108, ρ ¼ 0.0001 and
β ¼ 0.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Table 4.9 Initial points, value of function in initial points, number
of iterations. n ¼ 300 , ε ¼ 108, ρ ¼ 0.0001 and
β ¼ 0.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Table 4.10 Stationary solution of a chemical reactor. Initial point,
solution, functions values in these points. Newton with
finite-difference method. 5 iterations. ε ¼ 106 . . . . . . . . 149
Table 4.11 Robot kinematics problem. Initial point, solution,
functions values in these points. Newton with finite-
difference method. 4 iterations. ε ¼ 106 . . . . . . . . . . . . . . . 150
Table 4.12 Propan combustion in air – reduced variant. Initial
point, solution, functions values in these points.
Newton with finite-difference method. 10 iterations.
ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Table 4.13 Circuit design. Initial point, solution, functions values
in these points. Newton with finite-difference method.
4 iterations. ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Table 4.14 Stationary solution of a chemical reactor. Initial point,
solution, functions values in these points. Composite
Newton method of order 1. 4 iterations. ε ¼ 106 . . . . . . 166
Table 4.15 Robot kinematics problem. Initial point, solution,
functions values in these points. Composite Newton
method of order 1. 2 iterations. ε ¼ 106 . . . . . . . . . . . . . . . 166
Table 4.16 Propan combustion in air – reduced variant. Initial
point, solution, functions values in these points.
Composite Newton method of order 1. 7 iterations.
ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Table 4.17 Circuit design. Initial point, solution, functions values
in these points. Composite Newton method of order 1. 3
iterations. ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Table 5.1 Choices of βk in standard conjugate gradient
methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
List of Tables xxix

Table 5.2 Performances of HS, FR, and PRP for solving five
applications from the MINPACK-2 collection . . . . . . . . . . 217
Table 5.3 Performances of PRP+ and CD for solving five
applications from the MINPACK-2 collection . . . . . . . . . . 217
Table 5.4 Performances of LS and DY for solving five
applications from the MINPACK-2 collection . . . . . . . . . . 218
Table 5.5 Hybrid selection of βk based on the projection
concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Table 5.6 Performances of TAS, PRP-FR, and GN for solving five
applications from the MINPACK-2 collection . . . . . . . . . . 224
Table 5.7 Performances of HS-DY, hDY, and LS-CD for solving
five applications from the MINPACK-2 collection . . . . . 224
Table 5.8 Performances of NDLSDY for solving five applications
from the MINPACK-2 collection . . . . . . . . . . . . . . . . . . . . . . . . 231
Table 5.9 Performances of CG-DESCENT and CG-DESCENTaw
for solving five applications from the MINPACK-
2 collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Table 5.10 Performances of DESCONa for solving five
applications from the MINPACK-2 collection . . . . . . . . . . 244
Table 5.11 Performances of CONMIN for solving five applications
from the MINPACK-2 collection . . . . . . . . . . . . . . . . . . . . . . . . 251
Table 5.12 Performances of DK+w and DK+aw for solving five
applications from the MINPACK-2 collection . . . . . . . . . . 257
Table 5.13 Performances of CG-DESCENT and of
CG-DESCENTaw for solving five large-scale
applications from the MINPACK-2 collection . . . . . . . . . . 258
Table 5.14 Performances of DESCON and of DESCONa for
solving five large-scale applications from the
MINPACK-2 collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Table 5.15 Performances of DK+w and of DK+iw for solving five
large-scale applications from the MINPACK-
2 collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Table 6.1 Performances of L-BFGS for different values of m . . . . 293
Table 6.2 Performances of L-BFGS Elastic-plastic torsion.
n ¼ 40, 000. ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
Table 6.3 Performances of L-BFGS Pressure distribution in a
journal bearing. n ¼ 40, 000. ε ¼ 106 . . . . . . . . . . . . . . . . . . 294
Table 6.4 Performance of L-BFGS. Optimal design with com-
posite materials. n ¼ 40, 000. ε ¼ 106 . . . . . . . . . . . . . . . . . 295
Table 6.5 Performances of L-BFGS. Steady-state combustion.
n ¼ 40, 000. ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Table 6.6 Performances of L-BFGS. Minimal surfaces
with Enneper boundary conditions. n ¼ 40, 000.
ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Table 6.7 Performances of L-BFGS. Inhomogeneous
superconductors: 1-D Ginzburg-Landau. n ¼ 1000.
ε ¼ 106 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
xxx List of Tables

Table 6.8 Performances of MM-SR1gen versus MM-BFGS

(40,000 variables, cpu seconds) . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Table 6.9 Performances of L-BFGS (m ¼ 5) for solving five
applications from the MINPACK-2 collection (250,000
variables, cpu seconds) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Table 6.10 Performances of MM-SR1gen for solving five
applications from the MINPACK-2 collection (250,000
variables, cpu seconds) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
Table 7.1 Performances of TN for solving five applications from
the MINPACK-2 collection (40,000 variables) . . . . . . . . . 329
Table 7.2 Performances of TN for solving five applications from
the MINPACK-2 collection (250,000 variables) . . . . . . . . 329
Table 8.1 Performances of TRON for solving five applications
from the MINPACK-2 collection . . . . . . . . . . . . . . . . . . . . . . . . 351
Table 9.1 Performances of NELMED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Table 9.2 Performances of NEWUOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Table 9.3 Performances of DEEPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Table 9.4 Performances of NELMED for solving 16 small-scale
applications from the SMUNO collection . . . . . . . . . . . . . . . 367
Table 9.5 Performances of NEWUOA for solving 16 small-scale
applications from the SMUNO collection . . . . . . . . . . . . . . . 367
Table 9.6 Performances of DEEPS for solving 16 small-scale
applications from the SMUNO collection . . . . . . . . . . . . . . . 368
Table 9.7 Performances of NEWUOA for solving 10 large-scale
applications from the SMUNO collection . . . . . . . . . . . . . . . 368
Table 9.8 Performances of DEEPS for solving 10 large-scale
applications from the SMUNO collection . . . . . . . . . . . . . . . 369
Table 12.1 Elastic-plastic torsion. SPG. 40,000 variables. M ¼ 10 431
Table 12.2 Elastic-plastic torsion. L-BFGS-B. 40,000 variables.
m ¼ 5 .. . .. .. . .. . .. . .. . .. .. . .. . .. . .. . .. . .. .. . .. . .. . .. . .. .. . .. . 432
Table 12.3 Elastic-plastic torsion. TNBC. 40,000 variables . . . . . . . . 432
Table 12.4 Pressure distribution in a journal bearing. SPG. 40,000
variables. M ¼ 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
Table 12.5 Pressure distribution in a journal bearing. L-BFGS-B.
40,000 variables. m ¼ 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
Table 12.6 Optimal design with composite materials. SPG. 40,000
variables. M ¼ 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
Table 12.7 Optimal design with composite materials. L-BFGS-B.
40,000 variables. m ¼ 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434
Table 12.8 Steady-state combustion. SPG. 40,000 variables.
M ¼ 10 . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . 435
Table 12.9 Steady-state combustion. L-BFGS-B. 40,000 variables.
m ¼ 5 .. . .. .. . .. . .. . .. . .. .. . .. . .. . .. . .. . .. .. . .. . .. . .. . .. .. . .. . 435
Table 12.10 1-D Ginzburg-Landau problem. SPG. 1000 variables.
M ¼ 10 . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . 437
Table 12.11 1-D Ginzburg-Landau problem. L-BFGS-B. 1000
variables. m ¼ 5 . . . .. . . . . .. . . . . . .. . . . . .. . . . . .. . . . . .. . . . . . .. . 437
List of Tables xxxi

Table 13.1 Optimization process by QLD . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461

Table 13.2 Initial point, solution, and bounds on variables . . . . . . . . . 462
Table 13.3 Dynamic. Optimization process by QLD . . . . . . . . . . . . . . . . 463
Table 14.1 Augmented Lagrangian method . . . . . . . . . . . . . . . . . . . . . . . . . . 485
Table 14.2 Quadratic penalty method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
Table 14.3 Iterations generated by SPENBAR. σ 0 = 0.1, β = 0.9,
τ = 108 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . 500
Table 14.4 Iterations generated by SPENBAR. σ 0 = 0.1, β = 0.9,
τ = 108. Optimization of a heavy body . . . . . . . . . . . . . . . . . 501
Table 14.5 Performances of SPENBAR. σ 0 = 0.1, β = 0.9, τ = 108.
Application DES. Thomson problem . . . . . . . . . . . . . . . . . . . . 502
Table 14.6 Performances of SPENBAR. σ 0 = 0.1, β = 0.9, τ = 108.
Application HANG. Dog curve . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
Table 14.7 Performances of SPENBAR for solving nine
applications from the LACOP collection. σ 0 = 0.1,
β = 0.9, τ = 108 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
Table 14.8 MINOS for solving some large-scale linear
programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Table 14.9 MINOS with different initializations. Example 14.4 . . . 515
Table 14.10 MINOS with different initializations. Example 14.5 . . . 516
Table 14.11 Performances of MINOS for solving 12 applications
from the LACOP collection. Small-scale nonlinear
application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
Table 14.12 Performances of MINOS for solving six applications
from the LACOP collection. Large-scale nonlinear
applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
Table 14.13 Performances of MINOS for solving the HANG
application from the LACOP collection. Large-scale
nonlinear applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
Table 15.1 Performances of SNOPT for solving 12 applications
from the LACOP collection. Small-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 552
Table 15.2 Performances of SNOPT for solving 6 applications
from the LACOP collection. Large-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 552
Table 15.3 Comparison: MINOS versus SNOPT for solving 15
large-scale applications from the LACOP collection.
Large-scale nonlinear optimization applications . . . . . . . . 553
Table 15.4 Performances of NLPQLP for solving 8 applications
from the LACOP collection. Small-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 557
Table 15.5 Performances of NLPQLP for solving the ETA-
MACRO application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
Table 15.6 Performances of KNITRO/ACTIVE for solving 12
applications from the LACOP collection. Option 3.
Small-scale nonlinear optimization applications . . . . . . . . 564
xxxii List of Tables

Table 15.7 Performances of KNITRO/ACTIVE for solving 6

applications from the LACOP collection. Option 3.
Large-scale nonlinear optimization applications . . . . . . . . 564
Table 15.8 Performances of KNITRO/ACTIVE for solving the
HANG application from the LACOP collection. Option
3. Large-scale nonlinear optimization applications . . . . . 565
Table 16.1 Performances of CONOPT for solving 12 applications
from the LACOP collection. Small-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 594
Table 16.2 Performances of CONOPT for solving the PENICI
application from the LACOP collection . . . . . . . . . . . . . . . . . 594
Table 16.3 Performances of CONOPT for solving 6 applications
from the LACOP collection. Large-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 595
Table 16.4 Comparison between KNITRO/ACTIVE and CONOPT 595
Table 16.5 Performances of CONOPT for solving 6 applications
from the LACOP collection with SQP inhibited. Large-
scale nonlinear optimization applications . . . . . . . . . . . . . . . 596
Table 16.6 CONOPT with SQP versus CONOPT without SQP . . . . 596
Table 16.7 Performances of CONOPT with SQP for solving the
HANG application from the LACOP collection . . . . . . . . 597
Table 16.8 Performances of CONOPT without SQP for solving the
HANG application from the LACOP collection . . . . . . . . 597
Table 16.9 Comparison: MINOS, KNITRO/ACTIVE, SNOPT and
CONOPT. CPU computing time (seconds) . . . . . . . . . . . . . . 597
Table 17.1 Evolution of some elements of PDIP. Example 17.1 . . . 624
Table 17.2 Evolution of parameters of PDIP. Example 17.1 . . . . . . . 625
Table 17.3 Evolution of some elements of PDIP. Example 17.2 . . . 626
Table 17.4 Evolution of parameters of PDIP. Example 17.2 . . . . . . . 626
Table 17.5 Performances of PDIP for solving 4 applications from
the LACOP collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626
Table 17.6 Performances of KNITRO for solving 12 applications
from the LACOP collection. Option 0. Small-scale
nonlinear optimization applications . . . . . . . . . . . . . . . . . . . . . . 640
Table 17.7 Performances of KNITRO/INTERIOR-DIRECT for
solving 12 applications from the LACOP collection.
Option 1. Small-scale nonlinear optimization
applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
Table 17.8 Performances of KNITRO/INTERIOR-CG for solving
12 applications from the LACOP collection. Option 2.
Small-scale nonlinear optimization applications . . . . . . . . 641
Table 17.9 Performances of the KNITRO algorithms. Small-scale
nonlinear optimization applications . . . . . . . . . . . . . . . . . . . . . . 641
Table 17.10 Performances of KNITRO for solving 6 applications
from the LACOP collection. Option 0. Large-scale
nonlinear optimization applications . . . . . . . . . . . . . . . . . . . . . . 642
Rainfall Changes and Possible Linkages with the Synoptic Disturbances in Eastern India xxxiii

Table 17.11 Performances of KNITRO/INTERIOR-DIRECT for

solving 6 applications from the LACOP collection.
Option 1. Large-scale nonlinear optimization
applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
Table 17.12 Performances of KNITRO/INTERIOR-CG for solving
6 applications from the LACOP collection. Option 2.
Large-scale nonlinear optimization applications . . . . . . . . 643
Table 17.13 Performances of the KNITRO algorithms. Large-scale
nonlinear optimization applications . . . . . . . . . . . . . . . . . . . . . . 643
Table 17.14 Performances of KNITRO. Application PENICI
(n = 707, l = 602, m = 0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643
Table 17.15 Comparisons between KNITRO and CONOPT.
Small-scale nonlinear optimization applications . . . . . . . . 644
Table 17.16 Comparisons between KNITRO and CONOPT.
Large-scale nonlinear optimization applications . . . . . . . . 644
Table 18.1 Performances of filterSD for solving 8 applications
from the LACOP collection .. . . . . . . . . . . . .. . . . . . . . . . . .. . . . . 652
Table 18.2 Comparison of filterSD versus NLPQLP
and KNITRO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 652
Table 19.1 Performances of IPOPT for solving 12 applications
from the LACOP collection. Small-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 676
Table 19.2 Performances of IPOPT for solving 6 applications
from the LACOP collection. Large-scale nonlinear
optimization applications . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . 676
Table 19.3 Performances of IPOPT for solving the PENICI
application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677
Table 19.4 Performances of the IPOPT algorithm. Small-scale
nonlinear optimization applications . . . . . . . . . . . . . . . . . . . . . . 677
Table 19.5 Performances of the IPOPT algorithm. Large-scale
nonlinear optimization applications . . . . . . . . . . . . . . . . . . . . . . 677
Table 20.1 Performances of COBYLA. ρbeg = 1.5d0,
ρend = 1. d  8. . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . 684
Table 20.2 Performances of DFL . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . .. . . . . . . 689
 Introduction
1

This book is on modern continuous nonlinear optimization. Continuous nonlinear optimization

problems have a simple mathematical model and always refer to a real physical system, the running
of which we want to optimize. Firstly, a nonlinear optimization problem contains an objective
function which measures the performances or requirements of the system. Often, this function
represents a profit, a time interval, a level, a sort of energy or combination of different quantities
which have a physical significance for the modeler. The objective function depends on some
characteristics of the system, called variables or unknowns. The purpose of any optimization
problem is to find the values of these variables that minimize (or maximize) the objective function
subject to some constraints which the variables must satisfy. Constraints of an optimization problem
may have different algebraic expressions. There are static and dynamic constraints called functional
constraints. The difference between these types of constraints comes from the structure of their
Jacobian. The dynamic constraints always involve a Jacobian with a block-diagonal structure.
Besides, these functional constraints are nonlinear functions which may be equalities or inequalities,
or even range constraints. Another very important type of constraints is the simple bounds on
variables. The real applications of optimization include simple bounds on variables which express
the constructive engineering conditions and limits of the system to be optimized. The constraints of an
optimization problem define the so-called feasible domain of the problem. Both the objective function
and the constraints may depend on some parameters with known values which characterize the
system under optimization. The process of identifying the variables, parameters, simple bounds on
variables, the objective functions, and constraints is known as modeling, one of the finest intellectual
activities. In this book we assume that the variables can take real values and the objective function
and the constraints are smooth enough (at least twice continuously differentiable) with known first-
order derivatives. When the number of variables and the number of constraints is large, then the
optimization problem is quite challenging.
If the objective function and the constraints are all linear, then the problem is known as linear
programming, (see Dantzig (1963), Vanderbei (2001), and Andrei (2011d)) which represents a
special chapter in the mathematical programming area and is not material to this book. If the objective
function is a quadratic function and the constraints are all linear, we have a quadratic programming
problem with plenty of theoretical and computational results and applications (see Chap. 13 of this
book). There are optimization problems with only simple bounds on variables (see Chap. 12 of this
book). The general nonlinear optimization problems include both nonlinear functional constraints

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 1

N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_1
2 1 Introduction

and simple bounds on variables. The mathematical formulation of a general nonlinear optimization
problem is as follows:
minf ðxÞ,
x2ℝn
subject to ð1:1Þ
ei ðxÞ ¼ 0, i 2 E, cj ðxÞ  0, j 2 I, l  x  u,

where f : ℝn ! ℝ, E and I are the set of indices for equality and inequality constraints, respectively.
The equality constraints can be assembled as e(x) ¼ 0, where e:ℝn ! ℝm, and m ¼ card (E).
Similarly, the inequality constraints can be assembled as c(x)  0, where c : ℝn ! ℝp, and p ¼ card (I ).
The bounds l, u 2 ℝn satisfy the natural conditions  1 < li  ui < þ 1 for all i ¼ 1, . . ., n. li is
called the lower bound of variable xi, while ui is called the upper bound of xi. If li ¼ ui, then the
variable xi is fixed and can be eliminated from the problem.
Suppose there is an algorithm or a collection of algorithms able to solve general nonlinear
optimization problems. For recognizing whether the algorithm has found the optimal solution of
the problem, one needs some mathematical tools. The most elegant and effective one is the so-called
optimality conditions. These are expressed as a nonlinear algebraic system of equations which may be
used both to design nonlinear optimization algorithms and to check whether the current set of
variables is indeed the solution of the problem. There are two types of optimality conditions. The
necessary optimality conditions are conditions that must be satisfied by any solution point. The
sufficient optimality conditions are those that, if satisfied at a certain point, guarantee that this point is
in fact a solution. Observe the nuances of these two concepts (see Chap. 11 of this book). The
necessary conditions are also called the first-order conditions because they involve the properties of
both the gradients of the objective function and the constraints and they also show how these
gradients are related to each other at a solution. Actually, these conditions are called KKT
(Karush-Kuhn-Tucker) conditions and are the foundation of many algorithms for nonlinear optimi-
zation. The second-order conditions, both necessary and sufficient, use the second derivatives of the
functions defining the problem and examine these derivatives to see whether this extra information
can resolve the issue of increase or decrease of the objective function values in undecided directions.
If the optimality conditions are not satisfied at a current point, then they must be used to see how
the current solution estimation can be improved. This is the subject of sensitivity analysis, which
shows the sensitivity of the current solution to changes in the structure and in the data of the model of
the optimization problem.

1.1 Mathematical Modeling: Linguistic Models Versus

Mathematical Models

Normally, everyone is familiar with different representations of the surrounding world, the so-called
mental models, used in each moment of our existence. The decisions made in different situations are
not based on the real world, but on our mental images of the real world, on our mental images of the
relations among the components of the real world. The mental models represent our understanding of
the part of the creation that we want to know. Having in view that the support of our thinking is the
word, it follows that mental models are actually linguistic models. Therefore, in one way or another,
the human beings set forth their understanding of the real world as a linguistic description expressed
as a corpus of assertions (theorems) of the general form: if . . . then . . . .
The mental models have some advantages which recommend them to be used in our efforts to
understand the world. A mental model is flexible in the sense that it can take into consideration a
1.1 Mathematical Modeling: Linguistic Models Versus Mathematical Models 3

domain of information sensibly larger than the numerical one. Besides, a linguistic model may be
quickly adapted to some new situations and can be modified as soon as some new information is
available. The mental models are filters through which we can explain our experiences and evaluate
and select different actions. In a way, the greatest philosophical systems, political and economical
doctrines, theories of physics, or the literature itself are linguistic models.
However, the mental models have some disadvantages as well. They are not so easily understood
by the others. Their interpretation is dependent on their creator. Besides, the hypothesis used to
generate them is difficult to examine and even to accept. The ambiguities and contradictions
contained in these types of models can remain undetected, unexplained, and therefore unsolved. It
is quite natural to have difficulties in understanding the linguistic models suggested by the others.
Surprisingly, we are not so good at developing and understanding our own mental models or at using
them during the process of decision-making. Psychologists have shown that we are able to consider
only a very small number of factors in decision-making. In other words, the mental models we use in
decision-making are extremely simple. These are often imperfect because we frequently persist in
error when deducing the consequences from the suppositions on which they are based. These models
often express what we would like to happen and not what actually happens in reality.
But the greatest imperfection of the mental models lies in the fact that these intellectual
developments do not satisfy the criteria of completeness, minimality, and non-contradiction. It is
very likely that some very important assertions which change their significance may be omitted in a
mental (linguistic) model. Clearly, some new assertions may be introduced, which are in contradic-
tion with those we have previously considered in our reasoning. At the same time, in using the
linguistic models we are often faced with the very difficult problem of word rectification. This
problem dramatically limits the usage of linguistic models with different groups of people. Finally,
we must emphasize that during the linguistic models analysis and solving process, we are very likely
to be confronted with the circularity danger. The problem of the circularity of formal systems was
solved by G€odel (1931), who showed that expressing knowledge into a formal system and into logical
formal systems like those of Russel or of Zermelo-Fraenkel-Newmann is an illusion. There are
relatively simple assertions (theorems) which are impossible to be solved (decidable) in this kind of
formal systems.
A mathematical model is a representation in mathematical symbols of the relations between the
variables and the parameters belonging to the part of the creation we are interested in. The relations
describing the mathematical model often include variables and their derivatives, thus expressing the
local character of the model and that of predictability as well. The mathematical models have a
number of advantages versus the linguistic models. The most important is that mathematical models
do not have any imperfections while the linguistic models do have. They are explicit in the sense that
the hypothesis and the assumptions used in their development are public and subject to any criticism.
Besides, the (logical) consequences after solving them are very well justified mathematically. Finally,
the mathematical models are more comprehensive, being able to simultaneously consider an appre-
ciable multitude of factors. But the most important characteristic of mathematical models is that they
are written on the basis of the conservation laws. In this respect the Noether theorem shows that the
conservation laws are direct consequences of different symmetries (Andrei, 2008h). For example, the
conservation of energy is a consequence of the temporal symmetry, while the conservation of
momentum is a consequence of the symmetry of space. The entire physical world is depicted as
being governed according to mathematical laws.
4 1 Introduction

1.2 Mathematical Modeling and Computational Sciences

The theory and practice of mathematical modeling is based on computational sciences. Broadly,
computational sciences consist in the usage of computer systems for analyzing and solving scientific
problems. There is a sharp distinction between computer science and computational science. Com-
puter science focuses on the building and the analysis of computers as well as on computability. On
the other side, computational sciences emphasize the development of methods, techniques, and
algorithms for solving mathematical models and center on the convergence and complexity of
algorithms.
The purpose of computational sciences is to understand the evolution of a real-world system by
analyzing and solving the corresponding mathematical models associated to it and by using high
performance computers. By making use of mathematical symbols as well as of mathematical
accuracy and rigor, mathematical modeling has a central place in computational sciences. Mathemat-
ical modeling leads us to the maturation of the domain the mathematical model belongs to, and also to
the generation and the development of new mathematical methods and techniques able to solve new
problems. The place of the mathematical model within the general scheme for modeling and solving a
problem is shown in Fig. 1.1.
Observe that a mathematical model is synthesized from the real world, thus arriving into an
abstract world where mathematical rigor is present through mathematical concepts. The model is
solved by using mathematical theories irrespective of the physical significance of the elements of the
problem. The interpretation of the solution may lead us to some modifications of the model, which
again determines our addressing to the real world.
Notice that in the abstract world of the modeler we operate with concepts belonging to the
mathematical domain which the modeling process refers to. This is the question of the phenomenol-
ogy and the conservation laws that characterize the domain of interest. At the same time, certain
concepts from the theory of languages, of translators, and of compilers are used in order to develop
informatics technologies able to elaborate, update, and maintain the mathematical models.
On the other hand, in the abstract world of algorithms, developers operate with advanced
mathematical concepts in order to generate optimization algorithms and to study their convergence
and complexity. Additionally, some advanced languages like Fortran, C++, or Matlab are being used
for implementing algorithms in computer programs.

Fig. 1.1 The process of

modeling and solving
(optimizing) a problem
1.3 The Modern Modeling Scheme for Optimization 5

Therefore, we are faced with the following dilemma. Which is the better: to consider an approxi-
mate (simplified) mathematical model of the problem and then try to get an exact solution of it as
much as possible or to use a mathematical model as accurately as possible and then determine an
approximate solution of it? The practical experience recommends that the second alternative gives
better results. Indeed, the idea is that a mathematical model which follows to be optimized should be
developed by keeping a balance between the aims of improved accuracy in the model which involves
added complexity in its formulation and the increased ease of the optimization. This might be
achieved by optimizing successive more complicated versions of the model. In this way, the effects
of each refinement of the model on the optimization process can be monitored. Thus the difficulty
which the optimization algorithms have to face can be discovered much more quickly than if no
optimization were applied until the model was complete. In other words, it is better to start with a
simplified model and use it in an iterative scheme in which the model is alternatively optimized and
refined until an acceptable version of it is obtained both from its mathematical representation of
reality and its optimization viewpoints.

1.3 The Modern Modeling Scheme for Optimization

Formulating and developing a mathematical model of a given reality is one of the finest creative
intellectual activities. A short glimpse at the optimization domain shows a substantial progress of the
development of algorithms and computer codes to solve real large-scale nonlinear optimization
applications. From the application viewpoint, one can see that a large part of the time required for
developing, improving, and maintaining an optimization model is spent with data preparation and
transformation, as well as with report preparation and documentation. A nonlinear optimization
model involves time and intellectual effort to organize the data and write computer programs that
transform the data into a form required by the optimization codes. A typical example is the MPS (IBM
LP product) representation of a linear programming problem. In nonlinear optimization, the situation
is more complex because the user must prepare the functions defining the problem and their
derivatives in a form admitted by a particular optimizer. Of course, the user must also transmit to
the optimizer the simple bounds on variables, the initial point, and some other parameters associated
to a particular optimizer. Nonlinear optimization problems may be transmitted to the optimization
codes by using the so-called SIF – Standard Input Format (Bongartz, Conn, Gould, & Toint, 1995). It
is worth mentioning that such an approach of using an optimizer is only accessible to the specialist
who wrote the optimizer and not to the wide audience of analysts in charge of some projects involving
the optimizer. It should be emphasized that the optimizers have very different (and difficult)
communication facilities subject to transmitting the functions of the problem and their derivatives,
simple bounds on variables, the initial point, some other parameters, optimization conditions, etc.
Currently, the advanced modern industrial usage of optimization algorithms involves the algebraic
oriented languages that offer advanced capabilities for expressing an optimization model in a form
immediately accepted by an optimizer. The most important modeling technologies with optimization
facilities are GAMS and AMPL (see Table 1.1 below).
In the last two decades significant research activities have taken place in the area of local and
global optimization, including many theoretical, computational, and software contributions. The
access to this advanced optimization software needs more and more sophisticated modeling tools.
Algebraic oriented optimization modeling languages represent an important class of tools that
facilitate the communication of optimization models towards decision-making systems based on
the optimization paradigm. In a wider context, the algebraic oriented modeling tools evolve towards
6 1 Introduction

Table 1.1 Algebraic oriented modeling languages (partial list)

GAMS General Algebraic Modeling System (www.gams.com)
Development Research Center. The World Bank, 1818 H. Street, Washington D.C., USA.
Brooke, A., Kendrick, D., Meeraus, A., Raman, R., Rosenthal, R.E., GAMS A user’s guide. GAMS
Development Corporation, December 1998.
AMPL A Mathematical Programming Language (www.ampl.com)
Department of Industrial Engineering and Management Sciences. Northwestern University, Evanstone,
Illinois 60201, USA.
Fourer, R., Gay, M., Kernighan, B.W., AMPL: A modeling language for mathematical programming.
Second edition. Duxbury Press/Brooks/Cole Publishing Company, 2002.
ALLO A Language for Linear Optimization
(http://camo.ici.ro/projects/allo/allo.htm)
Research Institute for Informatics – Bucharest. 8-10 Bdl. Mareşal Alexandru Averescu, sector
1, 011455 Bucharest – Romania.
Andrei, N., The ALLO language for linear programming. ICI Technical Report No.1/2004, Bucharest,
Romania.
Andrei, N., Criticism of the linear programming algorithms reasoning. Academy Publishing House,
Bucharest 2011. (Chap. 17, Annex A6)
LPL A Structured Language for Modeling Linear Programs
(http://diuflx71.unifr.ch/lpl/mainmodel.html)
Institut of Informatics, University of Fribourg, Regina Mundi, rue de Faucigny 2, CH-1700 Fribourg,
Switzerland.
Hürlimann, T., The LPL modeling language: Highlights. University of Fribourg, Switzerland,
December 8, 2011.
AIMMS The Modeling System (www.aimms.com)
Paragon Decision Technology
B.V. P.O. Box 3277, 2001 DG Haarlem, The Netherlands
Bisschop, J., Roelofs, M., AIMMS – The user’s guide. Paragon Decision Technology, 1999.
MPL Modeling System (www.maximalsoftware.com)
Maximal Software, Inc., 2111 Wilson Blvd. Suite 700, Arlington, VA, USA.
Kristjansson, B., MPL User manual. Maximal Software Inc., Iceland, 1993.
LINDO Powerful Library of Optimization Solvers and Mathematical Programming Tools (http://www.
lindo.com)
Lindo Systems Inc., 1415 North Dayton Street, Chicago, Il, 60642, USA.
Schrage, L., Optimization Modeling with LINDO. 5th edition. Duxbury Press, 1997.
MOSEL An Extensible Environment for Modeling and Programming Solutions
(www.fico.com/en/Company)
FICO® Xpress Optimization Suite. 901 Marquette Avenue, Suite 3200, Minneapolis, MN 55402, USA.
(formerly DASH)
Colombani, Y., Heipcke, S., Mosel: An Overview, Xpress Team, FICO, Leamington Spa CV32 5YN,
UK. (http://www.fico.com/xpress)
TOMLAB TOMLAB Optimization Environment (tomopt.com/tomlab)
113 Cherry St Ste 95594. Seattle, WA 98104-2205, USA

fully integrated modeling and optimization management systems with access to databases,
spreadsheets, and graphical user interfaces.
As already known, at the fundamentals of every mathematical model lie the conservation laws
which are active in the domain to which the mathematical model belongs. Therefore, knowing these
conservation laws is essential, and their usage leads to models that satisfy the adequacy to real
principle. However, besides the conservation laws, additional empirical knowledge, principles,
different rules, or previous experience need to be considered in order to develop a mathematical
model of a given reality. In this chapter, certain aspects concerning the mathematical modeling
1.3 The Modern Modeling Scheme for Optimization 7

Fig. 1.2 The modern

modeling scheme of
nonlinear optimization
based on algebraic oriented
languages

process in the context of mathematical modeling technologies based on the algebraic oriented
languages are discussed.
Suppose that we have written the optimization mathematical model, i.e., we have specified the
objective function, the constraints, and the simple bounds on variables. The problem now is how to
transmit this model to the solver (optimizer) in order to get a solution. Since we consider mathemati-
cal models with thousands of variables and constraints, this problem appears to be quite challenging.
An elegant and very efficient solution is to use algebraic oriented languages for achieving an external
description of the model which can automatically be translated into the internal description of it by
means of a translator associated to the language used in this respect. For example, in case of linear
programming problems, the internal representation of the model is given by the MPS form. Therefore,
the translator associated to an algebraic oriented language generates the MPS form of the model from
its external description. In case of nonlinear optimization problems, the mechanism is more complex,
but mainly the same.
A deep analysis of the modeling and solving process (see Fig. 1.2) shows that mathematical
modeling involves more effort and time for data analysis, verification, and documentation, as well as
for updating and improving different variants of the model. The difficulties of the data management
for mathematical modeling are coming from the fact that nowadays we do not have a clear
methodology for mathematical modeling expressed as an algorithm or as a group of algorithms to
be used in a general context. However, understanding the linguistic representation of the process that
we need to represent through a mathematical model is crucial.
The development of a mathematical model is a complex process consisting in a mapping between
the objects (together with the relations among them) from the real world and the symbolic and
mathematical objects. This process involves both a very good knowledge of the reality we want to
represent in mathematical terms and a multitude of methods and techniques for mathematical
modeling. The correctness of a model is usually established after its solving. As a result of intensive
computational experiments, it is often necessary to reformulate the model by introducing some new
algebraic or differential equations (which were initially ignored) and also the corresponding data. The
mathematical modeling process is closely linked to the solving process of the model, supporting each
other. These two processes interact in order to build up a mathematical object placed in the
perspective of the infinite similarity with reality.
8 1 Introduction

Let us now try to detail the modeling process. At the very beginning, for the process we want to
represent into mathematical terms we have a linguistic description, actually a text expressed in the
natural language which describes the process. It should be pointed out that this linguistic description
is a scientific text, not a literary one. Obviously, this description includes references to the important
conservation laws which are active in the process domain under consideration. As a result of an
abstracting creation activity from this linguistic description and by using the conservation laws, we
get an algebraic-differential representation of the process, a mathematical model. This activity
mainly consists in identifying the variables and parameters of the process, together with the algebraic
and differential relations among these entities by using principles, empirical knowledge, different
rules, and some other additional knowledge of the process. Once having this algebraic-differential
representation of the process, the next step is to develop an internal representation of it, a represen-
tation directly admitted by any professional solver, in our case an optimizer. This is done through the
external representation of the model by making use of the so-called algebraic oriented mathematical
modeling languages. Therefore, the mathematical model has three forms of representation: an
algebraic-differential one with mathematical symbols, an external one by using an algebraic oriented
mathematical modeling language (like GAMS, AMPL, ALLO, etc.), and an internal representation
which is mainly an informatics description including different files and data bases directly admitted
by any solver (optimizer). Figure 1.2 presents the modern modeling scheme for nonlinear optimiza-
tion based on algebraic oriented languages.
Algebraic modeling languages represent a problem in a purely declarative way. Most of them
include some computational facilities to manipulate the data as well as certain control structures.
Many optimization models can be well represented declaratively in a very concise way, which also
favors the insight of the model. Besides, a mathematical model is stated in a declarative way by using
mathematical equalities and inequalities. This gives a clear documentation of the model which can be
parsed and interpreted by a compiler or a translator.
The algebraic languages are more declarative rather than imperative. A declarative language
emphasizes “what is being computed” more than “how it is being computed.” Furthermore, the
reasons for which we use a declarative presentation of models are conciseness, insight, and docu-
mentation. The most important algebraic oriented languages used for solving nonlinear optimization
problems are presented in Table 1.1.
As already mentioned, the algebraic oriented modeling languages use essentially declarative
statements as opposed to programming languages (Fortran, C++, etc.) in which the use of procedural
statements dominates over the use of the declarative ones. For instance, the few procedural statements
used in algebraic oriented languages are read/write data and solve commands. In order to
deal with more complicated nonlinear optimization models, these languages include if-then-
else, for, while commands. Such commands enable the modeler to write certain solution
algorithms directly in the modeling language (Bisschop, & Meeraus, 1982; Fourer, 1983; Hürlimann,
1999; Kallrath, & Wilson, 1997; Conejo, Castillo, Minguez, & Garcı́a-Bertrand, 2006; Castillo,
Conejo, Pedregal, Garcı́a, & Alguacil, 2001).

1.4 Classification of Optimization Problems

Plenty of problems of nonlinear optimization are known. The general nonlinear optimization
problems can be classified according to the presence or absence of the constraints (unconstrained
or constrained), the number of variables (small- or large-scale), the smoothness of the functions
defining the problem (differentiable or nondifferentiable), the type of variables (integer or real), the
knowledge of functions and parameters (deterministic or stochastic), the type of functions (linear,
1.4 Classification of Optimization Problems 9

quadratic, nonlinear), the convexity of functions (convex or nonconvex), etc. Observe that there are a
lot of optimization problems and consequently specialized optimization algorithms and software have
been developed. In this work we will consider only the continuous nonlinear optimization.

Linear and Nonlinear Optimization

A general optimization problem may have all the constraints as linear functions, in which case the
problem is called nonlinear optimization with linear constraints. If additionally the objective function
is also linear, then the problem is called linear programming. Linear programming is a special
domain of optimization with special algorithms: the simplex algorithm (Dantzig, 1963; Vanderbei,
1996; Andrei, 2011d; etc.), interior point algorithms (Karmarkar, 1984), affine scaling interior point
algorithms (Dikin, 1967, 1974), the affine scaling in null space (Nazareth, 1987; Kim, 1991; Kim, &
Nazareth, 1992, 1994), the primal-dual following the central trajectory (Lustig, Marsten, & Shanno,
1990, 1991), the predictor-corrector following the central trajectory (Lustig, Marsten, & Shanno
1992, 1994), homogeneous, self-dual methods (Mizuno, Todd, & Ye, 1993; Ye, Todd, & Mizuno,
1994), etc.
A special nonlinear optimization is the quadratic optimization also known as the quadratic
programming, in which the objective function is a quadratic function and all the constraints are
linear (see Chap. 13). If at least the minimizing function or some of the constraints are nonlinear
functions, then the problem is called nonlinear optimization or nonlinear programming.

Unconstrained and Constrained Optimization

By unconstrained optimization we understand the optimization problems for which E ¼ I ¼ ∅ in
(1.1). The unconstrained optimization arises in natural situations, but it is often the result of
reformulations of the constrained optimization problems in which the constraints are introduced by
penalization terms into the minimizing function in order to discourage constraints violations. Plenty
of unconstrained optimization algorithms are known: direct methods or the zero-th order methods like
the Nelder-Mead method (1965), the steepest descent (Cauchy, 1847), the Newton method (Nocedal,
& Wright, 2006; Sun, & Yuan, 2006), the conjugate gradient methods (Hestenes, 1980; Andrei,
2020a), the quasi-Newton and limited-memory quasi-Newton methods (Dennis, & Moré, 1977;
Dennis, & Schnabel, 1983; Nocedal, & Wright, 2006; Liu, & Nocedal, 1989), the inexact or truncated
Newton methods (Nash, 1984a, 1984b; Schlick, & Fogelson, 1992a, 1992b), the trust-region methods
(Nocedal, & Wright, 2006; Conn, Gould, & Toint, 2000), etc.
The constrained optimization is an optimization problem in which the variables have to satisfy at
least a functional constraint, or simple bounds. A special type of constrained optimization is the
so-called simple bounds optimization, in which the only constraints are simple bounds on variables.
These optimization problems are formulated and solved in a large range of activity domains,
particularly in physical sciences and engineering, industry and economics, management, finances,
agriculture, forestry, oceanography and environment, biology and medicine applications, atmo-
spheric sciences and weather forecasting, etc. Plenty of papers and books deal with nonlinear
optimization methods: (Luenberger, 1973, 1984; Gill, Murray, & Wright, 1981; Vavasis, 1991;
Bazaraa, Sherali, & Shetty, 1993; Mangasarian, 1969, 1995; Bertsekas, 1982b, 1999; Nocedal, &
Wright, 2006; Sun, & Yuan, 2006; Bartholomew-Biggs, 2008; Griva, Nash, & Sofer, 2009; Andrei,
2015a, 2017c), etc.

Continuous and Discrete Optimization

By continuous optimization we understand those optimization problems in which the variables can
take any real value. The feasible set for continuous optimization problems – the class of problems
considered in this book – is usually uncountable infinite. In contrast, in discrete optimization
10 1 Introduction

problems, the variables are restricted to take only integer values, that is, xi 2 ℤ, where ℤ is the set of
integers or binary values which have the form xi 2 {0, 1}. The problems with integer variables are
called integer programming. If some of the variables in the problem are not restricted to be integer or
binary variables, then they are called mixed integer programming. The characteristic of discrete
optimization problems is that the unknown x is drawn from a finite, but often very large, set. In other
words, the feasible set for the discrete optimization is a finite set. Normally, the continuous optimi-
zation problems are easier to solve because the smoothness of the functions defining the problem
makes it possible to use the information on the minimizing function and constraints at a particular
current point x to obtain information about these functions at all points in a vicinity of the current
point x. In discrete optimization problems the situation is completely different. The behavior of the
minimizing function and of the constraints may change significantly when moving from a feasible
point to another one, even if these feasible points are close enough. Discrete optimization problems
are not considered in this book. The interested reader may consult the texts by Papadimitriou and
Steiglitz (1982), Nemhauser and Wolsey (1988), and Wolsey (1998).

Global and Local Optimization

Usually, the algorithms for nonlinear optimization seek only a local solution that is a point at which
the value of the minimizing function is smaller than the values of function at all other nearby points.
Some algorithms seek the global solution, that is, a point where the value of the minimizing function
is smaller than the values of function at all feasible points. Any optimization algorithm needs an
initial point from where the computations start. Therefore, a local solution is a point near the initial
point where the function’s value is smaller than its value in all the other points near the initial point,
including the initial point. Although there are many practical optimization applications that require
global solutions, these are difficult to recognize and even difficult to compute. Global optimization
algorithms can be found in Pardalos and Rosen (1987), Floudas and Pardalos (1992), Horst, Pardalos,
and Thoai (2000), etc. On the other hand, the vast majority of books and papers on optimization
describe algorithms for determining local solutions.
There is a fundamental result in optimization: for linear programming and for convex program-
ming problems local solutions are also global solutions. Observe that general unconstrained or
constrained nonlinear optimization problems may have local solutions that are not global.

Convex Programming
The concept of convexity is crucial in optimization, and many practical optimization problems have
this property. The optimization problems with this property always have a very nice theory behind
them and are easier to solve. The term “convex” can be associated both with the sets and the
functions. A set C ⊂ ℝn is a convex set if a straight line segment connecting any two points in
C lies entirely inside C. In other words, C is a convex set if for any two points x, y 2 C, it follows that
αx þ (1  α)y 2 C, for all α 2 [0, 1]. A function f : C ! ℝ, where C ⊂ ℝn, is a convex function if its
domain C is a convex set and for any two points x, y 2 C, the following property holds:
f ðαx þ ð1  αÞyÞ  αf ðxÞ þ ð1  αÞf ðyÞ, for all α 2 ½0, 1: ð1:2Þ

Examples of convex sets include polyhedrons defined by a set of linear equalities and inequalities,
that is: {x 2 ℝn : Ax ¼ b, Dx  d}, where A and D are matrices of appropriate dimensions and b and
d are vectors, is a convex set. The unit ball {x 2 ℝn : kxk2  1} is a convex set. Examples of convex
functions are the linear functions f(x) ¼ cTx þ d, for any constant vector c and scalar d. Also the
quadratic functions f(x) ¼ xTHx, where H is a symmetric positive semidefinite matrix, are a convex
function.
1.4 Classification of Optimization Problems 11

A function f is a strictly convex function if the inequality (1.2) is strict whenever x 6¼ y and
α 2 (0, 1). A function f is concave if f is convex.
Convex programming – a branch of nonlinear optimization – defines a special case of the general
constrained optimization (1.1) in which the objective function is a convex function, the equality
constraints ei(x) ¼ 0, i 2 E, are all linear functions, and the inequality constraints functions cj(x)  0,
j 2 I, are all concave.
As it has been mentioned, if the objective function of the problem (1.1) and the feasible domain
defined by the constraints of (1.1) are both convex, then any local solution of the problem is also a
global solution.
A classical, comprehensive text on convex analysis is Rockafellar’s book (1970).

Deterministic and Stochastic Optimization

Deterministic optimization is characterized by the fact that the mathematical model describing the
problem depends on some parameters that are known with a given degree of accuracy. In other words,
the mathematical description of the problem is completely known as regards both the structure of the
problem and to the values of parameters defining the problem. In contrast, stochastic optimization
refers to those optimization problems with uncertain data in the model, problems that have an
incomplete description concerning both their structure and the values of their parameters. Stochastic
optimization methods are optimization methods that generate and use random variables. For stochas-
tic optimization problems, the random variables appear in the formulation of the problem itself,
which involves random objective functions or random constraints. The stochastic optimization is
described by Birge and Louveaux (1997), Kall and Wallace (1994), Spall (2003), etc.
Methods of optimization with uncertain data in the model of the optimization problem are chance-
constrained optimization and robust optimization. The chance-constrained optimization is a formu-
lation of an optimization problem that ensures that the variable x satisfies the constrained of the
problem with a probability above of certain level. Classical applications of the chance-constrained
optimization include water reservoir management, financial risk management, unmanned autono-
mous vehicle navigation (optimal navigation and reliable obstacle avoidance), as well as optimal
renewable energy generation. By robust optimization we understand that certain constraints are
required to be satisfied for all the possible values of the uncertain data in the model of the optimiza-
tion problem (Ben-Tal, El Ghaoui, & Nemirovski, 2009).

Small-Scale Versus Large-Scale Optimization

A measure of complexity and of course of difficulty of a nonlinear optimization problem is its size,
defined in terms of the number of variables and/or the number of constraints. Roughly speaking, we
distinguish two classes of problems: small-scale optimization problems with 5–10 unknowns and
constraints and large-scale optimization problems with hundreds or thousands or even millions of
variables and constraints. The theoretical developments of the nonlinear optimization, mainly the
necessary and sufficient conditions satisfied by a solution point including the Karush-Kuhn-Tucker
(KKT) conditions and their extensions, are irrespective of the size of the problem. However, when the
optimization algorithms turn to be implemented in computing programs, this classification as small-
scale or large-scale problems is crucial. Two aspects have to be taken into account for solving large-
scale optimization problems: structure and sparsity. For example, the unconstrained optimization
problems may be presented in generalized or extended forms given by the structure of their Hessian
matrix. The Hessian of the minimizing functions in generalized form has a block-diagonal structure,
while the Hessian of the minimizing functions in extended form has a band structure. Efficient
optimization algorithms for solving constrained optimization problems take into account the structure
12 1 Introduction

and the sparsity of the KKT conditions. It is well known that an important component of the vast
majority of optimization algorithms is to solve an algebraic system of linear equations. Advanced
techniques for solving linear systems are based on sparse matrix techniques.

1.5 Optimization Algorithms

By optimization we understand the design and analysis of algorithms (convergence and complexity)
for solving optimization problems and applications. All the optimization algorithms are iterative. For
solving an optimization problem, they start the computation from an initial point and generate a
hopefully convergent sequence of improved estimates of the solution until a specified termination
criterion has been satisfied. The optimization algorithms are differentiated by the strategy used to
move from a given iteration to the next one. These strategies use the values of the objective function
and of constraints and possibly the values of the first and of the second derivatives of these functions.
Some algorithms are based only on the local information of the functions at the current point, while
others accumulate information gathered from the previous iterations.
Good optimization algorithms should fulfill the following requirements: efficiency (they should
not require excessive computing time and storage), robustness (they should solve a wide variety of
optimization problems from different starting points), and accuracy (they should be able to generate a
solution with a specified precision without being sensitive to errors in data or to arithmetic rounding
errors). The most efficient and robust nonlinear optimization algorithms use combinations of some
techniques like the active-set methods, or the interior point methods with globalization strategies and
refining mechanisms, all of them in a frame of the advanced computational linear algebra.
The theory of iterative algorithms involves three aspects. The first one is the creation of the
algorithms themselves. The second aspect is the so-called global convergence analysis that addresses
the question of whether the algorithm, when initialized far away from the solution point, will
eventually converge to it. The third aspect is the local convergence analysis, which refers to the
rate at which the generated sequence of points {xk} converges to the solution. Two aspects of the
convergence-rate theory are important. The first is known as complexity analysis, which focuses on
how fast the algorithms converge to the solution. They differentiate themselves as polynomial-time
algorithms and non-polynomial-time algorithms. The second aspect includes a detailed analysis of
how fast the algorithms converge in their final stages.
Advanced optimization algorithms include some ingredients which make them reliable and
increase their numerical performances. The most important ingredients are as follows: automatic
differentiation (Griewank, 2000), finite-difference derivative estimates (Gill, Murray, & Wright,
1981), inertia correcting and singularity (Fletcher, 1987), sparse technology for solving linear
algebraic system of equations, quasi-Newton approximation of Hessian, stepsize computation by
line-search, tests for stopping the iterations, etc.

1.6 Collections of Applications for Numerical Experiments

In order to see the numerical performances of the nonlinear optimization algorithm described in this
book, a number of four collections of unconstrained and constrained optimization test problems and
applications are used. The unconstrained optimization algorithms are tested on the collection
SMUNO which includes a number of 16 applications with the maximum number of 10 variables
(see Appendix B). Also, the collection UOP that includes 80 large-scale unconstrained optimization
1.7 Comparison of Algorithms 13

problems in generalized or extended form, presented in Andrei (2013e, 2020a), is used to see the
performances of unconstrained optimization algorithms. For each test function from this collection,
we have considered 10 numerical experiments with the number of variables increasing as n ¼ 1000,
2000, . . ., 10000. Therefore, a set of 800 unconstrained optimization problems are solved in our
numerical experiments with unconstrained optimization algorithms. The constrained optimization
algorithms use the collection LACOP. This collection includes a number of 18 real constrained
nonlinear optimization applications (see Appendix C). Additionally, in Appendix D, we presented six
unconstrained optimization applications from the collection MINPACK-2 (Averick, Carter, & Moré,
1991; Averick, Carter, Moré, & Xue, 1992), each of them with 40,000 variables, or 250,000 variables,
which are used in our numerical experiments.

1.7 Comparison of Algorithms

Plenty of algorithms for unconstrained and constrained optimization are shown in this book. Some of
them accumulate information gathered at previous iterations, while others use only the local infor-
mation available at the current iteration. As it has already been mentioned, good algorithms must have
the following three properties: efficiency, robustness, and accuracy. To be reliable and to have a
guarantee that they find a solution, some properties and their convergence results have to be
discussed. However, this is not enough. The irony is that in optimization, algorithms for which
there are very well-established theoretical properties (descent, convergence, and complexity) prove to
have modest numerical performances. Having in view that the final test of a theory is its capacity to
solve the problems and the applications which originated it, our efforts have been directed to see the
numerical performances of the algorithms for solving the optimization problems and applications
from the abovementioned collections. In this book the algorithms have been compared in the
following environment.

Remark 1.1 All algorithms have been coded in double precision Fortran, compiled with f77 (default
compiler settings) and run on an Intel Pentium 4, 1.8 GHz workstation. For each problem from the
UOP collection, 10 numerical experiments with an increasing number of variables as n ¼ 1000, 2000,
. . ., 10000 have been performed. Hence, 800 problems have been solved in this set of numerical
experiments. Some optimization algorithms were tested for solving the applications from the
MINPACK-2 collection with 40,000 or 250,000 variables.
The algorithms compared in these numerical experiments find local solutions. Therefore, the
comparisons of the algorithms are given in the following context. Let f ALG1
i and f ALG2
i be the optimal
value found by ALG1 and ALG2 for problem i ¼ 1, . . ., 800, respectively. We say that, in the
particular problem i, the performance of ALG1 was better than the performance of ALG2 if:
 ALG1 
f  f ALG2  < 103 ð1:3Þ
i i

and if the number of iterations (#iter), or the number of function-gradient evaluations (#fg), or cpu the
CPU time of ALG1 was less than the number of iterations, or the number of function-gradient
evaluations, or the CPU time corresponding to ALG2, respectively.
The iterations are stopped if the inequality kgkk1  106 is satisfied, where k.k1 is the maximum
absolute component of a vector. The maximum number of iterations was limited to 2000.
To compare the performances of algorithms, the Dolan and Moré (2002) performance profiles are
used. For np problems and ns solvers, the performance profile P : ℝ ! [0, 1] is defined as follows. Let
P and S be the set of problems and the set of solvers, respectively. For each problem p 2 P and for
14 1 Introduction

each solver s 2 S, define tp,s¼ computing time (similarly for the number of iterations or the number of
function and its gradient evaluations) required to solve problem p by solver s. The idea is to compare
the performance of solver s on problem p with the best performance by any solver on this problem.
Therefore, the performance ratio is defined by:
tp,s
r p,s ¼  : ð1:4Þ
min s2S tp,s

With this, the performance profile may be defined by:

1 
Ps ð τ Þ ¼ size p 2 P : r p,s  τ , ð1:5Þ
np

for τ > 0, where for any set A, size(A) stands for the number of elements in A. The performance profile
Ps : ℝ ! [0, 1] for a solver s is a nondecreasing, piecewise constant function, continuous from the
right at each breakpoint. Ps(τ) is the probability for solver s 2 S so that the performance ratio rp, s is
within a factor τ > 0 of the best possible ratio. The function Ps is the cumulative distribution function
for the performance ratio. Observe that 1  Ps(τ) is the fraction of problems that the solver cannot
solve within a factor τ of the best solver. In our numerical experiments, in each figure, we used
np ¼ 800 and τ ¼ 16.
The performance profile plot of solvers may be computed as follows. Suppose that two solvers
(ns ¼ 2) have to be compared subject to a given metric (which can be the number of iterations, the
number of function and its gradient evaluations or the CPU computing time) for solving np problems.
Consider an integer value for parameter τ. Firstly, out of np problems, only the problems for which the
criterion (1.3) is satisfied are retained. Let np be the number of problems satisfying the criterion (1.3).
For each solver and for each problem, compute the performance ratio rp,s, for p ¼ 1, . . . , np and s ¼ 1,
2, corresponding to the metric selected. For each solver s ¼ 1, 2 and for i ¼ 1, . . ., τ, compute the
performance profile Ps(τ).
The percentage of problems for which an algorithm is the best is given on the left side of the plot.
On the other hand, the right side of the plot gives the percentage of the problems that are successfully
solved. In other words, for a given algorithm, the plot for τ ¼ 1, represents the fraction of problems for
which the algorithm was the most efficient over all algorithms. The plot for τ ¼ 1 represents the
fraction of problems solved by the algorithm irrespective of the required effort. Therefore, the plot for
τ ¼ 1 is associated to the efficiency of the algorithm, while the plot for τ ¼ 1 is associated to the
robustness of the algorithm. ♦

1.8 The Structure of the Book

This book has two parts. The first one includes Chaps. 2, 3, 4, 5, 6, 7, 8, and 9 that cover the theory and
practice of the unconstrained optimization. The second part contains Chaps. 12, 13, 14, 15, 16, 17, 18,
19, and 20 that are dedicated to the constrained optimization. Chapters 10 and 11 present an overview
on the constrained nonlinear optimization methods and the optimality conditions for nonlinear
optimization, respectively. Besides, the book has four appendices with a mathematical review
containing the main mathematical concepts and results used along the chapters, as well as three
collections with unconstrained and constrained applications on nonlinear optimization with which the
optimization algorithms presented in this book were tested.
We emphasize that the book presents the theoretical developments and the computational
performances of the modern unconstrained and constrained optimization algorithms for solving
1.8 The Structure of the Book 15

large-scale optimization problems and applications. Some other algorithms have not been considered
in this book: the conic model and colinear scaling (Davidon, 1980), tensor methods for solving system
of nonlinear equations (Schnabel, & Frank, 1984) and for unconstrained optimization (Schnabel, &
Chow, 1991), gradient flow methods for unconstrained optimization (Brown, & Bartholomew-Biggs,
1987, 1989), methods based on KKT optimality conditions (Abadie, & Guerrero, 1984), Frank-Wolfe
linearization (Frank, & Wolfe, 1956), successive linear programming (Griffith, & Stewart, 1961), the
convex simplex (Zangwill, 1967), feasible direction methods (Zoutendijk, 1960), the gradient
projection method (Rosen, 1960, 1961), the reduced gradient method (Wolfe, 1967), the generalized
reduced gradient (Abadie, 1978, 1979), generalized elimination methods, ellipsoid methods for
constrained optimization (Ecker, & Kupferschmid, 1983, 1985), etc.
Every chapter includes the description of a particular method for continuous nonlinear optimiza-
tion, the most important theoretical results on convergence, comparisons with other methods,
comments on the linear algebra used in implementations in computer programs, as well as the
numerical performances of the representative algorithms for solving optimization problems and
applications from appendices. The algorithms are tested for solving large-scale optimization
problems from the UOP collection up to 10,000 variables, as well as optimization applications
from the LACOP and MINPACK-2 collections with 40,000 or 250,000 variables. Finally, every
chapter terminates with Notes and References where some details, references, perspectives, and
historical comments are given.
As we know, for minimizing a sufficiently smooth function, at every iteration xk an optimization
algorithm based on the line-search strategy computes the search direction dk in the current point xk,
which must be a descent one, and a stepsize αk taken along the search direction to obtain a new
estimation of the minimum point, xkþ1 ¼ xk þ αkdk. The purpose of Chap. 2 is to present the most
important methods as well as their convergence properties for computing the stepsize αk, a crucial
aspect in any unconstrained optimization algorithm. Chapter 3 presents the steepest descent algorithm
for unconstrained optimization as well as its relaxed and accelerated variants. In Chap. 4 the Newton
method is detailed for solving both the nonlinear algebraic system of equations and the function
minimization. It is proved that if well initialized, the Newton method is quadratic convergent to a
minimum point of the minimizing function. Some modifications of the Newton method and the
composite Newton method are also shown. Chapter 5 is dedicated to the conjugate gradient methods
for unconstrained optimization. These methods are very well suited for solving large-scale uncon-
strained optimization problems and applications. The linear conjugate gradient methods are first
discussed, followed by the nonlinear conjugate gradient methods for unconstrained optimization,
insisting on standard, hybrid, modifications of the standard conjugate gradient schemes, and
memoryless BFGS preconditioned. The chapter illustrates the performances of the modern conjugate
gradient algorithms CG-DESCENT (Hager, & Zhang, 2005, 2006a), DESCON (Andrei, 2013c), and
CGOPT/DK (Dai, & Kou, 2013) for solving five applications from MINPACK2 collection, each of
them up to 250,000 variables. In Chap. 6 the quasi-Newton methods DFP, BFGS, and SR1 as well as
the limited memory BFGS method (L-BFGS) are discussed. Some modifications of the BFGS method
with one or two parameters are also presented. The new quasi-Newton method memoryless SR1 with
generalized secant equation is introduced and tested for solving five applications from the
MINPACK2 collection of applications, each of them up to 250,000 variables. Finally, the
performances of L-BFGS (m ¼ 5) for solving five applications from the MINPACK-2 collection,
each of them with 250,000 variables, illustrate that L-BFGS (m ¼ 5) is faster versus the modern
conjugate gradient algorithms CG-DESCENT, DESCON, and DK. The inexact or truncated Newton
method (TN) (Nash, 1984b, 1985) is considered in Chap. 7. Intensive numerical experiments show
that subject to the number of iterations and to the function calls TN is better than L-BFGS, but subject
to the CPU computing time metric, both L-BFGS and TN have similar performances, L-BFGS being
16 1 Introduction

slightly faster. Comparing the performances of TN subject to the CPU computing time versus the
modern conjugate gradient algorithms CG-DESCENT and DESCON, we see that the conjugate
gradient algorithms are top performers. Chapter 8 introduces another paradigm for solving uncon-
strained optimization problems – the trust-region method. The line-search methods presented in the
previous chapters generate a descent search direction d and then determine a suitable stepsize α along
this direction, hoping that the function values will reduce. On the other hand, the trust-region methods
define a region around the current iterate within which they trust the quadratic model to be an
adequate representation of the minimizing function and then choose the step to be the approximate
minimizer of the model in this region. Therefore, trust-region methods choose the direction and the
stepsize simultaneously. Of course, if a step is not acceptable, they reduce the size of the region and
find a new minimizer. For solving the applications from the MINPACK2 collection with 40,000
variables, we used the trust-region method implemented in TRON (Lin, & Moré, 1999) without
simple bounds. For solving these five applications, when comparing the performances of TRON
versus TN, we see that TN is top performer. In Chap. 9 the direct methods known as the zero-th order
methods for unconstrained optimization are described. These methods consider the case in which the
derivatives of the minimizing function are unavailable, impractical to obtain, or unreliable. From the
multitude of derivative-free optimization algorithms, we selected in this chapter only the algorithm of
Nelder-Mead (NELMED) (Nelder, & Mead, 1965), algorithm of Powell (NEWUOA) (Powell, 2003,
2004, 2006), and of Andrei (DEEPS) (Andrei, 2021a). The numerical experiments with these
algorithms for solving 16 real unconstrained optimization applications, included in the collection
SMUNO from Appendix B, show that NEWUOA is way more efficient than NELMED and than
DEEPS. In Chap. 10 an overview on constrained nonlinear optimization methods is given. Some
aspects concerning convergence tests, infeasible points, approximate sub-problems – local models
and their solving, globalization strategy – convergence from remote starting points, and refining the
local model are also discussed. It is shown that the most efficient and robust nonlinear optimization
algorithms use combinations of the active-set methods (local models based on sequential linear or
quadratic programming), or interior point methods with globalization strategies (augmented Lagrang-
ian, penalty and merit function, or filters methods) and refining mechanisms (line-search or trust-
region methods), all of them in a frame where advanced computational linear algebra techniques are
used. Chapter 11 is dedicated to the presentation of the optimality conditions for nonlinear optimiza-
tion. The key to understanding the nonlinear optimization are the Karush-Kuhn-Tucker (KKT)
optimality conditions. This is a major result which identifies an algebraic system of equalities and
inequalities that corresponds to the solution of any nonlinear optimization problem. For problems
with inequality constraints, the KKT approach generalizes the method of Lagrange multipliers, which
allows only equality constraints. For the development of the KKT optimality conditions, three
possible approaches can be used. One is based on the separation and support theorems from the
convex set theory. Another one uses the penalty functions while the third one comes from the theory
of Lagrange multipliers. Each of these approaches has its own virtues and provides its own insights
into the KKT Theorem. In this book we consider the optimality conditions for the continuous
nonlinear optimization (the mathematical programming), using the formalism of Lagrange. The
simple bounded optimization is the subject of Chap. 12. From the multitude of algorithms dedicated
to this problem, only the following are presented: the spectral projected gradient method (SPG) by
Birgin, Martı́nez, and Raydan (2000, 2001); the limited memory BFGS algorithm with gradient
projection (L-BFGS-B) by Byrd, Lu, Nocedal, and Zhu (1994, 1995); and the truncated Newton
with simple bounds (TNBC) by Nash (1984a, 1984b, 1985). Intensive numerical experiments showed
that L-BFGS-B (m ¼ 5) is more robust than SPG and than TNBC. Both SPG and L-BFGS-B are able
to solve large-scale simple bound optimization problems, L-BFGS-B being slightly faster. TNBC is
very sensitive to the interval in which the bounds on variable are defined. In Chap. 13, one of the most
1.8 The Structure of the Book 17

important nonlinear optimization problems is discussed, namely, the quadratic programming, in

which a quadratic objective function is minimized with respect to linear equality and inequality
constraints. Both the equality constrained and inequality constrained quadratic programming are
presented. For these problems, the interior point and the elimination of constraints are also discussed.
For the equality constrained quadratic programming, the following methods are presented: factoriza-
tion of the full KKT system, the Schur-complement method, the null-space method, the reduced
gradient for linear equality constraints, the preconditioned conjugate gradient for reduced systems,
and the projected conjugate gradient method. On the other hand, for the inequality constrained
quadratic programming, the primal active-set method, the active-set method with positive definite
Hessian, the reduced-gradient for inequality constraints, the reduced-gradient for simple bounds, and
the primal-dual active-set method are detailed. Penalty and augmented Lagrangian methods are the
subject of Chap. 14. The idea of these methods is to replace the original problem by a sequence of
sub-problems in which the constraints are expressed by terms added to the objective function. The
penalty concept is implemented in two different methods: the quadratic penalty method and the
nonsmooth exact penalty method. The augmented Lagrangian method or the multipliers method
explicitly uses Lagrange multiplier estimates in order to avoid the ill-conditioning of the quadratic
penalty method. The best known methods for solving nonlinear optimization problems combine the
penalty concept with the augmented Lagrangian in a penalty-barrier with quadratic approximation of
the inequality constraints (SPENBAR), (Andrei, 1996a, b, c, 1998a, 2015a, 2017c), or the minimiza-
tion of a modified augmented Lagrangian subject to linearized constraints (MINOS), (Murtagh, &
Saunders, 1978, 1980, 1982, 1987, 1995), or the minimization of the augmented Lagrangian subject
to simple bounds on variables (LANCELOT), (Conn, Gould, & Toint, 1992b). In this chapter, only
the theory behind the SPENBAR and MINOS algorithms together with some numerical results for
solving some applications from the LACOP collection are presented. Sequential Quadratic Program-
ming (SQP) methods are very effective for solving optimization problems with significant
nonlinearities in constraints. These methods are described in Chap. 15. They are active-set methods
and generate steps by solving quadratic programming sub-problems at every iteration. These methods
are used both in line-search and trust-region paradigm. Three implementations of these methods are
discussed: a SQP algorithm for large-scale constrained optimization (SNOPT) (Gill, Murray, &
Saunders 2002, 2005), a SQP algorithm with successive error restoration (NLPQLP) (Schittkowski,
1986, 2002, 2005, 2009, 2010), and active-set sequential linear-quadratic programming (KNITRO/
ACTIVE) (Byrd, Gould, Nocedal, & Waltz, 2004a). The performances of all these methods are
illustrated for solving large-scale problems from the LACOP collection. A very interesting and
profitable combination of the generalized reduced gradient with the sequential linear-programming
and with the sequential quadratic-programming, namely, CONOPT (Drud, 2011), is presented in
Chap. 16. Comparisons of CONOPT with SQP versus MINOS, KNITRO/ACTIVE, and versus
SNOPT for solving the applications from the LACOP collection subject to the total CPU computing
time show that CONOPT is top performer. One of the most powerful methods for solving nonlinear
optimization problems known as interior point methods is presented in Chap. 17. In this chapter we
consider the interior point method for general nonlinear optimization problems in line-search and
trust-region paradigms, respectively. Some aspects concerning the algorithmic developments for
solving the primal-dual system focus on direct and iterative methods. Both the line-search interior
point algorithm and the trust-region interior point algorithm illustrate the interior point algorithms. In
the last part of this chapter the KNITRO/INTERIOR algorithm is being described, together with its
numerical performances for solving large-scale general continuously nonlinear optimization
problems. KNITRO/INTERIOR provides two procedures for computing the steps within the interior
point approach. In the version INTERIOR-CG, each step is computed by using a projected conjugate
gradient iteration. It factors a projection matrix and uses the conjugate gradient method to
18 1 Introduction

approximately minimize a quadratic model of the barrier problem (Byrd, Hribar, & Nocedal, 1999).
In the version INTERIOR-DIRECT, the algorithm attempts to compute a new iterate by solving the
primal-dual KKT system using direct linear algebra (Waltz, Morales, Nocedal, & Orban, 2003). Four
variants of KNITRO were tested for solving the applications from the LACOP collection. It seems
that KNITRO which automatically chooses the algorithm based on the problem characteristics is the
best. The filter methods developed by Fletcher and Leyffer (2002) as a new technique for the
globalization of the nonlinear optimization algorithms are described in Chap. 18. The filter methods
can use sequential linear programming or sequential quadratic programming in the context of trust-
region methods. Chapter 19 presents an implementation of an interior point filter line-search
algorithm called IPOPT, for large-scale nonlinear programming, proposed by Wächter and Biegler
(2005a, 2005b). The idea of this algorithm is to combine the primal-dual interior point algorithms
with filter line-search. For solving 15 large-scale applications from the LACOP collection, IPOPT
(Table 19.2) needs 42.17 seconds. However, KNITRO (option 0) (Table 17.9) needs 17.81 seconds.
Chapter 20 includes two direct methods for solving constrained nonlinear optimization problems. The
first one is COBYLA by Powell (1993), while the second one is DFL by Liuzzi, Lucidi, and
Sciandrone (2010). At every iteration of COBYLA, the objective function and constraints are linearly
approximated by interpolation in vertices of a simplex structure. The corresponding linear program-
ming problem completed with a constraint of trust-region type is solved, thus obtaining a new
approximation of the solution of the original problem. DFL is an extension of the successive quadratic
penalty method which does not involve the derivative information. The idea of this method is to
approximately minimize a sequence of merit functions in which the penalization of the violation of
the constraints is progressively enlarged.
Our strategy for presenting the state-of-the-art of optimization is to describe the most important
modern methods for the unconstrained optimization (the steepest descent, the Newton method, the
conjugate gradient, the quasi-Newton methods, the inexact Newton and the trust-region methods) and
also for the constrained optimization (the quadratic programming, the penalty and the augmented
Lagrangian methods, the sequential quadratic programming, the interior point, the filter methods) and
to illustrate the computational performances of the most representative algorithms corresponding to
the considered methods. Besides, the linear algebra procedures and techniques implementing the
optimization algorithms are highly emphasized. As a final remark, the book deals with the basic
aspects of the optimization theory and with the numerical performances of the optimization
algorithms for solving large-scale optimization problems and real-applications.
The conclusion of the theoretical developments and of the intensive numerical studies is that there
is no algorithm able to solve any nonlinear unconstrained or constrained optimization problem or
application. The most modern, powerful nonlinear optimization algorithms presented and tested in
this book combine and integrate different optimization techniques (the modified augmented Lagrang-
ian, the sequential linear or quadratic programming, the interior point methods with filter line-search
or trust-region) and include advanced computational linear algebra procedures.

Notes and References

Continuous nonlinear optimization is practically present in any area of activity and is essential in
running engineering complex structures and complex economy aggregates. Optimization traces its
roots to calculus of variations and the work of Euler (1707–1783) and Lagrange (1736–1813).
Optimization is often called mathematical programming, a somewhat confusing term introduced by
Tjalling Koopmans (1910–1985) and consolidated inter alia by Leonid Kantorovich (1912–1986),
George Dantzig (1914–2005), and by some others (see Lenstra, Rinnooy Kan, and Schrijver (1991),
1.8 The Structure of the Book 19

Gr€otschel (2012)). The mathematical theory of optimization is found in many books: Luenberger
(1973, 1984), Bazaraa, Sherali, and Shetty (1993), Bertsekas (1999), Boyd and Vandenberghe (2004),
Nocedal and Wright (2006), Sun and Yuan (2006), Bartholomew-Biggs (2008), Luenberger and Ye
(2016), and Andrei (2013e, 2015a), to cite only some of them. Now, the professional optimization
algorithms work under the assistance of certain modern systems of modeling and optimization like
GAMS, AMPL, TOMLAB, ALLO, etc.
 Fundamentals on Unconstrained
Optimization. Stepsize Computation 2

Unconstrained optimization consists of minimizing a function which depends on a number of real

variables without any restrictions on their values. When the number of variables is large, this problem
becomes quite challenging. The methods for the unconstrained optimization are iterative. They start
with an initial guess of the variables and generate a sequence of improved estimates of the minimum
point until they terminate with a set of values for variables. At every iteration xk an optimization
algorithm computes the search direction dk in the current point, which must be a descent one, and a
stepsize αk taken along the search direction to obtain a new estimation of the minimum point,
xkþ1 ¼ xkþαkdk. The purpose of this chapter is to present the most important modern methods as
well as their convergence properties for computing the stepsize αk, which is a crucial component of
any unconstrained optimization algorithm. For checking if this set of values of variables is indeed the
solution of the problem, the optimality conditions should be used. If the optimality conditions are not
satisfied, they may be used to improve the current estimate of the solution. The algorithms described
in this book utilize the values of the minimizing function, of the first and possibly of the second
derivatives of this function. Other methods based only on function’s values (Golden-section search,
Fibonacci, compass search, dichotomous search, quadratic interpolation – see (Antoniou, & Lu,
2007) are not considered in this book. We emphasize and recommend that the reader study the
mathematical concepts and results included in Appendix A.

2.1 The Problem

In this book, the following unconstrained optimization problem

min f ðxÞ ð2:1Þ
x2ℝn

is considered, where f : ℝn ! ℝ is a real valued function f of n variables, smooth enough on ℝn. The
interest is to find a local minimizer of this function, that is a point x, so that
f ðx Þ  f ðxÞ for all x near x : ð2:2Þ

If f(x) < f(x) for all x near x, then x is called strict local minimizer, or simple minimizer of
function f. Often, f is referred to as the objective function, while f(x) as the minimum or the minimum
value.

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 21

N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_2
22 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

We emphasize that the local minimization problem is different from the global minimization
problem, where a global minimizer, i.e., a point x so that
f ðx Þ  f ðxÞ for all x 2 ℝn ð2:3Þ

is sought. This book deals with only the local minimization problems.
The function f in (2.1) may have any algebraic expression, and we suppose that it is twice
continuously differentiable on ℝn. Let us introduce ∇f(x) as the gradient of f and ∇2f(x) its Hessian.
Often, the gradient is denoted as g(x) ¼ ∇ f(x).
2 3 2 2 2 3
∂f ðxÞ ∂ f ðx Þ ∂ f ðx Þ
6 ∂x2 ⋯
6 ∂x1 7 6 ∂x1 ∂xn 77
6 7 6 1
7
gðxÞ  ∇f ðxÞ ¼ 66 ⋮ 7
7 , ∇ 2
f ð x Þ ¼ 6 ⋮ ⋱ ⋮ 7:
4 ∂f ðxÞ 5 6 7
4 ∂ 2 f ðx Þ ∂ f ðx Þ 5
2

∂xn ⋯
∂xn ∂x1 ∂x2n

For solving (2.1), plenty of methods are known (see Luenberger, 1973, 1984; Gill, Murray, &
Wright, 1981; Bazaraa, Sherali, & Shetty, 1993; Bertsekas, 1999; Nocedal, & Wright, 2006; Sun, &
Yuan, 2006; Bartholomew-Biggs, 2008; Griva, Nash, & Sofer, 2009; Andrei, 1999a, 2009e, 2015a,
2020a, 2021a). In general, for solving (2.1), the unconstrained optimization methods implement one
of the following two strategies: the line-search and the trust-region. Besides these two strategies, for
problems with a small number of variables (let us say 20), the direct methods are used. The direct
methods use only the values of the minimizing function without appealing to the differential
operators: the gradient or the Hessian. Often, the direct methods are called derivative-free optimiza-
tion or the zero-th order method. There are a lot of direct methods (Rios, & Shainidis, 2013; Larson,
Menickelly, Wild, 2019; Andrei, 2021a), etc.
In the line-search strategy, the corresponding algorithm chooses a direction dk and searches along
this direction from the current iterate xk for a new iterate with a lower function value. Specifically,
starting with an initial point x0, the iterations are generated as
xkþ1 ¼ xk þ αk dk , k ¼ 0, 1, . . . , ð2:4Þ

where dk 2 ℝn is the search direction along which the values of function f are reduced and αk 2 ℝ is
the stepsize determined by a line-search procedure. The main requirement is that the search direction
dk, at iteration k should be a descent direction. It is proved that the algebraic characterization of
descent directions is that

dTk gk < 0, ð2:5Þ

which is a very important criterion concerning the effectiveness of an algorithm. In (2.5), gk ¼ ∇ f(xk)
is the gradient of f in point xk. In order to guarantee the global convergence, sometimes it is required
that the search direction dk satisfy the sufficient descent condition

gTk dk  ckgk k2 , ð2:6Þ

where c is a positive constant.

In the trust-region strategy, the idea is to use the gathered information about the minimizing
function f to construct a model function mk whose behavior near the current point xk is similar to that
of the actual objective function f. In other words, the step p is determined by approximately solving
the following subproblem:
2.2 Fundamentals on the Convergence of the Line-Search Methods 23

min mk ðxk þ pÞ, ð2:7Þ

p

where the point xk þ p lies inside the trust region. If the step p does not produce a sufficient reduction
of the function values, then it follows that the trust-region is too large. In this case the trust-region is
shrinked and the model mk in (2.7) is re-solved. Usually, the trust-region is a ball defined by kpk2  Δ,
where the scalar Δ is known as the trust-region radius. Of course, elliptical and box-shaped trust
regions may be used.
Usually, the model mk in (2.7) is defined as a quadratic approximation of the minimizing
function f:
1
mk ðxk þ pÞ ¼ f ðxk Þ þ pT ∇f ðxk Þ þ pT Bk p, ð2:8Þ
2
where Bk is either the Hessian ∇2f(xk) or an approximation of it. Observe that whenever the size of the
trust-region, i.e., the trust-region radius, is reduced after a failure of the current iterate, the step from
xk to the new point will be shorter and usually run to a different direction from the previous point.
By comparison, the line-search and the trust-region differ in the order in which they choose the
search direction and the stepsize to move to the next iterate. The line-search starts with a descent
direction dk and then determines an appropriate distance along this direction, namely, the stepsize αk.
In the trust-region, the maximum distance is first chosen, that is the trust-region radius Δk. Then a
direction and a step pk that determine the best improvement of the function values subject to this
distance constraint are determined. If this step is not satisfactory, then the distance measure Δk is
reduced, and the process is repeated.
For solving the unconstrained optimization problems, the general principle of the methods based
on the differential operators (the gradient and the Hessian) consists in the construction in the current
point of an affine model or a quadratic model of the minimizing function, in solving this model and in
considering its solution as the next approximation of the minimum. This process is then repeated from
this new approximation of the minimum point. In the next section, in order to establish the validity of
this approach, it is essential to see the bounds of the errors corresponding to the models of the
minimizing function and to notice some fundamentals on the convergence of the line-search methods.

2.2 Fundamentals on the Convergence of the Line-Search Methods

From the very beginning, two forms of the fundamental theorem of calculus are presented: one for the
function-gradient and the other for the gradient-Hessian.

Theorem 2.1 Let f : ℝn ! ℝ be twice continuously differentiable in a neighborhood of a line

segment connecting the points x 2 ℝn and x 2 ℝn; then

ð1
f ðxÞ ¼ f ðx Þ þ ∇f ðx þ teÞedt

ð2:9Þ
0

and
24 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

ð1
∇f ðxÞ ¼ ∇f ðx Þ þ ∇2 f ðx þ teÞedt, ð2:10Þ
0

where e ¼ x  x. ♦

A direct consequence of Theorem 2.1 is the following form of Taylor’s theorem, which is one of
the most used results in optimization.

Theorem 2.2 Let f : ℝn ! ℝ be twice continuously differentiable in a neighborhood of a point

x 2 ℝn. Then for e 2 ℝn and kek sufficiently small it follows that
 
1
f ðxÞ ¼ f ðx Þ þ ∇f ðx ÞT e þ eT ∇2 f ðx Þe þ o kek2 : ♦ ð2:11Þ
2

Necessary Conditions for Optimality

The conditions which are satisfied at a local minimizer x are necessary conditions. Let f : ℝn ! ℝ be
twice continuously differentiable. In the following, based on Taylor’s theorem, we show that the
gradient of the minimizing function f vanishes at a local minimum point and the Hessian of f is
positive semidefinite. These are the necessary conditions for optimality. It is important to notice that
the necessary conditions of (2.1) are in close connection with a nonlinear algebraic system which may
be used to get fast and reliable algorithms for computing the minimum points of function f. The
following theorem gives the necessary conditions for the optimality of x.

Theorem 2.3 Let f : ℝn ! ℝ be twice continuously differentiable and let x be a local minimum of f.
Then
∇f ðx Þ ¼ 0:

Moreover, ∇2f(x) is positive semidefinite.

Proof Let u 2 ℝn be an arbitrary vector. Then Taylor’s theorem says that for all real t sufficiently
small

t2 T 2   
f ðx þ tuÞ ¼ f ðx Þ þ t∇f ðx ÞT u þ u ∇ f ðx Þu þ o t2 :
2
Since x is a local minimizer, then for t sufficiently small it follows that f(x þ tu)  f(x)  0.
Hence, for t sufficiently small and all u 2 ℝn
t
∇f ðx ÞT u þ uT ∇2 f ðx Þu þ oðtÞ  0: ð2:12Þ
2
Now, if we set t ¼ 0 and u ¼  ∇ f(x) we obtain

k∇f ðx Þk2 ¼ 0:

Setting in (2.12) ∇f(x) ¼ 0, dividing by t and setting t ¼ 0, it follows that

2.2 Fundamentals on the Convergence of the Line-Search Methods 25

1 T 2 
u ∇ f ðx Þu  0
2
for all u 2 ℝn. ♦

The condition ∇f(x) ¼ 0 is known as the first-order necessary optimality condition for uncon-
strained optimization, while a point satisfying this condition is called a stationary point or a critical
point of the minimizing function f. Observe that the condition ∇f(x) ¼ 0 is a nonlinear algebraic
system. Therefore, to compute the minimizers of a function f we can use fast algorithms for solving
nonlinear algebraic systems.

Sufficient Conditions for Optimality

The conditions which guarantee that x is a local minimizer are the sufficient conditions. Obviously,
the gradient of the minimizing function f also vanishes at a maximum point of f. In order to make a
clear distinction between minima and maxima, the sufficient conditions for optimality are introduced.
As it is known, to get a minimizer, the second derivative must be nonnegative. But, this alone is not
sufficient. To make sure that we have a minimum point, we must require that the second derivative is
strictly positive.

Theorem 2.4 Let f : ℝn ! ℝ be twice continuously differentiable in a neighborhood of x. Assume

that ∇f(x) ¼ 0 and that ∇2f(x) is positive definite. Then, x is a local minimizer of function f.

Proof Let u 2 ℝn be an arbitrary vector with u 6¼ 0. Then, for sufficiently small t we can write

t2 T 2   
f ðx þ tuÞ ¼ f ðx Þ þ t∇f ðx ÞT u þ u ∇ f ðx Þu þ o t2
2
t2  
¼ f ðx Þ þ uT ∇2 f ðx Þu þ o t2 :
2
Now, if λ > 0 is the smallest eigenvalues of ∇2f(x), it follows that for t sufficiently small
λ  
f ðx þ tuÞ  f ðx Þ  ktuk2 þ o t2 > 0:
2
Therefore x is a local minimizer of function f. ♦

Other details on the optimality conditions for the unconstrained optimization are given in
Chap. 11.

Bounds of the Distance Between the Approximations of Function f and the Function Value
in a Point
Let us consider function f twice continuously differentiable. In the following, let us present some
bounds on the distance between the local affine model (local affine approximation):

m1 ðxÞ ¼ f ðxÞ þ ∇f ðxÞT d, ð2:13Þ

or the local quadratic model (local quadratic approximation):

1
m2 ðxÞ ¼ f ðxÞ þ ∇f ðxÞT d þ dT ∇2 f ðxÞd ð2:14Þ
2
26 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

of the minimizing function and the function value in the point x þ d, where d is the searching
direction. From the very beginning, consider the differential function f : ℝ ! ℝ. Let D be an open,
convex set in ℝ.

Definition 2.1 A function f is Lipschitz continuous on D with the constant L if for any x, y 2 D,
|f(x)  f( y)|  Lkx  yk.
The following theorem shows a fundamental result from numerical computation by specifying that
if f 0(x) is Lipschitz continuous, then there is a bound of the distance between the local affine
approximation f(x) þ f 0(x)(y  x) and the function value f( y).

Theorem 2.5 Let f : D ⊂ ℝ ! ℝ be a function for which f0 is Lipschitz continuous on D with

constant L. Then, for any x, y 2 D the following estimation is true:
L
jf ðyÞ  f ðxÞ  f 0 ðxÞðy  xÞj  ðy  xÞ2 : ð2:15Þ
2

Proof Obviously
ðy
f ðyÞ  f ðxÞ ¼ f 0 ðzÞdz,
x

or
ðy
f ðyÞ  f ðxÞ  f ðxÞðy  xÞ ¼ ½f 0 ðzÞ  f 0 ðxÞdz:
0

Now, consider z ¼ x þ t(y  x), where dz ¼ (y  x)dt. Therefore,

ð1
f ðyÞ  f ðxÞ  f 0 ðxÞðy  xÞ ¼ ½f 0 ðx þ tðy  xÞÞ  f 0 ðxÞðy  xÞdt:
0

From the triangle inequality we get

Ð1
jf ðyÞ  f ðxÞ  f 0 ðxÞðy  xÞj  jy  xj Ljtðy  xÞjdt ¼ L2 jy  xj2 : ♦
0

Observe that (2.15) resembles the bound of the error from Taylor’s series with rest, in which the
00
Lipschitz constant L is a bound of the |f (ξ)|, for ξ 2 D. The main advantage of the Lipschitz
continuity is that, for the analysis of algorithms, the derivatives of high order need not be discussed.
In the following, let us present a bound of the distance between the local quadratic model and the
function value in a point.

Theorem 2.6 Let f : ℝn ! ℝ be a twice continuously differentiable function on the convex and open
set D ⊂ ℝn. Assume that ∇2f(x) is Lipschitz continuous in x 2 D with constant L  0. Then, for any
x þ d 2 D it follows that
2.2 Fundamentals on the Convergence of the Line-Search Methods 27

 h i
 T 1  L
f ðx þ dÞ  f ðxÞ þ ∇f ðxÞ d þ dT ∇2 f ðxÞd   kdk :
3
ð2:16Þ
2 6
The proof of this theorem is similar to the proof of Theorem 2.5, and it is left as an exercise for the
reader. ♦

Nonlinear Algebraic Systems

In the following, let us consider the simple case of nonlinear algebraic systems defined by the
vectorial function F : ℝn ! ℝm, where F ¼ [f1(x), . . ., fm(x)]T and m ¼ n. A continuous vectorial
function is continuously differentiable at x 2 ℝn if each component function fi(x), i ¼ 1, . . .m, is
continuous differentiable at x. The derivative of F at x is called the Jacobian matrix of F at x and is
denoted as F0(x) ¼ J(x) ¼ ∇ F(x)T.
Obviously

∂f i ðxÞ
½F0 ðxÞij ≜ ½J ðxÞij ¼ , i ¼ 1, . . . , m, j ¼ 1, . . . , n:
∂xj

For continuous vectorial functions, there is no mean value theorem. That is, in general, there may
not exist a point z 2 ℝn such that F(x þ d) ¼ F(x) þ J(z)d. Even if each component fi(x) satisfies
fi(x þ d ) ¼ fi(x) þ ∇ fi(zi)Td, the points zi may be different.
By using the definition of Jacobian, it follows that if F : ℝn ! ℝm is continuously differentiable on
an open set D ⊂ ℝn, then for any x, x þ d 2 ℝn, we have

ð1 ð
xþd

Fðx þ d Þ  FðxÞ ¼ J ðx þ tdÞddt ¼ F0 ðξÞdξ: ð2:17Þ

0 x

Definition 2.2 The function F : D ⊂ ℝn ! ℝm is H€ older continuous on D if there exists the

constants L  0 and p 2 (0, 1] such that for any x, y 2 D,

kFðxÞ  FðyÞk  Lkx  ykp :

If p ¼ 1, then F is Lipschitz continuous on D with the constant L.

For solving the algebraic nonlinear system F(x) ¼ 0 the same general principle is used to
approximate the function F by an affine or a quadratic model. The following results show the bounds
of the distance between the approximations of function F, affine or quadratic, and the function value
in a point.

Theorem 2.7 Let F : ℝn ! ℝm be continuously differentiable on the convex open set D ⊂ ℝn.
Assume that J is Lipschitz continuous at x 2 D with the constant L  0. Then, for any x þ d 2 D it
follows that:
L
kFðx þ dÞ  FðxÞ  J ðxÞdk  kd k2 : ð2:18Þ
2
28 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Proof We have

ð1
Fðx þ d Þ  FðxÞ  J ðxÞd ¼ J ðx þ td Þddt  J ðxÞd
0
ð1
¼ ½J ðx þ td Þ  J ðxÞddt:
0

Therefore,

ð1
kFðx þ dÞ  FðxÞ  J ðxÞdk  kJ ðx þ td Þ  J ðxÞkkdkdt
0
ð1 ð1
2 L
 Lktd kkdkdt ¼ Lkdk tdt ¼ kdk2 : ♦
2
0 0

Theorem 2.8 Let F : ℝn ! ℝm be a continuously differentiable function on the open convex set
D ⊂ ℝn. Then for any u, v, x 2 D it follows that
 
kFðuÞ  FðvÞ  J ðxÞðu  vÞk  sup kJ ðv þ tðu  vÞÞ  J ðxÞk ku  vk: ð2:19Þ
0t1

Moreover, assuming that J is Lipschitz continuous on D, then

kFðuÞ  FðvÞ  J ðxÞðu  vÞk  Lσ ðu, vÞku  vk ð2:20Þ

and
ku  x k þ kv  x k
kFðuÞ  FðvÞ  J ðxÞðu  vÞk  L ku  vk, ð2:21Þ
2
where σ(u, v) ¼ max {ku  xk, kv  xk}.

Proof From (2.17) and the mean value theorem it follows that
1
ð

kFðuÞ  FðvÞ  J ðxÞðu  vÞk ¼
½ J ðv þ t ð u  v Þ Þ  J ð x Þ ð u  v Þdt


0
ð1  
 kJ ðv þ tðu  vÞÞ  J ðxÞkku  vkdt  sup kJ ðv þ tðu  vÞÞ  J ðxÞk ku  vk,
0t1
0

which is exactly (2.19). Now, since J is Lipschitz on D, then, as above, we get

2.3 The General Algorithm for Unconstrained Optimization 29

ð1
kFðuÞ  FðvÞ  J ðxÞðu  vÞk  L kv þ tðu  vÞ  xkku  vkdt
0
¼ L sup kv þ tðu  vÞ  xkku  vk ¼ Lσ ðu, vÞku  vk
0t1

which is exactly (2.20). The result (2.21) is proved in the same manner. ♦

Theorem 2.9 Let F and J which satisfy the conditions of Theorem 2.7. Suppose that there exists
J(x)1. Then, there exist ε > 0 and β > α > 0, such that for all u, v 2 D, when max{ku  xk, kv  xk}  ε,
it follows that
αku  vk  kFðuÞ  FðvÞk  βku  vk: ð2:22Þ

Proof From the triangle inequality and from (2.20)

kFðuÞ  FðvÞk  kJ ðxÞðu  vÞk þ kFðuÞ  FðvÞ  J ðxÞðu  vÞk
 ½kJ ðxÞk þ Lσ ðu, vÞku  vk  ½kJ ðxÞk þ Lεku  vk:

Let β ¼ kJ(x)k þ Lε. Then, we obtain the inequality on the right side of (2.22). Similarly,
kFðuÞ  FðvÞk  kJ ðxÞðu  vÞk  kFðuÞ  FðvÞ  J ðxÞðu  vÞk
h i h i
 1= J ðxÞ1  Lσ ðu, vÞ ku  vk  1= J ðxÞ1  Lε ku  vk:

Therefore, if 1/(kJ(x)1kL ) > ε, then we obtain the inequality on the left side of (2.22), where
α ¼ 1/kJ(x)1k  Lε > 0. ♦

In conclusion, if x þ d 2 ℝn is close to x, then the affine approximation (2.13) or the quadratic

approximation (2.14) of function f in x þ d are very good approximations and may be used in an
iterative scheme for solving (2.1). The iterative scheme involves solving these approximations (affine
or quadratic), solution of which is considered the next approximations to the minimum point x. To
ensure the convergence of the iterative scheme, some conditions have to be introduced, which will be
presented and discussed in the following sections.

2.3 The General Algorithm for Unconstrained Optimization

Let f : ℝn ! ℝ be a twice continuous differentiable function bounded from below. As we have

already seen, at iteration k, given the current point xk and a descent search direction dk, any
optimization unconstrained algorithm computes the next approximation of the minimum point x
by the iterative scheme

xkþ1 ¼ xk þ αk dk , ð2:23Þ

where αk is the stepsize along the direction dk. We mention that a descent direction dk 6¼ 0 in the point
xk satisfies the condition ∇f(xk)Tdk < 0, called descent condition. Geometrically, the descent condition
means that the angle between dk and the negative gradient must be smaller than 90 . If αk 6¼ 0, we see
that (2.23) satisfies the general principle of numerical computation, which recommends that a new
30 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

estimation of the solution of a problem should be computed as a small highly accurate modification of
the previous estimation.
To start the computations, this iterative scheme must be initialized with an initial point x0. This
point has to be in domf and additionally it is required that the level set S ¼ {x 2 domf : f(x)  f(x0)} be
closed. This closeness of the level set ensures the elimination of the possibility that the algorithm
converges to a point from the frontier of domf.
The purpose of this section is to develop methods for an efficient computation of the scalar αk. Let

φðαÞ ¼ f ðxk þ αd k Þ: ð2:24Þ

Now, the problem is starting from xk, determine the stepsize αk in direction dk, such that
φ(αk) < φ(0). This is known as the line-search. For this line-search, two possibilities are more
important.

1. If αk is determined as the minimum of function f in the direction dk, that is

f ðxk þ αk dk Þ ¼ min f ðxk þ αdk Þ ð2:25Þ

α>0

or

φðαk Þ ¼ min φðαÞ, ð2:26Þ

α>0

then we say that the line-search is exact (or perfect).

2. On the other hand, if αk is selected such that an acceptable reducing of the objective is obtained
along the direction dk in point xk, i.e., f(xk)  f(xk þ αkdk) > 0 is acceptable, then we say that the
line-search is inexact (or acceptable).

Due to highly numerical costs, the exact line-search is not used for solving optimization problems,
apart from some particular cases, such as the quadratic objective. In the vast majority of situations, the
inexact line-search is used. The structure of an unconstrained optimization algorithm based on
descent directions is as follows.

Algorithm 2.1 General unconstrained optimization algorithm

1. Initialization. Consider an initial point x0 2 ℝn, as well as the convergence tolerance 0 < ε 1
sufficiently small. Set k ¼ 0
2. Compute the descent direction dk
3. Using the exact or the inexact line-search, compute the stepsize αk
4. Compute xkþ1 ¼ xk þ αkdk
5. Test a criterion for stopping the iterations. For example, if k∇f(xk)k  ε, then stop; otherwise set k ¼ k þ 1
and go to step 2 ♦

As we can see, the algorithm is very general and plenty of aspects remain to be clarified. For
example, in step 2 of the algorithm, it is not specified how the descent direction dk is computed.
However, this will be presented in the next chapters, where different procedures for the dk computa-
tion are discussed, for example, steepest descent, Newton, conjugate gradient, quasi-Newton, limited
memory L-BFGS, inexact (truncated) Newton, etc. In step 3, the stepsize αk can be computed by the
exact line-search or by the inexact line-search. In step 5 it is possible to implement other criteria for
stopping the iterations, etc.
2.4 Convergence of the Algorithm with Exact Line-Search 31

2.4 Convergence of the Algorithm with Exact Line-Search

In this section we are interested in the convergence of Algorithm 2.1, in which the stepsize is
computed by the exact line-search (2.25) or (2.26). Let φ(α) ¼ f(xk þ αdk). Observe that
φ(0) ¼ f(xk) and φ(α)  φ(0). Condition (2.25) is very exigent and involves the computation of the
global minimum of function φ(α), which is a very difficult task. Therefore, instead, we limit our
search to the first stationary point of φ, i.e., determine αk as
n o
αk ¼ min α  0 : ∇f ðxk þ αd k ÞT d k ¼ 0 : ð2:27Þ

Since by (2.25) the exact minimum of function f is obtained and by (2.27) a stationary point of
function φ is determined, then the conditions (2.25) and (2.27) determine the exact line-search.
Now, consider hdk,  ∇ f(xk)i as the angle between the vectors dk and  ∇ f(xk). Then

d Tk ∇f ðxk Þ
cos hd k , ∇f ðxk Þi ¼  : ð2:28Þ
kd k kk∇f ðxk Þk

The following theorem gives a bound of the reduction of the values of function f at every iteration
of Algorithm 2.1 with exact line-search.

Theorem 2.10 Let αk > 0 be solution of the problem (2.25) and assume that k∇2f(xk þ αdk)k  M for
any α > 0, where M is a positive constant. Then
1
f ðxk Þ  f ðxk þ αk d k Þ  k∇f ðxk Þk2 cos 2 hdk , ∇f ðxk Þi: ð2:29Þ
2M

Proof From the hypothesis of the theorem, for any α > 0, it follows that

α2
f ðxk þ αdk Þ  f ðxk Þ þ αdTk ∇f ðxk Þ þ M kd k k2 : ð2:30Þ
2
Now, define

dTk ∇f ðxk Þ
α¼ :
M kd k k2

Then, using (2.30) and (2.28) we get

α2
f ðxk Þ  f ðxk þ αk dk Þ  f ðxk Þ  f ðxk þ αdk Þ  αdTk ∇f ðxk Þ  M kd k k2
2
 T 2  T 2
1 d k ∇f ðxk Þ 1 2 dk ∇f ðxk Þ
¼ ¼ k∇f ðxk Þk
2 M kd k k2 2M kdk k2 k∇f ðxk Þk2
1
¼ k∇f ðxk Þk2 cos 2 hd k , ∇f ðxk Þi ♦
2M

Observe that the theorem says that, at every iteration, the reduction of the values of function f is
more emphasized when the angle hdk,  ∇ f(xk)i is more acute. The following two theorems are two
expressions of the same result that shows the convergence of Algorithm 2.1.
32 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Theorem 2.11 Let f(x) be a continuously differentiable function on the open set D ⊂ ℝn and
suppose that the direction dk is descent, i.e., dTk ∇f ðxk Þ  0, and the sequence {xk} generated by
Algorithm 2.1 with exact line-search satisfies a condition of monotony f(xkþ1)  f(xk). Let x 2 D be an
accumulation point of {xk} and a set K1 of indices with K 1 ¼ fk : lim xk ¼ xg. Suppose that there is a
positive constant M > 0 such that for all k 2 K1, kdkk < M. If d is an arbitrary accumulation point of
the sequence {dk}, then
T
d ∇f ðxÞ ¼ 0: ð2:31Þ

Moreover, if f(x) is twice continuous differentiable, then

T
d ∇2 f ðxÞd  0: ð2:32Þ

Proof We prove only (2.31). Obviously, (2.32) may be proved similarly. Consider K2 ⊂ K1 a set of
indices for which d ¼ lim k2K2 dk . If d ¼ 0, then (2.31) is trivial satisfied, otherwise consider the
following two cases.

(i) There is a set of indices K3 ⊂ K2 such that lim k2K 3 αk ¼ 0. Since αk is the exact stepsize, it
follows that dTk ∇f ðxk þ αk dk Þ ¼ 0. Moreover, since kdkk is uniformly bounded from above and
T
αk ! 0, at limit we have that d ∇f ðxÞ ¼ 0.
(ii) Now we refer to the case lim inf k2K2 αk ¼ α > 0. Let K4 ⊂ K2 be a set of indices for which
T
αk  α=2 . Suppose that (2.31) is not true, then d ∇f ðxÞ < δ < 0 . Therefore, there is a
neighborhood N ðxÞ of x and a set of indices K5 ⊂ K4 such that as soon as x 2 N ðxÞ and
k 2 K5, dTk ∇f ðxÞ  δ=2 < 0. Let bα be a positive number sufficiently small such that for all
0αb α and all k 2 K5, xk þ αdk 2 N ðxÞ. Consider α ¼ min fα=2, b αg. Then, by using the
monotony property of the algorithm, the exact line-search and the development in Taylor’s series
we have
P
1 P
f ðx Þ  f ðx 0 Þ ¼ ½f ðxkþ1 Þ  f ðxk Þ  ½f ðxkþ1 Þ  f ðxk Þ
k¼0 k2K 5
P
 ½f ðxk þ α dk Þ  f ðxk Þ
k2K 5
P ð2:33Þ
¼ α ∇f ðxk þ τk dk ÞT d k
k2K 5
P  
δ 
  α ¼ 1,
k2K 5 2

where 0  τk  α. The above contradiction shows that (2.31) is also true in case (ii).
The proof of (2.32) is similar. Instead of (2.33) it is sufficient to use the development in Taylor’s
series up to the second order to get
2.4 Convergence of the Algorithm with Exact Line-Search 33

P
f ðxÞ  f ðx0 Þ  ½f ðxk þ α d k Þ  f ðxk Þ
k2K 5
 
P  T ðα Þ2 T 2
¼ α ∇f ðxk Þ d k þ dk ∇ f ðxk þ τk d k Þdk
k2K 5 2
P ðα Þ2 T 2
 d k ∇ f ðxk þ τk d k Þdk
k2K 5 2
P h 1 δ  2 i
  ðα Þ ¼ 1:
k2K 5 2 2

Therefore, we get a contradiction which proves (2.32). ♦

Theorem 2.12 Let ∇f(x) be uniformly continuous on the level set S ¼ {x 2 ℝn : f(x)  f(x0)}. Also, let
θk ¼ hdk,  ∇ f(xk)i, where dk is the search direction generated by Algorithm 2.1 with exact line-
search. If
π
θk   μ, for certain μ > 0, ð2:34Þ
2
then either ∇f(xk) ¼ 0 for certain k, or f(xk) !  1 , or ∇f(xk) ! 0.

Proof Suppose that for all k, ∇f(xk) 6¼ 0 and the function f(xk) is bounded from below. Since {f(xk)} is
a sequence decreasing monotonically, then its limit exists. Hence,

f ðxk Þ  f ðxkþ1 Þ ! 0: ð2:35Þ

By contradiction, suppose that ∇f(xk) ! 0 is not true. Then, there exists an ε > 0 such that
k∇f(xk)k  ε. Therefore,

∇f ðxk ÞT dk
 ¼ k∇f ðxk Þk cos θk  ε sin μ ≜ ε1 ð2:36Þ
kd k k

Observe that

f ðxk þ αdk Þ ¼ f ðxk Þ þ α∇f ðξk ÞT d k

¼ f ðxk Þ þ α∇f ðxk ÞT d k þ α½∇f ðξk Þ  ∇f ðxk ÞT dk

 ð2:37Þ
∇f ðxk ÞT d k
 f ðxk Þ þ αkd k k þ k∇f ðξk Þ  ∇f ðxk Þk ,
kd k k

where ξk is on the line segment determined by xk and xk þ αdk. Since ∇f(x) is uniformly continuous on
the level set S, it follows that there exists an α such that when 0  αkd k k  α, then
1
k∇f ðξk Þ  ∇f ðxk Þk  ε1 : ð2:38Þ
2
From (2.35, 2.36, 2.37, and 2.38) we get



dk ∇f ðxk ÞT dk 1 1
f xk þ α  f ðxk Þ þ α þ ε1  f ðxk Þ  αε1 ,
kd k k kd k k 2 2

i.e.,
34 2 Fundamentals on Unconstrained Optimization. Stepsize Computation


d 1
f ðxkþ1 Þ  f xk þ α k  f ðxk Þ  αε1 ,
kd k k 2

which contradicts (2.35). Therefore, ∇f(xk) ! 0, thus proving the theorem. ♦

In the following, let us present the rate of convergence of Algorithm 2.1 with exact line-search. For
this, we need some technical results proved in the following three propositions.

Proposition 2.1 Let φ(α) be at least twice continuously differentiable on the closed interval [0, b]
and φ0(0) < 0. If the minimum of φ(α) on [0, b] is α 2 (0, b), then
α  e
α ¼ φ0 ð0Þ=M, ð2:39Þ
00
where M is a positive number such that φ (α)  M, for all α 2 [0, b].

Proof Consider the auxiliary function ψ(α) ¼ φ0(0) þ Mα which has a unique zero e
α ¼ φ0 ð0Þ=M.
00
But, on [0, b], φ (α)  M, therefore
ðα ðα
0 0 00 0
φ ðαÞ ¼ φ ð0Þ þ φ ðσ Þdσ  φ ð0Þ þ Mdσ ¼ ψðαÞ:
0 0

 0 
Now, taking α ¼ α and noticing that φ (α ) ¼ 0, from the above inequality it follows that
0  ψ(α) ¼ φ0(0) þ Mα, thus proving the proposition. ♦

Proposition 2.2 Let f(x) be twice continuously differentiable on ℝn. Then, for all x, d 2 ℝn and for
any scalar α it follows that

ð1

f ðx þ αdÞ ¼ f ðxÞ þ α∇f ðxÞT d þ α2 ð1  tÞ dT ∇2 f ðx þ tαdÞd dt: ð2:40Þ
0

Proof We have
ð1 ð1 h i
f ðx þ αdÞ  f ðxÞ ¼ df ðx þ tαdÞ ¼  α∇f ðx þ tαdÞT d dð1  tÞ
0 0
h i1 ð1 h i
¼  ð1  tÞα∇f ðx þ tαd Þ d þ ð1  tÞd α∇f ðx þ tαdÞT d
T
0
0
ð1

¼ α∇f ðxÞT d þ α2 ð1  tÞd T ∇2 f ðx þ tαd Þd dt ♦
0

Proposition 2.3 Let f(x) be a twice continuously differentiable function in a neighborhood of the
minimum x. Suppose that there exist the constants ε > 0 and M > m > 0 such that for any y 2 ℝn, as
soon as kx  xk < ε,
2.4 Convergence of the Algorithm with Exact Line-Search 35

mkyk2  yT ∇2 f ðxÞy  Mkyk2 : ð2:41Þ

Then
1 1
mkx  x k2  f ðxÞ  f ðx Þ  Mkx  x k2 ð2:42Þ
2 2
and

k∇f ðxÞk  mkx  x k: ð2:43Þ

Proof From Proposition 2.2 it follows that

ð1
f ðxÞ  f ðx Þ ¼ ∇f ðx Þ ðx  x Þ þ ð1  tÞðx  x ÞT ∇2 f ðtx þ ð1  tÞx Þðx  x Þdt
  T 

0
ð2:44Þ
ð1
¼ ð1  tÞðx  x Þ ∇ f ðtx þ ð1  tÞx Þðx  x Þdt:
 T 2

Now, from (2.41) and the mean value theorem we get

ð1 ð1
m kx  x k  2
ð1  tÞdt  ð1  tÞðx  x ÞT ∇2 f ðtx þ ð1  tÞx Þðx  x Þdt
0 0
ð2:45Þ
ð1
 Mkx  x k2 ð1  tÞdt:
0

From (2.44) and (2.45) it follows (2.42). For proving (2.43) observe that
ð1
∇f ðxÞ ¼ ∇f ðxÞ  ∇f ðx Þ ¼ ∇2 f ðtx þ ð1  tÞx Þðx  x Þdt:


Therefore,

ð1
k∇f ðxÞkkx  x k  ðx  x Þ ∇f ðxÞ ¼ ðx  x ÞT ∇2 f ðtx þ ð1  tÞx Þðx  x Þdt
  T

0
 m kx  x  k2 ,

which proves (2.43). ♦

The following theorem is central in establishing the rate of convergence of Algorithm 2.1 with
exact line-search. It is proved that, if the minimizing function f is twice continuously differentiable
with bounded Hessian, then the algorithm is at least linear convergent to a local minimum.

Theorem 2.13 Let {xk} be the sequence generated by Algorithm 2.1 with exact line-search,
convergent to the minimum point x of function f(x). Let f(x) be twice continuously differentiable in
36 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

a neighborhood of x and assume that there exist the positive constants ε > 0 and M > m > 0, such that
for all y 2 ℝn, as soon as kx  xk < ε,

mkyk2  yT ∇2 f ðxÞy  Mkyk2 :

Then, the sequence {xk} is at least linear convergent to x.

Proof Consider limk ! 1xk ¼ x. Therefore, for k sufficiently large, we can assume that kxk  xk  ε.
Since kxkþ1  xk < ε, then there exists a positive δ > 0 such that
kxk þ ðαk þ δÞdk  x k ¼ kxkþ1  x þ δdk k < ε: ð2:46Þ

Observe that φ(α) ¼ f(xk þ αdk) and φ0(α) ¼ ∇ f(xk þ αdk)Tdk. But, dk is a descent direction, then
φ (0) ¼ ∇ f(xk)Tdk < 0 and |φ0(0)|  k∇f(xk)kkdkk. Therefore, for a ρ 2 (0, 1) we can write
0

ρk∇f ðxk Þkkdk k  φ0 ð0Þ  k∇f ðxk Þkkdk k

and

φ00 ðαÞ ¼ d Tk ∇2 f ðxk þ αd k Þdk  Mkdk k2 :

From Proposition 2.1 we know that the minimum αk of φ(α) on [0, αk þ δ] satisfies
φ0 ð0Þ ρk∇f ðxk Þk
αk  e
αk ¼  ≜αk : ð2:47Þ
M kd k k2 M kd k k

Consider xk ¼ xk þ αk dk . From (2.46) it follows that kxk  x k < ε. Therefore,

f ðxk þ αk dk Þ f ðxk Þ  f ðxk þ αk d k Þ  f ðxk Þ
ð1

¼ αk ∇f ðxk ÞT dk þ α2k ð1  tÞ dTk ∇2 f ðxk þ tαk dk Þd k dt ðfrom Proposition 2:2Þ
0
1
 αk ðρÞk∇f ðxk Þkkdk k þ Mα2k kdk k2 ðfrom ð2:47ÞÞ
2
ρ2 ρ2 2
 2
k∇f ðxk Þk   m kx k  x  k2 ðfrom ð2:47Þ and ð2:43ÞÞ
2M  2M
  ρm
2 
M ðf ðx k Þ  f ðx ÞÞ ðfrom ð2:42ÞÞ

Hence,
f ðxkþ1 Þ  f ðx Þ ¼ ½hf ðxkþ1 Þ  if ðxk Þ þ ½f ðxk Þ  f ðx Þ
 2 ð2:48Þ
 1  ρm M ½f ðxk Þ  f ðx Þ:

Now, consider

 2 12
ρm
ω¼ 1 :
M

Obviously ω 2 (0, 1). Therefore, (2.48) can be rewritten as

f ðxk Þ  f ðx Þ  ω2 ½f ðxk1 Þ  f ðx Þ  ⋯  ω2k ½f ðx0 Þ  f ðx Þ:

2.5 Inexact Line-Search Methods 37

From (2.42) it follows that

2 2 2 M
kx k  x  k2  ½f ðxk Þ  f ðx Þ  ω2 ½f ðxk1 Þ  f ðx Þ  ω2 kxk1  x k2 ,
m m m 2
i.e.
rffiffiffiffiffi
 M
kxk  x k  ωkxk1  x k,
m

showing that the sequence {xk} is at least linear convergent to x. ♦

The next theorem establishes a bound of the reduction of the values of the minimizing function
f given by Algorithm 2.1 with exact line-search.

Theorem 2.14 Assume that the gradient of the minimizing function f(x) satisfies the following
condition:

ðx  zÞT ½∇f ðxÞ  ∇f ðzÞ  ηkx  zk2 ,

for any x, z 2 ℝn and let αk be the stepsize obtained by the exact line-search. Then
1
f ðxk Þ  f ðxk þ αk dk Þ  ηkαk d k k2 :
2

Proof Since αk is obtained by the exact line-search, it follows that

∇f ðxk þ αk dk ÞT d k ¼ 0:

Therefore, by the mean value theorem we get

αðk

f ðxk Þ f ðxk þ αk d k Þ ¼ d Tk ∇f ðxk þ td k Þdt

0
αðk αðk
2 1
¼ dTk ½∇f ðxk þ αk dk Þ  ∇f ðxk þ td k Þdt  kdk k ηðαk  tÞdt ¼ ηkαk dk k2 ,
2
0 0

proving the theorem. ♦

2.5 Inexact Line-Search Methods

In the previous section we discussed the convergence properties of the general unconstrained
optimization Algorithm 2.1 with exact line-search, i.e., the stepsize αk is computed as solution of
the following minimizing problem

f ðxk þ αk dk Þ ¼ min f ðxk þ αdk Þ, ð2:49Þ

α>0

or
38 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

n o
αk ¼ min α  0 : ∇f ðxk þ αd k ÞT d k ¼ 0 : ð2:50Þ
α>0

The problems (2.49) or (2.50) are not easy to solve, especially when the current point is far away
from the minimum point. On the other hand, for many optimization methods like the Newton or
quasi-Newton, their rate of convergence does not depend on the exact line-search. Usually, these
methods accept a unitary stepsize, and therefore the line-search in these methods is not critical.
Therefore, in Algorithm 2.1, as soon as a stepsize which determines an acceptable reducing of the
values of the minimizing function is obtained, then the exact line-search can be avoided, thus
significantly reducing the computational effort.
The purpose of this section is to present a number of inexact line-search procedures which proved
to be efficient and robust for solving unconstrained optimization problems.

1. Armijo Line-Search
Armijo (1966) introduced the following inexact line-search, known as Armijo rule. Consider the
following scalars β 2 (0, 1), τk ¼  (∇f(xk)Tdk)/kdkk2 and ρ 2 (0, 1/2). Set αk ¼ βmk τk , where mk is
the first nonnegative integer m for which

f ðxk Þ  f ðxk þ βm τk d k Þ  ρβm τk ∇f ðxk ÞT d k , ð2:51Þ

i.e., try m ¼ 0, 1, . . . until the above inequality is satisfied for certain m ¼ mk. The inequality (2.51) is
called the condition of Armijo. In other words, the stepsize is computed successively as αk ¼ τk, βτk,
β2τk, . . . until (2.51) is satisfied. The following theorem shows that there exists an interval for the
stepsizes α computed as above which satisfy the condition of Armijo.

Theorem 2.15 Consider xk, dk 2 ℝn such that dk 6¼ 0, ∇f(xk)Tdk < 0 and ρ 2 (0, 1). Then, there exists
ε ¼ ε(ρ) for which

f ðxk Þ  f ðxk þ αd k Þ  ρα∇f ðxk ÞT d k , ð2:52Þ

for any α 2 (0, ε].

Proof Obviously,
f ðxk þ αdk Þ  f ðxk Þ
0 6¼ ∇f ðxk ÞT dk ¼ lim :
α!0 α

Hence,

f ðxk þ αdk Þ  f ðxk Þ

lim ¼ 1:
α!0 α∇f ðxk ÞT d k

Therefore, there exists an ε > 0 such that for any α 2 (0, ε],

f ðxk þ αdk Þ  f ðxk Þ

0<  ρ,
α∇f ðxk ÞT d k

i.e., the conclusion of the theorem. ♦

2.5 Inexact Line-Search Methods 39

Now, suppose that Armijo line-search is initialized with the value αk ¼ ∇f ðxk ÞT dk =kdk k2. Then,
the following theorem ensures that in mild conditions, the stepsize given by Armijo rule is bounded
from below.

Theorem 2.16 Suppose that dk is a descent direction and ∇f(x) satisfies the Lipschitz continuity
k∇f( y)  ∇ f(x)k  Lky  xk for any x, y 2 S ¼ {x : f(x)  f(x0)}, where L is a positive constant. If the
line-search satisfies the condition (2.51) of Armijo, then
( )
βð1  ρÞ gTk d k
αk  min 1, :
L kd k k2

Proof Let K1 ¼ {k : αk ¼ τk} and K2 ¼ {k : αk < τk}. Then, for any k 2 K1 we have

f k  f kþ1  ρτk ∇f ðxk ÞT dk

and for any k 2 K2,

f k  f kþ1  ραk ∇f ðxk ÞT d k :

Using Armijo’s rule, since for any k 2 K2, αk/β  τk, it follows that for any k 2 K2,


α α
f k  f xk þ k dk < ρ k ∇f ðxk ÞT dk :
β β

Now, using the mean value theorem for the left side of the above inequality, it follows that there
exists a ξk 2 [0, 1] such that

T
αk ρ
∇f xk þ ξk dk dk > ∇f ðxk ÞT dk
β β

for any k 2 K2.

Using the Cauchy-Schwarz inequality, from the Lipschitz continuity it follows that the above
inequality gives


αk αk
Lkdk k  ∇f xk þ ξk d k  ∇f ðxk Þ
2
k d k k
β β
   T
 ∇f xk þ αβk ξk dk  ∇f ðxk Þ d k
 ρ∇f ðxk ÞT dk  ∇f ðxk ÞT d k ¼ ð1  ρÞ∇f ðxk ÞT dk

for all k 2 K2. Combining this with the inequality corresponding to k 2 K1, the conclusion of the
theorem is obtained. ♦

Theorem 2.17 (Termination of Armijo line-search) Let f be continuously differentiable with gradient
∇f(x) Lipschitz continuous with constant L > 0, i.e., k∇(x)  ∇ ( y)k  Lkx  yk, for any x, y from the
level set S ¼ {x : f(x)  f(x0)}. Let dk be a descent direction at xk, i.e., ∇f(x)Tdk < 0. Then, for fixed
γ 2 (0, 1):
40 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

1. The Armijo condition f ðxk þ αdk Þ  f ðxk Þ þ γαgTk dk (easily modified) is satisfied for all α 2

0, αkmax , where

2ðγ  1ÞgTk dk
αkmax ¼ :
Lkdk k22

2. For fixed τ 2 (0, 1) the stepsize generated by the backtracking-Armijo line-search terminates with
( )
2τ ð γ  1 Þg T
d k
αk  min α0k , k
,
Lkd k k22

where α0k is the initial stepsize at iteration k.

Proof Either α0k already satisfies the Armijo condition, or there is a second to the last step in the
Armijo backtracking algorithm which does not yet satisfy the Armijo condition. Therefore, the next
step will multiply this second to the last one by τ, which then satisfies the Armijo condition and the
algorithms stops with αk satisfying the above inequality. ♦

2. Goldstein Line-Search
Goldstein (1965) presented the following rule, known as Goldstein rule. Consider the interval
J ¼ fα > 0 : f ðxk þ αdk Þ < f ðxk Þg: ð2:53Þ

In order to have a sufficient reduction of the values of the minimizing function f, we should select a
value for α which is far away from the end points of the interval. The following two conditions seem
to be reasonable to reach this situation:

f ðxk þ αdk Þ  f ðxk Þ þ ρα∇f ðxk ÞT dk ð2:54Þ

and

f ðxk þ αdk Þ  f ðxk Þ þ ð1  ρÞα∇f ðxk ÞT d k , ð2:55Þ

where 0 < ρ < 1/2.

Obviously, these conditions may be written as

δ1 αk ∇f ðxk ÞT dk  f ðxk þ αk dk Þ  f ðxk Þ  δ2 αk ∇f ðxk ÞT d k , ð2:56Þ

where 0 < δ2 < 1/2 < δ1 < 1. (2.56) is known as Goldstein inexact line-search, or Goldstein rule, or
Goldstein conditions.
Let φ(α) ¼ f(xk þ αdk). Then, (2.54) and (2.55) can be expressed as

φðαk Þ  φð0Þ þ ραk φ0 ð0Þ, ð2:57Þ

φðαk Þ  φð0Þ þ ð1  ρÞαk φ0 ð0Þ: ð2:58Þ

2.5 Inexact Line-Search Methods 41

Observe that ρ < 1/2 is a necessary condition. Otherwise, if, for example, φ(α) is a quadratic
00
function which satisfies φ0(0) < 0 and φ (0) > 0, then the global minimum α of φ satisfies the relation
φðα Þ ¼ φð0Þ þ 12 α φ0 ð0Þ. Therefore, α satisfies (2.57) if and only if ρ < 1/2. This condition ρ < 1/2
has a great effect on the Newton and quasi-Newton methods, allowing α ¼ 1 in these methods and
therefore ensuring the superlinear convergence of these methods.

3. Wolfe Line-Search
It is quite possible that Goldstein conditions (2.54) and (2.55) be too strong and therefore exclude
the minimum points of function φ(α). That is why instead of (2.55), Wolfe (1969, 1971) introduced
the following condition:

∇f ðxk þ αk dk ÞT d k  σ∇f ðxk ÞT dk , ð2:59Þ

where σ 2 (ρ, 1). In terms of function φ, (2.59) can be written as

φ0 ðαk Þ ¼ ∇f ðxk þ αk dk ÞT d k  σ∇f ðxk ÞT dk ¼ σφ0 ð0Þ > φ0 ð0Þ:

The geometrical interpretation of (2.59) is that φ0(αk) computed in an acceptable point must be
greater or equal to a multiple σ 2 (0, 1) of φ0(0). The relations (2.54) and (2.59) define the weak Wolfe
inexact line-search, or the weak Wolfe rule, or the weak Wolfe conditions. In fact, these conditions are
a very subtle extension of Goldstein conditions. Observe that (2.59) can be obtained from the mean
value theorem and from (2.55). Indeed, let αk be a value of the stepsize which verifies (2.55), then

αk ∇f ðxk þ θk αk dk ÞT dk ¼ f ðxk þ αk dk Þ  f ðxk Þ  ð1  ρÞαk ∇f ðxk ÞT dk ,

where θk 2 (0, 1), which is exactly (2.59). Now, let us prove that there is an αk which satisfies the
inexact Wolfe line-search (2.54) and (2.59).

Theorem 2.18 Suppose that f is continuously differentiable and dk is a descent direction for f in xk.
Suppose that f is bounded from below along the ray {xk þ αdk : α  0}. If 0 < ρ < σ < 1, then there
exists an interval [a, b], 0 < a < b, such that the inexact weak Wolfe line-search

f ðxk þ αdk Þ  f ðxk Þ þ ρα∇f ðxk ÞT dk ,

∇f ðxk þ αdk ÞT dk  σ∇f ðxk ÞT d k ,

are satisfied for all α 2 [a, b].

Proof As above, let φ(α) ¼ f(xk þ αdk), α  0. Since φ0(0) < 0 and 0 < ρ < σ < 1, it follows that for
α sufficiently small, φ(α) < φ(0) þ ραφ0(0). But, since φ is bounded from below, this relation is
not satisfied for α sufficiently large. Therefore, if we define α ¼ supfb α > 0 : φðαÞ < φð0Þ þ
ραφ0 ð0Þ, 8α 2 ð0, b
αÞg, then α exists and is finite. Moreover, α satisfies (2.52) with equality, that is
φðαÞ ¼ φð0Þ þ ραφ0 ð0Þ. Using the mean value theorem, it follows that there exists β 2 ð0, αÞ such
that
φðαÞ  φð0Þ φð0Þ þ ραφ0 ð0Þ  φð0Þ
φ0 ðβÞ ¼ ¼ ¼ ρφ0 ð0Þ > σφ0 ð0Þ,
α α
42 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

since σ > ρ and φ0(0) < 0. But, from the continuity of φ0, there exists an interval ða, bÞ ⊂ ð0, αÞ
centered in β such that φ0(α)  σφ0(0) for all α 2 (a, b). Therefore, by construction, for any α 2 (a, b)
the following relations φ(α) < φ(0) þ ραφ0(0) and φ0(α)  σφ0(0) are verified, i.e., the weak Wolfe
conditions. ♦

The first weak Wolfe condition f(xkþαkdk)  f(xk) þ ραk ∇ f(xk)Tdk requires that the stepsize αk
achieve a sufficient reduction of f. That is why (2.54) is also called the condition of sufficient
reduction. In other words, this condition forces that the rate of reducing the function f in the direction
dk should be at least ρ ∇ f(xk)Tdk. In practice, ρ ¼ 104. Observe that any sufficiently small step may
satisfy the condition of sufficient reduction. Consequently, in order to avoid very small steps, totally
inefficient in the line-search, the second condition (2.59) ∇f(xk þ αkdk)Tdk  σ ∇ f(xk)Tdk called the
curvature condition is introduced. This condition ensures a bound of ∇f(xk þ αdk)Tdk. The typical
value of σ is 0.9. In general, the smaller the value of σ, the more accurate the line-search. However,
the smaller σ is, the bigger the computational effort is.

Proposition 2.4 Suppose that dk is a descent direction and ∇f satisfies the Lipschitz condition

k∇f ðxÞ  ∇f ðxk Þk  Lkx  xk k

for all x on the line segment connecting xk and xkþ1, where L is a constant. If the line-search satisfies
the Goldstein conditions (2.56), then
 T 
1  δ 1  gk d k 
αk  : ð2:60Þ
L kd k k2

If the line-search satisfies the weak Wolfe conditions, then

 T 
1  σ  gk d k 
αk  : ð2:61Þ
L kd k k2

Proof If the Goldstein conditions hold, then by (2.56) and by the mean value theorem, we have

δ1 αk gTk dk  f ðxk þ αk d k Þ  f ðxk Þ

¼ αk ∇f ðxk þ ξdk ÞT dk  αk gTk dk þ Lα2k kd k k2 ,

where ξ 2 [0, αk]. From the above inequality we get (2.60).

Subtracting ∇f(xk)Tdk from both sides of (2.59) and by using the Lipschitz condition, it follows that

ðσ  1Þ∇f ðxk ÞT dk  ð∇f ðxkþ1 Þ  ∇f ðxk ÞÞT dk  αk Lkdk k2 :

Observe that dk is a descent direction and σ < 1, therefore (2.61) follows from the above
inequality. ♦

As a comparison, a disadvantage of the Goldstein conditions in contrast to the Wolfe conditions is

that the first inequality from (2.56) may exclude all the minimizers of function φ. However, both these
methods have much in common, and their convergence theory is very similar.
It is worth mentioning that a stepsize computed by the weak Wolfe line-search conditions (2.54)
and (2.59) may not be sufficiently close to a minimizer of φk(α). In these situations, the strong Wolfe
2.5 Inexact Line-Search Methods 43

line-search conditions may be used, which consist of (2.54) and, instead of (2.59), the following
strengthened version
 
∇f ðxk þ αk dk ÞT dk   σ∇f ðxk ÞT dk ð2:62Þ

is used. From (2.50) we see that, if σ ! 0, then the stepsize which satisfies (2.54) and (2.62) tends to
be the optimal stepsize. Observe that, if a stepsize αk satisfies the strong Wolfe line-search, then it
satisfies the weak Wolfe conditions.
The strong Wolfe conditions allow σ to be chosen to vary the accuracy of the step. If ρ is fixed at a
value close to zero (ρ ¼ 104), then a value of σ close to ρ gives a more accurate step with respect to
closeness to a critical point of ∇f(xk þ αdk)Tdk. A value of σ close to 1 results in a more
approximate step.

Proposition 2.5 Suppose that the function f is continuously differentiable. Let dk be a descent
direction at point xk and assume that f is bounded from below along the ray {xk þ αdk : α > 0}.
Then, if 0 < ρ < σ < 1, there exists an interval of stepsizes α satisfying the weak Wolfe conditions and
the strong Wolfe conditions.

Proof Since φk(α) ¼ f(xk þ αdk) is bounded from below for all α > 0, the line l(α) ¼ f(xk) þ αρ ∇ f(xk)Tdk
must intersect the graph of φ at least once. Let α0 > 0 be the smallest intersection value of α, i.e.,

f ðxk þ α0 d k Þ ¼ f ðxk Þ þ α0 ρ∇f ðxk ÞT dk < f ðxk Þ þ ρ∇f ðxk ÞT dk : ð2:63Þ

Hence, a sufficient decrease holds for all 0 < α < α0.

00
Now, by the mean value theorem, there exists α 2 (0, α0) so that

f ðxk þ α0 dk Þ  f ðxk Þ ¼ α0 ∇f ðxk þ α00 d k Þ d k :

T
ð2:64Þ

Since ρ < σ and ∇f(xk)Tdk < 0, from (2.63) and (2.64) we get

∇f ðxk þ α00 dk Þ dk ¼ ρ∇f ðxk ÞT dk > σ∇f ðxk ÞT d k :

T
ð2:65Þ
00
Therefore, α satisfies the weak Wolfe line-search conditions (2.54) and (2.59) and the inequalities
00
are strict. By smoothness assumption on f, there is an interval around α for which the Wolfe
00
conditions hold. Since ∇f(xk þ α dk)Tdk < 0, it follows that the strong Wolfe line-search conditions
(2.54) and (2.62) hold in the same interval. ♦

We mention that the Wolfe line-search conditions are scale-invariant, i.e., multiplying the
minimizing function by a constant or making an affine change of variables does not alter them.
This line-search can be used in the most line-search methods for unconstrained optimization,
particularly in the implementation of Newton, quasi-Newton, conjugate gradient, etc.

4. Generalized Wolfe Line-Search

In the generalized Wolfe line-search, the absolute value in (2.62) is replaced by a pair of
inequalities:

σ 1 ∇f ðxk ÞT dk  ∇f ðxk þ αk d k ÞT dk  σ 2 ∇f ðxk ÞT dk , ð2:66Þ

where 0 < ρ < σ 1 < 1 and σ 2  0. The particular case in which σ 1 ¼ σ 2 ¼ σ corresponds to the strong
Wolfe line-search.
44 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

5. Hager-Zhang Line-Search
Hager and Zhang (2005) introduced the approximate Wolfe line-search

σ∇f ðxk ÞT dk  ∇f ðxk þ αk dk ÞT d k  ð2ρ  1Þ∇f ðxk ÞT dk , ð2:67Þ

where 0 < ρ < 1/2 and ρ < σ < 1. Observe that the approximate Wolfe line-search (2.67) has the same
form as the generalized Wolfe line-search (2.66), but with a special choice for σ 2. The first inequality
in (2.67) is the same as (2.59). When f is quadratic, the second inequality in (2.67) is equivalent to
(2.54).
In general, when φk(α) ¼ f(xk þ αdk) is replaced by a quadratic interpolating q(.) that matches
φk(α) at α ¼ 0 and φ0k ðαÞ at α ¼ 0 and α ¼ αk, (2.54) reduces to the second inequality in (2.67).
Observe that the decay condition (2.54) is a component of the generalized Wolfe line-search, while in
the approximate Wolfe line-search the decay condition is approximately enforced through the second
inequality in (2.67). As shown by Hager and Zhang (2005), the first Wolfe condition (2.54) limits the
accuracy of a conjugate gradient method to the order of the square root of the machine precision,
while with the approximate Wolfe line-search we can achieve accuracy to the order of the machine
precision.
The approximate Wolfe line-search is based on the derivative of φk(α). This can be achieved by
using a quadratic approximation of φk. The quadratic interpolating polynomial q that matches φk(α) at
α ¼ 0 and φ0(α) at α ¼ 0 and α ¼ αk (which is unknown) is given by
φ0k ðαk Þ  φ0k ð0Þ 2
qðαÞ ¼ φk ð0Þ þ φ0k ð0Þα þ α :
2αk

Observe that the first weak Wolfe condition (2.54) can be written as φk ðαk Þ  φk ð0Þ þ ραk φ0k ð0Þ:
Now, if φk is replaced by q in the first weak Wolfe condition, we get q(αk)  q(0) þ ρq0(αk), which is
rewritten as
φ0k ðαk Þ  φ0k ð0Þ
αk þ φ0k ð0Þαk  ραk φ0k ð0Þ,
2
and can be restated as

φ0k ðαk Þ  ð2ρ  1Þφ0k ð0Þ, ð2:68Þ

where ρ < min {0.5, σ}, which is exactly the second inequality in (2.67).
In terms of function φk(.) the approximate line-search aims at finding the stepsize αk which satisfies
the weak Wolfe conditions
φk ðαÞ  φk ð0Þ þ ρφ0k ð0Þα, and φ0k ðαÞ  σφ0k ð0Þ, ð2:69Þ

which are called LS1 conditions, or the conditions (2.68) together with
φk ðαÞ  φk ð0Þ þ εk , and εk ¼ εjf ðxk Þj, ð2:70Þ

where ε is a small positive parameter (ε ¼ 106), which are called LS2 conditions. εk is an estimate for
the error in the value of f at iteration k. With these, the approximate Wolfe line-search algorithm is as
follows:
2.5 Inexact Line-Search Methods 45

Algorithm 2.2 Hager and Zhang line-search algorithm

1. Choose an initial interval [a0, b0] and set k ¼ 0

2. If either LS1 or LS2 conditions are satisfied at ak, stop
3. Define a new interval [a, b] by using the secant2 procedure: [a, b]¼ secant2 (ak, bk)
4. If b  a > γ(bk  ak), then c ¼ (a þ b)/2 and use the update procedure: [a, b] ¼ update(a, b, c), where
γ 2 (0, 1). (γ ¼ 0.66)
5. Set [ak, bk] ¼ [a, b] and k ¼ k þ 1 and go to step 2 ♦


The update procedure changes the current bracketing interval [a, b] into a new one a, b by using
an additional point which is either obtained by a bisection step or by a secant step. The input data in
the procedure update are the points a, b, c. The parameter in the procedure update is θ 2 (0, 1)
(θ ¼ 0.5). The output data are a, b:

Algorithm 2.3 The update procedure

1. If c 2
= (a, b), then set a ¼ a, b ¼ b and return
2. If φ0k ðcÞ  0, then set a ¼ a, b ¼ c and return
3. If φ0k ðcÞ < 0 and φk(c)  φk(0) þ εk, then set a ¼ c, b ¼ b and return
4. If φ0k ðcÞ < 0 and φk(c) > φk(0) þ εk, then set ab ¼ a, bb ¼ c and perform the following steps:
(a) Set d ¼ ð1  θÞb b If φ0 ðd Þ  0, set b ¼ d, a ¼ ab and return
a þ θb: k
(b) If φ0k ðd Þ < 0 and φk(d )  φk(0) þ εk, then set ab ¼ d and go to step (a)
(c) If φ0k ðdÞ < 0 and φk(d ) > φk(0) þ εk, then set bb ¼ d and go to step (a) ♦


The update procedure finds the interval a, b so that
 
φk ðaÞ < φk ð0Þ þ εk , φ0k ðaÞ < 0 and φ0k b  0: ð2:71Þ

Eventually, a nested sequence of intervals [ak, bk] is determined, which converges to the point that
satisfies either LS1 (2.69) or LS2 (2.68) and (2.70) conditions.
The secant procedure updates the interval by secant steps. If c is obtained from a secant step based
on the function values at a and b, then we write
aφ0k ðbÞ  bφ0k ðaÞ
c ¼ secant ða, bÞ ¼ :
φ0k ðbÞ  φ0k ðaÞ

Since we do not know whether φ0 is a convex or a concave function, then a pair of secant steps is
generated by a procedure denoted secant2, defined as follows. The input data are the points a and b.

The outputs are a and b which define the interval a, b :

Algorithm 2.4 Procedure secant2

1. Set c¼ secant (a, b) and [A, B]¼ update (a, b, c)

2. If c ¼ B, then c ¼ secant (b, B)
3. If c ¼ A, then c ¼ secant (a, A)
 
4. If c ¼ A or c ¼ B, then a, b ¼ update ðA, B, cÞ: Otherwise, a, b ¼ ½A, B ♦
46 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

The Hager and Zhang line-search procedure finds the stepsize αk that satisfies either LS1 or LS2 in
a finite number of operations, as it is stated in the following theorem proved by Hager and
Zhang (2005).

Theorem 2.19 Suppose that φk(α) is continuously differentiable on an interval [a0, b0], where (2.71)
holds. If ρ 2 (0, 1/2), then the Hager and Zhang line-search procedure terminates at a point satisfying
either LS1 or LS2 conditions. ♦

Under some additional assumptions, the convergence analysis of the secant 2 procedure was given
by Hager and Zhang (2005), proving that the interval width generated by it is tending to zero, with the
pffiffiffi
root convergence order 1 þ 2: This line-search procedure is implemented in CG-DESCENT, one of
the most advanced conjugate gradient algorithms (see Chap. 5).

6. Dai and Kou Line-Search

In practical computations, the first Wolfe condition (2.54) may never be satisfied because of the
numerical errors, even for tiny values of ρ In order to avoid the numerical drawback of the weak
Wolfe line-search, Hager and Zhang (2005) introduced a combination of the original weak Wolfe
conditions and the approximate Wolfe conditions (2.67). Their line-search works well in numerical
computations, but in theory it cannot guarantee the global convergence of the algorithm. Therefore, in
order to overcome this deficiency of the approximate Wolfe line-search, Dai and Kou (2013)
introduced the so-called improved Wolfe line-search: “given a constant parameter ε > 0, a positive
sequence {ηk} satisfying ∑k1 ηk < 1 as well as the parameters ρ and σ satisfying 0 < ρ < σ < 1, Dai
and Kou (2013) proposed the following modified Wolfe condition:
   
f ðxk þ αd k Þ  f ðxk Þ þ min εgTk dk , ραgTk d k þ ηk : ” ð2:72Þ

The line-search satisfying (2.72) and (2.59) is called the improved Wolfe line-search. If f is
continuously differentiable and bounded from below, the gradient g is Lipschitz continuous and dk
is a descent direction (i.e., ∇f(xk)Tdk < 0), then there must exist a suitable stepsize satisfying (2.59) and
(2.72), since they are weaker than the weak Wolfe conditions.

7. Backtracking
Often, in practice, when a “good” search direction is known, then only the condition (2.54) is used
for the stepsize computation. The main idea of this algorithm is to start the computations with αk ¼ 1.
If the new point xk þ αkdk is not acceptable, then reduce the value of αk until the first condition of
(2.54) f(xk þ αkdk)  f(xk) þ ραk ∇ f(xk)Tdk is satisfied. This method is called inexact line-search with
backtracking. A simple backtracking algorithm is as follows.

Algorithm 2.5 Backtracking

1. Choose the scalars: ρ 2 (0, 1/2), 0 < l < u < 1 and set αk ¼ 1
2. Test if f(xk þ αkdk)  f(xk) þ ραk ∇ f(xk)Tdk
3. If f(xk þ αkdk)  f(xk) þ ραk ∇ f(xk)Tdk is not satisfied, then set αk ¼ βαk, where β 2 [l, u] and go to step 2;
otherwise, stop with αk obtained at previous step ♦
2.5 Inexact Line-Search Methods 47

Observe that backtracking is very similar to Armijo’s rule. Often, backtracking is used in the
Newton or quasi-Newton methods, where it is known that, for the vast majority of the steps of these
methods, their length is unitary, i.e., a single test of (2.54) is sufficient to get a convenient stepsize αk.
A more sophisticated variant of the inexact line-search with backtracking consists of determining
αk not by its simple reduction through a factor taken from the interval (0, 1), as in the above
algorithm, but using the quadratic interpolation. Indeed, let φ(α) ¼ f(xk þ αdk). From the very
beginning we know φ(0) ¼ f(xk) and φ0(0) ¼ ∇ f(xk)Tdk. Taking a unitary step, we find φ(1) ¼ f(xk þ dk).
If f(xk þ dk) does not satisfy (2.54), then the following quadratic model may be used in order to
approximate φ(α):

pðαÞ ¼ ðφð1Þ  φð0Þ  φ0 ð0ÞÞα2 þ φ0 ð0Þα þ φð0Þ,

which takes into account the above interpolating values. From the minimum condition p0(α) ¼ 0 we
get a new value for α as
φ 0 ð 0Þ
α¼ :
2ðφð1Þ  φð0Þ  φ0 ð0ÞÞ

This value may be used in an iterative process as above to get an acceptable stepsize. In order to
avoid too small stepsizes αk, some protections may be introduced. For example, if kαkdkk < η, where η
is a given parameter, then the computation of the stepsize is stopped. A simple variant of the inexact
line-search with backtracking is as follows.

Algorithm 2.6 Backtracking – variant

1. Choose the scalars: ρ 2 (0, 1/2), β 2 (0, 1) and set αk ¼ 1

2. If f(xk þ αkdk)  f(xk) þ ραk ∇ f(xk)Tdk, then, stop; otherwise go to step 3
3. Set αk ¼ βαk and go to step 2 ♦

Since dk is a descent direction, then ∇f(xk)Tdk < 0. Therefore, it follows that for sufficiently small
values αk we have:

f ðxk þ αk dk Þ ffi f ðxk Þ þ αk ∇f ðxk ÞT dk < f ðxk Þ þ ραk ∇f ðxk ÞT d k ,

which shows that the algorithm terminates in a finite number of iterations. The constant ρ is
interpreted as the fraction of reducing the values of f, predicted by linear extrapolation. Usually, ρ
is selected between 0.001 and 0.5 that means that we accept a reduction of function f between 1% and
50% from the prediction based on linear extrapolation. The parameter β is selected between 0.1 and
0.9. The following theorem shows the properties of the inexact line-search with backtracking.

Theorem 2.20 Let f : ℝn ! ℝ be a continuously differentiable function and the level set
S ¼ {x 2 ℝn : f(x)  f(x0)} a compact. Suppose that for any k, ∇f(xk)Tdk < 0. Then, Algorithm 2.6
determines in a finite number of iterations a value αk > 0 which satisfies the condition from step 2 of
the algorithm. The sequence xkþ1 ¼ xk þ αkdk satisfies f(xkþ1) < f(xk) for any k and

∇f ðxk ÞT d k
lim ¼ 0:
k!1 kd k k
48 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Proof We shall prove that the algorithm with backtracking cannot cycle indefinitely between steps
2 and 3. Indeed, if the algorithm cycles between these steps, then step 2 is never satisfied, and
therefore
 
f x k þ β j d k  f ðx k Þ
> ρ∇f ðxk ÞT dk :
βj

But β 2 (0, 1) and β j ! 0 when j ! 1 and therefore, for j ! 1 the above inequality becomes
∇f(xk)Tdk > ρ ∇ f(xk)Tdk, which is not possible since ρ 2 (0, 1/2) and ∇f(xk)Tdk 6¼ 0.
Therefore, there exists a j0 such that
 
f xk þ βj0 d k  f ðxk Þ  ρβj0 ∇f ðxk ÞT dk 6¼ 0:

Since ∇f(xk)Tdk 6¼ 0, it follows that for any k, f(xkþ1) < f(xk). Now, from the above inequality we get
 
f xk þ βj0 d k  f ðxk Þ ∇f ðxk ÞT dk
ρ :
β kd k k
j0 kd k k

Using the mean value theorem, it follows that

 T
∇f xk þ βj0 d k dk ∇f ðxk ÞT dk
ρ ,
kd k k kd k k

i.e., the conclusion of the theorem. ♦

It is worth mentioning that by using the same technique as above it is possible to obtain the inexact
line-search with approximative backtracking. This is exactly the last inequality from (2.67).
The condition from the inexact line-search with backtracking is sufficient to achieve an improve-
ment of the values of the minimizing function, but this is not strong enough to guarantee that any
accumulation point of the algorithm will be a stationary point of function f. In fact, if n  2, the
descent direction dk is very likely to be selected in such a way that the angle between dk and  ∇ f(xk)
should be very close to 90 . In this case, the cosine of this angle tends to zero, case in which the
general algorithm may converge to a nonstationary point. In order to avoid this situation, we require
that the cosine of the above angle should be uniformly bounded away from zero. In other words, we
require that the searching directions make an acute cone centered in  ∇ f(xk), i.e., they should satisfy
the so-called angle condition

∇f ðxk ÞT dk  τk∇f ðxk Þkkd k k,

where τ 2 (0, 1).

8. Nonmonotone Line-Search Grippo, Lampariello, and Lucidi

The nonmonotone line-search for Newton’s methods was introduced by Grippo, Lampariello, and
Lucidi (1986). In this method, the stepsize αk satisfies the following condition:
 
f ðxk þ αk d k Þ  max f xkj þ ραk gTk dk , ð2:73Þ
0jmðkÞ

where ρ 2 (0, 1), m(0) ¼ 0, 0  m(k)  min {m(k  1) þ 1, M} and M is a prespecified nonnegative
integer. Theoretical analysis and numerical experiments showed the efficiency and robustness of this
line-search for solving unconstrained optimization problems in the context of the Newton method.
2.5 Inexact Line-Search Methods 49

The r-linear convergence for the nonmonotone line-search (2.73), when the objective function f is
strongly convex, was proved by Dai (2002b).
Although these nonmonotone techniques based on (2.73) work well in many cases, there are some
drawbacks. Firstly, a good function value generated in any iteration is essentially discarded due to the
max in (2.73). Secondly, in some cases, the numerical performance is quite dependent on the choice
of M (see Raydan, (1997)). Furthermore, it was pointed out by Dai (2002b) that, although an iterative
method generates r-linearly convergent iterations for a strongly convex function, the iterates may not
satisfy the condition (2.73) for k sufficiently large and for any fixed bound M on the memory.

9. Nonmonotone Line-Search Zhang and Hager

Zhang and Hager (2004) proposed another nonmonotone line-search technique by replacing the
maximum function values in (2.73) with an average of function values. Suppose that dk is a descent
direction. Their line-search determines a stepsize αk as follows.

Algorithm 2.7 Zhang and Hager nonmonotone line-search

1. Choose a starting guess x0 and the parameters: 0  ηmin  ηmax  1, 0 < ρ < σ < 1 < β and μ > 0. Set
C0 ¼ f(x0), Q0 ¼ 1 and k ¼ 0
2. If k∇f(xk)k is sufficiently small, then stop
3. Line-search update: Set xkþ1 ¼ xk þ αkdk, where αk satisfies either the nonmonotone Wolfe conditions:

f ðxk þ αk dk Þ  Ck þ ραk gTk dk , (2.74)

∇f ðxk þ αk dk ÞT dk  σdTk gk , (2.75)

or the nonmonotone Armijo conditions: αk ¼ αk βhk , where αk > 0 is the trial step and hk is the largest
integer such that (2.74) holds and αk  μ
4. Choose ηk 2 [ηmin, ηmax] and set:
Qkþ1 ¼ ηkQk þ 1, (2.76)

Ckþ1 ¼ ηk Qk CQk þf ðxkþ1 Þ (2.77)

kþ1

5. Set k ¼ k þ 1 and go to step 2 ♦

Observe that Ckþ1 is a convex combination of Ck and f(xkþ1). Since C0 ¼ f(x0), it follows that
Ck is a convex combination of the function values f(x0), f(x1), . . ., f(xk). Parameter ηk controls the
degree of nonmonotonicity. If ηk ¼ 0 for all k, then this nonmonotone line-search reduces to the
monotone Wolfe or Armijo line-search. If ηk ¼ 1 for all k, then Ck ¼ Ak, where

1 X
n
Ak ¼ f ðxi Þ:
k þ 1 i¼0

Theorem 2.21 If ∇f(xk)Tdk  0 for each k, then, for the iterates generated by the nonmonotone line-
search of Zhang and Hager, we have f(xk)  Ck  Ak for each k. Moreover, if ∇f(xk)Tdk < 0 and f(x) is
bounded from below, then there exists αk satisfying either the Wolfe or the Armijo conditions of the
line-search update. ♦

Zhang and Hager (2004) proved the convergence of their algorithm.

50 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Theorem 2.22 Suppose that f is bounded from below and there exist the positive constants c1 and c2
such that ∇f(xk)Tdk   c1kgkk2 and kdkk  c2kgkk for all sufficiently large k. Then, under the weak
Wolfe line-search, if ∇f is Lipschitz continuous, then the iterates xk generated by the nonmonotone
line-search of Zhang and Hager have the property that liminfk!1k∇f(xk)k ¼ 0. Moreover, if ηmax < 1,
then limk!1 ∇ f(xk) ¼ 0. ♦

The numerical results reported by Zhang and Hager (2004) showed that this nonmonotone line-
search is superior to the nonmonotone technique (2.73).

10. Nonmonotone Line-Search Gu and Mo

A modified version of the nonmonotone line-search (2.73) was proposed by Gu and Mo (2008). In
this method, instead of an average of the successive objective function values introduced by Zhang
and Hager (2004), the current nonmonotone term is a convex combination of the previous
nonmonotone term and of the current value of the objective function, i.e., the stepsize αk is computed
to satisfy the following condition:

f ðxk þ αk dk Þ  Dk þ ραk gTk dk , ð2:78Þ

where

D0 ¼ f ðx0 Þ, k ¼ 0,
ð2:79Þ
Dk ¼ θk Dk1 þ ð1  θk Þf ðxk Þ, k  1,

with 0  θk  θmax < 1 and ρ 2 (0, 1). The theoretical and numerical results reported by Gu and Mo
(2008) in the frame of the trust-region method showed the efficiency of this nonmonotone line-search
scheme.

11. Nonmonotone Line-Search Huang, Wan, and Chen

Huang, Wan, and Chen (2014) proposed a new nonmonotone line-search as an improved version
of the nonmonotone line-search technique proposed by Zhang and Hager. Their algorithm
implementing the nonmonotone Armijo condition has the same properties as the nonmonotone
line-search algorithm of Zhang and Hager as well as some other properties that certify its conver-
gence in very mild conditions. Suppose that at xk the search direction is dk. The nonmonotone line-
search proposed by Huang, Wan, and Chen is as follows:

Algorithm 2.8 Nonmonotone line-search Huang, Wan, and Chen

1. Choose 0  ηmin  ηmax < 1 < β, δmax < 1, 0 < δmin < (1  ηmax)δmax, the convergence tolerance ε > 0 small
enough and μ > 0
2. If kgkk  ε, then the algorithm stops
3. Choose ηk 2 [ηmin, ηmax]. Compute Qkþ1 and Ckþ1 by (2.76) and (2.77), respectively. Choose
δmin  δk  δmax/Qkþ1. Let αk ¼ αk βhk  μ be a stepsize satisfying

Ckþ1 ¼ ηk Qk Ck þf
Q
ðxk þαk d k Þ
 Ck þ δk αk gTk d k , (2.80)
kþ1

where hk is the largest integer such that (2.80) holds and Qk, Ck, Qkþ1 and Ckþ1 are computed as in the
nonmonotone line-search of Zhang and Hager
4. Set xkþ1 ¼ xk þ αkdk. Set k ¼ k þ 1 and go to step 2 ♦
2.5 Inexact Line-Search Methods 51

If the minimizing function f is continuously differentiable and if gTk dk  0 for each k, then there
exists a trial step αk such that (2.80) holds. The convergence of this nonmonotone line-search is
obtained under the same conditions as in Theorem 2.22. The r-linear convergence is proved for
strongly convex functions.

12. Nonmonotone Line-Search Ou and Liu

Based on (2.78), a new modified nonmonotone memory gradient algorithm for unconstrained
optimization was elaborated by Ou and Liu (2017). Given ρ1 2 (0, 1), ρ2 > 0 and β 2 (0, 1) set
 
sk ¼  gTk dk =kd k k2 and compute the stepsize αk ¼ max {sk, skβ, skβ2, . . .} satisfying the line-
search condition:

f ðxk þ αk d k Þ  Dk þ ρ1 αk gTk dk  ρ2 α2k kdk k2 , ð2:81Þ

where Dk is defined by (2.79) and dk is a descent direction, i.e., gTk dk < 0: Observe that if ρ2 ¼ 0 and
sk  s for all k, then the nonmonotone line-search (2.81) reduces to the nonmonotone line-search
(2.78). The algorithm corresponding to this nonmonotone line-search presented by Ou and Liu is as
follows:

Algorithm 2.9 Ou and Liu nonmonotone line-search

1. Consider a starting guess x0 and select the parameters: convergence tolerance ε  0, 0 < τ < 1, ρ1 2 (0, 1),
ρ2 > 0, β 2 (0, 1) and an integer m > 0. Set k ¼ 0
2. If kgkk  ε, then stop
3. Compute the direction dk by the following recursive formula:

gk , if k  m,
dk ¼ P (2.82)
λk gk  mi¼1 λki d ki if k  m þ 1,

where
kgk k2
λki ¼ mτ kgk k2 þjgTk dki j
, i ¼ 1, . . . , m,
Pm
λk ¼ 1  i¼1 λki
4. Using the above procedure, determine the stepsize αk satisfying (2.82) and set xkþ1 ¼ xk þ αkdk
5. Set k ¼ k þ 1 and go to step 2 ♦

The algorithm has the following interesting properties. For any k  0, it follows that
 ð1  τÞkgk k2 : For any k  m, it follows that kdk k  max fkgk k, kd ki kg: Moreover, for
gTk dk
n o 1im

any k  0, kd k k  max gj :
0jk

Theorem 2.23 If the objective function is bounded from below on the level set S ¼ {x 2 ℝn : f(x)  f(x0)}
and the gradient ∇f(x) is Lipschitz continuous on an open convex set that contains S, then Algorithm
2.9 terminates in a finite number of iterates. Moreover, if the algorithm generates an infinite sequence
{xk}, then limk!þ1kgkk ¼ 0. ♦

The numerical results presented by Ou and Liu (2017) showed that this method is suitable for
solving large-scale unconstrained optimization problems and is more stable than other similar
methods.
52 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

13. Weak Wolfe Line-Search with Simple Bisection

The bisection method is a simple method to find a zero of a continuous function for which two
values of opposite sign are known. To determine a stepsize αk which satisfies the weak Wolfe line-
search conditions (2.54) and (2.59), the following simple algorithm based on the bisection concept
may be used.

Algorithm 2.10 Weak Wolfe line-search with simple bisection

high
1. Choose αk > 0 and set αlow k ¼ αk ¼0
2. If αk satisfies (2.54), then go to Step 4
 
3. Else (if αk does not satisfy (2.54)), then set: αk
high
¼ αk and αk ¼ αlow
high
k þ αk =2 and go to Step 2
4. If αk satisfies (2.59), then stop
5. Otherwise (if αk does not satisfy (2.59)), then set: αlow
k ¼ αk and
8
< 2αlow
k , if αhigh ¼ 0,
αk ¼   k
: αlow
k þ αhigh
k =2, if αhigh
k > 0,

and go to Step 2 ♦

Let us prove that the above Algorithm 2.10 determines a value for αk which satisfies both Wolfe
line-search conditions (2.54) and (2.59) in a finite number of steps.

Theorem 2.24 Suppose that function f is continuously differentiable on ℝn and bounded from below
on the half-line {xk þ αdk : α > 0}. Then, Algorithm 2.10 terminates in finite time and generates a
value for αk that satisfies the weak Wolfe conditions (2.54) and (2.59).

Proof Let us define φ(α) ¼ f(xk þ αdk) and introduce the following two sets:
S1 ¼ fα > 0 : ð2:54Þ holdsg,
S2 ¼ fα > 0 : ð2:59Þ holdsg:

Observe that both S1 and S2 are closed in ℝ+. Moreover, for α sufficiently small since φ0 is
continuous and ρ < 1,
ðα ðα
φðαÞ ¼ φð0Þ þ φ ðτÞdτ < φð0Þ þ ρφ0 ð0Þdτ:
0

0 0

Therefore, there exists δ1 > 0 such that [0, δ1] ⊂ S1, i.e., there exists α > 0 satisfying the first
Wolfe condition (2.54). Now, consider the second Wolfe condition (2.59). Let α > 0 and two
n o n o
low,½i high,½i
sequences αk ⊂ S1 and αk ⊂ S2 be such that
ℕ ℕ

low,½i low,½i
αk < α for any i 2 ℕ, αk ! α, ð2:83Þ
i!1

high,½i high,½i
αk >α for any i 2 ℕ, αk ! α: ð2:84Þ
i!1
2.5 Inexact Line-Search Methods 53

= S2 and hence φ0(α)  σφ0(0). Then, since φ0 is

With this, let us prove that α 2 S2. Assume that α 2
continuous and ρ < σ, there exists a value δ2 > 0 such that for any θ 2 [0, δ2] it follows that
φ0(α þ θ)  σφ0(0). Therefore, for all θ 2 [0, δ2] we have
ð
αþθ

φðα þ θÞ ¼ φðαÞ þ φ0 ðτÞdτ < φð0Þ þ ðα þ θÞσφ0 ð0Þ:

α

high,½i
But, since αk converges to α from the right, it follows that there exists an index j large enough
high,½j high,½j
so that αk 2 ½α, α þ δ2 , thus contradicting the assumption that αk 2 S1 : Therefore, α 2 S2.
Now, let us prove that the algorithm terminates in finite time. If the algorithm terminates in finite
time, then αk generated by it satisfies both weak Wolfe line-search conditions. For this, let us define
low,½i high,½i ½i high
αk , αk , and αk as the values of αlow
k , αk , and αk at the beginning of the iteration i of the
algorithm. The following properties can be observed:

low,½i ½i ½0

1. Observe that for all i it is impossible that αk ¼ 0, because in this case αk ¼ 2i αk and hence
½i ½i
αk 2 ½0, δ1  and αlow is updated to αk > 0 (see Step 5 of the algorithm).
n k o
low,½i low,½i ½i
2. The sequence αk is an increasing one in S1 such that for all i, αk < αk : αlow
k can only be
ℕ
low,½iþ1 ½i
updated in Step 5 of Algorithm 2.10. It will be increased to the strictly larger value αk ¼ αk
½iþ1 ½i
and as we can see in Step 5 of the algorithm, αk takes on a strictly larger value than αk .
high,½i ½i 
3. Initially, for a few iterations αk ¼ 0: But, once it has taken on a value αk 0 > 0 at some
iteration i0, then this can only happen in Step 3 of Algorithm 2.10. Starting with the iteration i0, the
high,½i high
elements of the sequence αk with i  i0 decrease in S1 because αk is only updated in Step 3 of
the algorithm to a value of αk that is strictly smaller than αhigh
k and αk is itself updated to a strictly
smaller value.
high,½i low,½i ½0
4. All in all, only two possibilities can appear. Either αk ¼ 0 for all i and then αk ¼ 2i1 αk
for all i, in which case Algorithm 2.10 finds that function f is unbounded. However, this case must
be excluded since f is bounded in the assumptions of the theorem. Or, there exists an index i0 2 ℕ
  n o
high,½i  ½i low,½i high,½i low,½i
such that αk 0 > 0 and therefore αk ¼ αk þ αk =2, the sequence αk is
n o h i ℕ
high,½i low,½i high,½i
increasing, the sequence αk is decreasing, and the interval αk , αk is halved in
ℕ
low,½i
length at every iteration. Therefore, it follows that αk converges to a point αk from S1 and
high,½i low,½i
αk converges to the same point in S1. In conclusion αk 2 S1 \ S2. Therefore, αk 2 S1 \ S2
for i sufficiently large, proving that the algorithm terminates with this value. ♦

Observe that this is a very simple procedure for computing a value of αk which satisfies the weak
Wolfe line-search conditions (2.54) and (2.59) without any safeguarding to the too large or too small
values of αk.

14. Barzilai and Borwein Line-Search

A special nonmonotone line-search is the Barzilai and Borwein (1988) method. In this method, the
next approximation to the minimum is computed as xkþ1 ¼ xk  Dkgk, k ¼ 0, 1, . . ., where
Dk ¼ αkI, I being the identity matrix. The stepsize αk is computed as solution of the problem
54 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

T 
min ksk  Dk yk k or as solution of min D1
k sk  yk : In the first case, αk ¼ sk yk =kyk k and in the
2
αk αk
 
second one, αk ¼ ksk k2 = sTk yk , where sk ¼ xkþ1  xk and yk ¼ gkþ1  gk. Barzilai and Borwein
proved that their algorithm is super-linearly convergent. Many researchers studied the Barzilai and
Borwein algorithm, including Raydan (1997), Grippo and Sciandrone (2002), Dai, Hager,
Schittkowski, and Zhang (2006), Dai and Liao (2002), Narushima, Wakamatsu, and Yabe, (2008),
Liu and Liu (2019), etc.
The nonmonotone line-search methods were investigated by many authors; for example, see Dai
(2002b) and the references therein. Observe that all these nonmonotone line-searches concentrate on
modifying the first Wolfe condition (2.54). Also the approximate Wolfe line-search (2.67) of Hager
and Zhang and the improved Wolfe line-search (2.72) and (2.59) of Dai and Kou modify the first
Wolfe condition, responsible for a sufficient reduction of the objective function value. No numerical
comparisons among the abovementioned nonmonotone line-searches were given.
As regards the stopping of the iterative scheme (2.4), one of the most popular criteria is kgkk  ε,
where ε is a small positive constant and k.k is the Euclidian or l1 norm.

2.6 Convergence of the Algorithm with Inexact Line-Search

In this section we present the convergence of the general algorithm for unconstrained optimization
with inexact line-search, i.e., Algorithm 2.1, in which the stepsize is computed by using the inexact
line-searches. For this, the quality of the search direction needs to be studied. The quality of the search
direction is defined by the angle between the search direction and the negative gradient. In order to
ensure the convergence of Algorithm 2.1, the situations in which the search direction sk ¼ αkdk
is orthogonal to the negative gradient must be avoided. In other words, the angle θk between sk
and  ∇ f(xk) is uniformly bounded away from 90 , i.e., for any k:
π
θk   μ, ð2:85Þ
2
where μ > 0. The inequality (2.85) is called angle condition. Consider gk ¼ ∇ f(xk), then the angle
θk 2 [0, π/2] is defined as

gTk sk
cos θk ¼  : ð2:86Þ
kgk kksk k

Observe that all the procedures for the stepsize computation (except for the Barzilai-Borwein) are
based on the Armijo condition (2.51) or on the Wolfe condition (2.54) and (2.59). Therefore, the
convergence analysis of Algorithm 2.1 with inexact line-search will be separately presented for the
Armijo line-search and for the Wolfe line-search. The following theorem shows the convergence of
Algorithm 2.1 in which the stepsize is computed by using the Armijo line-search (2.51) (Griva, Nash,
and Sofer, 2009).

Theorem 2.25 Let f be a real-valued function continuously differentiable on ℝn. Let x0 be the initial
point and consider the sequence {xk}, defined by xkþ1 ¼ xk þ αkdk, where αk  0 is the stepsize, a
scalar, and dk is the search direction, a vector of dimension n. Assume that:

(i) The level set S ¼ {x : f(x)  f(x0)} is bounded.

(ii) The gradient ∇f(x) is Lipschitz continuous for all x, i.e., k∇f(x)  ∇ f( y)k  Lkx  yk for some
constant 0 < L < 1 .
2.6 Convergence of the Algorithm with Inexact Line-Search 55

(iii) The vectors dk satisfy the sufficient descent condition

dTk ∇f ðxk Þ
  ε > 0:
kdk kk∇f ðxk Þk

(iv) The search directions dk are gradient related as

kd k k  mk∇f ðxk Þk

for all k, where m > 0, and bounded in norm

kd k k  M

for all k.
(v) The stepsize αk satisfies the Armijo line-search

f ðxk þ αd k Þ  f ðxk Þ þ ραk dTk ∇f ðxk Þ,

where 0 < ρ < 1.

Then,

lim k∇f ðxk Þk ¼ 0:

k!1

Proof Before presenting the proof, some comments are needed. Observe that the above theorem
makes several assumptions to guarantee the convergence of Algorithm 2.1 with Armijo line-search.
Two of them refer to the minimizing function: to have a bounded level set and its gradient to be
Lipschitz continuous. The other three assumptions refer to the algorithm: the search direction has to
satisfy the sufficient descent condition and is gradient related, and the stepsize has to satisfy the
Armijo line-search.
The boundedness of the level set ensures that the function takes on its minimum value in a finite
point. The Lipschitz continuity of the gradient of f is a technical assumption.
The search direction must first of all be a descent direction, that is, dTk ∇f ðxk Þ < 0: However, the
danger is that dk satisfying dTk ∇f ðxk Þ < 0 might become arbitrarily close to being orthogonal to ∇f(xk)
while still remaining a descent direction, thus dramatically slowing down the convergence of the
algorithm. To avoid this situation, we assume that

dTk ∇f ðxk Þ
  ε > 0,
kdk kk∇f ðxk Þk

for all k, where ε > 0 is a specified tolerance. From (2.86), this condition can also be written as
cos θk  ε > 0,

where θk is the angle between the search direction dk and the negative gradient  ∇ f(xk). In other
words, the search direction satisfies the angle condition.
The search directions dk are gradient related if kdkk  mk∇f(xk)k for all k, where m > 0 is a
constant. In other words, this condition says that the norm of the search direction cannot become too
much smaller than that of the gradient.
56 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

These conditions on the search direction can normally be guaranteed by the algorithm for
minimizing function f.
Now, the assumptions that the stepsize produces a sufficient decrease in function f and it is not too
small are ensured by the Armijo line-search. This ensures a nontrivial reduction in the function value
at each iteration. Nontrivial is measured in terms of the Taylor series. More specifically, we require
that the stepsize αk should produce a decrease in the function value that is at least a certain fraction of
the decrease predicted by the linear approximation given by the Taylor series, i.e.,

f ðxk þ αk dk Þ  f ðxk Þ þ ραk dTk ∇f ðxk Þ,

where 0 < ρ < 1. When ρ is near zero, this condition is easier to satisfy since only a small decrease in
the function value is required. (Usually, ρ ¼ 0.0001.)
With these comments, let us prove the theorem. The proof has five steps. Firstly, it is shown that
f is bounded from below on S. Secondly, it is proved that limk ! 1f(xk) exists. Thirdly, we prove that
the following result limk!1αkk∇f(xk)k2 ¼ 0 holds. Then we show that if αk < 1, then αk  γk∇f(xk)k2
for an appropriate constant γ > 0. Finally, it is proved that lim k∇f ðxk Þk ¼ 0:
k!1

1. f is bounded from below on S. Since f is continuous, it follows that the level set S is closed. But, by
the assumption (i), the set S is bounded. Therefore, S is a compact. A continuous function on a
compact set takes on its minimum value at some point in that set, that is, f(x)  C, for some
constant C (see Theorem A3.1 – Weierstrass Extreme Value Theorem).
2. limk!1f(xk) exists. The sufficient descent condition ensures that f(xkþ1)  f(xk)  f(x0) so that
xk 2 S for all k. The sequence {f(xk)} is monotone decreasing and bounded from below, so it has a
limit, let us say f :
3. limk!1αkk∇f(xk)k2 ¼ 0. We have

f ðx0 Þ  f ¼ ½f ðx0 Þ  f ðx1 Þ þ ½f ðx1 Þ  f ðx2 Þ þ ⋯

P1 P1
¼ ½f ðxk Þ  f ðxkþ1 Þ   ραk dTk ∇f ðxk Þ
k¼0 k¼0
P1 P1
 ραk εkd k kk∇f ðxk Þk  ραk εmk∇f ðxk Þk2 :
k¼0 k¼0

Since f ðx0 Þ  f  f ðx0 Þ  C < 1 it follows that the above final summation is convergent, that is,
the terms in this summation go to zero:

lim ραk εmk∇f ðxk Þk2 ¼ 0:

k!1

Since ρ, ε, and m are fixed nonzero constants, it follows that limk!1αkk∇f(xk)k2 ¼ 0.

4. If αk < 1, then αk  γk∇f(xk)k2 for an appropriate constant γ. To prove this, we use the backtracking
line-search. If αk < 1, then the sufficient descent condition is violated when the stepsize 2αk is tried,
that is,

f ðxk þ 2αk dk Þ  f ðxk Þ > 2ραk dTk ∇f ðxk Þ:

Since ∇f(x) is Lipschitz continuous, then by Theorem 2.7 it follows that

2.6 Convergence of the Algorithm with Inexact Line-Search 57

1
f ðxk þ 2αk dk Þ  f ðxk Þ  2αk dTk ∇f ðxk Þ  Lk2αk dk k2 :
2
That is,

f ðxk Þ  f ðxk þ 2αk dk Þ  2αk dTk ∇f ðxk Þ  2Lkαk d k k2 :

Combining this inequality with the violated condition when the stepsize was 2αk, it follows that

αk Lkdk k2  ð1  ρÞd Tk ∇f ðxk Þ:

The sufficient descent condition (iii) and the gradient relatedness condition (iv) give

αk Lkdk k2  ð1  ρÞεkdk kk∇f ðxk Þk  ð1  ρÞεmk∇f ðxk Þk2 :

Since kdkk  M, it follows that αk  γk∇f(xk)k2 with

ð1  ρÞεm
γ¼ > 0:
M2 L

5. limk!1k∇f(xk)k ¼ 0. Observe that either αk ¼ 1 or αk  γk∇f(xk)k2. Therefore,

n o
αk  min 1, γ k∇f ðxk Þk2 :

Hence,
h n oi
αk k∇f ðxk Þk2  min 1, γ k∇f ðxk Þk2 k∇f ðxk Þk2  0:

But, from step 3 above we know that limk!1αkk∇f(xk)k2 ¼ 0. Since γ > 0, it follows that
limk!1k∇f(xk)k ¼ 0. ♦

Now, we come back to see the convergence of Algorithm 2.1 in which the stepsize is determined
by the Wolfe line-search (2.54) and (2.59). Let us show a very useful result, due to Zoutendijk (1970).
From the proof of this result, we can get a clear idea how the properties of the minimizing function are
linked to the Wolfe conditions.

Theorem 2.26 Suppose that function f is bounded from below on ℝn and it is continuous differen-
tiable in a neighborhood N of the level set S ¼ {x 2 ℝn : f(x)  f(x0)}. Suppose that the gradient g(x)
of f is Lipschitz continuous, i.e., there exists a positive constant L > 0 such that

kgðxÞ  gðyÞk  Lkx  yk ð2:87Þ

for any x, y 2 N. Consider an iteration xkþ1 ¼ xk þ αkdk, where dk is a descent direction and αk
satisfies the weak Wolfe line-search (2.54) and (2.59). Then
X
cos 2 θk kgk k2 < 1: ð2:88Þ
k1
58 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Proof From (2.59) we have

 T
gkþ1  gk dk  ðσ  1ÞgTk d k :

On the other hand, the Lipschitz condition (2.87) shows that

 T
gkþ1  gk dk  αk Lkdk k2 :

Combining these two relations, we have

 
σ  1 gTk d k
αk  : ð2:89Þ
L kd k k2

Now, from the first Wolfe condition (2.54) and (2.89), it follows that
  2
σ  1 gTk d k
f ðxkþ1 Þ  f ðxk Þ þ ρ :
L kd k k2

From (2.86), the above relation can be rewritten as

f ðxkþ1 Þ  f ðxk Þ þ c cos 2 θk kgk k2 ,

where c ¼ ρ(σ  1)/L. Summing these relations for k  1 and since function f is bounded from below,
we get the conclusion of the theorem. ♦

The relation (2.88) is called Zoutendijk condition (Zoutendijk, 1970). It is worth mentioning that
this condition is directly involved in proving the global convergence of the general Algorithm 2.1. As
we can see, if for any k the iterations xkþ1 ¼ xk þ αkdk are in such a way that cosθk  δ > 0, then from
(2.88) we get limk!1kgkk ¼ 0. In other words, if the search direction does not tend to be orthogonal
to the gradient, then the sequence of the gradients is convergent to zero. Observe that the Zoutendijk
condition is a condition on the search direction. If, for example, the search direction is exactly gk,
i.e., the steepest descent algorithm introduced for the first time by Cauchy (1847), then cosθk ¼ 1.
This shows that, in order to obtain the global convergence of the Cauchy algorithm, we need to
execute an adequate line-search. This proves the importance of the line-search in the frame of the
general Algorithm 2.1 with descent directions.
In this context, the following corollary of the Zoutendijk condition can be proved.

Corollary 2.1 Let f : ℝn ! ℝ be bounded from below on ℝn and continuously differentiable in a

neighborhood N of the level set S ¼ {x 2 ℝn : f(x)  f(x0)}. If {xk} is the sequence generated by the
general Algorithm 2.1 with weak Wolfe line-search, then

cos θk k∇f ðxk Þk ! 0: ♦

If cosθk  δ > 0 for all k, then, from the Zoutendijk condition, it follows that k∇f(xk)k ! 0, i.e., the
general Algorithm 2.1 with weak Wolfe line-search converges to a stationary point of function f.
The limit limk!1kgkk ¼ 0 is the best global convergence result we may obtain. Obviously, there is
no guarantee that the algorithm converges to the minimum point. However, there is a guarantee that
the iterates of the algorithm enter in a domain of attraction of the stationary points.
The following theorem shows the global convergence of Algorithm 2.1 with inexact line-search.
2.6 Convergence of the Algorithm with Inexact Line-Search 59

Theorem 2.27 Suppose that in Algorithm 2.1 the search direction dk satisfies the angle
condition (2.85) and αk is computed by the Goldstein conditions (2.54) and (2.55) or by the weak
Wolfe conditions (2.54) and (2.59). If ∇f(x) exists and is uniformly continuous on the level set
S ¼ {x 2 ℝn : f(x)  f(x0)}, then ∇f(xk) ¼ 0 for a k or f(xk) !  1 or ∇f(xk) ! 0.

Proof Firstly, consider αk computed by the Goldstein conditions (2.54) and (2.55). Suppose that for
all k, gk ¼ ∇ f(xk) 6¼ 0 and f(xk) is bounded from below. It follows that f(xk)  f(xkþ1) ! 0, that is, from
(2.54) we have gTk sk ! 0. Now, suppose that gk ! 0 is not true. Then, there exists a subsequence
such that kgkk  ε and kskk ! 0. Since θk  π/2  μ, we get
 
π
cos θk  cos  μ ¼ sin μ,
2
i.e.,

gTk sk  kgk kksk k sin μ  εksk k sin μ:

Using Taylor’s series, we can write

f ðxkþ1 Þ ¼ f ðxk Þ þ ∇f ðξk ÞT sk ,

where ξk is on the line segment connecting xk with xkþ1. From the uniform continuity it follows that
∇f(ξk) ! gk when sk ! 0. Therefore,

f ðxkþ1 Þ ¼ f ðxk Þ þ gTk sk þ oðksk kÞ:

Hence,
f ðxk Þ  f ðxkþ1 Þ
! 1,
gTk sk

which contradicts the second Goldstein condition (2.55). Therefore, gk ! 0, thus proving the global
convergence when αk is determined by the Goldstein conditions.
The global convergence for the case in which the stepsize is computed by the weak Wolfe
conditions is proved similarly. Observe that from the uniform continuity of ∇f(x), it follows that

gTkþ1 dk ¼ gTk d k þ oðkdk kÞ,

such that

gTkþ1 dk
! 1:
gTk dk

But, this contradicts the relation gTkþ1 d k =gTk dk  σ < 1 from (2.59). Therefore, gk ! 0, thus
proving the global convergence when αk is determined by the Wolfe conditions. ♦

The convergence of the general unconstrained optimization Algorithm 2.1 with weak Wolfe
inexact line-search is presented in the next theorem. Mainly, it is proved that the sequence generated
by the algorithm satisfies the Zoutendijk condition.

Theorem 2.28 Let f : ℝn ! ℝ be a continuously differentiable function bounded from below.

Suppose that ∇f(x) is uniformly continuous on the level set S ¼ {x 2 ℝn : f(x)  f(x0)} and αk is
60 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

determined by the weak Wolfe conditions (2.54) and (2.59). Then the sequence generated by
Algorithm 2.1 satisfies

∇f ðxk ÞT sk
lim ¼ 0, ð2:90Þ
k!þ1 ks k k

i.e.,

k∇f ðxk Þk cos θk ! 0: ð2:91Þ

Proof Since ∇f(xk)Tsk < 0 and f is bounded from below, it follows that the sequence {xk} is well
defined and {xk} ⊂ S. Moreover, since {f(xk)} is a decreasing sequence, then it is convergent.
We prove (2.90) by contradiction. Suppose that (2.90) is not true. Then, there exists an ε > 0 and a
subsequence indexed after the set of indices K such that for any k 2 K

∇f ðxk ÞT sk
  ε:
ks k k

But, from (2.54) we have


∇f ðxk ÞT sk
f ðxk Þ  f ðxkþ1 Þ  ρksk k   ρksk kε:
ks k k

Since {f(xk)} is convergent, it follows that the sequence {sk : k 2 K} is also convergent to zero. On
the other hand, from (2.59), for any k  0 we have
 
ð1  σ Þ ∇f ðxk ÞT sk  ð∇f ðxk þ sk Þ  ∇f ðxk ÞÞT sk :

Therefore, for any k 2 K

∇f ðxk ÞT sk 1
ε  k∇f ðxk þ sk Þ  ∇f ðxk Þk: ð2:92Þ
ks k k 1σ

Since we proved that {sk : k 2 K} ! 0, it follows that the right hand side of (2.92) tends to zero
by the uniform continuity of ∇f on set S. But this is a contradiction which actually proves
the theorem. ♦

Observe that (2.90) involves k∇f(xk)k cos θk ! 0, which is exactly the Zoutendijk condition
expressed in (2.88) in another form. The importance of the Zoutendijk condition is if cosθk  δ > 0,
then limk!1k∇f(xk)k ¼ 0. If the hypothesis of uniform continuity is replaced with the Lipschitz
condition of the gradient, then another result on the convergence of Algorithm 2.1 with inexact weak
Wolfe line-search is obtained. But, before proving such a result, let us develop a technical result
which gives a bound of the reduction of the values of the minimizing function f.

Proposition 2.6 Let f : D ! ℝ be a continuously differentiable function on the open set D ⊂ ℝn

with gradient ∇f(x) Lipschitz continuous, i.e., k∇f( y)  ∇ f(x)k  Lky  xk, where L > 0 is constant. If
f(xk þ αdk) is bounded from below and α > 0, then for any αk > 0 which satisfies the weak Wolfe
conditions (2.54) and (2.59), it follows that
2.6 Convergence of the Algorithm with Inexact Line-Search 61

f ðxk Þ  f ðxk þ αk dk Þ  γ k∇f ðxk Þk2 cos 2 hdk , ∇f ðxk Þi, ð2:93Þ

where γ > 0 is a constant.

Proof From the Lipschitz continuity of ∇f and (2.59) we get

αk Lkdk k2  d Tk ð∇f ðxk þ αk d k Þ  ∇f ðxk ÞÞ  ð1  σ ÞdTk ∇f ðxk Þ,

i.e.,
1σ
αk kdk k  kd kk∇f ðxk Þk cos hd k , ∇f ðxk Þi
Lkdk k k
1σ
¼ k∇f ðxk Þk cos hdk , ∇f ðxk Þi:
L
Now, using (2.54) it follows that

f ðxk Þ  f ðxk þ αk dk Þ  ραk d Tk ∇f ðxk Þ

¼ ραk kdk kk∇f ðxk Þk cos hdk , ∇f ðxk Þi
1σ
 ρk∇f ðxk Þk cos hdk , ∇f ðxk Þi k∇f ðxk Þk cos hdk , ∇f ðxk Þi
L
ρð 1  σ Þ
¼ k∇f ðxk Þk2 cos 2 hdk , ∇f ðxk Þi,
L
which is exactly (2.93) with γ ¼ ρ(1  σ)/L. ♦

Theorem 2.29 Let f(x) be a continuously differentiable function on ℝn for which its gradient ∇f(x)
satisfies the Lipschitz condition k∇f( y)  ∇ f(x)k  Lky  xk, where L > 0 is a constant. Let αk be the
stepsize computed by the weak Wolfe conditions (2.54) and (2.59). If the angle condition (2.85) is
satisfied, then, for the sequence {xk} generated by Algorithm 2.1, ∇f(xk) ¼ 0 for a k or f(xk) !  1 or
∇f(xk) ! 0.

Proof Suppose that for any k, ∇f(xk) 6¼ 0. From Proposition 2.6 it follows that

f ðxk Þ  f ðxkþ1 Þ  γ k∇f ðxk Þk2 cos 2 θk ,

where γ ¼ ρ(1  σ)/L is a positive constant independent of k. Then, for any k > 0 we can write

X
k1 X
k1
f ðx0 Þ  f ðxk Þ ¼ ðf ðxi Þ  f ðxiþ1 Þ  γ min k∇f ðxi Þk2 cos 2 θi : ð2:94Þ
0ik
i¼0 i¼0

Now, since θk satisfies the angle condition (2.85), it follows that

X
1
cos 2 θk ¼ þ1:
k¼0

Therefore, from (2.94), ∇f(xk) ! 0 or f(xk) !  1 , thus proving the theorem. ♦

The above theorem is a direct consequence of the Zoutendijk and of the angle conditions.
Moreover, all these results have been obtained in mild conditions: function f is continuously
differentiable and its gradient ∇f(x) is Lipschitz continuous.
62 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

In the following, we point out a result on the reduction of the values of the minimizing function in
conditions in which it is uniformly convex. At every iteration Proposition 2.6 gives a bound of the
reduction of a general function with the gradient Lipschitz continuous. The estimation (2.93) is
expressed in terms of the Zoutendijk condition. Uniform convexity leads us to a new estimation
which depends on the bounds of the Hessian of the minimizing function. As it is known, a function
f(x) is uniformly convex (or strong convex) if there exists a constant η > 0 such that

ðy  zÞT ð∇f ðyÞ  ∇f ðzÞÞ  ηky  zk2 , ð2:95Þ

or there are the positive constants m and M (m < M ) such that

mkyk2  yT ∇2 f ðxÞy  Mkyk2 : ð2:96Þ

Theorem 2.30 Let f(x) be a uniformly convex function and Algorithm 2.1 in which αk satisfies (2.54).
Then
ρη
f ðxk Þ  f ðxk þ αk dk Þ  pffiffiffiffiffiffiffiffiffiffi kαk d k k2 : ð2:97Þ
1 þ M=m

Proof The following two cases must be considered.

(i) Suppose that d Tk ∇f ðxk þ αk d k Þ  0. In this case, we can write

αðk

f ðxk Þ  f ðxk þ αk dk Þ ¼ d Tk ∇f ðxk þ tdk Þdt

0
αðk

 d Tk ð∇f ðxk þ αk d k Þ  ∇f ðxk þ td k ÞÞdt

0
αðk
1
 kdk k2 ηðαk  tÞdt ¼ kαk d k k2 η
2
0
ρη
 pffiffiffiffiffiffiffiffiffiffi kαk dk k2 :
1þ M=m

(ii) Suppose that dTk ∇f ðxk þ αk dk Þ > 0. Then there exists 0 < α < αk such that

dTk ∇f ðxk þ α d k Þ ¼ 0:

Now, from the uniform convexity (2.96) it follows that

1
f ðxk Þ  f ðxk þ α dk Þ  Mkα dk k2 ð2:98Þ
2
and
2.6 Convergence of the Algorithm with Inexact Line-Search 63

1
f ðxk þ αk d k Þ  f ðxk þ α dk Þ  mkðαk  α Þdk k2 : ð2:99Þ
2
Since f(xk þ αkdk) < f(xk), from (2.98) and (2.99) it follows that
rffiffiffiffiffi!
M 
αk  1 þ α :
m

Therefore,

f ðxk Þ  f ðxk þ αk dk Þ  ραk dTk ∇f ðxk Þ

 ραk dTk ð∇f ðxk þ α dk Þ  ∇f ðxk ÞÞ
ρη
 ηραk α kd k k2  pffiffiffiffiffiffiffiffiffiffi kαk dk k2 :
1 þ M=m

Hence, (2.97) is true in both cases. ♦

The convergence of the general unconstrained optimization algorithm with inexact line-search
presented in this section is based on the descent condition ∇f(xk)Tdk < 0 and on the hypothesis that the
search direction dk is bounded in norm away from zero. Moreover, the convergence is proved by
assuming that the minimizing function f is at least twice continuously differentiable and its gradient is
Lipschitz continuous. Therefore, for smooth enough functions, as we can find in real optimization
applications, the above results ensure the convergence of Algorithm 2.1 with inexact line-search.
In the following, we consider another approach of the convergence of Algorithm 2.1 with inexact
line-search for uniformly (strong) convex functions when the search direction satisfies a sufficient
descent condition (Shi, 2004). Indeed, let us assume that the search direction dk satisfies the sufficient
descent condition

gTk dk  ckgk k2 , ð2:100Þ

where c > 0 is a constant.

Theorem 2.31 If:

(i) function f is bounded from below on the level set S ¼ {x 2 ℝn : f(x)  f(x0)}, where x0 2 ℝn is the
initial point,
(ii) the gradient g(x) ¼ ∇ f(x) is Lipschitz continuous on a convex set N which includes S, i.e.
there exists L > 0 such that kg(x)  g( y)k  Lkx  yk, for any x, y 2 N,
(iii) dk is a descent direction which satisfies gTk dk < 0,

then the general unconstrained optimization Algorithm 2.1 with inexact line-searches given by
Armijo, Goldstein, the weak Wolfe, or the strong Wolfe generates an infinite sequence {xk} that
satisfies the Zoutendijk condition

T 2
gk dk
lim ¼ 0: ð2:101Þ
k!1 kd k k

Proof For the algorithm of Armijo with inexact line-search (2.51) let K1 ¼ {k : αk ¼ τk} and
K2 ¼ {k : αk < τk}. Then, from (2.51) it follows that
64 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

f ðxk Þ  f ðxk þ αk dk Þ  ρτk gTk d k , k 2 K1,

f ðxk Þ  f ðxk þ αk dk Þ  ραk gTk dk , k 2 K2:

Since for any k 2 K2, αk/β  τk, we have

f ðxk Þ  f ðxk þ ðαk =βÞdk Þ < ραk gTk dk =β, k 2 K 2 :

By the mean value theorem applied to the left side member of the above inequality, there exists a
θk 2 [0, 1] such that

αk ∇f ðxk þ αk θk dk =βÞT d k =β < ραk gTk d k =β, k 2 K2,

i.e., for any k 2 K2

∇f ðxk þ αk θk d k =βÞT dk > ρgTk dk :

From (ii) and from the Cauchy-Schwarz inequality, it follows that for any k 2 K2

αk Lkdk k2 =β  k∇f ðxk þ αk θk dk =βÞ  ∇f ðxk Þkkdk k

 ð∇f ðxk þ αk θk d k =βÞ  ∇f ðxk ÞÞT d k  ð1  ρÞgTk d k :

Therefore, for any k 2 K2

βð1  ρÞgTk dk
αk   :
Lkdk k2

Hence, for any k 2 K2

2
βρð1  ρÞ gTk dk
f ðxk Þ  f ðxk þ αk dk Þ > :
L kd k k

But, τk ¼  (∇f(xk)Tdk)/kdkk2. Hence, for any k 2 K1

T 2
gk dk
f ðxk Þ  f ðxk þ αk dk Þ  ρ :
kd k k

If we set η ¼ min {ρ, βρ(1  ρ)/L}, then, from the above relations we get

T 2
gk dk
f ðxk Þ  f ðxk þ αk dk Þ  η :
kd k k

From (i) we get (2.101), since the search direction is a descent one.
For the algorithm of Goldstein (2.55), by using the mean value theorem as well as the Cauchy-
Schwarz inequality, it follows that there exists θk 2 [0, 1], such that

ð1  ρÞαk ∇f ðxk ÞT dk  f ðxk þ αk d k Þ  f ðxk Þ ¼ αk ∇f ðxk þ αk θk d k ÞT d k

¼ αk ½∇f ðxk þ αk θk d k Þ  ∇f ðxk ÞT d k þ αk ∇f ðxk ÞT dk
 αk ∇f ðxk ÞT dk þ αk k∇f ðxk þ αk θk dk Þ  ∇f ðxk Þk kdk k
 αk ∇f ðxk ÞT dk þ α2k Lkdk k2 :

Therefore,
2.6 Convergence of the Algorithm with Inexact Line-Search 65

!
ρ gTk d k
αk  :
L kd k k2

Now, taking into consideration (2.54), we get

2
ρ2 gTk d k
f ðx k Þ  f ðx k þ α k d k Þ  :
L kd k k

Therefore, from (i) it follows that the relation (2.101) is true for Algorithm 2.1 with Goldstein
inexact line-search.
For the general algorithm with weak Wolfe inexact line-search, from (ii), from (2.59) and from the
Cauchy-Schwarz inequality we get

αk Lkd k k2  k∇f ðxk þ αk d k Þ  ∇f ðxk Þk kdk k

 ½∇f ðxk þ αk dk Þ  ∇f ðxk ÞT d k  ð1  σ Þ gTk dk :

Therefore,
!
1  σ gTk dk
αk  :
L kd k k2

Now, from (2.54) it follows that

2
ρð1  σ Þ gTk d k
f ðxk Þ  f ðxk þ αk dk Þ  :
L kd k k

Therefore, from (ii) it follows that (2.101) is also true for Algorithm 2.1 with inexact weak Wolfe
line-search. ♦

In conclusion, for the inexact line-search procedures of Armijo, Goldstein, and Wolfe, we proved
that

1. the stepsize αk is lower bounded, and

2. for any k

2
gTk dk
f ðxk þ αk dk Þ  f ðxk Þ  η0 , ð2:102Þ
kd k k

where η0 > 0. These two results are very important and are to be used in proving the convergence
of the general Algorithm 2.1.

In order to establish the convergence of Algorithm 2.1, in the following we shall consider the class
of strongly convex functions. We mention that the strongly convex functions represent a natural class
of functions with very good properties that may be used to prove convergence theorems. If f is twice
continuously differentiable and strongly convex on ℝn, then the hypotheses (i) and (ii) of Theorem
2.31 are satisfied, f(x) has an unique minimum point x, and there exist the constants 0 < m  M such
that for any x, y 2 ℝn
66 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

mkyk2  yT ∇2 f ðxÞy  Mkyk2 , ð2:103Þ

1 1
mkx  x k2  f ðxÞ  f ðx Þ  Mkx  x k2 , ð2:104Þ
2 2

mkx  yk2  ð∇f ðxÞ  ∇f ðyÞÞT ðx  yÞ  Mkx  yk2 , ð2:105Þ

i.e.,

mkx  x k2  ∇f ðxÞT ðx  x Þ  Mkx  x k2 : ð2:106Þ

From the Cauchy-Schwarz inequality and the mean value theorem applied to (2.106), we get
mkx  x k  k∇f ðxÞk  Mkx  x k: ð2:107Þ

Theorem 2.32 Suppose that f : ℝn ! ℝ is a strongly convex function, twice continuously differen-
tiable and

gTk dk  τkgk kkd k k, ð2:108Þ

where 0 < τ  1, then the general Algorithm 2.1 with inexact line-search procedures of Armijo,
Goldstein, and weak Wolfe generates an infinite sequence {xk} such that

lim kgk k ¼ 0, ð2:109Þ

k!1

and {xk} converges at least linearly to x.

Proof Since f is strongly convex, it follows that there exist the constants 0 < m  M such that for any
x 2 ℝn
Mkx  x k  k∇f ðxÞk  mkx  x k:

From (2.105) observe that m  L. From (2.102), by using the new descendent condition (2.108),
we can write
 T 2  T 2
g d g d
f ðxk Þ  f ðxk þ αk d k Þ  η0 kdkk kk ¼ η0 kdk kkkgk k kgk k2
k
2
 η0 τ kgk k  η0 τ m kxk  x k2  η0 τ2 m2 ðf ðxk Þ  f ðx ÞÞ:
2 2 2 2
M
Denote:
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
η τ 2 m2 pffiffiffiffiffiffiffiffiffiffiffiffiffi
γ¼ 2 0 and ζ ¼ 1  γ 2 :
M

Hence

f ðxk Þ  f ðxk þ αk d k Þ  γ 2 ðf ðxk Þ  f ðx ÞÞ: ð2:110Þ

Now, let us prove that for the Armijo, Goldstein, and weak Wolfe inexact line-search, γ  1.
For the Armijo inexact line-search, since
2.6 Convergence of the Algorithm with Inexact Line-Search 67

η0 ¼ min fρ, βρð1  ρÞ=Lg  βρð1  ρÞ=L,

it follows that

η0 τ2 m2 2βρð1  ρÞτ2 m2 β ð1  ρÞτ2 m

γ2 ¼ 2    βτ2 ð1  ρÞ < 1:
M LM M
For the Goldstein inexact line-search, η0 ¼ ρ2/L, then

η0 τ2 m2 2ρ2 τ2 m2 ρτ2 m
γ2 ¼ 2 ¼   τ2 ρ < 1:
M LM M
For the weak Wolfe inexact line-search, η0 ¼ ρ(1  σ)/L, and therefore

η0 τ2 m2 2ρð1  σ Þτ2 m2 ð1  σ Þτ2 m

γ2 ¼ 2 ¼   ð1  σ Þτ2 < 1:
M LM M
Therefore, for all these inexact line-search procedures, γ  1. Now, from (2.110) it follows that

f ðxkþ1 Þ  f ðxk Þ  γ 2 ðf ðxk Þ  f ðx ÞÞ,

or

f ðxk Þ  f ðxk1 Þ  γ 2 ðf ðxk1 Þ  f ðx ÞÞ:

Subtracting f(x) from both members of the above inequality, we get

f ðxk Þ  f ðx Þ  ð1  γ 2 Þðf ðxk1 Þ  f ðx ÞÞ

¼ ζ 2 ðf ðxk1 Þ  f ðx ÞÞ  ⋯  ζ 2k ðf ðx0 Þ  f ðx ÞÞ:

But,
2 2
kx k  x  k2  ðf ðxk Þ  f ðx ÞÞ  ζ 2k ðf ðx0 Þ  f ðx ÞÞ
m m
2ðf ðx0 Þ  f ðx ÞÞ 2k
¼ ζ ¼ ω2 ζ 2k ,
m
where

2ð f ð x 0 Þ  f ð x  Þ Þ
ω2 ¼ :
m
Therefore, kxk  xk  ωζ k, thus establishing the linear convergence of the sequence {xk}
to x. ♦

The above theorem shows that for strongly convex continuously differentiable functions, if the
search direction dk satisfies the descent condition (2.108), then the general unconstrained optimiza-
tion algorithm with inexact line-search procedures of Armijo, Goldstein, and Wolfe for minimizing
the function f is convergent in the sense of (2.109). It is possible to prove that the theorem is
also true in case in which the condition (2.108) is replaced by the sufficient descent condition
gTk dk  c kgk k2 , where c is a positive constant.
68 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

2.7 Three Fortran Implementations of the Inexact Line-Search

For the stepsize computation, the most used algorithms are based on the inexact line-search. In these
algorithms αk is computed to approximately minimize f along the ray {xk þ αdk : α  0} or at least to
sufficiently reduce f. Plenty of inexact line-search procedures were proposed: Goldstein (1965),
Armijo (1966), Wolfe (1969, 1971), Powell (1976), Dennis and Schnabel (1983), Fletcher (1987),
Potra and Shi (1995), Lemaréchal (1981), Shanno (1983), Moré and Thuente (1990), Hager and
Zhang (2005), and many others.
Any line-search procedure requires an initial estimate α0 of the stepsize and generates a sequence
{αk} that converges to a stepsize satisfying the conditions imposed by the user (e.g., Armijo,
Goldstein, Wolfe, etc.), or it informs that such a stepsize does not exist. Mainly, line-search
procedures have two phases: a bracketing phase that finds an interval [a, b] containing an acceptable
value for the stepsize and a selection phase that locates the final stepsize. The selection phase reduces
the bracketing interval during the search and usually interpolates some values of the function and its
gradient obtained from the earlier steps of searching in order to guess the location of the minimum.
In the following, we shall present three implementations of the inexact line-search: backtracking
(Armijo), the Wolfe line-search with simple bisection, and the Wolfe line-search with safeguarding
and cubic interpolation. They enable the reader to see the subtlety and refinement of the implemen-
tation in the computing programs of line-search algorithms.

1. Backtracking (Armijo’s line-search). Figure 2.1 presents a Fortran implementation of the Armijo
line-search.

The Armijo line-search with backtracking is not efficient and robust for solving large-scale
unconstrained optimization problems. This was introduced here just to show that it is an important
component of the Wolfe line-search conditions. The Armijo line-search does not guarantee reason-
able progress of the line-search. Theorem 2.15 only shows that (2.52) is always satisfied for
sufficiently small αk, provided that dk is a descent direction, but we are not sure that αk becomes
unacceptably small. In order to avoid such cases, the curvature condition has to be introduced.

2. Wolfe line-search with simple bisection. Figure 2.2 presents a Fortran implementation of the Wolfe
line-search with simple bisection. This is a simple procedure for computing a value of αk which
satisfies the Wolfe line-search conditions (2.54) and (2.59) without any safeguarding against the
too large or too small values of αk.

3. Wolfe line-search with safeguarding and cubic interpolation. This is a variant of a line-search
procedure which is simple enough to generate safeguarded stepsizes satisfying the Wolfe
conditions (2.54) and (2.59) (proposed by Shanno, (1983), with minor modifications by Andrei,
1995). Suppose that we are at the iteration k. In order to have a simple interpretation of the
procedure and a clear description, a Fortran version of it is presented in Fig. 2.3.

The inputs of this procedure are n the number of variables, x ¼ xk a vector with the current values
of variables, f ¼ f(xk) the value of the minimizing function in x, d ¼ dk the current search direction,
gtd ¼ ∇ f(xk)Tdk the scalar product of the current gradient and the search direction, and dnorm ¼ kdkk
the l2-norm of the search direction. The outputs of the procedure are alpha ¼ αk the stepsize satisfying
the Wolfe line-search conditions, xnew ¼ xkþ1 ¼ xk þ αkdk the new point, fnew ¼ f(xkþ1) the function
value in the new point, gnew ¼ ∇ f(xkþ1) the gradient of the minimizing function in the new point,
2.7 Three Fortran Implementations of the Inexact Line-Search 69

fgcnt the number of function and its gradient calls, lscnt indicates that the line-search procedure
performed a number of iterations, and lsflag indicates that the number of iterations in the line-search
procedure is greater than a prespecified threshold.
In Fig. 2.3, maxls is the maximum number of iterations in the line-search procedure, epsm is the
epsilon machine, and evalfg(n,xnew,fnew,gnew) is the subroutine that implements the
algebraic expressions of the minimizing function and its gradient. In the input, this subroutine has
n as the number of variables and xnew as the new point. In the output, it computes fnew as the value
of function f in the new point and gnew as the gradient of f in the new point.
We can see that a line-search procedure is complicated. To be reliable, it must incorporate a lot of
features. Firstly, observe that the Wolfe conditions are implemented in a complicated form, which
takes into consideration both the ratio between the rate of the decrease of f in the direction dk at the
new point and the rate of the decrease in the direction dk at the current point xk. It also takes some
precautions to avoid too small or too large values of the stepsize. Observe that in the selection phase
of the procedure, the cubic interpolation is used. The cubic interpolation provides a good model for

the minimizing function in the searching interval. Suppose we have an interval a, b containing the
desirable stepsize and two previous stepsizes estimates αi1 and αi in this interval. A cubic function is
used to interpolate the values, φk(αi1), φk(αi), φ0k ðαi1 Þ, and φ0k ðαi Þ, where φk(α) ¼ f(xk þ αdk).
Bulirsch and Stoer (1980) proved that such a cubic function always exists and is unique. The

minimizer of this cubic function in a, b , that is, a new estimation of the stepsize, is either at one
of the endpoints or in the interior, case in which it is given by

*
Subroutine back(n,x,d,fx,gtd,rho,sigma, alpha,xnew,fxnew,gradnew,
* fgcnt,lscnt, nexp)
*-----------------------------------------------------------------------.
* Subroutine back implements the backtracking algorithm. (Armijo)
*
* Input parameters:
* =================
* n (integer) the number of variables in the function to be
* minimized.
* x (double) vector containing the current estimation to the
* minimizer.
* d (double) vector containing the current searching direction.
* fx (double) function value in the current point.
* gtd (double) scalar product: (gradient)T * direction.
* rho (double) rho parameter in backtracking.
* sigma(double) parameter for reducing in backtracking.
* nexp (integer) the number of the numerical experiment.
*
* Output parameters:
* ==================
* alpha (double) steplength satisfying the backtracking condition.
* xnew (double) vector containing the new estimation to the minimum
* of function: xnew(i) = x(i) + alpha*d(i).
* fxnew (double) function value in xnew.
* gradnew (double) the gradient in xnew.
* fgcnt (integer) the number of function and its gradient evaluations.
* lscnt (integer) the number of linear seaching.
*
* Calling subroutines:
* ====================
* call evalfg(n,x,f,g)
* This is a user supplied subroutine which calculates the function and
* its gradient at x and places them in f and g respectively.
*
* Remark:
Fig. 2.1 Subroutine back which implements backtracking (Armijo)
70 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

* =======
* Usually, the subroutine back is called with the following values of
* rho and sigma parameters: rho = 0.0001 and sigma = 0.8.
* These values do not have a significative influence on the performances
* of the searching subroutine.
*-----------------------------------------------------------------------.

real*8 x(n),d(n), xnew(n), gradnew(n)

real*8 fx, fxnew, fy, gtd
real*8 rho, sigma, alpha
real*8 y(100000), gy(100000)
integer fgcnt, lsiter
*
alpha=1.d0
lsiter=1
*
1 continue
do i=1,n
y(i)=x(i) + alpha*d(i)
end do
c1 ------------------------------------------ ! function evaluation
call evalfg(n,y,fy,gy, nexp)
fgcnt = fgcnt + 1

if(fy .le. fx + rho*alpha*gtd) go to 10

if(fy .gt. fx + rho*alpha*gtd) then

alpha = alpha * sigma
lsiter=lsiter+1
go to 1
end if
*
10 continue

do i=1,n
xnew(i) = x(i) + alpha*d(i)
end do
c2 -------------------------------------------- ! function evaluation
call evalfg(n,xnew,fxnew,gradnew, nexp)
fgcnt = fgcnt + 1

if ( lsiter .ne. 0 ) then

lscnt = lscnt + 1
end if

return
end
Fig. 2.1 (continued)

 
φ0k ðαi Þ þ b  a
αiþ1 ¼ αi  ðαi  αi1 Þ , ð2:111Þ
φ0k ðαi Þ  φ0k ðαi1 Þ þ 2b

where
φk ðαi1 Þ  φk ðαi Þ
a ¼ φ0k ðαi1 Þ þ φ0k ðαi Þ  3 , ð2:112Þ
αi1  αi
 1=2
b ¼ a2  φ0k ðαi1 Þφ0k ðαi Þ : ð2:113Þ

In Fig. 2.3 the new estimate αiþ1 is computed as alphatemp. The interpolation process can be
repeated by discarding the data at one of the stepsizes αi1 or αi and replacing it by φk(αiþ1) and
φ0k ðαiþ1 Þ: Observe that the interpolation step that determines a new estimation to the stepsize is
2.7 Three Fortran Implementations of the Inexact Line-Search 71

*
subroutine LSbis(n,x,f,d,gtd,alpha,xnew,fnew,gnew,fgcnt, nexp)
*-----------------------------------------------------------------------.
* Subroutine LSbis implements the Wolfe line-search conditions with
* bisection.(Andrei)
*
* Input parameters:
* =================
* n (integer) the number of variables in the function to be
* minimized.
* x (double) vector containing the current estimation to the
* minimizer.
* f (double) the function value in current point.
* d double) vector containing the current searching direction.
* gtd (double) scalar product: (gradient)T * direction.
* nexp (integer) the number of the numerical experiment.
*
* Output parameters:
* ==================
* alpha (double) steplength satisfying the backtracking condition.
* xnew (double) vector containing the new estimation to the minimum
* of function: xnew(i) = x(i) + alpha*d(i).
* fxnew (double) function value in xnew.
* gnew (double) the gradient in xnew.
* fgcnt (integer) the number of function and gradient evaluations.
*
* Calling subroutines:
* ====================
* call evalfg(n,x,f,g)
* This is a user supplied subroutine which calculates the function and
* its gradient at x and places them in f and g respectively.
*
* Remark:
* =======
* The Wolfe line-search conditions use two parameters.In this subroutine
* these parameters are cw1=0.0001d0 used in the first Wolfe condition
* and cw2=0.8d0 used in the second Wolfe condition.
*-----------------------------------------------------------------------.

parameter(ia=50000)
double precision x(n), d(n), xnew(n), gnew(n)
double precision xtemp(ia), ftemp, gtemp(ia)
double precision f, fnew, gtd, gtd1
double precision alpha, alphalow, alphahigh
integer fgcnt, nexp, ils

c alpha = 1.d0
alphalow = 0.d0
alphahigh = 0.d0

cw1 = 0.0001d0
cw2 = 0.8d0

ils = 0

10 continue
ils = ils + 1
if(ils .gt. 20) return

do i=1,n
xtemp(i) = x(i) + alpha*d(i)
end do
call evalfg(n,xtemp,ftemp,gtemp, nexp)
fgcnt = fgcnt + 1

c Test W1 (The First Wolfe condition)

if(ftemp .le. f + cw1*alpha*gtd) then

go to 50

else

Fig. 2.2 Subroutine LSbis which generate stepsizes satisfying the weak Wolfe line-search with simple bisection
72 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

alphahigh = alpha
alpha = (alphalow + alphahigh)/2.d0
go to 10
end if

50 continue

gtd1 = 0.d0
do i=1,n
gtd1 = gtd1 + gtemp(i)*d(i)
end do

c Test W2 (The second Wolfe condition)

if(gtd1 .ge. cw2*gtd) then

go to 100
else
alphalow = alpha
if(alphahigh .eq. 0.d0) alpha = 2.d0*alphalow
if(alphahigh .gt. 0.d0) alpha = (alphalow+alphahigh)/2.d0
end if

go to 10

100 continue

do i=1,n
xnew(i) = x(i) + alpha*d(i)
end do

call evalfg(n,xnew,fnew,gnew, nexp)

fgcnt = fgcnt + 1

return
end

Fig. 2.2 (continued)

*
subroutine wolfeLS (n,x,f,d,gtd,dnorm, alpha,xnew,fnew,gnew,
+ fgcnt,lscnt,lsflag, nexp)
*-----------------------------------------------------------------------.
* Subroutine wolfeLS implements the weak Wolfe line-search conditions.
*
C SCALAR ARGUMENTS
integer n,fgcnt,lscnt,lsflag
double precision f,gtd,dnorm,alpha,fnew

C ARRAY ARGUMENTS
double precision x(n),d(n),xnew(n),gnew(n)

C LOCAL SCALARS
integer i,lsiter
double precision alphap,alphatemp,fp,dp,gtdnew,a,b

lsflag = 0
* Maximum number of Line-Search is maxls (now=20)
maxls=20

Fig. 2.3 Subroutine wolfeLS which generate safeguarded stepsizes satisfying the inexact weak Wolfe line-search with
cubic interpolation
2.7 Three Fortran Implementations of the Inexact Line-Search 73

alphap = 0.0d0
fp = f
dp = gtd

do i = 1,n
xnew(i) = x(i) + alpha * d(i)
end do
c1 ---------------------------------------- ! function evaluation
call evalfg(n,xnew,fnew,gnew, nexp)
fgcnt = fgcnt + 1

gtdnew = 0.0d0
do i = 1,n
gtdnew = gtdnew + gnew(i) * d(i)
end do

lsiter = 0

* Test whether the Wolfe line-seach conditions have been met.

10 if ( alpha * dnorm .gt. 1.0d-30 .and. lsiter .lt. maxls .and.

+ .not. ( gtdnew .eq. 0.0d0 .and. fnew .lt. f ) .and.
+ ( ( fnew .gt. f + 1.0d-04 * alpha * gtd .or.
+ abs( gtdnew / gtd ) .gt. 0.9d0 ) .or. ( lsiter .eq. 0 .and.
+ abs( gtdnew / gtd ) .gt. 0.5d0 ) ) ) then

* Thest whether the new point has a negative slope and a higher function
* value than that corresponding to alpha=0. In this case, the search has
* passed through a local max and is heading for a distant local minimum.
* Reduce alpha, compute the new corresponding point, its function value
* and gradient, as well as gtdnew.
* Repeat this test until a good point has been found.

20 if ( alpha * dnorm .gt. 1.0d-30 .and. fnew .gt. f .and.

+ gtdnew .lt. 0.0d0 ) then

alpha = alpha / 3.0d0

do i = 1,n
xnew(i) = x(i) + alpha * d(i)
end do
c2 ------------------------------------------------ ! function evaluation
call evalfg(n,xnew,fnew,gnew, nexp)
fgcnt = fgcnt + 1

gtdnew = 0.0d0
do i = 1,n
gtdnew = gtdnew + gnew(i) * d(i)
end do

alphap = 0.0d0
fp = f
dp = gtd

goto 20

end if

* Cubic interpolation to find a new trial point corresponding to alphatemp.

a = dp + gtdnew - 3.0d0 * ( fp - fnew ) / ( alphap - alpha )

b = a ** 2 - dp * gtdnew

if ( b .gt. 0.0d0 ) then

b = sqrt( b )
else
b = 0.0d0
end if
alphatemp = alpha - ( alpha - alphap ) * ( gtdnew + b - a ) /
+ ( gtdnew - dp + 2.0d0 * b )

Fig. 2.3 (continued)

74 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

* Test whether the line minimum has been bracketed.

if ( gtdnew / dp .le. 0.0d0 ) then

* Here the minimum has been bracketed.

* Test whether the trial point lies sufficiently within the bracketed
* interval. If it does not, choose alphatemp as the midpoint of the
* interval.

if ( 0.99d0 * max( alpha, alphap ) .lt. alphatemp .or.

+ alphatemp .lt. 1.01d0 * min( alpha, alphap ) ) then
alphatemp = ( alpha + alphap ) / 2.0d0
end if

else

* Here the minimum has not been bracketed.

* The trial point is too small, double the largest prior point.

if ( gtdnew .lt. 0.0d0 .and.

+ alphatemp .lt. 1.01d0 * max( alpha, alphap ) ) then
alphatemp = 2.0d0 * max( alpha, alphap )
end if

* The trial point is too large, halve the smallest prior point.

if ( ( gtdnew .gt. 0.0d0 .and.

+ alphatemp .gt. 0.99d0 * min( alpha, alphap ) ) .or.
+ alphatemp .lt. 0.0d0 ) then
alphatemp = min( alpha, alphap ) / 2.0d0
end if

end if

* Save and continue the search.

alphap = alpha
fp = fnew
dp = gtdnew

alpha = alphatemp

do i = 1,n
xnew(i) = x(i) + alpha * d(i)
end do
c3 ----------------------------------------------- ! function evaluation
call evalfg(n,xnew,fnew,gnew, nexp)
fgcnt = fgcnt + 1

gtdnew = 0.0d0
do i = 1,n
gtdnew = gtdnew + gnew(i) * d(i)
end do

lsiter = lsiter + 1

goto 10

c------------------------------ End if for stopping criteria

end if
if ( lsiter .ge. maxls ) then
lsflag = 1
end if

if ( lsiter .ne. 0 ) then

lscnt = lscnt + 1
end if
return
end

Fig. 2.3 (continued)

2.8 Numerical Studies: Stepsize Computation 75

safeguarded in order to ensure that the new stepsize is not too close to the endpoints of the interval.
Some more details may be found, for example, in (Dennis, & Schnabel, 1983), (Shanno, 1983), and
(Nocedal, & Wright, 2006).

2.8 Numerical Studies: Stepsize Computation

As we have already seen, for the stepsize αk computation along the descent direction dk, there are a lot
of algorithms for which plenty of theoretical developments subject to their definition and conver-
gence were presented. The problem we face, problem which is one of the greatest provocations of any
theory, is that of its implementation in computer programs able to solve a large diversity of
optimization problems with different structures and complexities. Therefore, in the following, we
shall consider three line-search algorithms for the stepsize computation: backtracking (see subroutine
back in Fig. 2.1), the weak Wolfe line-search (Shanno-Phua) (Shanno, & Phua, 1976, 1980), (Shanno,
1983), (see subroutine WolfeLS in Fig. 2.3), and the strong Wolfe line-search (Moré-Thuente) (Moré,
& Thuente, 1992; Andrei, 2021b) (see subroutines MTlines in Andrei, 2021b, pp. 15–22). We want to
see their performances in the context of two unconstrained optimization algorithms: the steepest
descent and the Hestenes-Stiefel conjugate gradient. We mention that these unconstrained optimiza-
tion algorithms will be analyzed in Chaps. 3 and 5, respectively.
Firstly, let us consider the steepest descent algorithm equipped with the above three line-searches.
The search direction of the steepest descent is dk ¼  gk.

Example 2.1 Consider the function

!2
X
n1
2
X
n
f 1 ðxÞ ¼ ð x i  1Þ þ x2j  0:25
i¼1 j¼1

with the initial point x0 ¼ [1, 2, . . ., n], and n ¼ 100. The performances of the steepest descent
algorithm with the above three line-searches are presented in Table 2.1.

Example 2.2 For the function

 2 X
n1  2  2
f 2 ðxÞ ¼ 3x1  2x21 þ 3xi  2x2i  xi1  2xiþ1 þ 1 þ 3xn  2x2n  xn1 þ 1
i¼2

with the initial point x0 ¼ [1, 1, . . ., 1] and n ¼ 100, Table 2.2 shows the performances of the
steepest descent algorithm equipped with three line-searches.

Example 2.3 Now, consider the function

X
n=2
 2  2
f 3 ðxÞ ¼ ð1:5  x2i2 þ x2i1 x2i Þ2 þ 2:25  x2i1 þ x2i1 x22i þ 2:625  x2i1 þ x2i1 x32i
i¼1

with the initial point x0 ¼ [1, 0.8, . . ., 1, 0.8] and n ¼ 100. Table 2.3 presents the performances of the
steepest descent algorithm equipped with three line-searches.
76 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Table 2.1 Performances of the inexact line-searches: backtracking, weak Wolfe, and strong Wolfe. Steepest descent
algorithm. Function f1(x)

Line-search #iter #fg cpu(s) f* g

Backtracking 137 2836 62.47 75 0.953999e-06
Weak Wolfe 14 48 1.18 75 0.680075e-06
Strong Wolfe 20 103 2.39 75 0.248833e-06

Table 2.2 Performances of the inexact line-searches: backtracking, weak Wolfe, and strong Wolfe. Steepest descent
algorithm. Function f2(x)

Line-search #iter #fg cpu(s) f* g

Backtracking 607 13665 330.81 0.397067 0.827037e-06
Weak Wolfe 78 195 4.58 0.222e-13 0.826909e-06
Strong Wolfe 93 289 6.85 0.1575e-13 0.797402e-06

Table 2.3 Performances of the inexact line-searches: backtracking, weak Wolfe, and strong Wolfe. Steepest descent
algorithm. Function f3(x)

Line-search #iter #fg cpu(s) f* g

Backtracking 1142 19617 477.28 0.1254e-12 0.984514e-06
Weak Wolfe 1087 2218 52.27 0.5976e-12 0.681328e-06
Strong Wolfe 300 898 21.73 0.1337e-11 0.928115e-06

Table 2.4 Performances of the inexact line-searches: backtracking, weak Wolfe, and strong Wolfe. Hestenes-Stiefel
conjugate gradient algorithm. Function f1(x)

Line-search #iter #fg cpu(s) f* g

Backtracking 229 7977 132.15 9453.238852 0.606498e-04
Weak Wolfe 115 2912 52.94 9453.238852 0.127230e-05
Strong Wolfe 23 107 2.20 9453.238852 0.576972e-05

In the next numerical experiment, let us see the performances of the three line-search algorithms
for the stepsize αk computation (backtracking, weak Wolfe, strong Wolfe) in the context of the
Hestenes-Stiefel conjugate gradient algorithm. The search direction in this algorithm is computed as
dkþ1 ¼  gk þ βkdk, in which the conjugate gradient parameter βk is computed as βk ¼ gTkþ1 yk =yTk d k ,
where yk ¼ gkþ1  gk. The Hestenes-Stiefel conjugate gradient will be presented and analyzed in
Chap. 5.

Example 2.4 Table 2.4 shows the performances of the line-search algorithms, backtracking, weak
Wolfe, and strong Wolfe, for the minimization of function f1(x) presented in Example 2.1 with the
same initial point and n ¼ 10000 and using the Hestenes-Stiefel conjugate gradient algorithm.
2.8 Numerical Studies: Stepsize Computation 77

Table 2.5 Performances of the inexact line-searches: backtracking, weak Wolfe, and strong Wolfe. Hestenes-Stiefel
conjugate gradient algorithm. Function f2(x)

Line-search #iter #fg cpu(s) f* g

Backtracking 264 5070 9.39 1.961211 0.627392e-05
Weak Wolfe 44 82 0.82 0.302616e-13 0.189612e-05
Strong Wolfe 68 164 1.48 0.712527 0.322331e-05

Table 2.6 Performances of the inexact line-searches: backtracking, weak Wolfe, and strong Wolfe. Hestenes-Stiefel
conjugate gradient algorithm. Function f3(x)

Line-search #iter #fg cpu(s) f* g

Backtracking 84 1075 10.87 0.581314e-10 0.691753e-04
Weak Wolfe 12 22 0.39 0.103860e-14 0.319524e-07
Strong Wolfe 16 34 0.60 0.147450e-12 0.104290e-05

In the above tables: #iter represents the number of iterations to get a solution, #fg is the number of function and its
gradient evaluations, cpu(s) is the CPU computing time in seconds, f  is the value of the minimizing function in the
minimum point, kgk is the norm of the gradient in the minimum point

Example 2.5 Table 2.5 shows the performances of the line-search algorithms, backtracking, weak
Wolfe, and strong Wolfe, for the minimization of function f2(x) given in Example 2.2 with the same
initial point and n ¼ 10000 and using the Hestenes-Stiefel conjugate gradient algorithm.

Example 2.6 Table 2.6 presents the performances of the line-search algorithms, backtracking, weak
Wolfe, and strong Wolfe, for the minimization of function f3(x) with n ¼ 10000 and using the
Hestenes-Stiefel conjugate gradient algorithm.

From the above tables we can see that the inexact weak and the inexact strong Wolfe line-searches
are the best line-search procedures for the stepsize computation. Observe that the weak Wolfe line-
search is more efficient than the strong Wolfe line-search. However, from this limited set of
numerical experiments, it cannot be concluded which version of the Wolfe line-search (weak or
strong) is the better. That is why we next illustrate the running of some conjugate gradient algorithms
for solving the problems from the UOP collection (Andrei, 2020a).

Numerical Studies
1. In the following, we consider the Hestenes-Stiefel conjugate gradient algorithm in which the
stepsize is computed by the weak Wolfe line-search with cubic interpolation (Shanno) (see
Fig. 2.3) and the strong Wolfe line-search Moré-Thuente (see Andrei, 2021b) for solving the
problems from the UOP collection (Andrei, 2020a, pp. 455–466). The UOP collection includes
80 functions in generalized or extended form. For each test function ten numerical experiments
with the number of variables n ¼ 1000, 2000, . . ., 10000 have been considered, thus obtaining a
number of 800 problems. All the algorithms have been coded in double precision Fortran and
compiled with f77 (default compiler settings) and run on an Intel Pentium 4, 1.8 GHz workstation.
For the comparison of algorithms, we follow the Remark 1.1
78 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Fig. 2.4 Performances of the Hestenes-Stiefel conjugate gradient algorithm with the inexact weak Wolfe line-search
versus the inexact strong Wolfe line-search

The iterations are stopped if the inequality kgkk1  106 is satisfied, where k.k1 is the maximum
absolute component of a vector. In line-search algorithms, we set ρ ¼ 0.0001 and σ ¼ 0.8. The
maximum number of iterations was limited to 2000.
Figure 2.4 presents the Dolan and Moré (2002) performances profiles of the Hestenes-Stiefel
conjugate gradient algorithm in which the stepsize is computed by the weak Wolfe line-search with
cubic interpolation (Shanno) and the strong Wolfe line-search (Moré-Thuente).

In a performance profile plot, the top curve corresponds to the method that solved the most
problems in a time that was within a given factor of the best time. The percentage of the test problems
for which a method is the fastest is given on the left axis of the plot. The right side of the plot gives the
percentage of the test problems that were successfully solved by these algorithms, respectively. The
left side of the plot is a measure of the efficiency of an algorithm, while the right side is a measure of
the robustness of an algorithm. It is obvious that the Hestenes-Stiefel with inexact weak Wolfe line-
search is more efficient and more robust versus the same algorithm with strong Wolfe line-search.
From Fig. 2.4, when comparing weak Wolfe versus strong Wolfe line-searches subject to the
number of iterations or subject to the CPU time metric, we can see that the weak Wolfe is top
performer. The table inside the plot of Fig. 2.4 shows that, subject to the number of iterations, the
weak Wolfe is better in 366 problems (i.e., it achieved the minimum number of iterations in
366 problems). The strong Wolfe is better in 222 problems and they achieve the same number of
iterations in 92 problems, etc. Out of 800 problems considered in this numerical study, only for
680 problems does the criterion (1.3) hold. Therefore, subject to the number of iterations metric or
subject to the CPU time metric, on average, the weak Wolfe appears to slightly generate the best
stepsizes.

2. Now, let us report some numerical results obtained with a Fortran implementation of the Hager
and Zhang conjugate gradient algorithm (Hager, & Zhang, 2005), in which the stepsize is
computed by using the weak Wolfe line-search with a simple bisection technique (see Fig. 2.2),
the weak Wolfe line-search with cubic interpolation (Shanno) (see Fig. 2.3), and the strong Wolfe
line-search (Moré-Thuente) (see Andrei, 2021b) for solving 800 problems from the UOP collec-
tion with the number of variables in the range [1000, 10000].
The conjugate gradient parameter in the Hager and Zhang conjugate gradient algorithm (HZ) is
computed as
2.8 Numerical Studies: Stepsize Computation 79

 
βk ¼ max βNk , ηk ,

T
1 ky k k2
βk ¼ T
N
y 2 T gkþ1 , ð2:114Þ
yk d k k yk d k
1
ηk ¼ , η ¼ 0:01:
kdk k min fη, kgk kg

(This algorithm will be presented in Chap. 5.) Figure 2.5 presents the Dolan and Moré performance
profiles of these algorithms: HZ with weak Wolfe line-search (HZWOLFE), HZ with weak Wolfe
line-search with simple bisection (HZBIS), and HZ with strong Wolfe line-search (HZSWOLFE).

From Fig. 2.5 we can see that the strong Wolfe line-search is more expensive than the weak Wolfe
line-search. For example, subject to the CPU computing time, HZWOLFE is faster in solving
476 problems (out 664), and HZSWOLFE is faster in solving only 89 problems. However, the strong
Wolfe line-search has the advantage that, by decreasing σ, we have a direct control of the quality of
the search by forcing the accepted value of the stepsize α to be near enough to a local minimum. This
is important in nonlinear conjugate gradient methods for solving large-scale unconstrained
optimization.

Fig. 2.5 Performance profiles of the Hager-Zhang algorithm with different line-search procedures
80 2 Fundamentals on Unconstrained Optimization. Stepsize Computation

Table 2.7 Global performances of HZ for solving 800 unconstrained optimization problems from the UOP collection
with different line-searches

Algorithm and line-search procedure #iter #fg cpu(s)

HZ with weak Wolfe line-search with simple bisection 941932 5608849 8459.79
HZ with weak Wolfe line-search with cubic interpolation 422540 2243545 1654.15
HZ with strong Wolfe line-search 534378 4618333 8230.91

It is interesting to notice the global performances of the Hager-Zhang conjugate gradient algorithm
(HZ) with three line-search procedures: the weak Wolfe line-search with bisection, the weak Wolfe
line-search with cubic interpolation, and the strong Wolfe line-search. Table 2.7 presents the global
performances of HZ for solving this set of 800 unconstrained optimization problems with different
line-searches.
The conclusion of these intensive numerical studies is that the weak Wolfe line-search is more
efficient and more robust versus the backtracking and versus the Wolfe line-search with bisection and
versus the strong Wolfe line-search. A comparison between the subroutine WolfeLS which
implements the weak Wolfe line-search with safeguarded stepsize and cubic interpolation
(Fig. 2.3) on one side and the subroutines LSbis (Fig. 2.2) and back (Fig. 2.1) shows the importance
of the line-search in the frame of unconstrained optimization algorithms. The strong Wolfe line-
search in the implementation of Moré and Thuente (see subroutine MTlines in Andrei, (2021b),
pp. 15–22) is more sophisticated. It implements the safeguarding of stepsizes by introducing some
large limits on them, uses the quadratic or the cubic interpolation, possibly taking the average of these
quadratic and cubic steps. However, in our numerical experiments, we found that the weak Wolfe
line-search with safeguarding and cubic interpolation implemented in the subroutine WolfeLS is more
efficient and more robust than the strong Wolfe line-search implemented in the subroutine MTlines.

Notes and References

This chapter was dedicated to the stepsize computation in the frame of unconstrained optimization
algorithms. The stepsize computation is an important component of any optimization algorithm.
Starting from xk, the stepsize computation is to determine αk, in the direction dk such that φ(αk) < φ(0),
where φ(α) ¼ f(xk þ αdk). This is a one-dimensional minimization and can be achieved by using the
traditional methods: dichotomous search, Fibonacci and golden-section search, Powell quadratic
(1964) and Davidon cubic interpolation (1959), two-point interpolation, etc. These methods have
not been discussed in this book, but the interested reader can find them, for example, in Antoniou and
Lu (2007) or Walsh (1975).
The modern line-searches presented in Sect. 2.5 are based on the Armijo and Wolfe line-search
conditions. No numerical studies on their performances are known. An excellent presentation of the
theoretical developments concerning the convergence of the general algorithm with descent direction
for unconstrained optimization with both exact and inexact line-searches was given by Sun and Yuan
(2006). We followed their presentation. Details on the Wolfe line-search are also given by Dennis and
Schnabel (1983) and Nocedal and Wright (2006). See also (Pulkkinen, 2008), (Bartholomew-Biggs,
2008), and (Andrei, 2009e). Nocedal and Wright (2006) presented important details on computing the
stepsize by the strong Wolfe line-search concerning the initial stepsize selection, cubic interpolation,
selection of the bracketing interval [αlo, αhi], selection of the stepsize αj in this interval and
replacement of one of the endpoints by αj, stopping test, etc.
More details on the stepsize computation for unconstrained optimization algorithms are given by
(Gill, & Murray, 1974a; Murray, & Overton, 1979; Gill, Murray, & Wright, 1981; Lemaréchal, 1981;
Fletcher, 1987; Dennis, & Schnabel, 1983; Kelley, 1999; Shi, & Shen 2015; Gould, 2006; Nocedal, &
Wright, 2006; Yuan, 2006; Sun, & Yuan, 2006; Griva, Nash, & Sofer, 2009; Andrei, 2020a; etc.).
 Steepest Descent Methods
3

The steepest descent method was designed by Cauchy (1847) and is the simplest of the gradient
methods for the optimization of general continuously differential functions in n variables. Its
importance is due to the fact that it gives the fundamental ideas and concepts of all unconstrained
optimization methods. It introduces a pattern common to many optimization methods. In this pattern,
an iteration consists of two parts: the choice of a descent search direction dk followed at once by a
line-search to find a suitable stepsize αk. The search direction in the steepest descent method is exactly
the negative gradient.
Actually, this is not an efficient method for solving large-scale and complex unconstrained
optimization problems and applications. This simple strategy of moving along the negative gradient
works well for particular functions with near-circular contours, but practical optimization problems or
real applications may involve functions with narrow curving valleys which need a more sophisticated
approach.
It is common practice in optimization algorithms to introduce the steepest descent when the search
direction is not well defined. In fact, one point that emphasizes the importance of the steepest descent
in optimization algorithms is exactly this one. All the respectable optimization algorithms include this
stratagem: when the search direction is not well defined, try the steepest descent. This stratagem is
often used as protection to make sure the algorithm is running appropriately.

3.1 The Steepest Descent

The unconstrained problem considered is

min ff ðxÞ : x 2 ℝn g, ð3:1Þ

where f : ℝn ! ℝ is a continuously differentiable function lower bounded. If xk is the current point

which represents an approximation to the minimum point x, then f(x) can be approximated by its
affine model

f ðxk þ dÞ  f ðxk Þ þ ∇f ðxk ÞT d,

where d 2 ℝn is small, i.e., kdk is small.

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 81

N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_3
82 3 Steepest Descent Methods

Observe that, if the above approximation of function f is good enough, then we want to choose the
direction d so that ∇f(xk)Td is as small as possible. Let us normalize the direction d so that kdk ¼ 1.
Then, among all the directions d with kdk ¼ 1, the direction
∇f ðxk Þ
de ¼
k∇f ðxk Þk

makes the smallest inner product with the gradient ∇f(xk). Indeed, this follows from the following
inequalities:
∇f ðxk Þ e
∇f ðxk ÞT d  k∇f ðxk Þkkd k ¼ ∇f ðxk ÞT ¼ ∇f ðxk ÞT d:
k∇f ðxk Þk

Therefore, the un-normalized direction dk ¼  ∇ f(xk) is called the direction of steepest descent at
point xk.
Observe that dk ¼  ∇ f(xk) is a descent direction as long as ∇f(xk) 6¼ 0. To see this, we can write
∇f(xk)Tdk ¼  ∇ f(xk)T ∇ f(xk) < 0, so long as ∇f(xk) 6¼ 0. With these, the following algorithm, called
the steepest descent algorithm, may be presented.

Algorithm 3.1 Steepest descent

1. Choose an initial point x0 and a convergence tolerance ε > 0 sufficiently small for stopping the iterations.
Set k ¼ 0
2. Compute ∇f(xk). Set dk ¼  ∇ f(xk)
3. If kdkk  ε, then stop; otherwise, continue with step 4
4. Compute the stepsize αk by the exact or by the inexact line-search procedures
5. Set xkþ1 ¼ xk þ αkdk, k ¼ k þ 1 and go to step 2 ♦

Since dk ¼  ∇ f(xk) is a descent direction, it follows that f(xk+1) < f(xk). In step 4, the exact or the
inexact line-search may be used. Let us consider a numerical example to see the running of the
steepest descent algorithm.

Example 3.1 Consider the function

 
f 1 ðxÞ ¼ ð0:5 þ 0:5x1 Þ4 x42 exp 2  ð0:5 þ 0:5x1 Þ4  x42 ,

illustrated in Fig. 3.1.

The local minimum point of this function is x ¼ [1, 1], in which f 1 ¼ 1: For the initial point
x0 ¼ [0.1, 0.8] the steepest descent algorithm, in which in step 4 the stepsize is computed by
backtracking, gives the solution in 49 iterations  and 544 evaluations of the minimizing function
and its gradient. The evolutions of the error f 1 ðxk Þ  f 1  as well as of kgkk2 are presented in
Fig. 3.2a, b, respectively.
 
Observe that the error f 1 ðxk Þ  f 1  converges to zero as a geometrical series, i.e., the convergence
 
is linear. From Fig. 3.2a we can see that the error f 1 ðxk Þ  f 1  is drastically reduced in the first few
iterations, then the algorithm slowly converges to the solution. This is a characteristic of the steepest
descent algorithm. If instead of backtracking the weak Wolfe line-search is used, then the solution is
obtained in 13 iterations.
The evolution of the iterations in the steepest descent algorithm is in “zigzag.” Indeed, if the line
search is exact, then gTkþ1 gk ¼ d Tkþ1 dk ¼ 0: Therefore, the successive gradients are orthogonal, i.e.,
3.1 The Steepest Descent 83

200
180
0
160
−0.2
140
−0.4 120
−0.6 100
−0.8 80
−1 60
300
200 250 40
200
100 150
100 20
50
0 0
20 40 60 80 100 120 140 160 180 200

Fig. 3.1 Representation of function f1(x)

 
Fig. 3.2 (a) Evolution of the error f 1 ðxk Þ  f 1 . (b) Evolution of kgkk2

the successive search directions are orthogonal. On the other hand, if the line-search is inexact, kgkk
becomes smaller and smaller along the iterations. Therefore, from Taylor’s development, we have
f ðxk  αgk Þ ¼ f ðxk Þ  αgTk gk þ oðkαgk kÞ, where the first order term αgTk gk ¼ αkgk k2 is very small.
Hence, the reduction of the values of the minimizing function will be very small. This is the main
drawback of the steepest descent algorithm.

Convergence of the Steepest Descent Method for Quadratic Functions

Consider the quadratic function

1
f ðxÞ ¼ xT Qx  bT x, ð3:2Þ
2
where Q 2 ℝn  n is a symmetric and positive definite matrix. Let λmin and λmax be the smallest and the
largest eigenvalues of Q. As we know, a method of descent directions generates a sequence
xkþ1 ¼ xk þ αk d k ,
84 3 Steepest Descent Methods

where dk is a descent direction for which

∇f ðxk ÞT dk < 0,

such that f(xk þ αdk) < f(xk) for small values of α. For the above quadratic function, the value of the
stepsize α which minimizes f along the line through xk in the direction dk can be analytically
computed as

dTk r k
α¼ , ð3:3Þ
dTk Qd k

where ek ¼ x  xk is the error at iteration xk and

r k ¼ Qek ¼ b  Qxk ð3:4Þ

is the residual from iteration k.

In the steepest descent method,
dk ¼ ∇f ðxk Þ ¼ r k : ð3:5Þ

Now, defining kxk2Q ¼ xT Qx, then the Q norm of the error is

kekþ1 k2Q ¼ kx  ðxk þ αk d k Þk2Q ¼ kek  αk dk k2Q ¼

¼ kek k2Q  2αk eTk Qd k þ α2k kdk k2Q
 T 2
rT rk r rk
¼ kek k2Q  2 Tk eTk Qr k þ  k 2 r Tk Qr k ð3:6Þ
r k Qr k r Tk Qr k
 T 2 !
2 r rk
¼ kek kQ 1  T k T 1
r k Qr k r k Q r k

In order to prove the convergence of the steepest descent method, we need a theoretical result
known as the inequality of Kantorovich.

Inequality of Kantorovich

Let λmax and λmin be the largest and the smallest eigenvalues of a symmetric and positive definite
matrix Q, respectively. Then
 T  T 1 
d Qd d Q d ðλ max þ λ min Þ2
β¼  T 2  :
d d 4λ max λ min

Proof Let Q ¼ RDRT. Then Q1 ¼ RD1RT, where R ¼ RT is an orthonormal matrix and the
eigenvalues of Q are 0 < λmin ¼ λ1  λ2  ⋯  λn ¼ λmax, and D ¼ diag (λ1, . . ., λn). Then
 T  
d RDRT d d T RD1 RT d vT DvvT D1 v
β¼  T
  T
 ¼ ,
T
d RR d d RR d T vT vvT v
Pn
where v ¼ RTd. Now, let γ i ¼ v2i =ðvT vÞ: Then, γ i  0 and i¼1 γ i ¼ 1: Hence,
3.1 The Steepest Descent 85

Pn
X
n X
n   γ i ð1=λi Þ
1 i¼1
β¼ γ i λi γi ¼ P :
λi n
i¼1 i¼1 1= γ i λi
i¼1

The largest value of β is obtained when γ 1 þ γ n ¼ 1. Therefore,

γ 1 ð1=λ min Þ þ γ n ð1=λ max Þ ðγ 1 λ min þ γ n λ max Þðγ 1 λ max þ γ n λ min Þ
β ¼
1=ðγ 1 λ min þ γ n λ max Þ λ min λ max
ðλ min =2 þ λ max =2Þðλ max =2 þ λ min =2Þ ðλ max þ λ min Þ2
 ¼ : ♦
λ min λ max 4λ max λ min

At this moment, applying the Kantorovich inequality to the right side term of (3.6) and noticing
that

4λ min λ max ðλ max  λ min Þ2

1 ¼ ,
ðλ min þ λ max Þ2 ðλ max þ λ min Þ2

it follows that
λ max  λ min κ1
kekþ1 kQ  ke k  ke k , ð3:7Þ
λ max þ λ min k Q κ þ 1 k Q
where κ ¼ λmax/λmin is the condition number of Q. In other words, the convergence of the steepest
descent method for quadratic functions is linear.

Example 3.2 Let us see the running of the steepest descent algorithm for minimizing a quadratic
function. Consider the function
1
f 2 ðxÞ ¼ xT Qx,
2
where Q ¼ diag (1, 2, . . ., n) . Observe that Q is a positive definite matrix with the eigenvalues 1, 2,
. . ., n. The condition number of Q is n. For this function, ∇f(x) ¼ Qx and the stepsize is computed as

gTk gk
αk ¼ , where gk ¼ Qxk :
gTk Qgk

As we know, for this optimal selection of αk, the rate of convergence of the steepest descent
algorithm is linear
λ max  λ min
kxk  x kQ  kx  x  kQ ,
λ max þ λ min k1
where kzkQ ¼ zTQz, and λmin and λmax are the smallest and the largest eigenvalues of Q, respectively.
The solution of this problem is vector zero, while the value of the function in the minimum point is
also zero. Considering n ¼ 500 , ε ¼ 106 and the initial point x0 ¼ [0.5, 0.5, ⋯, 0.5] , then, along
the iterations, the steepest descent algorithm gives the following evolution of the error |f2(xk)  f2(x)|
from Fig. 3.3.
86 3 Steepest Descent Methods

Fig. 3.3 Evolution of the

error |f2(xk)  f2(x)| .
n ¼ 500

Fig. 3.4 (a) Evolution of the error |f2(xk)  f2(x)| . n ¼ 2. (b) Zigzag evolution

This solution is obtained in 3342 iterations. Observe the linear convergence of this algorithm,
especially in the final part of the iterations. For large values of n, it follows that (λmax  λmin)/
(λmax þ λmin) tends to 1, thus proving the linear convergence.
Now, considering Q ¼ diag (1, 10), then the error |f2(xk)  f2(x)| has the evolution from Fig. 3.4a.
The trajectory of variables is illustrated in Fig. 3.4b. In Fig. 3.4a we can see the linear convergence of
the algorithm, while in Fig. 3.4b the trajectory of variables is in zigzag in wright angles.
The rate of convergence of the steepest descent algorithm depends on the ratio between the largest
axis and the smallest axis of the ellipsoid associated to the matrix Q. The bigger this ratio, the slower
the convergence. Details on the convergence of the steepest descent algorithm for quadratic functions
are presented in the following theorem.

Theorem 3.1 Consider the problem

1
minn f ðxÞ  xT Qx, ð3:8Þ
x2ℝ 2

where Q is a symmetric and positive definite matrix with λmin and λmax the smallest and the largest
eigenvalues, respectively. Let x be the solution of the problem (3.8). Then, the sequence {xk}
generated by the steepest descent algorithm converges to x, the rate of convergence is at least
linear, and the following relations hold:
3.1 The Steepest Descent 87

f ðxkþ1 Þ  f ðx Þ ðκ  1Þ2 ðλ max  λ min Þ2

 ¼ , ð3:9Þ
f ðxk Þ  f ðx Þ ðκ þ 1Þ2 ðλ max þ λ min Þ2
 
kxkþ1  x kQ κ  1 λ max  λ min
 ¼ , ð3:10Þ
kx k  x  kQ κþ1 λ max þ λ min
rffiffiffiffiffiffiffiffiffiffi 
kxkþ1  x k pffiffiffi κ  1 λ max λ max  λ min
 κ ¼ , ð3:11Þ
kx k  x  k κþ1 λ min λ max þ λ min

where κ ¼ λmax/λmin.

gT g
Proof Consider the steepest descent algorithm xkþ1 ¼ xk  αkgk, where αk ¼ gTkQgk and gk ¼ Qxk.
k k

Then
1 T 1 T
f ðxk Þ  f ðxkþ1 Þ 2 xk Qxk  2 ðxk  αk gk Þ Qðxk  αk gk Þ
¼
f ðxk Þ 1 T
x Qx
2 k k
1  T 2
αk gTk Qxk  α2k gTk Qgk gk gk
¼ 2 ¼ T  :
1 T
xk Qxk gk Qgk gTk Q1 gk
2
Now, from the Kantorovich inequality, we get
 T 2  2
f ðxkþ1 Þ gk gk 4λ max λ min λ max  λ min
¼1 T  T 1  ¼ 1  ¼ ,
f ðx k Þ gk Qgk gk Q gk ðλ max þ λ min Þ2 λ max þ λ min

which is exactly (3.9).

Now, let ek ¼ xk  x. Since Q is symmetric and positive definite, it follows that

λ min eTk ek  eTk Qek  λ max eTk ek :

Moreover, since x ¼ 0, we get

kxk  x k2Q ¼ eTk Qek ¼ xTk Qxk ¼ 2f ðxk Þ:

Now, using the above inequalities, it follows that for any k  0,

λ min kxk  x k2  2f ðxk Þ  λ max kxk  x k2 :

Therefore,
 2  2
λ min kxkþ1  x k2 kxkþ1  x kQ λ max  λ min
  ,
λ max kxk  x k2 kxk  x k2Q λ max þ λ min

which gives (3.10) and (3.11). ♦

This is one of the best estimations we can obtain for the steepest descent in certain conditions.
From (3.9) we can write

2
κ ðQ Þ  1
f ðxkþ1 Þ  f ðx Þ  ðf ðxk Þ  f ðx Þ,
κ ðQ Þ þ 1

where κ(Q) is the condition number of Q. (See Appendix A.)

88 3 Steepest Descent Methods

For general nonlinear functions, it is possible to show that the steepest descent method with an
exact line-search also converges linearly, with a constant rate that is bounded by the fundamental
ratio [(κ(∇2f(x))  1)/(κ(∇2f(x)) þ 1)]2.
For strongly convex functions for which the gradient is Lipschitz continuous, Nemirovsky and
Yudin (1983) define the global estimate of the rate of convergence of an iterative method as f(xk+1) 
f(x)  ch(x1  x, m, L, k), where c is a constant, h(.) is a function, m is a lower bound on the
smallest eigenvalue of the Hessian ∇2f(x), L is the Lipschitz constant, and k is the iteration number.
The faster the rate at which h converges to zero as k ! 1 , the more efficient the algorithm.
Now, we have an algorithm for minimizing continuously differentiable functions, the steepest
descent. In step 4 of this algorithm, for stepsize computation, both the exact or the inexact line-search
procedures may be used. In the following, before discussing the convergence properties of the
steepest descent algorithm, let us present a numerical study referring to a comparison between the
steepest descent with backtracking versus the steepest descent with weak Wolfe line-searches. This
study is relevant for the importance of the line-search in gradient descent algorithms.

Numerical Study

Consider the problems from the UOP collection (Andrei, 2020a), which includes 80 large-scale
unconstrained optimization problems in generalized or extended form. For each test function from
this collection, we have considered 10 numerical experiments with the number of variables increasing
as n ¼ 100, 200, . . ., 1000. Therefore, a set of 800 unconstrained optimization problems are solved in
our numerical experiments, both with steepest descent with backtracking (SDB) and steepest descent
with weak Wolfe (SDW). Both SDB and SDW use the same stopping criterion given by k∇f(xk)k2  εg
 
or αk gTk dk   εf jf ðxkþ1 Þj, where εg ¼ 106 and εf ¼ 1016. In backtracking, ρ ¼ 0.0001, β ¼ 0.8.
Figure 3.5 presents the performances of these algorithms subject to the CPU computing time. From
Fig. 3.5 we can see that SDW is more efficient and more robust than SDB. This is not a surprise,
because the backtracking (see Fig. 2.1) is a very crude line-search procedure. In contrast, the weak

Fig. 3.5 Steepest descent

with weak Wolfe (SDW)
versus steepest descent
with backtracking (SDB)
3.1 The Steepest Descent 89

Wolfe (see Fig. 2.3) is more advanced, being able to generate an acceptable (inexact) stepsize which
emphasizes the minimization of function f. For example, the table inside the plot shows that, subject
to the CPU time, SDW is faster in 455 problems, while SDB is faster only in solving 5 problems. Out
of 800 problems considered in this numerical study, only for 541 problems does the criterion (1.3)
hold.

Convergence of the Steepest Descent Method for General Functions

In the following, let us discuss the global convergence and the local rate of convergence of the
steepest descent method for general continuously differentiable functions.

Theorem 3.2 Assume that f : ℝn ! ℝ is convex and differentiable and additionally ∇f is Lipschitz
continuous, i.e., k∇f(x)  ∇ f( y)k  Lkx  yk, for any x, y, with constant L > 0. Then the steepest
descent with fixed stepsize δ  1/L satisfies

kx 0  x  k2
f ðxk Þ  f ðx Þ  :
2δk

Proof The theorem says that the steepest descent algorithm has the convergence rate O(1/k), that is,
in order to get f(xk)  f(x)  ε, it needs O(1/ε) iterations. The convergence is more emphasized if the
initial point x0 is close to the minimum point x.
From the Lipschitz continuity with constant L, it follows that
L
f ðyÞ  f ðxÞ þ ∇f ðxÞT ðy  xÞ þ ky  xk2 ,
2
for all x, y. Now, in the above inequality, when introducing y ¼ x  δ ∇ f(x), we get
 
Lδ
f ðy Þ  f ðx Þ  1  δk∇f ðxÞk2 :
2
Considering x+ ¼ x  δ ∇ f(x) and taking 0 < δ  1/L, it follows that
δ
f ðxþ Þ  f ðx Þ þ ∇f ðxÞT ðx  x Þ  k∇f ðxÞk2
 2 
1
¼ f ðx  Þ þ kx  x k2  kxþ  x k :
2
2δ
Summing over iterations we get
X
k  
1 1
ðf ðx i Þ  f ðx  ÞÞ  kx 0  x  k2  kx k  x  k2  kx 0  x  k2 :
i¼1
2δ 2δ

Since {f(xk)} is a nonincreasing sequence, it follows that

1X kx  x  k2
k
f ðx k Þ  f ðx  Þ  ðf ðxi Þ  f ðx ÞÞ  0 : ♦
k i¼1 2δk
90 3 Steepest Descent Methods

Theorem 3.3 Assume that f : ℝn ! ℝ is convex and differentiable and ∇f is Lipschitz continuous
with constant L > 0. Then the steepest descent with backtracking line-search satisfies

kx 0  x  k2
f ðxk Þ  f ðx Þ  ,
2δ min k

where δmin ¼ min {1, β/L} and β is as in Algorithm 2.5 or Algorithm 2.6.

Proof The proof is the same as the proof of Theorem 3.2. ♦

Observe that if β is not too small, then we do not lose much in the convergence compared to the
fixed stepsize (β/L versus 1/L ).
Now, let us discuss the convergence of the steepest descent algorithm for minimizing strongly
convex functions. Strong convexity of f means that for some m > 0, ∇2f(x)  mI for any x. For strongly
convex functions, we have a better lower bound than the one from the usual convexity
m
f ðyÞ  f ðxÞ þ ∇f ðxÞT ðy  xÞ þ ky  x k2 ,
2
for all x, y.

Theorem 3.4 Assume that f : ℝn ! ℝ is strongly convex, and its gradient is Lipschitz continuous,
i.e., k∇f(x)  ∇ f( y)k  Lkx  yk for any x, y, with constant L > 0. Then the steepest descent with fixed
stepsize δ  2/(m þ L ) or with backtracking line-search satisfies
L
f ðxk Þ  f ðx Þ  ck kx0  x k2 ,
2
where 0 < c < 1. ♦

The theorem says that under strong convexity, the rate of convergence is O(ck), i.e., exponentially
fast. In other words, to get f(xk)  f(x)  ε, the algorithm needs O(log(1/ε)) iterations. The constant
c depends on the condition number L/m. A higher condition number involves a slower rate of
convergence.
Observe that these results are obtained under the Lipschitz continuity of the gradient of the
minimizing function and under the strong convexity of the minimizing function. A question arises:
how realistic are these conditions? The Lipschitz continuity of ∇f means that ∇2f(x)  LI, where L is
the Lipschitz constant and I is the identity matrix. For example, consider f ðxÞ ¼ 12 ky  Axk2 : Then,
∇2f(x) ¼ ATA, so ∇f is Lipschitz with L ¼ σ 2max ðAÞ ¼ kAk2 : On the other hand, strong convexity
means that ∇2f(x)  mI, where I is the identity matrix. For example, consider again the function
f ðxÞ ¼ 12 ky  Axk2 : Then, ∇2f(x) ¼ ATA and m ¼ σ 2min ðAÞ:

Theorem 3.5 Let function f : ℝn ! ℝ be continuously differentiable. Then, any accumulation point
of the sequence {xk} generated by the steepest descent Algorithm 3.1 with exact line-search is a
stationary point of function f.

Proof Consider x an accumulation point of the sequence {xk} and K an infinite set of indices such
that lim k2K xk ¼ x. Let dk ¼  ∇ f(xk). Since f is continuously differentiable, the sequence {dk : k 2 K}
is uniformly bounded and kdkk ¼ k∇f(xk)k. Since the hypotheses of Theorem 2.11 are satisfied, it
follows that k∇f ðxÞk2 ¼ 0, that is, ∇f ðxÞ ¼ 0. ♦
3.1 The Steepest Descent 91

Theorem 3.6 Let f : ℝn ! ℝ be a twice continuously differentiable function on ℝn and k∇2f(x)k  M,

where M is a positive constant. Let x0 be the initial point and ε > 0 a positive constant sufficiently
small. Then, the sequence generated by the steepest descent Algorithm 3.1 satisfies the condition of
termination after a finite number of iterations or limk!1f(xk) ¼  1 or limk!1 ∇ f(xk) ¼ 0.

Proof Consider the infinite case. From Algorithm 3.1 and Theorem 2.10, it follows that
1
f ðxk Þ  f ðxkþ1 Þ  k∇f ðxk Þk2 :
2M
Hence,

X
k1
1 X
k1
f ðx 0 Þ  f ðx k Þ ¼ ½f ðxi Þ  f ðxiþ1 Þ  k∇f ðxi Þk2 :
i¼0
2M i¼0

Therefore, limk!1f(xk) ¼  1, or limk!1 ∇ f(xk) ¼ 0. ♦

The next theorem establishes the rate of convergence of the steepest descent algorithm with exact
line-search.

Theorem 3.7 Let f : ℝn ! ℝ which satisfies the hypothesis of Theorem 2.12. If the sequence {xk}
generated by the steepest descent algorithm converges to x, then the rate of convergence is linear.

Proof The theorem is a direct consequence of Theorem 2.12.

The rate of convergence of the steepest descent algorithm for general twice continuously differ-
entiable functions is established by the following theorem.

Theorem 3.8 Let f : ℝn ! ℝ be a twice continuously differentiable function in a neighborhood of x

with ∇f(x) ¼ 0 and the Hessian ∇2f(x) a positive definite matrix. Let λmax and λmin be the largest and
the smallest eigenvalues of ∇2f(x), respectively, which satisfy 0 < m  λmin  λmax  M. Let {xk} be
the sequence generated by the steepest descent Algorithm 3.1 convergent to x. Let

f ðxkþ1 Þ  f ðx Þ
¼ βk : ð3:12Þ
f ðxk Þ  f ðx Þ

Then, for any k, βk < 1 and

Mm
lim supβk  < 1: ð3:13Þ
k!þ1 M

Proof From Theorem 2.10 it follows that

½f ðxk Þ  f ðx Þ  ½f ðxkþ1 Þ  f ðx Þ ¼ f ðxk Þ  f ðxkþ1 Þ

1
 k∇f ðxk Þk2 :
2M
Using the definition of βk, we have
1
ð1  βk Þ½f ðxk Þ  f ðx Þ  k∇f ðxk Þk2 :
2M
92 3 Steepest Descent Methods

Therefore, from the hypothesis of the theorem, we get

k∇f ðxk Þk2

βk  1  < 1: ð3:14Þ
2M½f ðxk Þ  f ðx Þ

Suppose that
xk  x
! d:
kxk  x k

Obviously,
 2 
k∇f ðxk Þk2 ¼ kxk  x k2 ∇2 f ðx Þd  þ oð1Þ

and
 
1 T
f ðxk Þ  f ðx Þ ¼ kxk  x k2 d ∇2 f ðx Þd þ oð1Þ :
2
From the above inequalities, it follows that
 2
k∇f ðxk Þk2 2∇2 f ðx Þd
lim ¼ T  2m: ð3:15Þ
k!1 f ðxk Þ  f ðx Þ d ∇2 f ðx Þd

Hence, from (3.14) and (3.15) we obtain

k∇f ðxk Þk2 m

lim supβk  1  lim inf  1  < 1,
k!1 k!1 2M½f ðxk Þ  f ðx Þ M

thus proving the theorem. ♦

We may conclude that:

♦ The steepest descent algorithm is linear convergent, that is, the error f(xk)  f  tends to zero like a geometric series.
♦ The rate of convergence depends on the condition number of the Hessian of the minimizing function (which is
unknown). The convergence may be very slow even for functions relatively well conditioned. When the condition
number of the Hessian is large, then the steepest descent algorithm converges so slowly that it does not have any
practical value.
♦ The steepest descent with weak Wolfe line-search is way more efficient and more robust than the steepest descent
with backtracking.
♦ The parameters used in backtracking do not have a significant effect on the convergence of the algorithm. The exact
line-search improves the convergence, but without any significant effect.

3.2 The Relaxed Steepest Descent

It is interesting to see what happens when the stepsize αk computed by a line-search procedure is
randomly modified in [0, 1], that is, the new estimation to the minimum x is computed as
xkþ1 ¼ xk þ θk αk dk , ð3:16Þ

where dk ¼  ∇ f(xk) and θk is a random variable uniformly distributed in the interval [0, 1]. Thus, a
new algorithm called relaxed steepest gradient algorithm is obtained (Andrei, 2004b, 2005a).
3.2 The Relaxed Steepest Descent 93

Fig. 3.6 Evolution of the

error |f(xk)  f(x)| for
steepest descent versus
relaxed steepest descent

Example 3.3 Consider the function

1
f ðxÞ ¼ xT Qx,
2
where Q ¼ diag (1, 2, . . ., n) and n ¼ 500, from Example 3.2. Taking the initial point
x0 ¼ [0.5, 0.5, ⋯, 0.5] and ε ¼ 106 in the criterion for stopping the iterations, then, along the
iterations, the steepest descent and the relaxed steepest descent algorithms give the following
evolution of the error |f(xk)  f(x)| from Fig. 3.6. ♦

In this case of quadratic positive definite functions, the relaxed steepest descent algorithm is more
efficient than the classical steepest descent algorithm. Indeed, for solving the problem, the steepest
descent algorithm needs 3342 iterations, while the relaxed steepest descent only 499 iterations. A
slightly better evolution subject to the number of iterations is obtained when θk is selected as a
random variable uniformly distributed in the interval [0, 2] . For this quadratic function, taking
different values for the number of variables n, Table 3.1 shows the number of iterations for the
steepest descent algorithm and its relaxed version, where θk 2 [0, 1] and θk 2 [0, 2].
This behavior of the relaxed steepest descent algorithm illustrates a very serious limitation of the
optimal stepsize selection as well as a cognitive misleading, since a small modification of the stepsize
determines major changes in the performances of the steepest descent algorithm. In fact, for quadratic
positive definite functions, in very mild conditions on θk 2 [0, 2], the following convergence result of
the relaxed steepest descent algorithm may be established (Raydan & Svaiter, 2002).

Theorem 3.9 For the quadratic problem

1
min f ðxÞ ¼ xT Qx  bT x,
2
where Q 2 ℝn  n is a symmetric and positive definite matrix, if the sequence θk 2 [0, 2] has an
accumulation point θ 2 [0, 2], then the sequence {xk}:
94 3 Steepest Descent Methods

Table 3.1 Number of iterations of the steepest descent and of the relaxed steepest descent for the quadratic problem
from Example 3.3

Steepest Relaxed steepest Relaxed steepest

n descent θk ∈ [0,1] descent θk ∈ [0,2]
descent
1000 6682 812 860
2000 13358 950 1676
3000 20034 1627 1684
4000 26710 1407 1780
5000 33386 1615 1962
6000 40062 2198 1992
7000 46738 2684 1850
8000 534414 2440 2348
9000 60092 1765 3028
10000 66768 3005 3231

gTk gk
xkþ1 ¼ xk  θk αk gk ; αk ¼ , gk ¼ Qxk ,
gTk Qgk

generated by the relaxed steepest descent algorithm is linearly convergent to x.

Proof Observe that, for any k , the function

Φ k ð θ Þ ¼ f ð x k  θ α k gk Þ

is a convex polynom of second order which has a minimum in point θ ¼ 1. Moreover,

Φk(0) ¼ Φk(2) and for any θ 2 [0, 2], Φk(θ)  Φk(0). Therefore, for all k, f(xkþ1)  f(xk). Since f is
lower bounded, it follows that
lim ðf ðxk Þ  f ðxkþ1 ÞÞ ¼ 0:
k!1

We can write
   gT g  2
1 k k
f ðxk  θ αk gk Þ ¼ f ðxk Þ  θ  θ2 :
2 gTk Qgk

Observe that for θ 2 [0, 2], θ  θ2/2  0. Therefore, {f(xk)} is a decreasing sequence convergent
to f(x).
There is a ξ 2 (0, 1) such that ξ < θ < 2  ξ. Since Φk(θ) is convex for all θ 2 (ξ, 2  ξ) we
have Φk(θ) < Φk(ξ). But,

Φk ðξÞ < Φk ð0Þ  ξðΦk ð0Þ  Φk ð1ÞÞ:

Since
 T 2
1 gk gk
Φ k ð 0Þ  Φ k ð 1Þ ¼ ,
2 gTk Qgk

it follows that
3.2 The Relaxed Steepest Descent 95

 2
ξ gTk gk
Φk ðξÞ < Φk ð0Þ  :
2 gTk Qgk

Therefore,
 2
ξ gTk gk ξ gTk gk
Φ k ð 0Þ  Φ k ð ξ Þ >  ,
T
2 gk Qgk 2 λ max

that is
ξ
f ðxk Þ  f ðxk  ξαk gk Þ > kg k2 :
2λ max k 2

But, f(xk)  f(xk  ξαkgk) ! 0, showing that gk ! 0 and hence x ! x, solution of the problem.
Since {f(xk)} is a decreasing sequence, this tends to f(x) , that is x ! x. ♦

The relaxed steepest descent algorithm can be used for solving general continuously differentiable
functions (Andrei, 2004b). The following example illustrates the performances of the steepest descent
algorithm (SDB) versus the relaxed steepest descent (RSDB), where the stepsize in both algorithms is
computed by backtracking.

Example 3.4 Consider the function

n 
X 2
f ðxÞ ¼ ðx1  3Þ2 þ x 1  3  2ð x 1 þ x 2 þ ⋯ þ x i Þ 2 :
i¼2

Let x0 ¼ [0.001, . . ., 0.001] be the initial point and the criterion for stopping the iterations
k∇f(xk)k  εg, where εg ¼ 106. Then the number of iterations and the average stepsize required
by SDB and by RSDB with backtracking ( ρ ¼ 0.0001 and σ ¼ 0.8 ), for different values of n, is
as in Table 3.2.

These numerical examples show the limits of the steepest descent algorithm. Observe that a very
simple, small multiplicative modification of the stepsize through a random variable uniformly
distributed in the interval [0, 1] determines major changes in the behavior of the steepest descent
algorithm. In other words, the steepest descent algorithm has very reduced robustness at the variations
of the stepsize. In fact, for strongly convex functions, the following result can be proved (Raydan &
Svaiter, 2002).

Table 3.2 Number of iterations and the average stepsize for SDB and RSDB with backtracking for Example 3.4

SDB RSDB
n #iter Average stepsize #iter Average stepsize
10 186997 0.103413 40462 0.327121
20 253806 0.047808 101595 0.217972
30 410108 0.031707 105885 0.172842
40 780362 0.024489 122293 0.146184
50 829749 0.020759 144316 0.126295
96 3 Steepest Descent Methods

Theorem 3.10 If the sequence {θk} has an accumulation point θ 2 (0, 1), then the sequence {xk}
generated by the relaxed steepest descent algorithm converges to x.

Proof Consider the function φ(θ) ¼ f(xk  θ αkgk), where gk ¼ ∇ f(xk). We can write
1
f ðxk  θ αk gk Þ ¼ f ðxk Þ  θ αk gTk gk þ θ2 α2k gTk ∇2 f ðxk Þgk :
2
Since f is strongly convex, it follows that φ(θ) is a convex function and φ(0) ¼ f(xk). From the
strong convexity of f we have
 
Mαk 2
f ðxk  θ αk gk Þ  f ðxk Þ  θ  θ αk kgk k22 :
2

2 θ is a concave function and negative on (0, 2/Mαk) with maximum value 1/2Mαk
But, θ  Mα k 2

in 1/Mαk. Therefore, for all k, f(xkþ1)  f(xk) . Since f is lower bounded, it follows that

lim ðf ðxk Þ  f ðxkþ1 ÞÞ ¼ 0: ♦

k!1

In the following, let us assume that function f is strongly convex and the level set S ¼ {x 2 ℝn :
f(x)  f(x0)} is closed. Then, the following theorem can be proved, which shows the linear conver-
gence of the relaxed steepest descent algorithm with backtracking.

Theorem 3.11 For strongly convex functions, the steepest descent algorithm relaxed through the
sequence {θk}, where θk is a random variable uniformly distributed in the interval [0,1] and where the
stepsize is computed by backtracking with parameters ρ and β, is linear convergent and
!
Y
k1

f ðxk Þ  f  ci ðf ðx0 Þ  f  Þ, ð3:17Þ
i¼0

where

ci ¼ 1  min f2mρθi , 2mρθi β=Mg < 1: ð3:18Þ

Proof Observe that the backtracking uses the parameters ρ and β as in Algorithm 2.5 or as in
Algorithm 2.6. Consider 0 < θ < 1, then
 
Mαk 2
f ðxk  θ αk gk Þ  f ðxk Þ  θ  θ αk kgk k22 :
2

2 θ is a concave function and for all 0  θ  Mαk ,

Note that θ  Mα k 2 1

Mαk 2 θ
θ θ  :
2 2
Therefore,
θ
f ðxk  θ αk gk Þ  f ðxk Þ  αk kgk k22  f ðxk Þ  ρθαk kgk k22 ,
2
3.2 The Relaxed Steepest Descent 97

since ρ  1/2. Hence, the backtracking terminates with αk ¼ 1 or with a value αk  β/M. With this, at
iteration k a lower bound of the reduction of function f is obtained. In the first case we have

f ðxkþ1 Þ  f ðxk Þ  ρθk kgk k22 ,

while in the second one,

β
f ðxkþ1 Þ  f ðxk Þ  ρθk kg k2 :
M k 2
Therefore,

f ðxkþ1 Þ  f ðxk Þ  min fρθk , ρθk β=Mgkgk k22 :

Obviously,

f ðxkþ1 Þ  f   f ðxk Þ  f   min fρθk , ρθk β=Mgkgk k22 :

But kgk k22  2mðf ðxk Þ  f  Þ, (see Appendix A). Combining these relations we find that
f ðxkþ1 Þ  f   ð1  min f2mρθk , 2mρθk β=MgÞðf ðxk Þ  f  Þ:

Denote ck ¼ 1  min {2mρθk, 2mρθkβ/M} . Therefore, for any k ¼ 0, 1, . . .

f ðxkþ1 Þ  f   ck ðf ðxk Þ  f  Þ,

which proves the suboptimality from iteration k. Since ck < 1 the sequence {f(xk)} converges to f 
like a geometrical series with a factor which depends on the bound on the condition number M/m , the
backtracking parameters, the sequence {θk} of the random numbers uniformly distributed in the
interval [0,1] as well as on the initial suboptimality. Therefore, the relaxed steepest descent algorithm
is linear convergent. ♦

Numerical Study: SDB Versus RSDB

This numerical study is devoted to comparing the performances of the steepest descent algorithm
(SDB) with the relaxed steepest descent algorithm (RSDB), both of them being equipped with
backtracking. To achieve this, let us consider the collection UOP (Andrei, 2020a), which includes
80 large-scale unconstrained optimization problems in generalized or extended form. For each test
function from this collection, we have considered 10 numerical experiments with the number of
variables increasing as n ¼ 100, 200, . . ., 1000. Therefore, a set of 800 unconstrained optimization
problems are solved in our numerical experiments with SDB and RSDB unconstrained optimization
algorithms. Figure 3.7 presents the performances of these algorithms subject to the CPU computing
time. Observe that the relaxed steepest descent with backtracking (RSDB) is way more efficient and
more robust than the steepest descent with backtracking (SDB).
For example, subject to the CPU computing time, we can see that RSDB is faster in solving
414 problems, while SDB is faster in solving only 44 problems. Out of 800 problems solved in this
numerical study, only for 550 does the criterion (1.3) hold.
98 3 Steepest Descent Methods

Fig. 3.7 Relaxed steepest

descent with backtracking
(RSDB) versus steepest
descent with backtracking
(SDB)

3.3 The Accelerated Steepest Descent

Let us present in this section the accelerated steepest descent algorithms for solving the uncon-
strained optimization problem (3.1) (Andrei, 2005b, 2006a). Suppose that function f is twice
continuously differentiable. At the current iteration k, xk, fk, gk, and the search direction dk ¼  gk
are known. Now, by the weak Wolfe line-search (2.54) and (2.59), the stepsize αk can be computed,
and thus the following point z ¼ xk  αkgk is determined. The first Wolfe condition (2.54) shows that
the stepsize αk > 0 satisfies

f ðzÞ ¼ f ðxk  αk gk Þ  f ðxk Þ  ραk gTk gk :

With these, let us introduce the accelerated steepest descent algorithm by means of the following
iterative scheme:

xkþ1 ¼ xk  θk αk gk , ð3:19Þ

where θk > 0 is a parameter which is to be determined in such a way as to improve the algorithm. Now,
we have
 
1
f ðxk  αk gk Þ ¼ f ðxk Þ  αk gTk gk þ α2k gTk ∇2 f ðxk Þgk þ o kαk gk k2 :
2
On the other hand, for θ > 0 it follows that
 
1
f ðxk  θαk gk Þ ¼ f ðxk Þ  θαk gTk gk þ θ2 α2k gTk ∇2 f ðxk Þgk þ o kθαk gk k2 :
2
Therefore,
f ðxk  θαk gk Þ ¼ f ðxk  αk gk Þ þ Ψk ðθÞ, ð3:20Þ
3.3 The Accelerated Steepest Descent 99

where
1 2 
Ψ k ðθ Þ ¼ θ  1 α2k gTk ∇2 f ðxk Þgk þ ð1  θÞαk gTk gk
2
   
þθ2 αk o αk kgk k2  αk o αk kgk k2 : ð3:21Þ

Let us denote

ak  αk gTk gk  0,
bk  α2k gTk ∇2 f ðxk Þgk ,
 
ε k  o α k kgk k2 :

Observe that ak  0 and for convex functions, bk  0. Therefore,

1 2 
Ψk ð θ Þ ¼ θ  1 bk þ ð1  θÞak þ θ2 αk εk  αk εk : ð3:22Þ
2
But, Ψ0k ðθÞ ¼ ðbk þ 2αk εk Þθ  ak and Ψ0k ðθm Þ ¼ 0 for
ak
θm ¼ : ð3:23Þ
bk þ 2αk εk

Observe that Ψ0k ð0Þ ¼ ak  0: Therefore, assuming that bk þ 2αkεk > 0, then Ψk(θ) is a convex
quadratic function with the minimum value in point θm and

ðak  ðbk þ 2αk εk ÞÞ2

Ψk ð θ m Þ ¼   0:
2ðbk þ 2αk εk Þ

Considering θ ¼ θm in (3.20) and since bk  0, it follows that for every k,

ðak  ðbk þ 2αk εk ÞÞ2

f ðxk  θm αk gk Þ ¼ f ðxk  αk gk Þ   f ðxk  αk gk Þ,
2ðbk þ 2αk εk Þ

which is a possible improvement of the values of function f (when ak  (bk þ 2αkεk) 6¼ 0).
Therefore, using this simple multiplicative modification of the stepsize αk as θkαk, where
θk ¼ θm ¼ ak/(bk þ 2αkεk), it follows that

ða  ðbk þ 2αk εk ÞÞ2

f ðxkþ1 Þ ¼ f ðxk  θk αk gk Þ  f ðxk Þ  ραk gTk gk  k
2ðbk þ 2αk εk Þ

 ð3:24Þ
ðak  ðbk þ 2αk εk ÞÞ2
¼ f ðxk Þ  ρak þ  f ðxk Þ,
2ðbk þ 2αk εk Þ

since

ðak  ðbk þ 2αk εk ÞÞ2

ρak þ  0,
2ðbk þ 2αk εk Þ

where ρ 2 (0, 1/2).

Now, neglecting the contribution of εk in (3.24), an improvement of the function values is still
obtained as
100 3 Steepest Descent Methods


ð a  bk Þ 2
f ðxkþ1 Þ  f ðxk Þ  ρak þ k  f ðxk Þ: ð3:25Þ
2bk

In order to get the algorithm, a procedure for the computation of bk is needed. For this, at point
z ¼ xk  αkgk,
1
f ðzÞ ¼ f ðxk  αk gk Þ ¼ f ðxk Þ  αk gTk gk þ α2k gTk ∇2 f ðxek Þgk ,
2
where xek is a point on the line segment connecting xk and z. On the other hand, at point xk ¼ z þ αkdk,
1
f ðxk Þ ¼ f ðz þ αk gk Þ ¼ f ðzÞ þ αk gTz gk þ α2k gTz ∇2 f ðxk Þgz ,
2
where gz ¼ ∇ f(z) and xk is a point on the line segment connecting xk and z. Having in view the local
character of the searching and that the distance between xk and z is small enough, we can consider
xek ¼ xk ¼ xk : So, by adding the above equalities, the following value for bk is obtained:

bk ¼ αk yTk gk , ð3:26Þ

where yk ¼ gz  gk. Observe that the computation of bk needs an additional evaluation of the gradient
in point z. Therefore, neglecting the contribution of εk and considering θk ¼ θm ¼ ak/bk in (3.19), the
following algorithm can be presented.

Algorithm 3.2 Accelerated steepest descent (with Wolfe line-search)

1. Choose a starting point x0 2 dom f and compute: f0 ¼ f(x0) and g0 ¼ ∇ f(x0). Select εA > 0 sufficiently
small and positive values 0 < ρ < σ < 1 used in the Wolfe line-search conditions. Set d0 ¼  g0 and k ¼ 0
2. Test a criterion for stopping the iterations. If the test is satisfied, then stop; otherwise, continue with step 3
3. Using the weak Wolfe line-search conditions (2.54) and (2.59), determine the stepsize αk
4. Update the variables xkþ1 ¼ xk  αkgk and compute fkþ1 and gkþ1. Compute sk ¼ xkþ1  xk and
yk ¼ gkþ1  gk
5. Acceleration scheme:
(a) Compute: z ¼ xk  αkgk, gz ¼ ∇ f(z) and yk ¼ gz  gk
(b) Compute: ak ¼ αk gTk gk , and bk ¼ αk yTk gk
(c) if |bk| < εA, then go to step 6
(d) if |bk|  εA, then compute θk ¼ ak/bk. Compute z ¼ xk  θkαkgk, gz ¼ ∇ f(z), sz ¼ z  xk and
yz ¼ gz  gk. If yTz sz > 0, then set xkþ1 ¼ z and compute fkþ1 ¼ f(xkþ1), gkþ1 ¼ ∇ f(xkþ1), sk ¼ xkþ1  xk
and yk ¼ gkþ1  gk. Go to step 6. Otherwise, if yTz sz  0, then go to step 6
6. Set k ¼ k þ 1 and go to step 2 ♦

Usually, εA is epsilon machine. The un-accelerated steepest descent algorithm can immediately be
obtained from the above algorithm by skipping step 5. If f is bounded along the direction gk, then
there exists a stepsize αk satisfying the Wolfe line-search conditions (2.54) and (2.59). Under
reasonable assumptions, the Wolfe conditions are sufficient to prove the global convergence of the
algorithm.
The algorithm is equipped with an acceleration scheme (see step 5) introduced by Andrei (2005b,
2006a). This scheme modifies the stepsize determined by the Wolfe line-search conditions in such a
way as to improve the reduction of the minimizing function values along the iterations. It is proved
that this acceleration scheme is linear convergent. However, when the acceleration scheme is used, in
some unconstrained optimization algorithms, it is necessary to monitor whether yTk sk > 0: The Wolfe
line-search implemented in step 3 of the algorithm ensures that yTk sk > 0, which is crucial in quasi-
3.3 The Accelerated Steepest Descent 101

Newton unconstrained optimization algorithms. If ∇2f(xk) is positive definite, it is easy to see that in
the new point xkþ1 ¼ xk  θkαkgk we have yTk sk > 0 for k ¼ 0, 1, . . . . In general, for solving the
unconstrained optimization problems by using, for example, the accelerated conjugate gradient or the
accelerated quasi-Newton methods, if the new yTz sz  0, then the algorithm updates xkþ1 ¼ xk þ αkdk.
Otherwise, if yTz sz > 0, then the algorithm updates xkþ1 ¼ xk þ θkαkdk, where the acceleration factor
θk is computed as in Algorithm 3.2. Having in view that the contribution of the acceleration to
reducing the values of the minimizing function is small, it follows that by acceleration, the condition
yTk sk > 0 is conserved along the iterations and hence the above scheme is very seldom used.
Observe that, if |ak| > bk, then θk > 1. In this case, θkαk > αk and it is also possible that θkαk  1 or
θkαk > 1. Hence, the stepsize θkαk can be greater than 1. On the other hand, if |ak|  bk, then θk  1. In
this case, θkαk  αk, so the stepsize θkαk is reduced. Therefore, if |ak| 6¼ bk, then θk 6¼ 1 and the stepsize
αk computed by the Wolfe conditions will be modified by its increasing or its decreasing through
factor θk.
Neglecting εk in (3.22), we can see that Ψk(1) ¼ 0 and if |ak|  bk/2, then Ψk(0) ¼  ak  bk/2  0
and θk < 1. Therefore, for any θ 2 [0, 1], Ψk(θ)  0. Consequently, for any θ 2 (0, 1) it follows that
f(xk þ θαkdk) < f(xk). In this case, for any θ 2 [0, 1], θkαk  αk. However, in our algorithm we selected
θk ¼ θm as the point achieving the minimum value of Ψk(θ).
In the following, for strongly convex functions, let us prove the linear convergence of the
acceleration scheme (Andrei, 2006a). For strongly convex functions, it is easy to prove that,
k∇f(x)k2  2m( f(x)  f(x)), for all x 2 S ¼ {x 2 ℝn : f(x)  f(x0)}, where x is a local solution of
(3.1). (See Appendix A.)

Proposition 3.1 Suppose that f is a strongly convex function on the level set S ¼ {x : f(x)  f(x0)}.
Then, the sequence {xk} generated by the accelerated steepest descent algorithm 3.2 converges
linearly to x, solution to the problem (3.1).

Proof From (3.24), f(xkþ1)  f(xk) for all k. Since f is bounded below, it follows that
lim ðf ðxk Þ  f ðxkþ1 ÞÞ ¼ 0:
k!1

Now, since f is strongly convex, there exist positive constants m and M so that mI  ∇2f(x)  MI on
S. Suppose that xk  αgk 2 S and xk  θmαgk 2 S for all α > 0, where θm is the acceleration factor.
Therefore,

ðak  bk Þ2
f ðxk  θm αgk Þ  f ðxk  αgk Þ  : ð3:27Þ
2bk
But, from the strong convexity, the following quadratic upper bound of f(xk  αgk) is true
1
f ðxk  αgk Þ  f ðxk Þ  αkgk k22 þ Mα2 kgk k22 :
2
Observe that α þ Mα2/2 is a convex function. Therefore, for any 0  α  1/M, it follows that

α þ Mα2 =2  α=2:
Hence,
1
f ðxk  αgk Þ  f ðxk Þ  αkgk k22 þ Mα2 kgk k22
2
α
 f ðxk Þ  kgk k22  f ðxk Þ  ραkgk k22 ,
2
since ρ  1/2.
102 3 Steepest Descent Methods

The line-search with backtracking terminates either with s ¼ 1 , or with a value α  β/M. This
gives a lower bound on the reduction of the minimizing function f. For α ¼ 1 we have

f ðxk  αgk Þ  f ðxk Þ  ρ kgk k22

and for α  β/M

ρβ
f ðxk  αgk Þ  f ðxk Þ  kg k2 :
M k 2
Therefore, for 0  α  1/M, we have

ρβ
f ðxk  αgk Þ  f ðxk Þ  min ρ, kgk k22 : ð3:28Þ
M

On the other hand,

 2
ð ak  bk Þ 2 αkgk k22  α2 Mkgk k22 ð1  αMÞ2
 ¼ kgk k22 :
2bk 2α2 Mkgk k2 2
2M

Now, as above, for α ¼ 1

ðak  bk Þ2 ð1  MÞ2
 kgk k22 :
2bk 2M

For α  β/M

ðak  bk Þ2 ð1  βÞ2
 kgk k22 :
2bk 2M

Hence,

ðak  bk Þ2 ð1  M Þ2 ð1  β Þ2
 min , kgk k22 : ð3:29Þ
2bk 2M 2M

Form (3.27), (3.28) and (3.29) we get


ρβ
f ðxk  θm αgk Þ  f ðxk Þ  min ρ, kgk k22
M

ð1  M Þ2 ð1  β Þ2
 min , kgk k22 : ð3:30Þ
2M 2M

Therefore,

 
ρβ ð1  M Þ2 ð1  β Þ2
f ðxk Þ  f ðxkþ1 Þ  min ρ, þ min , kgk k22 :
M 2M 2M

But, f(xk)  f(xkþ1) ! 0 and consequently gk tends to zero, that is, xk converges to x. Having in
view that f(xk) is a nonincreasing sequence, it follows that f(xk) converges to f(x). From (3.30) we
see that
3.3 The Accelerated Steepest Descent 103

 
ρβ ð1  M Þ2 ð1  β Þ2
f ðxkþ1 Þ  f ðxk Þ  min ρ, þ min , kgk k22 :
M 2M 2M

Combining this with

kgk k22  2mðf ðxk Þ  f  Þ

and subtracting f  from both sides of the above inequality, we have

f ðxkþ1 Þ  f   cðf ðxk Þ  f  Þ, ð3:31Þ

where
 
2mρβ ð1  M Þ2 m ð1  β Þ2 m
c ¼ 1  min 2mρ,  min , < 1:
M M M

Therefore, f(xk) converges to f(x) at least as fast as a geometric series with a factor that depends on
the parameter ρ in the first Wolfe condition and on the bounds m and M, i.e., the convergence is at
least linearly. ♦

Remark 3.1 Basically, the acceleration scheme modifies the stepsize αk in a multiplicative way to
improve the reduction of the function values along the iterations. In the accelerated algorithm, instead
of xkþ1 ¼ xk þ αkdk, the new estimation of the minimum point is computed as
xkþ1 ¼ xk þ ηk αk dk , ð3:32Þ

where the acceleration factor ηk is computed as

ak
ηk ¼  , ð3:33Þ
bk

where ak ¼ αk gTk gk , bk ¼  αk(gz  gk)Tgk, gz ¼ ∇ f(z) and z ¼ xk  αkgk. Hence, if |bk|  εA, where
εA > 0 is sufficiently small, then the new estimation of the solution is computed as xkþ1 ¼ xk þ ηkαkdk;
otherwise xkþ1 ¼ xk þ αkdk. ♦

Example 3.5 In the following, let us see the running of the accelerated steepest descent with
backtracking for minimizing the function
! !2
X
n Xn
f ðx Þ ¼ n cos xj þ ið1  cos xi Þ  sin xi :
i¼1 j¼1

Consider n ¼ 100, the initial point x0 ¼ [0.2, . . ., 0.2] and the parameters ρ ¼ 0.0001 and β ¼ 0.8
in the backtracking procedure. The criterion for stopping the iterations is k∇f(xk)k2  εg. For
minimizing this function, the algorithm needs 150 iterations to get the solution. In Fig. 3.8 observe
the evolution of the error |f(xk)  f | and the evolution of the contribution (ak  bk)2/2bk þ ρak at the
reduction of the function f; see (3.25). Figure 3.9 presents the evolution of the same elements, but this
time for ρ ¼ 0.01. In this case, for minimizing this function, the algorithm needs 114 iterations.

Observe that the quantity (ak  bk)2/2bk þ ρak (ak < 0) is indeed much smaller than the error
|f(xk)  f |, but it contributes to the reduction of the minimizing function even if ρ is small in the first
weak Wolfe condition.
104 3 Steepest Descent Methods

Fig. 3.8 Accelerated

steepest descent with
backtracking. ρ ¼ 0.0001.
Error |f(xk)  f | and
contribution (ak  bk)2/
2bk + ρak

Fig. 3.9 Accelerated

steepest descent with
backtracking. ρ ¼ 0.01.
Error |f(xk)  f | and
contribution (ak  bk)2/
2bk + ρak

Numerical Study

This numerical study presents a comparison of the accelerated steepest descent with backtracking
(ASDB) versus the steepest descent with backtracking (SDB) and versus the relaxed steepest descent
with backtracking (RSDB). For this, the collection UOP (Andrei, 2020a), which includes 80 large-
scale unconstrained optimization problems in generalized or extended form, is used. For each test
function from this collection, 10 numerical experiments have been considered, with the number of
variables increasing as n ¼ 100, 200, . . ., 1000. Therefore, a set of 800 unconstrained optimization
problems are solved. Figure 3.10 presents the performance profiles of ASDB versus SDB. We can see
3.3 The Accelerated Steepest Descent 105

Fig. 3.10 Accelerated

steepest descent with
backtracking (ASDB)
versus steepest descent
with backtracking (SDB)

Fig. 3.11 Accelerated

steepest descent with
backtracking (ASDB)
versus relaxed steepest
descent with backtracking
(RSDB) and versus
steepest descent with
backtracking (SDB)

that the accelerated steepest descent algorithm ASDB is more efficient and more robust than its
un-accelerated variant SDB. Subject to the CPU computing time, ASDB is faster in solving
272 problems, while SDB is faster in solving only 170 problems. Out of 800 problems, only for
549 of them does the criterion (1.3) hold.
Figure 3.11 presents the performance profiles of ASDB versus SDB and versus RSDB. Observe
that RSDB is more efficient and more robust than SDB, but ASDB is the best.
In conclusion, if we want to minimize the continuous differentiable function f and if the current
point is xk, then, the best direction of moving to find better points is the negative gradient  ∇ f(xk),
that is, the steepest descent direction. However, as soon as we move away from xk along the steepest
106 3 Steepest Descent Methods

descent direction for obtaining xkþ1, even if the value of the minimizing function decreases, this
direction ceases to be a good one. The purpose of the acceleration is to find points on the steepest
descent direction for which the reduction of the value of the minimizing function is better.

3.4 Comments on the Acceleration Scheme

The acceleration scheme presented in Remark 3.1 was developed for the steepest descent algorithm,
but it can be applied to any other unconstrained optimization algorithms based on the descent
direction, for example, the conjugate gradient, the quasi-Newton, the limited-memory BFGS, and
the truncated Newton. In general, the acceleration scheme is recommended for those unconstrained
optimization methods for which the evolution of the stepsize is very hieratic, having up to one or two
orders of magnitude along the iterations, like the conjugate gradient algorithms. The value of ηk
computed as in (3.33) is small, and so the variables are not dramatically modified by acceleration.
When the acceleration scheme is used in conjunction with the conjugate gradient or the quasi-Newton
or the limited-memory quasi-Newton BFGS methods, it is necessary to monitor whether yTk sk > 0: If
∇2f(x) is positive definite, then in the new point xkþ1 ¼ xk þ ηkαkdk it follows that yTk sk > 0 for all
iterations k ¼ 0, 1, . . . . For solving unconstrained optimization problems by using the accelerated
scheme, Algorithm 3.2 may consider
(
xk þ ηk αk dk , if yTk sk > 0,
xkþ1 ¼ ð3:34Þ
xk þ αk dk , if yTk sk  0:

However, our intensive numerical experiments with large-scale optimization problems have
proved that (3.34) does not need to be implemented. Having in view that the contribution
(ak  bk)2/2bk þ ρak is small, it follows that the condition yTk sk > 0 is conserved along the iterations
by acceleration.
The accelerated scheme proved to be a major improvement of the unconstrained optimization
algorithms (Dener, Denchfield, & Munson, 2019). We should emphasize that another acceleration
scheme of the optimization algorithms is preconditioning. Notice that preconditioning is quite
dependent on the matrix of preconditioning. If the preconditioning matrix contains useful information
about the inverse Hessian of the objective function, it is better to use it into a quasi-Newton context
than into a preconditioned conjugate gradient one. For example, for solving large-scale unconstrained
optimization problems, preconditioning the nonlinear conjugate gradient methods based on the Perry-
Shanno scheme remains an open question with very little consensus (Andrei, 2019d) (see Chap. 5).
On the other hand, the acceleration of the conjugate gradient algorithms based on the modification of
the stepsize is well understood and has proved to be extremely effective in practice (see Chap. 5).

Notes and References

The steepest descent algorithm designed by A.L. Cauchy (1789–1857) is not an efficient minimization
method. This simple strategy of proceeding along the negative gradient works well for functions with
circular or near-circular contours, but practical optimization problems or real applications may
involve functions with narrow curving valleys which need more sophisticated approaches, as we
are going to see in the next chapters of this book. We have insisted on presenting the steepest descent
method as it emphasizes the pattern of line-search methods. As we said, in this pattern, a particular
iteration k consists of two operations: the choice of a descent search direction dk followed by a line-
search along this direction to get a suitable (inexact) stepsize αk.
3.4 Comments on the Acceleration Scheme 107

The relaxed steepest descent algorithm was defined by Andrei (2005a). The development of the
accelerated conjugate gradient algorithms was presented by Andrei (2009c, 2009d, 2010a).
Discussions and comments on the accelerated scheme (3.32) and (3.33) were given by Babaie-
Kafaky and Rezaee (2018) and by Sun, Liu, and Liu (2021). A class of accelerated conjugate gradient
methods based on a modified secant equation was presented by Ou and Lin (2020).
 The Newton Method
4

In the panoply of the optimization methods and in general, for solving problems that have an
algebraic mathematical model, the Newton method has a central position. The idea of this method
is to approximate the mathematical model through a local affine or a local quadratic model. This
chapter is dedicated to presenting the Newton method for solving algebraic nonlinear systems on the
one hand and to minimizing smooth enough functions on the other one. It is proved that, initialized
near solution, the Newton method is quadratic convergent to a minimum point of the minimizing
function. Some modifications of the Newton method and the composite Newton method are also
presented.

4.1 The Newton Method for Solving Nonlinear Algebraic Systems

Let us consider an algebraic nonlinear system F(x) ¼ 0, where under F : ℝn ! ℝn is a continuous

differentiable vectorial function with the components fi(x), i ¼ 1, . . ., n, scalar functions. We are
interested in finding a point x such that F(x) ¼ 0. Let J(x) be the Jacobian matrix of F in point x
defined as the n  n matrix
2 3
∂f 1 ∂f 1 ∂f 1
6 ∂x ∂x ⋯ ∂x 7
6 1 2 n7
6 7
6 ∂f 2 ∂f 2 ∂f 2 7
6 ⋯ 7
J ðxÞ ¼ 6 ∂x1 ∂x2 ∂xn 7:
6 7
6 ⋮ ⋮ ⋱ ⋮ 7
6 7
4 ∂f n ∂f n ∂f n 5
⋯
∂x1 ∂x2 ∂xn

Then, given the estimation xk of x, the Newton method computes the next estimation xkþ1 by
solving a local linear approximation of F (local affine model) in xk, that is,
Mk ðxÞ  Fðxk Þ þ J ðxk Þðx  xk Þ ¼ 0: ð4:1Þ

Supposing that for any k, the Jacobian matrix J(xk) is nonsingular, then xkþ1 is defined as

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 109
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_4
110 4 The Newton Method

xkþ1 ¼ xk  J ðxk Þ1 Fðxk Þ, k ¼ 0, 1, 2, . . . ð4:2Þ

This is the Newton method for solving nonlinear algebraic system of equations. If J is a singular
matrix, then the Newton method is not defined, and some transformations have to be considered in
order to overcome this situation. In the following, suppose that J(x) is nonsingular. By the continuity
of J it follows that J(xk) is nonsingular for any xk near enough to x.
For solving the system F(x) ¼ 0, an initial point x0 from dom ( fi), i ¼ 1, . . ., n, must be specified
and then apply (4.2) until a criterion for stopping the iterations has been satisfied.

Algorithm 4.1 Newton for F(x) ¼ 0

1. Initialization. Consider the initial point x0 2 ℝn as well as the convergence tolerance ε > 0 sufficiently
small. Set k ¼ 0.
2. ComputeF(xk) and J(xk) ¼ ∇ F(xk)
3. Solve the linear algebraic system J(xk)dk ¼  F(xk)
4. If kdkk1  ε, then stop; otherwise continue with step 5
5. Set xkþ1 ¼ xk þ dk, k ¼ k þ 1 and continue with step 2 ♦

Observe that the most delicate operations in this algorithm are in step 2, where the Jacobian matrix
must be evaluated in the current point, and also in step 3, where a linear system has to be solved,
known as the Newton system. In step 4, some other stopping criteria could be used. However, we will
prove that the Newton method initialized with a “good” starting point is quadratic convergent to a
local solution. Therefore, for stopping the iterations, the following criterion kxkþ1  xkk  ε, is also
convenient.

Example 4.1 Consider the vectorial function F defined as

 
x21 þ x22  2
FðxÞ ¼ :
exp ðx1  1Þ  x2

Applying the Newton method with initial point x0 ¼ [1.4 0.8], the following results are obtained
as in Tables 4.1 and 4.2. Observe that only six iterations are needed to get the solution.
From Table 4.2 we can see that
kx   x 2 k kx  x3 k kx   x 4 k
¼ 0:477, ¼ 0:499, ¼ 0:5

kx  x 1 k 2 
kx  x 2 k 2
kx   x 3 k2

thus showing that

kx  xkþ1 k
kx   x k k2

is asymptotically constant when k ! 1.

Since the error kx  xkk is very fast reduced to zero, this cannot be numerically verified. ♦
4.1 The Newton Method for Solving Nonlinear Algebraic Systems 111

Table 4.1 Iterations generated by the Newton method

k x1k x k2
0 .1400000000000E+01 .8000000000000E+00
1 .1070918449595E+01 .1000892713209E+01
2 .1002402519583E+01 .9999422607128E+00
3 .1000002882268E+01 .9999999985242E+00
4 .1000000000004E+01 .1000000000000E+01
5 .1000000000000E+01 .1000000000000E+01

Table 4.2 Evolution of kx  xkk and kF(xk)k

k x * – xk F ( xk )
0 .44721359549995 8E+00 .9157627488965E+00
1 .709240680574689E– 01 .1654342180140E+00
2 .240321330115156E– 02 .5302195270017E– 02
3 .288226844402807E– 05 .6442977199172E– 05
4 .415378842433256E–11 .9288153280959E– 11
5 .000000000000000E+00 .0000000000000E+00

Observe that the local affine model Mk(x) which approximates F(x) in the current point xk is very
efficient for the determination of the next estimation of the solution. In the following theorem, the
quadratic convergence is proved, which is the main result of the Newton method.

Theorem 4.1 Let F : ℝn ! ℝn be a vectorial continuous differentiable function and F(x) ¼ 0. If:

(i) Jacobian J(x) of F in x is nonsingular.

(ii) Jacobian J(x) is continuously Lipschitz in a neighborhood of x,

then, for all x0 near enough to x, the Newton method defined by (4.2) generates a sequence {xk}
which is quadratic convergent to x.

Before proving this theorem, we need some technical results as follows. We can say that
J : ℝn ! ℝnn is Lipschitz continuous on the open set D 2 ℝn if there exists a positive constant L,
called the Lipschitz constant, such that kJ(x)  J( y)k  Lkx  yk for all x, y 2 D. In other words, the
difference J(x)  J( y) is proportional with x  y. Observe that the Lipschitz condition on J is stronger
than the continuity of J, but weaker than the twice differentiability of J. Mainly, L gives a measure of
the nonlinearity of function F in x.
Another important point is the quadratic convergence. It is important from two points of view. On
the one hand, the quadratic convergence guarantees that, if the Newton method can be initialized near
solution, then it is rapid convergent to the solution. On the other hand, a very simple stopping
criterion, namely, kxk  xk1k < ε, can be associated to the Newton method, where ε > 0 is a
sufficiently small convergence tolerance.
112 4 The Newton Method

Proposition 4.1 Let F : ℝn ! ℝn be continuously differentiable and a, b 2 ℝn. Then

ð1
FðbÞ ¼ FðaÞ þ J ða þ tðb  aÞÞðb  aÞdt, ð4:3Þ
0

where J is the Jacobian matrix of F.

Proposition 4.2 Let F : ℝn ! ℝnbe integrable on the interval [a, b], then
b  b
ð  ð
 
 FðtÞdt  kFðtÞkdt: ð4:4Þ
 
 
a a

Proposition 4.3 Let J : ℝm ! ℝnm be a continuously function with matrices values. If J(x) is
nonsingular, then there exists δ > 0 such that for all x 2 ℝm with kx  xk < δ, J(x) is nonsingular and
kJ ðxÞk  2kJ ðx Þk, ð4:5Þ
   
J ðxÞ1  < 2J ðx Þ1 , ð4:6Þ

1 
J ðx Þ1 1 kx  x k  kFðxÞk  2kJ ðx Þkkx  x k: ð4:7Þ
2

Proof Let δ > 0 be sufficiently small. Then, for any x for which kx  xk < δ, we have
kJ(x)k  k J(x)k þ Lkx  xk, where L is the Lipschitz constant associated to J(x). Therefore,
(4.5) holds if Lδ < kJ(x)k.

Result (4.6) is a direct consequence of the Banach lemma (see Appendix A) if

kI  J(x)1 J(x)k < 1/2. This follows from
  1 1
J ðx Þ 
δ<
2L
since
   
I  J ðx Þ1 J ðxÞ ¼ J ðx Þ1 ðJ ðx Þ  J ðxÞÞ
  1    ð4:8Þ
 LJ ðx Þ kx  x k  LδJ ðx Þ1   1=2:

For proving (4.7) observe that if kx  xk < δ, then tkx  xk < δ holds for any 0  t  1. From
(4.5) it follows that

ð1
kFðxÞk  kJ ðx þ tðx  x ÞÞkkx  x kdt  2kJ ðx Þkkx  x k,
0

which is the right inequality in (4.7). For proving the left inequality observe that
4.1 The Newton Method for Solving Nonlinear Algebraic Systems 113

ð1
J ðx Þ1 FðxÞ ¼ J ðx Þ1 J ðx þ tðx  x ÞÞdt
0
ð1  
¼ ðx  x Þ 
I  J ðx Þ1 J ðx þ tðx  x ÞÞ ðx  x Þdt,
0

and therefore, from (4.8) we get

0 1 1
ð   
  1   
J ðx Þ FðxÞ  kx  x k@1   I  J ðx Þ1 J ðx þ tðx  x ÞÞ dtA  1 kx  x k
  2
 
0

Therefore,
1    
kx  x k  J ðx Þ1 FðxÞ  J ðx Þ1 kFðxÞk,
2
which completes the proof. ♦

Proposition 4.3 shows that the set of nonsingular matrices is an open set. The second part of the
proposition follows from the fact that if the application x ! J(x) is continuous, then x ! J(x)1 is
also continuous as soon as J(x)1 is definite. With these results, let us prove Theorem 4.1.

Proof of Theorem 4.1 Suppose that the estimation xk is sufficiently closed to x and J(xk) is
nonsingular. Consider the iteration

xkþ1 ¼ xk  J ðxk Þ1 Fðxk Þ:

Now, subtracting x from both members we get

xkþ1  x ¼ xk  x  J ðxk Þ1 Fðxk Þ:

But F(x) ¼ 0, therefore

xkþ1  x ¼ xk  x  J ðxk Þ1 ðFðxk Þ  Fðx ÞÞ:

Now, from Proposition 4.1 we can estimate the difference F(xk)  F(x) as

ð1
Fðxk Þ  Fðx Þ ¼ J ðx þ tðxk  x ÞÞðxk  x Þdt


0
ð1 ð1
¼ J ðx Þðxk  x Þdt þ ðJ ðx þ tðxk  x ÞÞ  J ðx ÞÞðxk  x Þdt
 

0 0
ð1
¼ J ðx Þðxk  x Þ þ ðJ ðx þ tðxk  x ÞÞ  J ðx ÞÞðxk  x Þdt:
0
114 4 The Newton Method

Therefore,
kFðxk Þ  Fðx Þ  J ðx Þðxk  x Þk
1 
ð 
 
¼  ðJ ðx þ tðxk  x ÞÞ  J ðx ÞÞðxk  x Þdt
    

 
0
ð1
 kðJ ðx þ tðxk  x ÞÞ  J ðx ÞÞðxk  x Þkdt
0
ð1
 kJ ðx þ tðxk  x ÞÞ  J ðx Þk kxk  x kdt
0
ð1
L
 L t kxk  x k2 dt ¼ kxk  x k2 :
2
0

With these

xkþ1  x ¼ xk  x  J ðxk Þ1 ðFðxk Þ  Fðx ÞÞ

¼ xk  x  J ðxk Þ1 ðJ ðx Þðxk  x Þ þ Fðxk Þ  Fðx Þ  J ðx Þðxk  x ÞÞ
 
¼ I  J ðxk Þ1 J ðx Þ ðxk  x Þ  J ðxk Þ1 ðFðxk Þ  Fðx Þ  J ðx Þðxk  x ÞÞ:

Therefore,
  
 
kxkþ1  x k   I  J ðxk Þ1 J ðx Þ ðxk  x Þþ
 
J ðxk Þ1 ðFðxk Þ  Fðx Þ  J ðx Þðxk  x ÞÞ
 
 I  J ðxk Þ1 J ðx Þ kxk  x kþ
 
J ðxk Þ1  kFðxk Þ  Fðx Þ  J ðx Þðxk  x Þk
  L 
 I  J ðxk Þ1 J ðx Þ kxk  x k þ J ðxk Þ1  kxk  x k2 :
2
From the Lipschitz continuity, we obtain the following estimation:
   
I  J ðxk Þ1 J ðx Þ ¼ J ðxk Þ1 ðJ ðxk Þ  J ðx ÞÞ
   
 J ðxk Þ1  kJ ðxk Þ  J ðx Þk  LJ ðxk Þ1  kxk  x k:

That is,
3L  
J ðxk Þ1  kxk  x k2 :
kxkþ1  x k  ð4:9Þ
2
Now, using Proposition 4.3, it follows that for all xk near enough to x we have kJ(xk)1k  2M,
where M ¼ kJ(x)1k. Therefore, for xk sufficiently closed to x,

kxkþ1  x k  3LMkxk  x k2 , ð4:10Þ

thus proving the quadratic convergence of the Newton method. ♦

4.1 The Newton Method for Solving Nonlinear Algebraic Systems 115

Fig. 4.1 The number of

iterations for solving the
system from Example 4.1 8
when the Newton method is
initialized in points from 6
the domain [1, 3]  [1, 3]
4

0
30
20 25
20
15
10 10
5
0 0

We can see that if kxk  x k  6LM

1
, then kxkþ1  x k  12 kxk  x k, which is the progress of the
method near solution.

Example 4.2 Figure 4.1 shows the number of iterations needed by the Newton method initialized in
a lattice of points in the domain [1, 3]  [1, 3] for solving the nonlinear system from Example 4.1.

Note that near the solution, the method requires a relative small number of iterations. Moreover, a
number of “plateaus” with initial points may be identified. For every point from these plateaus, the
number of iterations is the same.
The Newton method has excellent local convergence properties. Far away from the solution, there
is no guarantee that the quadratic convergence property is conserved. It is quite possible for
the method not to be well defined in the sense that there exists an estimation xk in which the
Jacobian J(xk) is a singular matrix. In such circumstances it is necessary to modify the method in
order to obtain its global convergence. Theorem 4.1 makes us sure that, if the Jacobian matrix of F(x)
in point x is nonsingular and the Lipschitz continuous and the initial point is in a neighborhood of the
solution, then the Newton method is quadratic convergent in the sense of (4.10). The next theorem
emphasizes this behavior of the Newton method.

Theorem 4.2 Let F : ℝn ! ℝn be a continuously differentiable function on a convex set D ⊂ ℝn.

Suppose that there exist x 2 ℝn solution of the system F(x) ¼ 0 and the constants b η, γ > 0 such that
ηÞ centered in x of radius b
the ball Bðx , b η is included in D. Moreover, assume that there exists J(x)1
which satisfies the condition kJ(x)1k  γ and J(x) is Lipschitz continuous on the ball Bðx , b ηÞ with
constant L. Then, there exists ε > 0 such that for any x0 2 B(x, ε), the sequence {xk}

xkþ1 ¼ xk  J ðxk Þ1 Fðxk Þ, k ¼ 0, 1, . . . ð4:11Þ

generated by the Newton method is well definite, convergent to x and satisfies

kxkþ1  x k  γLkxk  x k2 , k ¼ 0, 1, . . . ð4:12Þ

Proof Select ε such that J(x) is nonsingular for any x 2 B(x, ε). Let us prove that the convergence is
quadratic since the local error given by the local affine model used to generate each iteration of the
Newton method is at most O(kxk  xk2). Define
116 4 The Newton Method

 
1
ε ¼ min bη, : ð4:13Þ
2γL

By induction after k we prove that (4.12) is satisfied at each iteration and

1
kxkþ1  x k  kxk  x k, ð4:14Þ
2
that is,

xkþ1 2 Bðx , εÞ: ð4:15Þ

Let us now prove that J(x0) is nonsingular. From kx0  xk  ε, the Lipschitz continuity of J in x
and from (4.12) it follows that
  1   
J ðx Þ ½J ðx0 Þ  J ðx Þ  J ðx Þ1 kJ ðx0 Þ  J ðx Þk
 γLkx0  x k  γLε  1=2:

Now, from the lemma of Banach (see Appendix A), we have that J(x0) is nonsigular and
  1 
  J ðx Þ   
J ðx0 Þ1      2J ðx Þ1   2γ:
1  J ðx Þ ½J ðx0 Þ  J ðx Þ
1

Therefore, x1 is well definite and

x1  x ¼ x0  x  J ðx0 Þ1 Fðx0 Þ

¼ x0  x  J ðx0 Þ1 ½Fðx0 Þ  Fðx Þ
¼ J ðx0 Þ1 ½Fðx Þ  Fðx0 Þ  J ðx0 Þðx  x0 Þ:

Note that F(x)  F(x0)  J(x0)(x  x0) is exactly the difference between F(x) and the affine
model evaluated in x. Therefore, from Theorem 2.7 and the above relation, it follows that

kx1 x k  kJ ðx0 Þ1 k½Fðx ÞFðx0 Þ  J ðx0 Þðx  x0 Þk
L
 2γ kx0  x k2 ¼ γLkx0  x k2 ,
2
thus proving (4.12). Since kx0  xk  1/(2γL), it follows that kx1  x k  12 kx0  x k , that is,
x1 2 B(x, ε), thus completing the case k ¼ 0. As above, the proof continues by induction. ♦

In the above theorem the constants γ and L can be concentrated in a single constant Lrel ¼ γL. Since
  1   
J ðx Þ ½J ðxÞ  J ðx Þ  J ðx Þ1 kJ ðxÞ  J ðx Þk  γLkx  x k ¼ Lrel kx  x k

for x 2 Bðx , b
ηÞ, we can see that Lrel is a Lipschitz constant which measures the relative nonlinearity
of function F in point x. In this context, Theorem 4.2 says that the convergence radius of the Newton
method is inversely proportional to the relative nonlinearity of F in point x. The relative nonlinearity
is a fundamental concept which defines the behavior of the Newton method. All the convergence
results can be easily reformulated by using this concept. However, we preferred to use the absolute
nonlinearity, that is, the Lipschitz constant L, since this gives us a possibility for a deep analysis of the
Newton method.
4.1 The Newton Method for Solving Nonlinear Algebraic Systems 117

Theorem 4.2 is very powerful. The bound ε given by (4.13) is an estimation of the most
disadvantageous situation, in which the domain of the quadratic convergence extends in the direction
from x in which F is the most nonlinear. On the other hand, in the direction from x in which F is the
least nonlinear, the domain of the quadratic convergence is much bigger.
If the condition J(x) is Lipschitz continuous on Bðx , b ηÞ with constant L is replaced with the
condition of H€older’s continuity, that is, kJ(x)  J( y)k  Lkx  ykδ, where δ 2 (0, 1], then Theorem
4.2 remains true, where this time (4.12) is expressed as

kxkþ1  x k  γLkxk  x k1þδ , k ¼ 0, 1, . . .

Applications: The Newton Method for Solving Nonlinear Algebraic Systems of Equations
In the following, let us present the running and the performances of the Newton method for solving
some applications from the collection SMUNO (see Appendix B) as well as from the MINPACK-II
collection.

Application S5 (Stationary solution of a chemical reactor) The application S5 from the SMUNO
collection is expressed as a nonlinear algebraic system and solved with the Newton method:
1  x1  k1 x1 x6 þ r 1 x4 ¼ 0,
1  x2  k2 x2 x6 þ r 2 x5 ¼ 0,
x3 þ 2k3 x4 x5 ¼ 0,
k1 x1 x6  r 1 x4  k3 x4 x5 ¼ 0,
1, 5ðk2 x2 x6  r 2 x5 Þ  k3 x4 x5 ¼ 0,
1  x4  x5  x6 ¼ 0,

where k1 ¼ 31.24 k2 ¼ 0.272 k3 ¼ 303.03 r1 ¼ 2.062 r2 ¼ 0.02.

Table 4.3 contains the initial point x0, the functions values in the initial point F(x0), the solution x
of this application given by the Newton method in 5 iterations, and the functions values in x.
Figure 4.2 presents the evolution of kF(xk)k.

Table 4.3 Stationary solution of a chemical reactor. Initial point, solution, functions values in these points. Newton
method. 5 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 1.09 – 35.164864 0.97424361 0.244e–14
2 1.05 – 0.2403040 0.98282907 – 0.241e–16
3 3.05 3626.94637 0.05151276 – 0.104e–10
4 0.99 – 1779.9233 0.93567106 0.524e–11
5 6.05 – 1814.7127 0.00009083 0.524e–11
6 1.09 – 7.13 0.06423809 – 0.416e–16
118 4 The Newton Method

Fig. 4.2 Evolution of kF(xk)k for applications S5, S6, S9, and S14

Application S6 (Robot kinematics problem) The application S6 from the SMUNO collection is
considered as a nonlinear algebraic system and solved with the Newton method:

4:731 103 x1 x3  0:3578x2 x3 

0:1238x1 þ x7  1:637 103 x2  0:9338x4  0:3571 ¼ 0,

0:2238x1 x3 þ 0:7623x2 x3 þ
0:2638x1  x7  0:07745x2  0:6734x4  0:6022 ¼ 0,

x6 x8 þ 0:3578x1 þ 4:731 103 x2 ¼ 0,

0:7623x1 þ 0:2238x2 þ 0:3461 ¼ 0,
x21 þ x22  1 ¼ 0,
x23 þ x24  1 ¼ 0,
x25 þ x26  1 ¼ 0,
x27 þ x28  1 ¼ 0:

Table 4.4 shows the initial point x0, the functions values in the initial point F(x0), the solution x of
this application given by the Newton method in 3 iterations, and the functions values in x. Figure 4.2
presents the evolution of kF(xk)k.
4.1 The Newton Method for Solving Nonlinear Algebraic Systems 119

Table 4.4 Robot kinematics problem. Initial point, solution, functions values in these points. Newton method.
3 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 0.164 0.0026823 0.16443166 0.111e–15
2 – 0.98 – 0.0098180 – 0.98638847 – 0.111e–15
3 – 0.94 0.1050028 – 0.94706369 – 0.433e–17
4 – 0.32 0.0017588 – 0.32104573 0.555e–16
5 – 0.99 – 0.0127040 – 0.99823316 – 0.111e–15
6 – 0.056 – 0.014 0.05941842 – 0.111e–15
7 0.41 – 0.016764 0.41103315 0.522e–09
8 – 0.91 – 0.0038 – 0.91162039 0

Table 4.5 Propan combustion in air – reduced variant. Initial point, solution, functions values in these points. Newton
method. 9 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 10 109.85000 0.003114068 – 0.160e–09
2 10 209.54267 34.59792428 – 0.515e–09
3 0.05 – 0.3487418 0.065041778 – 0.368e–09
4 50.5 5098.5172 0.859378096 0.158e–06
5 0.05 2659.2930 0.036951859 0.788e–07

Application S9 (Propan combustion in air – reduced variant) This application from the SMUNO
collection is written here as a nonlinear algebraic system:
x1 x2 þ x1  3x5 ¼ 0,
2x1 x2 þ x1 þ 2R10 x22 þ x2 x23 þ R7 x2 x3 þ R9 x2 x4 þ R8 x2  Rx5 ¼ 0,
2x2 x23 þ R7 x2 x3 þ 2R5 x23 þ R6 x3  8x5 ¼ 0,
R9 x2 x4 þ 2x24  4Rx5 ¼ 0,
x1 x2 þ x1 þ R10 x22 þ x2 x23 þ R7 x2 x3 þ R9 x2 x4 , þ R8 x2 þ R5 x23 þ R6 x3 þ x24  1 ¼ 0

where
R5 ¼ 0:193 R6 ¼ 0:4106217541E  3 R7 ¼ 0:5451766686E  3
R8 ¼ 0:44975E  6 R9 ¼ 0:3407354178E  4 R10 ¼ 0:9615E  6
R ¼ 10:

Table 4.5 contains the initial point x0, the functions values in the initial point F(x0), the solution x
of this application given by the Newton method in 9 iterations, and the functions values in x.
Figure 4.2 illustrates the evolution of kF(xk)k corresponding to this application.
120 4 The Newton Method

Application S14 (Circuit design) This application taken from the SMUNO collection is as follows:

 
ð1  x1 x2 Þx3 exp x5 g1k  g3k x7 103  g5k x8 103  1
þg4k x2  g5k ¼ 0, k ¼ 1, . . . , 4,

 
ð1  x1 x2 Þx4 exp x6 g1k  g2k  g3k x7 103  g4k x9 103  1
þg4k  g5k x1 ¼ 0, k ¼ 1, . . . , 4,
x1 x3  x2 x4 ¼ 0,

where
2 3
0:4850 0:7520 0:8690 0:9820
6 7
6 0:3690 1:2540 0:7030 1:4550 7
6 7
g¼6
6 5:2095 10:0677 22:9274 20:2153 77:
6 7
4 23:3037 101:7790 111:4610 191:2670 5
28:5132 111:8467 134:3884 211:4823

Table 4.6 contains the initial point x0, the functions values in the initial point F(x0), the solution x
of this application given by the Newton method in 4 iterations, and the functions values in x.
Figure 4.2 illustrates the evolution of kF(xk)k corresponding to this application.

Application N1 Solid fuel ignition. Application from the MINPACK-II collection (Averick,
Carter, & Moré, 1991), (Averick, Carter, Moré, & Xue, 1992) (Bratu’s problem).
A steady-state model of solid fuel ignition can be described in terms of the solution uλ of the
boundary value problem

ΔuðxÞ ¼ λ exp ½uðxÞ, x 2 D, uðxÞ ¼ 0 for x 2 ∂D

where Δ is the Laplace operator, D is a domain in ℝ2 with boundary ∂D, and λ 2 ℝ is a parameter.
This problem is known as Bratu’s problem. The model simulates a thermal reaction process in a rigid
material (Aris, 1975). On the rectangular domain D ¼ (l1, u1)  (l2, u2) the problem can be

Table 4.6 Circuit design. Initial point, solution, functions values in these points. Newton method. 4 iterations.
e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 0.7 0.95525047 0.89999995 0.124e–13
2 0.5 3.04155543 0.44998747 0.206e–12
3 0.9 – 4.16621316 1.00000648 0.973e–12
4 1.9 – 2.11846214 2.00006854 0.167e–11
5 8.1 5.636125856 7.99997144 – 0.355e–14
6 8.1 22.31397917 7.99969268 0.0
7 5.9 24.95943321 5.00003127 0.255e–12
8 1 42.18784331 0.99998772 0.284e–13
9 1.9 – 0.32000000 2.00005248 0.277e–14
4.1 The Newton Method for Solving Nonlinear Algebraic Systems 121

lambda=5 30

0.7 25
0.6
0.5 20
0.4
0.3 15
0.2
0.1
10
0
30
20 30
5
20
10 10
0 0 5 10 15 20 25 30
Fig. 4.3 Solution of Bratu’s problem

represented as a finite dimensional version using a finite-difference formulation. The vertices zi, j 2 D
are determined by choosing the grid spacings hx and hy and by defining
 
zi,j ¼ l1 þ ihx , l2 þ jhy , 0  i  nx þ 1, 0  j  ny þ 1

such that znx þ1,ny þ1 ¼ ðu1 , u2 Þ. The approximations ui, j to u(zi, j) can be obtained by using central
differences to approximate the Laplacian operator. This leads to a system of n ¼ nxny nonlinear
equations:
hy
h

2ui,j  uiþ1,j  ui1,j þ x 2ui,j  ui,jþ1  ui,j1 ¼ λhx hy exp ui,j

hx hy

where 1  i  nx and 1  j  ny. The solution of Bratu’s problem requires the determination of a path
u(.) such that f(u(λ), λ) ¼ 0, where f : ℝn  ℝ ! ℝn is the mapping that defines the finite dimensional
version of Bratu’s problem.

Considering nx ¼ 30, ny ¼ 30, then a nonlinear algebraic system with n ¼ 900 variables is
obtained. For λ ¼ 5, the Newton method determines a solution of the above system with an accuracy
smaller than 106 in 2 iterations. Figure 4.3 presents the solution of this application.

Application N2 Flow in a drive cavity. Application from the MINPACK-II collection (Averick,
Carter, & Moré, 1991), (Averick, Carter, Moré, & Xue, 1992).
The steady flow of a viscous incompressible fluid in a planar region D is described by the Navier-
Stokes equations
νΔu þ ðu ∇Þu þ ∇p ¼ 0, ∇ u ¼ 0,

where u : D ! ℝ2 is the velocity field of the fluid, p : D ! ℝ is the pressure, and ν is the viscosity
parameter (the reciprocal of the Reynolds number R). In a classical driven cavity problem, the region
D is the unit square in ℝ2, and the boundary conditions are

ð0, 1Þ, ξ2 ¼ 1,
uð ξ 1 , ξ 2 Þ ¼
ð0, 0Þ, 0  ξ2 < 1:
122 4 The Newton Method

One of the main difficulties associated with this formulation is that there are no boundary
conditions on the pressure p. In the stream function-vorticity formulation, the pressure is eliminated,
and the problem is expressed in terms of the stream function φ and vorticity ω. The components
u1 and u2 of the velocity vector u are expressed in terms of the stream function φ by u1 ¼ ∂yφ,
u2 ¼  ∂xφ. Thus, the incompressibility condition ∇ u ¼ 0 is automatically satisfied. The vorticity
ω  ∂xu2  ∂yu1 is thus given by Δφ ¼ ω. The stream function-vorticity formulation of the driven
cavity problem is then



νΔ2 φ þ ∂y φ ð∂x ωÞ  ð∂x φÞ ∂y ω ¼ 0,
Δφ ¼ ω:

The formulation of the driven cavity problem in terms of the stream function requires the solution
of the following boundary value problem



Δ2 φ  Re ∂y φ ð∂x φÞ  ð∂x φÞ ∂y φ ¼ 0:

with the boundary conditions


1, if ξ2 ¼ 1,
φðξ1 , ξ2 Þ ¼ ∂x φðξ1 , ξ2 Þ ¼ 0, ∂y φðξ1 , ξ2 Þ ¼
0, if 0  ξ2 < 1:

(Re is the Reynolds number.) The finite dimensional formulation of the driven cavity problem follows
Schreiber and Keller (1983). The vertices zi, j 2 D are determined by choosing the grid spacings hx and
hy and by defining
 
zi,j ¼ ihx , jhy , 0 < i  nx þ 1, 0 < j  ny þ 1,

such that znx þ1,ny þ1 ¼ ð1, 1Þ: The approximations ui,j to φ(zi,j) are obtained by using central
differences to approximate the Laplacian operator and the partial derivatives ∂yφ and ∂xφ. These
approximations lead to a system of n ¼ nxny equations in the n unknown ui,j of the form
f(u) ¼ Au þ b  ReΦ(u), where A is the discrete biharmonic operator, b contains boundary
information, and Φ is the discrete representation of the nonlinear term.
An interesting feature of this formulation is that the discrete biharmonic approximation becomes
poorly conditioned as the dimension n increases. Thus, the problem becomes difficult to solve even
for moderate values of the Reynolds number Re. This difficulty can be overcome by preconditioning
the problem. Thus, f is preconditioned with the fast biharmonic solver bihar. The preconditioned
problem A1f(u) ¼ 0 with (nx ¼ 30, ny ¼ 30), that is, n ¼ 900 variables, is solved for various values
of the Reynolds number. Figure 4.4 illustrates the streamlines for three Reynolds numbers:
Re ¼ 200, Re ¼ 400, and Re ¼ 1000, respectively. Considering nx ¼ 30, ny ¼ 30, and Re ¼ 200,
the Newton method determines a solution of the above system in 4 iterations, with an accuracy
smaller than 106. The graphical representation of the solution is given in Fig. 4.4a. For nx ¼ 30,
ny ¼ 30, and Re ¼ 400, the Newton method gives a solution in 5 iterations, like in Fig. 4.4b.
Figure 4.4c shows the solution of the problem for nx ¼ 30, ny ¼ 30, and Re ¼ 1000 obtained by the
Newton method in 7 iterations.
4.2 The Gauss-Newton Method 123

Re=200 30

0 25

−0.02
20
−0.04

−0.06 15

−0.08
10
−0.1
30
20 30 5
20 Re=200
10 10
0 0 5 10 15 20 25 30
Re=400 30

0.02 25
0
−0.02 20
−0.04
−0.06 15

−0.08
10
−0.1
30
20 30
20 5
10 10 Re=400
0 0
5 10 15 20 25 30
Re=1000 30

0.02 25

0
20
−0.02

−0.04 15
−0.06
10
−0.08
30
20 30
5
20
10 10 Re=1000
0 0 5 10 15 20 25 30
Fig. 4.4 (a) Flow in a drive cavity (Re ¼ 200). (b) Flow in a drive cavity (Re ¼ 400). (c) Flow in a drive cavity
(Re ¼ 1000)

4.2 The Gauss-Newton Method

Many systems of nonlinear equations are expressed in the form of a vector of m independent
functions, each of them depending on n variables: f1(x), . . ., fm(x), where x 2 ℝn. Let f ¼ [f1(x), . . .,
fm(x)]T. The solution of the system fi(x) ¼ 0, i ¼ 1, . . ., m, can be obtained by forming a real-valued
function
124 4 The Newton Method

X
m
FðxÞ ¼ f i ðxÞ2 : ð4:16Þ
i¼1

If F(x)is minimized by using an unconstrained optimization algorithm, then the individual

functions fi(x), i ¼ 1, . . ., m, are minimized in the least-squares sense. A method for solving the
above class of problems, known as the Gauss-Newton method, can be developed by applying the
Newton method for minimizing function F.
Let J(x) be the Jacobian matrix of F in point x defined as the m  n matrix
2 3
∂f 1 ∂f 1 ∂f 1
6 ∂x ⋯
6 1 ∂x2 ∂xn 77
6 7
6 ∂f 2 ∂f 2 ∂f 2 7
6 ⋯ 7
J ðxÞ ¼ 6 ∂x1 ∂x2 ∂xn 7:
6 7
6 ⋮ ⋮ ⋱ ⋮ 7
6 7
4 ∂f m ∂f m ∂f m 5
⋯
∂x1 ∂x2 ∂xn

In this case, the Jacobian matrix need not be a square matrix. By differentiating Fwith respect
to xj, we get

∂F X
m
∂f
¼ 2f i ðxÞ i , for j ¼ 1, . . . , n: ð4:17Þ
∂xj i¼1
∂xj

Therefore, if ∇F(x) is the gradient of F in point x, it follows that the gradient of F may be expressed
as

∇F ¼ 2J T f : ð4:18Þ

Assuming that the functions fi(x), i ¼ 1, . . ., m, are twice continuously differentiable, then (4.17)
yields
2
∂ F X ∂f ∂f
m X m 2
∂ fi
¼2 i i
þ2 f i ðxÞ ,
∂xj ∂xk i¼1
∂x j ∂x k i¼1
∂x j ∂xk

for j, k ¼ 1, . . ., n.
If the second derivatives of fi(x) are neglected, we have
2
∂ F X ∂f ∂f
m
ffi2 i i
:
∂xj ∂xk i¼1
∂xj ∂x k

Thus, the Hessian of F can be approximated as

∇2 FðxÞ ffi 2J T ðxÞJ ðxÞ: ð4:19Þ

Since the gradient and the Hessian of F in the current point xk are known and assuming that the
Hessian (4.19) is nonsingular, then the Newton method can be applied for solving this problem as
4.3 The Newton Method for Function Minimization 125

 1  
xkþ1 ¼ xk  2J ðxk ÞT J ðxk Þ 2J ðxk ÞT f ðxk Þ
 1  
¼ x k  J ðx k ÞT J ðx k Þ J ðxk ÞT f ðxk Þ , k ¼ 0, 1, . . .

When the current point xk is in the neighborhood of the solution x, then the functions fi(xk),
i ¼ 1, . . ., m, can be accurately represented by the linear approximation of the Taylor series and
therefore, the matrix (4.19) becomes an accurate representation of the Hessian of F(xk). In this case,
the method converges very rapidly. If the functions fi(x), i ¼ 1, . . ., m, are linear, then F(x) is
quadratic, the matrix (4.19) is the exact Hessian, and the problem is solved in one iteration. Of course,
the Gauss-Newton method breaks down if the Hessian (4.19) becomes singular. In this case, the
remedy is the modification of the Gauss-Newton method by using the methods described in Sect. 4.5.

4.3 The Newton Method for Function Minimization

Let f : ℝn ! ℝ be a twice continuously differentiable function and consider the minimizing problem

minn f ðxÞ: ð4:20Þ

x2ℝ

Obviously, in the current point xk we can compute f(xk), the gradient ∇f(xk), and the Hessian
∇2f(xk), where by definition
2 2 2 2 3
∂ f ðx Þ ∂ f ðx Þ ∂ f ðx Þ
6 ∂x2 ⋯
6 1 ∂x1 ∂x2 ∂x1 ∂xn 77
6 2 7
6 ∂ f ðxÞ ∂2 f ðxÞ ∂ f ðx Þ 7
2
6 ⋯ 7
∇ f ðx Þ ¼ 6
2
6 ∂x2 ∂x1 ∂x22 ∂x2 ∂xn 77:
6 7
6 ⋮ ⋮ ⋱ ⋮ 7
6 2 7
4 ∂ f ðxÞ ∂2 f ðxÞ ∂ f ðx Þ 5
2
⋯
∂xn ∂x1 ∂xn ∂x2 ∂x2n

If f is a smooth function, then ∇2f(x) is a symmetric matrix. Suppose that with a minimal effort we
can get the analytical expression of the Hessian. With these elements the quadratic model of function f
in point xk can be defined as
1
m2k ðxÞ ¼ f ðxk Þ þ ∇f ðxk ÞT ðx  xk Þ þ ðx  xk ÞT ∇2 f ðxk Þðx  xk Þ: ð4:21Þ
2
For solving the problem (4.20), the main idea of the Newton method is quite simple: in an iterative
manner, minimize the quadratic model (4.21) and take its solution xkþ1 as a new approximation to the
minimum point. In this new point xkþ1 compute again a new quadratic model and take it as a new
approximation to x, etc.
Minimizing (4.21) leads to the iterative formula

xkþ1 ¼ xk  ∇2 f ðxk Þ1 ∇f ðxk Þ, ð4:22Þ

known as the Newton method.

126 4 The Newton Method

Algorithm 4.2 Newton for minf(x)

1. Initialization. Consider an initial point x0 2 ℝn as well as the convergence tolerance ε > 0 sufficiently
small for stopping the iterations. Set k ¼ 0
2. Compute ∇f(xk) and ∇2f(xk)
3. If k∇f(xk)k  ε, then stop; otherwise, continue with step 4
4. Solve the linear algebraic system ∇2f(xk)dk ¼  ∇ f(xk)
5. Set xkþ1 ¼ xk þ dk, k ¼ k þ 1 and continue with step 2 ♦

Note that in this algorithm, the most difficult operations are in steps 2 and 4, in which the Hessian
matrix has to be evaluated in the current point and a linear algebraic system needs to be solved.
However, the Newton method has some very important properties which define its efficiency. The
most important is that its convergence is quadratic when initialized near the solution. All the
modifications of this method refer to either the evaluation of the Hessian or to the solving of the
linear algebraic system ∇2f(xk)dk ¼  ∇ f(xk), known as the Newton system.

Example 4.3 Let us compute the minimum point of the function

f ðxÞ ¼ x1 exp x22 þ x21 þ x1 þ x22  x2

Figure 4.5 shows the representation of the function in the domain [1, 1]  [0, 2].
The solution of this problem is x ¼ [0.966616, 0.262870] and f(x) ¼  1.1281165. Consider-
ing a lattice of the initial points in the domain [1, 1]  [0, 2], then the number of the iterations
required by the Newton method for minimizing function f is shown in Fig. 4.6. We can see that near
the solution, the method needs a relatively small number of iterations. Moreover, there are some
plateaus with initial points for which the number of iterations is the same.
The following theorem shows that the Newton method (4.22) is well defined and its rate of
convergence is quadratic. The theorem is based on the Lipschitz constant of the Hessian.

Theorem 4.3 Let f : ℝn ! ℝ be twice continuously differentiable and the current point xk sufficiently
close to the solution x of the problem (4.20) for which ∇f(x) ¼ 0. If the Hessian matrix ∇2f(x) is
positive definite and ∇2f(x) is Lipschitz continuous, that is,

20
18
5
4 16
3 14
2
12
1
0 10
−1 8
−2
30 6
20 25
20 4
15
10 10 2
5
0 0 2 4 6 8 10 12 14 16 18 20
Fig. 4.5 Representation of function f
4.3 The Newton Method for Function Minimization 127

Fig. 4.6 Number of

iterations of the Newton
method for different initial
points from the domain 7
[1, 1]  [0, 2]
6

iteratii
4

1
30
30
20
20
10
10
X2 0 0
X1

 
 2 
∇ f ðxÞij  ∇2 f ðyÞij   Lkx  yk, for all 1  i, j  n

where ∇2f(x)ij is the (i, j)- element of the Hessian and L is a positive constant, then for any k the
Newton iteration (4.22) is well defined and the sequence {xk} generated by the algorithm is quadratic
convergent to x.

Proof Let hk ¼ xk  x. From the Taylor development, it follows that

 
0 ¼ ∇f ðx Þ ¼ ∇f ðxk Þ  ∇2 f ðxk Þhk þ O khk k2 :

Since f is twice continuously differentiable, xk is sufficiently close to x and ∇2f(x) is positive

definite, it follows that it is quite reasonable to suppose that the points xk are in a neighborhood of x,
∇2f(xk) are positive definite, and ∇2f(xk)1are upper bounded. Therefore, at the iteration k the Newton
method is defined. Multiplying the above relation by ∇2f(xk)1 we get
 
0 ¼ ∇2 f ðxk Þ1 ∇f ðxk Þ  hk þ O khk k2
   
¼ d k  hk þ O khk k2 ¼ hkþ1 þ O khk k2

By definition of O(.), there exists a constant C such that

khkþ1 k  Ckhk k2 : ð4:23Þ

If xk 2 Ω ¼ {x : khk  γ/C, h ¼ x  x, γ 2 (0, 1)}, then

khkþ1 k  γ khk k  γ 2 =C < γ=C: ð4:24Þ

Therefore, xkþ1 2 Ω. By induction, it follows that the Newton iteration is well defined for all k and
khkk ! 0 when k ! 1, thus proving the convergence of the Newton method. From (4.23) we can see
that the rate of convergence of the Newton method is quadratic. ♦
128 4 The Newton Method

The following theorem shows the quadratic convergence of the Newton method, but this time the
theorem is based on the assumption that the Hessian matrix is bounded.

Theorem 4.4 Let f be a twice continuously differentiable function defined over ℝn. Assume that:

• there exists μ > 0 such that

∇2 f ðxÞ  μI for any x 2 ℝn , ð4:25Þ

• there exists M > 0 such that

 2 
∇ f ðxÞ  ∇2 f ðyÞ  Mkx  yk for any x, y 2 ℝn : ð4:26Þ
2 2

Let {xk} be the sequence generated by the Newton method (4.22), and let x be the unique
minimizer of f over ℝn. Then
M
kxkþ1  x k2  kx  x k22 : ð4:27Þ
2μ k

If kx0  x k2  Mμ , then
 k
2μ 1 2
kx k  x  k2  : ð4:28Þ
M 2

Proof Firstly, let us prove (4.27). By the fundamental theorem of calculus, it follows that
 1
xkþ1  x ¼ xk  ∇2 f ðxk Þ ∇f ðxk Þ  x
 1
¼ xk  x þ ∇2 f ðxk Þ ð∇f ðx Þ  ∇f ðxk ÞÞ
ð
 1 1  2 
¼ xk  x þ ∇2 f ðxk Þ ∇ f ðxk þ tðx  xk ÞÞ ðx  xk Þdt
ð 0
 2 1 1  2 
¼ ∇ f ðx k Þ ∇ f ðxk þ tðx  xk ÞÞ  ∇2 f ðxk Þ ðx  xk Þdt:
0

Then,
 ð 1 
 1 
  2   
kxkþ1  x k2   ∇2 f ðxk Þ    ∇ f ð xk þ tð x 
 x k Þ Þ  ∇ 2
f ð x k Þ ð x  x k Þdt

2
  ð1 0 2
 2 1   2   
  ∇ f ðx k Þ   
∇ f ðxk þ tðx  xk ÞÞ  ∇ f ðxk Þ ðx  xk Þ 2 dt
2 
2 0
  ð1
 1   
  ∇2 f ðxk Þ  ∇2 f ðxk þ tðx  xk ÞÞ  ∇2 f ðxk Þ2 kx  xk k2 dt
2 0
ð
1 1
 Mtkxk  x k22 dt
μ 0
M
¼ kxk  x k22 :
2μ
4.4 The Newton Method with Line-Search 129

For proving (4.28) observe that for k ¼ 0,

 0
μ 2μ 1 2
kx 0  x  k2  ¼ :
M M 2
1
2k
Suppose that for an integer k, (4.28) is satisfied, i.e., kxk  x k2  2μ
M 2 : Now, let us show that
this holds for k þ 1. We have
!
 k 2   kþ1
 M  2 M 2μ 1 2 2μ 1 2
kxkþ1  x k2  kxk  x k2  ¼ : ♦
2μ 2μ M 2 M 2

The interpretation of the condition (4.25) is as follows. As we know, the Taylor series of f near xk is
1
f ðxÞ ¼ f ðxk Þ þ ∇f ðxk ÞT ðx  xk Þ þ ðx  xk ÞT ∇2 f ðxk Þðx  xk Þ þ . . . ð4:29Þ
2
The condition ∇2f(x)  μI determines that the quadratic approximation of f in xk is consistent in the
sense that the quadratic term in (4.29) is large enough such that the iterative formula for the Newton
method xkþ1 ¼ xk  ∇2f(xk)1gk, k ¼ 0, 1, . . ., is well defined.
On the other hand, the condition k∇2f(x)  ∇2f( y)k2  Mkx  yk2 can be rewritten as
 2 
∇ f ðxÞ  ∇2 f ðyÞ
2
 M,
kx  yk2

where 0 < M < 1 . Therefore, this condition tells us that the tensor of the derivative of the third order
of f in xk is small enough such that the rest of the Taylor series can be very well neglected. In
conclusion, for the iterative process xkþ1 ¼ xk  ∇2f(xk)1gk, k ¼ 0, 1, . . ., to work, both these
conditions (4.25) and (4.26) require that the quadratic approximation of f in xk should be a good one.

4.4 The Newton Method with Line-Search

The Newton method is a local method. When the initial point x0 is far away from the solution, we are
not sure that ∇2f(xk) is positive definite, and there is no guarantee that the Newton direction dk is
descendent. Since, as we know, the line-search is a strategy of globalization, then we can define the
Newton method with line-search, (also called the damped Newton method) which guarantees the
global convergence.
The Newton method with line-search is defined as

dk ¼ ∇2 f ðxk Þ1 ∇f ðxk Þ, ð4:30Þ

xkþ1 ¼ xk þ αk dk , ð4:31Þ

where αk is the stepsize, which can be determined either by the exact line-search, or by the inexact
line-search. The Newton algorithm with line-search is as follows:
130 4 The Newton Method

Algorithm 4.3 Newton with line-search for minf(x)

1. Initialization. Consider an initial point x0 2 ℝn as well as the convergence tolerance ε > 0 sufficiently
small for stopping the iterations. Set k ¼ 0
2. Compute ∇f(xk) and ∇2f(xk)
3. If k∇f(xk)k  ε, then stop; otherwise, continue with step 4
4. Solve the linear algebraic system ∇2f(xk)dk ¼  ∇ f(xk)
5. Compute αk by the exact line-search
f ðxk þ αk d k Þ ¼ min f ðxk þ αdk Þ
α0
or by the inexact line-search
αk ¼ argminf ðxk þ αdk Þ
α0
6. Set xkþ1 ¼ xk þ αkdk, k ¼ k þ 1 and continue with step 2 ♦

The following theorem shows the global convergence of Algorithm 4.3.

Theorem 4.5 Let f : ℝn ! ℝ be a twice continuously differentiable function on the open and convex
set D ⊂ ℝn. Suppose that for any x0 2 D there exists a constant m > 0 such that f(x) satisfies

uT ∇2 f ðxÞu  mkuk2 , for any u 2 ℝn and x 2 S, ð4:32Þ

where S ¼ {x : f(x)  f(x0)} is the level set corresponding to the initial point. Then the sequence {xk}
generated by Algorithm 4.3 with exact line-search satisfies:

(i) If the sequence {xk} is finite, then ∇f(xk) ¼ 0 for a certain k.

(ii) If the sequence {xk} is infinite, then {xk} is convergent to the unique minimum xof f.

Proof From (4.32) it follows that f(x) is a strict convex function on ℝn. Therefore, its stationary point
is the unique global minimum. Also, the level set S is convex, bounded, and closed. Since the
sequence {f(xk)} is monotonously decreasing, then {xk} ⊂ S, that is, {xk} is bounded. Therefore,
there is a limit point x 2 S such that xk ! x and f ðxk Þ ! f ðxÞ. Now, since f is twice continuously
differentiable, from Theorem 2.12 it follows that ∇f ðxk Þ ! ∇f ðxÞ ¼ 0. Finally, observe that, since
the stationary point is unique, then the sequence {xk} is convergent to x, the unique minim. ♦

Now, let us see the convergence of the Newton algorithm with inexact line-search. If the weak
Wolfe line-search (2.54) and (2.59) are used, then from (2.93) in Proposition 2.6, it follows that

f ðxk Þ  f ðxk þ αk dk Þ  ηk∇f ðxk Þk2 cos 2 hdk , ∇f ðxk Þi, ð4:33Þ

where η is a constant independent of k. With these, the global convergence of the Newton method
with inexact line-search can be proved.

Theorem 4.6 Let f : ℝn ! ℝ be a twice continuously differentiable function on the open and convex
set D ⊂ Rn. Suppose that for any x0 2 ℝn there exists the constant m > 0 such that f(x) satisfies (4.32)
on the level set S. If the line-search satisfies (4.33), then the sequence {xk} generated by Algorithm 4.3
with inexact line-search satisfies
lim k∇f ðxk Þk ¼ 0 ð4:34Þ
k!1

and {xk} is convergent to the unique minimum xof function f.

4.4 The Newton Method with Line-Search 131

Proof Since f(x) satisfies (4.32), then f(x) is uniformly convex on S. Also, from (4.33) it follows that
f(x) is strictly monotonously decreasing and the sequence {xk} is bounded. Therefore, there exists a
constant M > 0 such that for any k,
 2 
∇ f ðxk Þ  M: ð4:35Þ

Then, from (4.33) we get

dTk ∇f ðxk Þ ∇f ðx ÞT ∇2 f ðxk Þ1 ∇f ðxk Þ

cos hdk , ∇f ðxk Þi ¼ ¼  2 k 1 
kdk kk∇f ðxk Þk ∇ f ðxk Þ ∇f ðxk Þk∇f ðxk Þk
d Tk ∇2 f ðxk Þdk m
¼  
kdk k∇2 f ðxk Þdk  M

Again, from (4.33) it follows that

X
1 X1
m2
1> ½f ðxk Þ  f ðxkþ1 Þ  η 2 k∇f ðxk Þk2 , ð4:36Þ
k¼0 k¼0 M

thus proving (4.34). Note that f(x) is uniformly convex, which means that it has only a stationary
point. Therefore, (4.34) shows that the sequence {xk} is convergent to the unique minimum x of
function f. ♦

The following result emphasizes the local rate of the convergence of the Newton method. Recall
that for all x from a vicinity of a minimum point x such that ∇2f(x) is positive definite the Hessian
∇2f(x) will also be positive definite. The Newton method will be well defined in this vicinity and will
converge quadratically, provided that the stepsizes αk are always 1 eventually.

Theorem 4.7 Suppose that f is twice continuously differentiable and that its Hessian ∇2f(x) is
Lipschitz continuous in a neighborhood of a solution x at which the sufficient optimality conditions
(Theorem 11.6) are satisfied. Consider the iteration xkþ1 ¼ xk þ d Nk , where dNk ¼ ∇2 f ðxk Þ1 gk :
Then:

(i) If the starting point x0 is sufficiently close to x, then the sequence {xk} generated by the Newton
method converges to x.
(ii) The rate of convergence of {xk} is quadratic.
(iii) The sequence of the gradients norms {kgkk} converges quadratically to zero.

Proof From the optimality condition ∇f(x) ¼ 0 and by the definition of the Newton method, it
follows that

xk þ d Nk  x ¼ xk  x  ∇2 f ðxk Þ1 gk
 
¼ ∇2 f ðxk Þ1 ∇2 f ðxk Þðxk  x Þ  ð∇f ðxk Þ  ∇f ðx ÞÞ :

But
132 4 The Newton Method

ð1
∇f ðxk Þ  ∇f ðx Þ ¼ ∇2 f ðxk þ tðx  xk ÞÞðxk  x Þdt:
0

Therefore,
 2 
∇ f ðxk Þðxk  x Þ  ð∇f ðxk Þ  ∇f ðx ÞÞ
1 
ð 
  2  
¼  ∇ f ðxk Þ  ∇ f ðxk þ tðx  xk ÞÞ ðxk  x Þdt
 2  

 
0
ð1
 
 ∇2 f ðxk Þ  ∇2 f ðxk þ tðx  xk ÞÞkxk  x kdt
0
ð1
1
 kx k  x k  2
Ltdt ¼ Lkxk  x k2 ,
2
0

where L is the Lipschitz constant of ∇ f(x) for x near x. Since ∇2f(x) is nonsingular, there is a radius
2

r > 0 such that k∇2f(xk)1k  2k∇2f(x)1k (see Theorem A2.5) for all xk with kxk  xk  r.
Therefore,
   
xk þ dN  x   L∇2 f ðx Þ1 kxk  x k2 ¼ Lekxk  x k2 ,
k

 
where Le ¼ L∇2 f ðx Þ1  and dNk ¼ ∇2 f ðxk Þ1 ∇f ðxk Þ: Choosing x0 so that kx0  x k 
n  o
min r, 1= 2Le , then the above inequality may be used inductively to deduce that the sequence
{xk} converges to x and the rate of convergence is quadratic.
To prove that the gradient norms converge to zero quadratically, we use the relations xkþ1 
xk ¼ dNk and ∇f ðxk Þ þ ∇2 f ðxk Þd Nk ¼ 0, to obtain that
 
k∇f ðxkþ1 Þk ¼ ∇f ðxkþ1 Þ  ∇f ðxk Þ  ∇2 f ðxk Þd Nk 
1 
ð 


 2 N 2 N
¼  ∇ f xk þ td k ðxkþ1  xk Þdt  ∇ f ðxk Þd k 
 
0
ð1

 
 ∇2 f xk þ td Nk  ∇2 f ðxk Þd Nk dt
0
1  2 1  2
 LdNk   L∇2 f ðxk Þ1  k∇f ðxk Þk2
2  2
 2L∇2 f ðx Þ1  k∇f ðxk Þk2 ,
2

showing (iii). ♦

The following theorem shows that in some mild conditions, when the iterates generated by the
Newton method are approaching the solution, then the Wolfe (or the Goldstein) line-search
conditions will accept the stepsize αk ¼ 1 for all large k.
4.5 Analysis of Complexity 133

Theorem 4.8 Suppose that f : ℝn ! ℝ is twice continuously differentiable. Consider the iteration
xkþ1 ¼ xk þ αkdk, where dk is a descent direction and αk satisfies the Wolfe line-search (2.54) and
(2.59) with ρ  1/2. If the sequence {xk} converges to a point x such that ∇f(x) ¼ 0 and ∇2f(x) is
positive definite and if the search direction satisfies
 
∇f ðxk Þ þ ∇2 f ðxk Þd k 
lim ¼ 0,
k!1 kd k k

then:

(i) The stepsize αk ¼ 1 is admissible for all k greater than a certain index k0.
(ii) If αk ¼ 1 for all k > k0, then {xk} converges to x superlinearly. ♦

Observe that if ρ > 1/2, then the line-search would exclude the minimizer of a quadratic and the
unit stepsize may not be admissible.

4.5 Analysis of Complexity

This section is devoted to analyzing the complexity of the Newton method, namely, an estimation of
the number of iterations for solving an unconstrained optimization problem. From the very beginning
let us introduce some concepts. At the same time, in order to simplify the notation, instead of
xkþ1 ¼ xk þ αkdk, the following notation will be used x+ ¼ x þ αd. The Newton method is defined
by two concepts: the Newton step and the Newton decrement. The Newton decrement can be used as a
criterion for stopping the iterations (Andrei, 2004a).

The Newton Step For x 2 dom f, the vector

d nt ¼ ∇2 f ðxÞ1 ∇f ðxÞ ð4:37Þ

is called the Newton step for function f in point x. If ∇2f(x) is positive definite, it follows that, if
∇f(x) 6¼ 0, then

∇f ðxÞT dnt ¼ ∇f ðxÞT ∇2 f ðxÞ1 ∇f ðxÞ < 0:

Therefore, if x is not the minimum point, then the Newton step is always a descent direction.
Therefore, in order to minimize the second order approximation (4.21) of f in x, the idea is to add the
Newton step at the current point. Observe that if f is quadratic, then x þ dnt is the exact minimum of f.
If f is close to a quadratic, then our intuition tells us that x þ dnt is a very good estimation of the
minimum x of f. Moreover, since f is strictly convex and twice continuously differentiable, the
quadratic model of f will be very exact when x is near x. Therefore, as soon as x is near enough to x,
it results that the point x þ dnt is a very good approximation of x, i.e., it is very close to x. If the
optimality condition ∇f(x) ¼ 0 is linearized around x, then

∇f ðx þ dÞ ffi ∇f ðxÞ þ ∇2 f ðxÞd ¼ 0

and, as we can see, this is a linear algebraic system in d with solution d ¼ dnt. Therefore, the Newton
step dnt is exactly what is necessary to add to x in order satisfy the optimality condition.
134 4 The Newton Method

A property of the Newton step concerning its invariance at the linear coordinate or affine
transformations needs to be mentioned. Suppose that T 2 ℝnn is a nonsingular matrix and define
f ðyÞ ¼ f ðTyÞ: Then

∇f ðyÞ ¼ T T ∇f ðxÞ, ∇2 f ðyÞ ¼ T T ∇2 f ðxÞT,

where x ¼ Ty. With these, the Newton step for f in the variables y is

1 T

dnt ¼  T T ∇2 f ðxÞT T ∇f ðxÞ

¼ T 1 ∇2 f ðxÞ1 ∇f ðxÞ ¼ T 1 dnt ,

where dnt is the Newton step of f in the variables x. Therefore, the Newton steps for f and f are

connected by the same linear transformation and x þ d nt ¼ T y þ dnt .

The Newton Decrement The Newton decrement in point x is defined by the quantity
 1=2
r ðxÞ ¼ ∇f ðxÞT ∇2 f ðxÞ1 ∇f ðxÞ : ð4:38Þ

The Newton decrement has a very important role in the analysis of the Newton method and can be
used as a criterion for stopping the iterations in the algorithm of this method. Indeed, the quantity
f ðxÞ  inf y fbðyÞ where fb is the quadratic approximation of f in x can be expressed as

1
f ðxÞ  inf y fbðyÞ ¼ f ðxÞ  fbðx þ d nt Þ ¼ r ðxÞ2 :
2
Therefore, r2/2 is an estimation of the suboptimality f(x)  f  based on the quadratic approxima-
tion of f in x. Observe that the Newton decrement can be expressed as

1=2
r ðxÞ ¼ d Tnt ∇2 f ðxÞdnt , ð4:39Þ

where dnt is the Newton step. Therefore, it follows that r is exactly the norm of the Newton step
subject to the Hessian of function f

1=2
r ðxÞ ¼ kd nt k∇2 f ðxÞ ¼ d Tnt ∇2 f ðxÞdnt : ð4:40Þ

Finally, we mention that the Newton decrement is affine invariant. In other words, the Newton
decrement of f ðyÞ ¼ f ðTyÞ in y, where T is nonsingular, is the same as the Newton decrement of f in
x ¼ Ty.
In order to have a general analysis of the convergence of the Newton method, we consider that a
line-search is performed along the Newton direction. This leads to the Newton method with line-
search, which is different from the pure Newton method, in which at every iteration the stepsize is
one. In the Newton algorithm with line-search, at every iteration the Newton step and the Newton
decrement are computed. These involve the inverse of the Hessian matrix of the minimizing function
f. Numerically, this is the most delicate operation. However, from the viewpoint of the convergence,
this operation does not have any importance since it is the bounds of the Hessian which are the most
important for the strongly convex functions considered in our analysis.
Suppose that f is a twice continuous differentiable function, strongly convex with the constant m,
that is, for any x 2 S, ∇2f(x)  mI. As we know, this involves that there is another constant M > 0 such
4.5 Analysis of Complexity 135

that, for any x 2 S, ∇2f(x)  MI. Additionally, suppose that the Hessian matrix of f is Lipschitz
continuous on S with the constant L, that is, for all x, y 2 S, k∇2f(x)  ∇2f( y)k2  Lkx  yk2. The
Lipschitz constant L can be interpreted as a bound of the third order derivative of function f, which is
zero for quadratic functions. Moreover, L gives a measure of how well function f can be approximated
by a quadratic model around the current point. The smaller the Lipschitz constant, the better the
approximation of f in the current point by the quadratic model, i.e., the Newton method will work
better.

Theorem 4.9 Let f be a twice continuous differentiable function, strongly convex with constants m
and M, for which the Hessian matrix is Lipschitz continuous with the constant L. Then, there exist the
real numbers η and γ with the properties 0 < η  m2/L and γ > 0, such that the following two
conditions hold:

• If k∇f(x)k2  η, then

f ðxkþ1 Þ  f ðxk Þ  γ: ð4:41Þ

• If k∇f(x)k2 < η, then the backtracking line-search selects αk ¼ 1 and

 2
L L
2 k∇f ðxkþ1 Þk2  2 k∇f ðxk Þk2 ð4:42Þ
2m 2m

Before proving the theorem, some comments are needed (Andrei, 2004a). Let us analyze the above
implications. Indeed, the first condition shows that even if the norm of the gradient is greater than a
positive constant, then the algorithm achieves a reduction of the values of the minimizing function.
The second condition is stronger and has the following interpretation. Suppose that at iteration k we
have k∇f(xk)k2 < η. Since η  m2/L, it follows that k∇f(xkþ1)k2 < η, that is, the second condition is also
satisfied at iteration k þ 1. Recursively, continuing this interpretation, we conclude that as soon as the
second condition holds, it will be satisfied for all the next iterations, i.e., for all j  k we have
k∇f(xj)k2 < η. Therefore, for all j  k, the algorithm will take a Newton step of unitary steplength
αj ¼ 1 and
 
  2
L 

∇f xjþ1   L ∇f xj 
2 2 2 2
ð4:43Þ
2m 2m
Applying this inequality recursively, for all j  kwe get
 2jk  2jk
L 

∇f xj   L k∇f ðxk Þk 1
2 2  :
2m2 2m2 2
Therefore,
3  2jkþ1

1 

∇f xj 2  2m 1
f xj  f   ð4:44Þ
2m 2
L2 2
This inequality shows that as soon as the second condition has been satisfied, the convergence of
the algorithm is extremely rapid. Keeping in mind that in this situation the stepsizes are all equal to
136 4 The Newton Method

one, then from (4.44) we can see that the error is quadratically convergent to zero, that is, at every
iteration the number of the correct digits in the value of the minimizing function is doubled.
Every iteration of the Newton algorithm may be divided into two stages. The first one is called the
damped Newton stage, since the algorithm may select the stepsize α < 1. The second one appears as
soon as k∇f(x)k2 < η has been satisfied and is called the pure Newton stage or the Newton quadratic
convergent stage. In this stage, the stepsize is one.
With these, let us determine the complexity of the Newton algorithm with line-search. At first
determine an upper bound of the number of iterations in the damped Newton stage. Indeed, since at
every iteration, f decreases at least with γ, it follows that the number of iterations in the damped
Newton stage should not be larger than

f ðx 0 Þ  f 
:
γ

For finding a bound of the number of iterations corresponding to the pure Newton stage, consider
(4.44). Indeed, we can see that f(x)  f   ε after not more than
 
ε
log 2 log 2 0
ε
iterations, where ε0 ¼ 2m3/L2.
Therefore, the total number of iterations after which f(x)  f   ε is upper bounded by
f ðx0 Þ  f   
ε
þ log 2 log 2 0 ð4:45Þ
γ ε

Note that the term log2log2(ε0/ε), which upper bounds the number of iterations in the Newton
quadratic convergent stage, increases extremely slowly with the accuracy ε. Practically, if we
consider, for example, ε ¼ 5 1020ε0, then the number of iterations corresponding to the pure
Newton stage is not larger than 6.
Hence, we can say that the total number of iterations associated to the Newton method for
minimizing function f with a very good approximation is upper bounded by
f ðx0 Þ  f 
þ 6: ð4:46Þ
γ

Now, let us prove the above theorem. This consists of analyzing the stages of the Newton method.

Proof of Theorem 4.9 Damped Newton stage. We need to establish (4.41). Suppose that
k∇f(x)k2  η. At the very beginning let us establish a lower bound on the stepsize computed by
backtracking. From the strong convexity, it follows that on S we have ∇2f(x)  MI. Hence,

Mkd nt k22 2
f ðx þ αd nt Þ  f ðxÞ þ α∇f ðxÞT dnt þ α
2
2 M 2 2
 f ðxÞ  αr ðxÞ þ α r ðxÞ
2m

Observe that b
α ¼ m=M verifies the termination condition of the backtracking, since
m
f ðx þ b
αdnt Þ  f ðxÞ  r ðxÞ2  f ðxÞ  ρ b
αr ðxÞ2 :
2M
4.5 Analysis of Complexity 137

Therefore, the backtracking gives a stepsize of length α  βm/M, thus involving a reduction of the
function values
m
f ðxþ Þ  f ðxÞ  ραr ðxÞ2  ρβ r ðxÞ2
M
m m
 ρβ 2 k∇f ðxÞk22  ρβη2 2 ,
M M
That is, (4.41) is satisfied for
m
γ ¼ ρβη2 : ð4:47Þ
M2
Newton quadratic convergent stage. Now, let us establish (4.42). Suppose that k∇f(x)k2 < η. Let us
prove that, if

m2
η  3ð1  2ρ Þ ,
L
then the backtracking selects stepsizes of unitary length. But the Lipschitz continuity of the Hessian
matrix
 2 
∇ f ðxÞ  ∇2 f ðyÞ  Lkx  yk
2 2

determines that for all α  0

αLkd nt k32  dTnt ∇2 f ðx þ αdnt Þ  ∇2 f ðxÞ dnt  αLkd nt k32 :

This gives the possibility to determine an upper bound for φ(α) ¼ f(x þ αdnt). Indeed,

αLkdnt k32  φ00 ðαÞ  φ00 ð0Þ  αLkdnt k32 :

By integration twice we get

α2 00 α3
φðαÞ  φð0Þ þ αφ0 ð0Þ þ φ ð0Þ þ Lkd nt k32 , ð4:48Þ
2 6
and

α2 T 2 α3
f ðx þ αd nt Þ  f ðxÞ þ α∇f ðxÞT d nt þ d ∇ f ðxÞd nt þ Lkdnt k32 : ð4:49Þ
2 nt 6
Analyzing the first three terms in (4.49), observe that they are the second order approximation of
f(x þ αdnt) around α ¼ 0. The fourth gives an upper bound of the deviation from the quadratic
function. The bound (4.49) holds for any direction d. Therefore, applied to the Newton direction, we
get
 
α α3  3
f ðx þ αdnt Þ  f ðxÞ  α 1  ∇f ðxÞT ∇2 f ðxÞ1 ∇f ðxÞ þ L∇2 f ðxÞ1 ∇f ðxÞ2
2 6 ð4:50Þ
 
α 2 L 3
 f ðxÞ  α 1  r ðxÞ þ α3
r ðxÞ :
2 6m3=2
Suppose that k∇f(x)k2  η  3(1  2ρ )m2/L. From the strong convexity of function f, it follows
that
138 4 The Newton Method

r ðxÞ  3ð1  2ρ Þm3=2 =L

and
1 L
f ðx þ dnt Þ  f ðxÞ  r ðxÞ2 þ 3=2 r ðxÞ3
2 6m
 f ðxÞ  ρ r ðxÞ2 ¼ f ðxÞ þ ρ∇f ðxÞT d nt ,

that is, the stepsize α ¼ 1 is accepted by backtracking.

Now, let us examine the rate of convergence. From the Lipschitz continuity we have
 
k∇f ðxþ Þk2 ¼ ∇f ðx þ d nt Þ  ∇f ðxÞ  ∇2 f ðxÞdnt 2
1 
ð 
 2

¼  ∇ f ðx þ αd nt Þ  ∇ f ðxÞ dnt dα
 2
 ð4:51Þ
 
0 2
L L 2 L
 kdnt k22 ¼ ∇2 f ðxÞ1 ∇f ðxÞ2  k∇f ðxÞk22 ,
2 2 2m2
that is exactly (4.42). ♦

In conclusion, the Newton algorithm selects unitary stepsizes and satisfies (4.42) if k∇f(x)k2 < η,
where

m2
η ¼ min f1, 3ð1  2ρÞg : ð4:52Þ
L
Introducing (4.47) and (4.52) in (4.46), we obtain that the total number of iterations corresponding
to the Newton method is bounded by
f ðx 0 Þ  f 

2 þ 6: ð4:53Þ
ρβ min f1, 3ð1  2ρ Þg mL
2 m
M2

Observe that:

♦ The estimation (4.53) depends on three constants m, M, and L which are unknown for the vast
majority of practical optimization applications. Moreover, this estimation is valid only for twice
continuously differentiable, strongly convex functions.
♦ The number of iterations depends on the initial point x0. (See Figs. 4.1 and 4.6.) The closer to x
the point x0 is, that is the smaller the initial suboptimality f(x0)  f  is, the smaller the number of
iterations in the damped Newton stage will be. This emphasizes once more the local character of
the Newton method as well as the importance of choosing a good initial point.
♦ The number of iterations in the Newton method does not explicitly depend on the number of
variables of the problem.
♦ The parameters ρ and β from backtracking are not important in the convergence of the damped
Newton stage.

Therefore, the bound (4.53) does not have a practical value, but a conceptual one, showing that for
twice continuously differentiable, strongly convex, functions the Newton algorithm converges
quadratically to the solution if the initial point is in a neighborhood of the minimum point. If the
initial point is far way from the solution, nothing can be said about convergence.
4.5 Analysis of Complexity 139

Example 4.4 Consider the minimization of the function

n 
X 2
f ðxÞ ¼ ðx1  3Þ2 þ x 1  3  2ð x 1 þ x 2 þ ⋯ þ x i Þ 2
i¼2

The solution of this problem is x ¼ [3, 3, 0, ⋯, 0], for which f(x) ¼ 0. Observe that the
function is complex, and its Hessian matrix is a full matrix.
Taking n ¼ 50 and different initial points, then the initial value of function f and the number of
iterations (#iter) for the Newton method with backtracking are as in Table 4.7
For n ¼ 100, the number of iterations of the Newton method for minimizing function f are
presented in Table 4.8.
For n ¼ 300, the Newton method gives the results from Table 4.9.
From the Tables 4.7–4.9 , we can see that the bigger the initial suboptimality f(x0)  f , the bigger
the number of iterations, which confirms the structure of (4.53).

Table 4.7 Initial points, value of function in initial points, number of iterations. n ¼ 50 , e ¼ 108, r ¼ 0.0001
and b ¼ 0.8

Nr. x0 f ( x0 ) #iter
1 [0.001, ... , 0.001] 450.215229 9
2 [0.01, ... , 0.01] 500.96877 16
3 [0.1, ... , 0.1] 31666.3496 28
4 [1, ... , 1] 263010248.01 44

Table 4.8 Initial points, value of function in initial points, number of iterations. n ¼ 100 , e ¼ 108, r ¼ 0.0001
and b ¼ 0.8

Nr. x0 f ( x0 ) #iter
1 [0.001, ... , 0.001] 903.46713 13
2 [0.01, ... , 0.01] 1380.68874 17
3 [0.1, ... , 0.1] 860222.816 40
4 [1, ... , 1] 0.82040405 – 1010 54

Table 4.9 Initial points, value of function in initial points, number of iterations. n ¼ 300 , e ¼ 108, r ¼ 0.0001
and b ¼ 0.8

Nr. x0 f ( x0 ) #iter
1 [0.001, ... , 0.001] 2808.6649 16
2 [0.01, ... , 0.01] 33102.2686 23
3 [0.1, ... , 0.1] 0.197075349 –10 9 66
4 [1, ... , 1] 0.196030836 – 10
13 85
140 4 The Newton Method

Advantages and Disadvantages of the Newton Method

The Newton method has some very important advantages which establish its central position in the
frame of unconstrained optimization methods:

• In general the convergence of the Newton method is rapid, being quadratic near x. The algorithm
is immediately in the attraction basin of x, that is, the iterations are beyond the damped stage. The
algorithm gives a very accurate solution in 5 or 6 iterations at most.
• The Newton method is affine invariant, i.e., it is not sensitive to any change of the coordinate
system or to the condition number of the level sets.
• The backtracking parameters ρ and β have little influence over the performances of the Newton
method. The exact line-search does not significantly improve the performances of the method. In
the most practical implementations of backtracking, ρ ¼ 0.0001 and β ¼ 0.8.
• The Newton method is indifferent to the dimension of the problem. Its performances in solving
problems with a large number of variables are similar to its performances in solving problems with
a small number of variables.

The most important disadvantages of the Newton method are:

• The method depends on the initial point. The difficulties of convergence appear in the damped
stage. In general, for nonconvex functions, if the initial point is far away from x, then the method
can be divergent. The hypothesis of convexity is crucial in the Newton method.
• The practical implementation of this method requires the computation and storage of the Hessian
of the minimizing function as well as the solving of the Newton system. These are very difficult
tasks, especially for complicated, strongly nonlinear unconstrained optimization problems or for
problems with a large number of variables. The quasi-Newton methods, which inherit some
properties of the Newton method, can be used for solving these difficult problems, but at a
superlinear rate of convergence. In other words, what we gain in the computational effort we
lose in the convergence.

4.6 The Modified Newton Method

As known, given the initial point x0, the Newton method is defined by the following computational
scheme:
xkþ1 ¼ xk þ αk dk , ð4:54Þ

dk ¼ ∇2 f ðxk Þ1 ∇f ðxk Þ, ð4:55Þ

for k ¼ 0, 1, . . . . The Newton method is well definite if and only if the following conditions hold:
(i) the Hessian ∇2f(xk) is nonsingular and (ii) the approximation (4.21) is valid. If condition (i) is
violated, then (4.55) may have an infinite number of solutions or no solution at all. On the other hand,
if condition (ii) is violated, then dk may not yield the solution in one iteration and, if ∇2f(xk) is not
positive definite, dk may not even yield a reduction in the minimizing function.
Therefore, one of the most important difficulties with this method is that the Hessian matrix ∇2f(xk)
might not be positive definite. In this case we have no certainty that the quadratic model
approximating function f around xk has a minimum point. If ∇2f(xk) is not defined, then the quadratic
4.6 The Modified Newton Method 141

model m2k ðxÞ is unbounded. In order to overcome these situations, some modifications of the Newton
method have been suggested, the most important ones discussed in this section being Levenberg
(1944) and Marquardt (1963), Goldfeld, Quandt, and Trotter (1966), Zwart (1969), Matthews and
Davies (1971), Goldstein and Price (1967), and Gill and Murray (1972). Other modified Newton
methods like Fiacco and McCormick (1968), Fletcher and Freeman (1977), second order Armijo step
rule by Goldfarb (1980), and by Moré and Sorensen (1979) were analyzed by Sun and Yuan (2006).
Some other papers on using the negative curvature directions in optimization were given by
(Forsgren, Gill, & Murray, 1995), (Facchinei, & Lucidi, 1998), (Gill, Kungurtsev, & Robinson,
2017a, 2017b), and (Goldfarb, Mu, Wright, & Zhou, 2017).

(a) The Levenberg and Marquardt Method

A simple technique to ensure that the search direction is a descent one is given by the Levenberg-
Marquardt modification of the Newton method:

1
xkþ1 ¼ xk  ∇2 f ðxk Þ þ μk I ∇f ðxk Þ, k ¼ 0, 1, . . . , ð4:56Þ

where μk  0.
Consider the symmetric matrix ∇2f(x), which may not be positive definite and let λ1, . . ., λn be its
eigenvalues with eigenvectors v1, . . ., vn. Obviously, the eigenvalues λ1, . . ., λn are real, but may not
all be positive. Now, let us consider the matrix G ¼ ∇2f(x) þ μI, where μ  0. The eigenvalues of
G are λ1 þ μ, . . ., λn þ μ. Indeed,

Gvi ¼ ∇2 f ðxÞ þ μI vi ¼ ∇2 f ðxÞvi þ μIvi

¼ λi vi þ μvi ¼ ðλi þ μÞvi ,

which shows that for all i ¼ 1, . . ., n, vi is also an eigenvector of ∇2f(x) with eigenvalue λi þ μ.
Therefore, if the parameter μ is sufficiently large, then we are sure that all the eigenvalues of G are
positive, that is, G is positive definite. In other words, if the parameter μk in the Levenberg-Marquardt
modification of the Newton method (4.56) is sufficiently large, then the search direction
dkþ1 ¼  (∇2f(xk) þ μkI)1 ∇ f(xk) always is a descent direction. If μk ! 0, then the Levenberg-
Marquardt modification of the Newton method (4.56) approaches the behavior of the pure Newton
method. On the other hand, if μk ! 1 , then the algorithm approaches a pure gradient method with
small stepsizes. In practice, the algorithm starts with a small value of μk, and then slowly it is
increased until a descent iteration is obtained, that is, until f(xkþ1) < f(xk).

(b) The Goldfeld, Quandt, and Trotter Method

When the Hessian Gk  ∇2f(xk) becomes nonpositive, this method replaces the Hessian by the
n  n identity matrix In. This approach can be implemented by considering

bk ¼ Gk þ γI n ,
G ð4:57Þ
1þγ

where γ is set to a large value if Gk is nonpositive definite or to a small value if Gk is positive definite.
bk  I n and therefore dk   ∇ f(xk). As known, a nonpositive definite
Obviously, if γ is large, then G
Gk arises at points far from the solution, where the steepest descent method is most effective in
reducing the value of f(x). Therefore, the above modification (4.57) leads to an algorithm that
combines the properties of the Newton and steepest descent methods.
142 4 The Newton Method

(c) The Zwart Method

The idea of this method is to form a modified matrix
bk ¼ U T Gk U þ ε,
G ð4:58Þ

where U is a real unitary matrix (i.e., UTU ¼ In) and ε is a diagonal n  n matrix with diagonal
elements εi, i ¼ 1, . . ., n. Obviously, for the real symmetric matrix Gk, there exists a real unitary
(or orthogonal) matrix U such that UTGkU is a diagonal matrix with diagonal elements λi, i ¼ 1, . . ., n,
where λi are the eigenvalues of Gk. Therefore, from (4.58), we can see that G bk is a diagonal matrix
with the elements λi þ εi, i ¼ 1, . . ., n. If εi, i ¼ 1, . . ., n, are selected as

0, λi > 0,
εi ¼
δ  λi , λi  0,

where δ is a positive constant, then Gbk will be positive definite. The matrix UTGkU can be computed
by solving the characteristic equation det(Gk  λIn) ¼ 0. This method involves a minimal disturbance
of Gk, and therefore the convergence properties of the Newton method are preserved. However, the
difficulty with this method is the determination of the n roots of the characteristic polynomial of Gk.

(d) The Matthews and Davies Method

This method simultaneously determines the modification of Gk and the Newton search direction dk.
As known, for the matrix Gk, a diagonal matrix D can be computed as D ¼ LGkLT,where
L ¼ Ln1⋯L2L1, is a unit lower triangular matrix and L1, L2, . . ., Ln1 are the elementary matrices
(see Appendix A). If Gk is positive definite, then D is positive definite and vice-versa. If D is not
b can be formed by replacing each zero or
positive definite, then a positive definite diagonal matrix D
bk can be formed
negative element in D by a positive element. In this way, a positive definite matrix G
as

b LT 1 :
bk ¼ L1 D
G

bk dk ¼ ∇f ðxk Þ, that

Therefore, the search direction can be computed as solution of the system G
is,

b LT 1 dk ¼ ∇f ðxk Þ:
L1 D ð4:59Þ

b LT 1 dk ¼ zk , then (4.59) can be expressed as L1zk ¼  ∇ f(xk). Therefore,

Denoting D

zk ¼ L∇f ðxk Þ ¼ Ln1 . . . L2 L1 ∇f ðxk Þ:

1
b zk :
With this, the search direction dk can be computed as dk ¼ LT D

(e) The Goldstein and Price Method

When ∇2f(xk) is not positive definite, then, instead of the direction dk from (4.55), the steepest
descent method is used. From the angle rule
π
θ μ
2
for a certain μ > 0 where θ is the angle between dk and gk, then the search direction in this method is
computed as
4.6 The Modified Newton Method 143

(
∇2 f ðxk Þ1 gk , cos θ  η,
dk ¼
gk , otherwise,

where η > 0 is a given constant. Obviously, the direction dk satisfies the condition cosθ  η, thus
obtaining the convergence of the method.

(f) The Gill and Murray Method

In this method, when ∇2f(xk) is not positive definite, then the Hessian matrix ∇2f(xk) is modified
as ∇2f(xk) þ νkI, where νk > 0 is selected in such a way that the matrix ∇2f(xk) þ νkI is positive definite
and well conditioned. At iteration k the following steps are performed: set Gk ¼ ∇2 f ðxk Þ þ νk I ,
where νk ¼ 0 if ∇2f(xk) is positive definite or νk > 0 otherwise; solve the modified Newton system
Gk dk ¼ gk subject to dk and update the variables as xkþ1 ¼ xk þ dk.
For the selection of the parameter νk, Gill and Murray (1974a) suggested the following procedure
based on the Cholesky factorization of Gk  ∇2f(xk) as Gk þ E ¼ LDLT, where E is a diagonal matrix
with nonnegative elements. If E ¼ 0, then consider νk ¼ 0. If E 6¼ 0, then, by using the Gerschgorin
circles, compute an upper bound a1 of νk as
 ( )
 X    

   
a1 ¼  min ðGk Þii  ðGk Þij     min λi :
1in j6¼i
 i

Observe that
a2 ¼ max feii g
i

is also an upper bound of νk, where eii is the i-th diagonal element of E. With these, an estimation of
νk is given by
νk ¼ min fa1 , a2 g,

which gives a positive definite matrix Gk .

In the following, let us present a Cholesky factorization which is numerically stable and can be
used in the Newton method as well as in the frame of some other optimization methods. The Cholesky
factorization Gk ¼ LDLT of the positive definite matrix Gk is expressed as

X
j1
d jj ¼ gjj  dpp l2jp , ð4:60Þ
p¼1

!
1 X
j1
lij ¼ gij  dpp ljp lip , i  j þ 1, ð4:61Þ
d jj p¼1

where gij represents the elements of Gk and djj are the diagonal elements of D. At this moment, for
defining a stable Cholesky factorization for the factors L and D, the following conditions are imposed:

(a) All the diagonal elements of D are positive

(b) The elements of the matriceal factors L and D are uniformly bounded, that is,
144 4 The Newton Method

dkk > δ > 0 for any k and jr ik j  β for i > k, ð4:62Þ

pffiffiffiffiffiffi
where r ik ¼ lik d kk , β is a given positive number, and δ is a positive number sufficiently small.
In the following, let us present the step j of this factorization. Suppose that the first j  1 columns
of the factors L and D have been computed, that is, for k ¼ 1, . . ., j  1, the elements dkk and lik, i ¼ 1,
. . ., n, have been determined so that they satisfy the requirements from (4.62). Now, compute
 
 X
j1 
 2
γ j ¼ ξj  d pp ljp ,
 p¼1


where ξj is gjj and set


d ¼ max γ j , δ :
pffiffiffi
In order to establish that d can be accepted as the j-th element of D, test if r ij ¼ lij d satisfies
pffiffiffiffiffi
(4.62). If it does, set d jj ¼ d and compute the j-th column of L where lij ¼ r ij = djj : Otherwise, if rij
does not satisfy (4.62), then set
 
 X
j1 
 2
d jj ¼ ξj  dpp ljp ,
 p¼1


where ξj ¼ gjj þ ejj, in which ejj is selected in such a way so that max|rij| ¼ β, with which, as above,
compute the j-th column of L.
At step n of this procedure, we obtain a Cholesky factorization of Gk as

Gk ¼ LDLT ¼ Gk þ E,

where E is a diagonal matrix with diagonal elements eii nonnegative. For a given matrix Gk, the
elements of the matrix E depend on β. Gill and Murray (1974a) proved that if n > 1, then
 2
ξ

kEðβÞk1  þ ðn  1Þβ þ 2 γ þ ðn  1Þβ2 þ δ,

β

where ξ and γ are the maximum of the absolute value of the nondiagonal elements of Gk and the
pffiffiffiffiffiffiffiffiffiffiffiffiffi
maximum of the diagonal elements of this matrix, respectively. Since for β2 ¼ ξ= n2  1 the above
bound is minimized, then we can consider
n pffiffiffiffiffiffiffiffiffiffiffiffiffi o
β2 ¼ max γ, ξ= n2  1, εM , ð4:63Þ

where εM represents the accuracy of the machine. Introducing εM in (4.63) is motivated by the case in
which kGkk is too small. With these developments, the following modified Cholesky factorization
algorithm may be presented, in which cip ¼ lipdpp, p ¼ 1, . . ., j; i ¼ j, . . ., n are the parameters stored in
Gk for reducing the memory requirements.
4.7 The Newton Method with Finite-Differences 145

Algorithm 4.4 Modified Cholesky factorization – Gill and Murray

1. Compute β as in (4.63). Set: j ¼ 1, cii ¼ gii, i ¼ 1, . . ., n. Consider a value for δ

2. Determine the smallest index q such that |cqq| ¼ maxj  i  n|cii|. Permute line q with line i and column
q with column i
3. Compute line i of L and determine the maximum module of lijdjj
Set ljp ¼ cjp/dpp, p ¼ 1, . . ., j  1
P
j1
Compute cij ¼ gij  ljp cip , i ¼ j þ 1, . . ., n
p¼1
Determine θj ¼ maxjþ1in |cij|, (if j ¼ n, then θj ¼ 0)
n   o
4. Compute the j-th diagonal element of D: djj ¼ max δ, cjj , θ2j =β2 and update the element ejj as ejj ¼
djj – cjj
5. Update cii ¼ cii  c2ij =djj , i ¼ j þ 1, . . ., n. Set j ¼ j þ 1 and continue with step 2 ♦

The modified Cholesky factorization is important in the unconstrained optimization, particularly

in the modified Newton method. Note that if Gk ¼ Gk þ Ek is positive definite and the condition
  
 1 
number of this matrix is uniformly bounded, that is, Gk G   κ, where κ  0, then we have
k

∇f ðxk ÞT sk 1
  k∇f ðxk Þk:
ks k k κ

With these, under the Wolfe inexact line-search, from Theorem 2.27 it follows that the sequence
{∇f(xk)} is convergent to zero. In other words, for the modified Newton method, the following result
holds.

Theorem 4.10 Let f : D ⊂ ℝn ! ℝ be a twice continuously differentiable function on the open set D
and the level set S ¼ {x : f(x)  f(x0)} a compact. If the sequence {xk} is generated by the modified
Newton method, then
lim ∇f ðxk Þ ¼ 0: ♦
k!1

We can see that in order to get a well-defined Newton method, there are a lot of procedures for
modifying the Hessian matrix of the minimizing function. We point out that there are no computa-
tional studies about the efficiency of all these modifications of the Newton method and about their
classification for seeing which one is the best.

4.7 The Newton Method with Finite-Differences

As we have already seen, at every iteration the Newton method requires the evaluation of the Hessian
matrix of the minimizing function f and the solving of the Newton system ∇2f(xk)dk ¼  ∇ f(xk) for
determining the Newton search direction dk. For the evaluation of the Hessian matrix ∇2f(xk) in the
current point xk, we have two possibilities. The first one consists of the analytical determination of the
elements of the Hessian. The second one is automatic differentiation. The automatic differentiation is
a powerful solution and is included in some professional systems for modeling and optimization like
GAMS (Brooke, Kendrick, Meeraus, Raman, & Rosenthal, 1998), (Brooke, Kendrick, Meeraus, &
146 4 The Newton Method

Raman, 2005), and AMPL (Fourer, Gay, & Kernighan, 2002). An alternative for these two
possibilities is to use the finite-difference approximation of the derivatives.
In this section, the Newton method with finite-differences is presented, both for solving nonlinear
algebraic systems of equations and for functions minimization. At the start, let us discuss the
approximation of derivatives by finite-differences. For function f : ℝ ! ℝ the estimation of f 0(x)
by finite differencing using only values of f(x) is discussed in Appendix A.
Let F : ℝn ! ℝm. Then the element (i, j) of the Jacobian matrix J(x) of function F(x) can be
approximated as

∂f i ðxÞ f i x þ hej  f i ðxÞ

ffi aij ¼ , ð4:64Þ
∂xj h

where fi(x) is the i-th component of F(x), ej is the j-th column of the unity matrix, and h is a small
perturbation of x.
If we denote with A.j as the j-th column of the matrix A which approximates the Jacobian, then

F x þ hej  FðxÞ
A:j ¼ : ð4:65Þ
h

Theorem 4.11 Let F : ℝn ! ℝm be continuously differentiable on the open and convex set D ⊂ ℝn
with J(x) Lipschitz continuous with the constant L for any x 2 D. Then, in the norm k.k which satisfies
kejk ¼ 1, for any j ¼ 1, . . ., n it follows that
 
  L
A:j  J ðxÞ:j   jhj, ð4:66Þ
2
and in the norm l1,
L
kA  J ðxÞk1  jhj: ð4:67Þ
2

Proof As we know (see Theorem 2.7, (2.18)), for any x, x þ d 2 D

L
kFðx þ dÞ  FðxÞ  J ðxÞdk  kd k2 :
2

Considering d ¼ hej, from the above inequality, it follows that


  
F x þ hej  FðxÞ  J ðxÞhej   L hej 2 ¼ L jhj2 :
2 2
m  
P
Dividing by h, we get (4.66). For proving (4.67) we remember that kAk1 ¼ max aij  , thus
1jn i¼1
immediately getting (4.67). ♦

An efficient calculation of the sparse Jacobian matrix for nonlinear systems of equations using
finite differences was presented by Andrei (1983).
Let us now consider the function f : ℝn ! ℝ. An approximation to the gradient ∇f(x) in x can be
obtained by the forward finite-difference formula
4.7 The Newton Method with Finite-Differences 147

∂f ðxÞ f ðx þ hei Þ  f ðxÞ

ffi : ð4:68Þ
∂xi h

Obviously, for the evaluation of the gradient, the function f has to be evaluated in n þ 1 points:
x and x þ hei, i ¼ 1, . . ., n. Therefore, from (2.18),
∂f ðxÞ f ðx þ hei Þ  f ðxÞ
¼ þ δh , ð4:69Þ
∂xi h

where
L
jδh j  h: ð4:70Þ
2
Hence, the error in approximation by the forward finite-difference is of order O(h).
A highly accurate approximation is obtained by using the central finite-difference
∂f ðxÞ f ðx þ hei Þ  f ðx  hei Þ
ffi : ð4:71Þ
∂xi 2h

The following theorems give bounds on the errors in the approximation of the gradient and of the
Hessian by central finite-difference.

Theorem 4.12 Let f : ℝn ! ℝ be a twice continuous differentiable function on the open and convex
set D ⊂ ℝn for which the Hessian ∇2f(x) is Lipschitz continuous with the constant L for any x 2 D.
Consider the norm k.k which satisfies keik ¼ 1 for all i ¼ 1, . . ., n. Suppose that for any i ¼ 1, . . ., n,
x þ hei, x  hei 2 D. Let a 2 ℝn be a vector with the components ai, i ¼ 1, . . ., n, defined as

f ðx þ hei Þ  f ðx  hei Þ
ai ¼ : ð4:72Þ
2h
Then
 
ai  j∇f ðxÞj   L h2 : ð4:73Þ
i 6
If the infinite norm l1 is used, then
L
ka  ∇f ðxÞk1  h2 : ð4:74Þ
6

Proof Consider Theorem 2.6 and define α and β as

1  
α ¼ f ðx þ hei Þ  f ðxÞ  h½∇f ðxÞi  h2 ∇2 f ðxÞ ii , ð4:75Þ
2
1  
β ¼ f ðx  hei Þ  f ðxÞ þ h½∇f ðxÞi  h2 ∇2 f ðxÞ ii : ð4:76Þ
2

Then, taking d ¼  hei in (2.16), we get

148 4 The Newton Method

L L
jαj  h3 , jβj  h3
6 6
From the triangle inequality,
L
j α  β j  h3 :
3
With this, from (4.75), (4.76), and (4.72), it follows that α  β ¼ 2h(ai  [∇f(x)]i), which
proves (4.73). Next, using the definition of l1 from (4.73), we get (4.74) at once, which proves the
theorem. ♦

Theorem 4.13 Let f : ℝn ! ℝ be a twice continuously differentiable function on the convex and open
set D ⊂ ℝn for which the Hesian ∇2f(x) is Lipschitz continuous with the constant L for any x 2 D.
Consider the norm k.k which satisfies keik ¼ 1 for all i ¼ 1, . . ., n. Suppose that x, x þ hei, x þ hej,
x þ hei þ hej 2 D for any i, j ¼ 1, . . ., n. Let A 2 Rnn be a matrix with the elements

f x þ hei þ hej  f ðx þ hei Þ  f x þ hej þ f ðxÞ

aij ¼ ð4:77Þ
h2
Then,
    5

aij  ∇2 f ðxÞ ij   Lh ð4:78Þ
3
If l1, l1, or the Frobenius norm are used, then
 
A  ∇2 f ðxÞ  5 Lhn: ð4:79Þ
3

Proof Like in the proof of the above theorem, define

T 1
T

α ¼ f x þ hei þ hej  f ðxÞ  hei þ hej ∇f ðxÞ  hei þ hej ∇2 f ðxÞ hei þ hej ,
2
1
β ¼ f ðx þ hei Þ  f ðxÞ  ðhei ÞT ∇f ðxÞ  ðhei ÞT ∇2 f ðxÞðhei Þ,
2


T 1
T

γ ¼ f x þ hej  f ðxÞ  hej ∇f ðxÞ  hej ∇2 f ðxÞ hej :

2

Then,
   
α  β  γ ¼ h2 aij  ∇2 f ðxÞ ij ð4:80Þ

On the other hand,

L 3 L L  3 5
jα  β  γ j  jαj þ jβj þ jγ j  hei þ hej  þ khei k3 þ hej   Lh3 :
6 6 6 3
This inequality together with (4.80) proves (4.78). The inequality (4.79) follows from (4.78) and
from the definition of the norms. ♦
4.7 The Newton Method with Finite-Differences 149

The Newton method with finite-difference for solving nonlinear algebraic systems of equations
F(x) ¼ 0, where F : ℝn ! ℝn is continuous differentiable, may be presented as follows.

Algorithm 4.5 Newton with finite-difference for F(x) ¼ 0

1. Initialization. Consider an initial point x0 2 ℝn as well as the convergence tolerance ε > 0 sufficiently
small for stopping the iterations. Set k ¼ 0
2. Compute F(xk) and for j ¼ 1, . . ., n, compute the j-th column
Fðxk þhk ej ÞFðxk Þ
ðAk Þ:j ¼ hk
of matrix Ak, where hk is a small perturbation of xk
3. Solve the linear algebraic system Akdk ¼  F(xk)
4. If kdkk2  ε, then stop; otherwise continue with step 5
5. Set xkþ1 ¼ xk þ dk, k ¼ k þ 1 and go to step 2 ♦

Applications Solved by the Newton Method with Finite-Difference

In the following, let us illustrate the running of Algorithm 4.5 for solving the applications S5, S6, S9,
and S14 from the SMUNO collection by means of the Newton method with finite-difference, where at
every iteration hk ¼ 106.

Application S5 (Stationary solution of a chemical reactor) Table 4.3 presents the initial point, the
functions values in the initial point F(x0), solution of this application obtained by the Newton method
with ε ¼ 106 in 5 iterations, and the functions values in x. On the other hand, Table 4.10 shows the
initial point, solution of this application given by the Newton method with finite-difference with
ε ¼ 106 in 5 iterations, and the functions values in these points.

Application S6 (Robot kinematics problem) Table 4.4 shows the initial point x0, the functions values
in the initial point F(x0), the solution x of this application given by the Newton method with ε ¼ 106
in 3 iterations, and the functions values in x. Table 4.11 shows the initial point x0, the solution x of
this application given by the Newton method with finite-difference with ε ¼ 106 in 4 iterations, and
the functions values in these points.

Table 4.10 Stationary solution of a chemical reactor. Initial point, solution, functions values in these points. Newton
with finite-difference method. 5 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 1.09 –35.164864 0.97424361 0.176600e– 07
2 1.05 –0.2403040 0.98282907 0.102566e– 09
3 3.05 3626.94637 0.05151277 –0.771388e– 04
4 0.99 –1779.9233 0.93567119 0.385517e– 04
5 6.05 –1814.7127 0.00009070 0.385692e– 04
6 1.09 – 7.13 0.06423810 0.949240e–14
150 4 The Newton Method

Table 4.11 Robot kinematics problem. Initial point, solution, functions values in these points. Newton with finite-
difference method. 4 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 0.164 0.0026823 0.16443166 0.0
2 – 0,98 – 0.0098180 – 0.98638847 0.1110223e–15
3 – 0,94 0.1050028 – 0.94706369 0.3469446e–17
4 – 0,32 0.0017588 – 0.32104573 0.0
5 – 0,99 – 0.0127040 – 0.99823316 – 0.4440892e–15
6 –0.056 – 0.014 0.05941842 – 0.4440892e–15
7 0.41 – 0.016764 0.41103315 0.499000e–09
8 – 0.91 – 0.0038 – 0.91162039 0

Table 4.12 Propan combustion in air – reduced variant. Initial point, solution, functions values in these points.
Newton with finite-difference method. 10 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 10 109.85000 0.0031141022 –0.1600708e–09
2 10 209.54267 34.597924281 –0.4749359e–09
3 0,05 – 0.3487418 0.0650417788 –0.2873626e–09
4 50,5 5098.5172 0.8593780968 0.15903885e–06
5 0,05 2659.2930 0.0369518591 0.79210239e–07

Application S9 (Propan combustion in air – reduced variant) Table 4.5 contains the initial point x0,
the functions values in the initial point F(x0), the solution x of this application given by the Newton
method with ε ¼ 106 in 9 iterations, and the functions values in x. Table 4.12 shows the initial point,
the solution x of this application obtained by the Newton method with finite-difference with ε ¼ 106
in 10 iterations, and the functions values in these points.

Application S14 (Circuit design) Table 4.6 contains the initial point x0, the functions values in the
initial point F(x0), the solution x of this application given by the Newton method with ε ¼ 106 in
4 iterations, and the functions values in x. Table 4.13 shows the initial point x0, the solution x of this
application given by the Newton method with finite-difference with ε ¼ 106 in 4 iterations, and the
functions values in these points.

Observe that the performances of the Newton method with finite-difference are similar to those of
the Newton method.

Theorem 4.14 Let F : ℝn ! ℝn be continuously differentiable on the convex and open set D ⊂ ℝn.
Suppose that there are x 2 ℝn and the constants r, β > 0 such that B(x, r) ⊂ D, F(x) ¼ 0, J(x)1
exists and kJ(x)1k  β, where J is Lipschitz continuous on B(x, r) ¼ {x 2 ℝn : kx  xk < r}
centered in x of radius r. Then, there are ε, h > 0 such that if x0 2 B(x, r) and {hk} is a sequence of
real numbers with 0 < |hk|  h, then the sequence {xk} generated by Algorithm 4.5 is well defined and
4.7 The Newton Method with Finite-Differences 151

Table 4.13 Circuit design. Initial point, solution, functions values in these points. Newton with finite-difference
method. 4 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 0.7 0.95525047 0.89999995 0.427648e –06
2 0.5 3.04155543 0.44998743 0.352005e–05
3 0.9 – 4.16621316 1.00000645 0.101280e–04
4 1.9 – 2.11846214 2.00006862 0.144598e–04
5 8.1 5.636125856 7.99997146 –0.1617286e–06
6 8.1 22.31397917 7.99969192 0.581500e –07
7 5.9 24.95943321 5.00003074 –0.1969207e–05
8 1 42.18784331 0.99998770 0.181683e–06
9 1.9 – 0.32000000 2.00005229 0.700407e–08

converges linearly to x. If lim hk ¼ 0, then the convergence is superlinear. If there is a constant c1
k!1
such that

jhk j  c1 kxk  x k, ð4:81Þ

or there is another constant c2 such that

jhk j  c2 kFðxk Þk, ð4:82Þ

then the convergence is quadratic.

Proof Select ε and h such that for xk 2 B(x, ε), the matrix Ak is nonsingular and |hk| < h. Consider
ε  r and
1
εþh : ð4:83Þ
2βL

By induction we prove that

1
kxkþ1  x k  kxk  x k, ð4:84Þ
2
that is,
xkþ1 2 Bðx , εÞ:

For k ¼ 0, let us prove that A0 is nonsingular. From Theorem 4.11 it follows that
Lh
kAð x Þ  J ð x Þ k  :
2
Therefore,
  1    

J ðx Þ ðA0  J ðx ÞÞ  J ðx Þ1  A  J ðx0 Þ þ ðJ ðx0 Þ  J ðx ÞÞ
  0
Lh 1
β þ Lε  :
2 2
152 4 The Newton Method

Now, from the von Neumann theorem (see Appendix A), it follows that A0 is nonsingular and
 1 
A   2β: ð4:85Þ
0

Therefore, x1 is well defined and

x1  x ¼ x0  A1
0 Fð x 0 Þ  x

1
¼ A0 f½Fðx Þ  Fðx0 Þ  J ðx0 Þðx  x0 Þ þ½ðJ ðx0 Þ  A0 Þðx  x0 Þg:


By using Theorem 2.7 (see (2.18)), from (4.85) and (4.83), we get
 
kx1  x k  A1 fkFðx Þ  Fðx0 Þ  J ðx0 Þðx  x0 Þk þ kJ ðx0 Þ  A0 kkx  x0 kg
n
0 o
L  L
 2β kx  x0 k2 þ hkx0  x k ð4:86Þ
2 2
1
 βLðε þ hÞkx  x0 k  kx0  x k,

2
that is, (4.84) is true for k ¼ 0. Now, supposing that this result is true for k ¼ j and by using the same
technique, we can prove that the result is true for k ¼ j þ 1. Therefore, (4.83), (4.84) are true, thus
proving the linear convergence of the sequence {xk}.
For showing the superlinear and the quadratic convergence, a finer and more improved bound for
kA0  J(x0)k should be used. When limk!1hk ¼ 0, then the second term in the bracket of (4.86) tends
to zero, thus proving that

kxkþ1  x k
! 0, when k ! 1,
kx k  x  k

that is, the superlinear convergence of {xk}.

Similarly, when (4.81) is satisfied, then from (4.78) we get the quadratic convergence of {xk}
to x. ♦

For the optimization problem minn f ðxÞ, when the gradient ∇f(x) is available, as in the case of the
x2ℝ
vast majority of applications, then a finite-difference approximation of the Hessian ∇2f(x) can be
obtained by using either the forward finite-difference formula or the central finite-difference formula.
At iteration k, the Newton method with central-difference for minimizing function f is

∇f xk þ hj ej  ∇f ðxk Þ
ðAÞ:j ¼ , j ¼ 1, . . . , n,
hj

A þ AT
Ak ¼ ,
2
xkþ1 ¼ xk  A1
k ∇f ðxk Þ,

where
pffiffiffi   

hj ¼ η max xj , xej sign xj ;

xej is an estimation of xj and η is a sufficiently small number, close to the ε machine.

4.7 The Newton Method with Finite-Differences 153

If the hypotheses of Theorem 4.14 are satisfied and if hj is of order

hj ¼ Oðkxk  x kÞ,

then the Newton method with the finite-difference xkþ1 ¼ xk  A1 k ∇f ðxk Þ has a quadratic rate of
convergence. An appropriate choice of the difference parameter hj can be difficult. The rounding
errors overwhelm the calculation if hj is too small, while the truncation errors dominate if hj is too
large. The Newton codes rely on forward differences, since they often yield sufficient accuracy for
reasonable values of hj. The central differences are more accurate, but they require the work twice (2n
gradient evaluations against n evaluations).
Many algorithms for optimization use the product of the Hessian ∇2f(xk) and a given vector dk. In
this case, an approximation of the product ∇2f(xk)dk is given by
∇f ðxk þ hd k Þ  ∇f ðxk Þ
∇2 f ðxk Þdk ffi ,
h
where
pffiffiffiffiffi
2 εm ð1 þ kxkþ1 kÞ
h¼ ,
kd k k

for which the error is of order O(h). Observe that for obtaining this approximation, only a single
evaluation of the gradient in point xk þ hdk is necessary.
If the gradient is not available, then an approximation of the Hessian can be computed by using the
values of the minimizing function. From (4.71) we get the following approximation of the Hessian:
  
 

f xk þ hi ei þ hj ej  f ðxk þ hi ei Þ  f xk þ hj ej  f ðxk Þ
b
Ak ¼ ,
ij hi hj

where
p ffiffiffi   

hj ¼ 3
η max xj , xej sign xj ,

or
p ffiffiffi
hj ¼ eεxj ,
3

where eε is the accuracy of the machine. In this case, the forward or the central finite-difference
formula give the following approximations to the gradient:

f x k þ hj e j  f ð x k Þ
ðgbk Þj ¼ , j ¼ 1, . . . , n,
hj

and

f xk þ hj ej  f xk  hj ej
ðgbk Þj ¼ , j ¼ 1, . . . , n,
2hj
 
respectively. The errors of these approximations are of order O(hj) and O h2j , respectively. In this
case, the Newton method with finite-difference is expressed as
1
xkþ1 ¼ xk  Abk gbk ,
154 4 The Newton Method

where Abk and gbk are approximations by the finite-difference of the Hessian ∇2f(xk) and of the gradient
∇f(xk), respectively. In the standard hypothesis given by Theorem 4.14, we can prove that
 1    
  L  
kxkþ1  x k  Abk  kxk  x k2 þ Abk  ∇2 f ðxk Þ kxk  x k þ kgbk  ∇f ðxk Þk : ð4:87Þ
2
Observe that the difference between (4.87) and (4.86) is the additional term kgbk  ∇f ðxk Þk. To get
 
the quadratic convergence, we must obviously have kgbk  ∇f ðxk Þk ¼ O kxk  x k2 , thus involv-
ing the selection hj ¼ O(kxk  xk2). Therefore, with this selection of hj the convergence of the
Newton method with central-difference is quadratic.
It is worth mentioning that although the error of the Newton method with central-difference is of
 
order O h2j , which is smaller than the error corresponding to the forward-difference O(hj), the cost
of the central-difference scheme is twice greater than the forward-difference. This is the reason why
the central-difference scheme is very rarely used in practice, only for problems for which the
evaluation of the minimizing function is relatively simple or for problems requiring solutions of
great accuracy. Stewart (1967) gave some practical finite-difference schemes based on commuting
from forward-difference to central-difference schemes.

4.8 Errors in Functions, Gradients, and Hessians

The unconstrained optimization problem becomes more difficult to solve in the presence of errors in
the evaluations of the function and its gradient. Suppose that we can compute only f approximately.
If we compute fb ¼ f þ εf rather than f, then a forward-difference gradient with the difference
increment h

fbðx þ hÞ  fbðxÞ
Dh f ðxÞ ¼
h
 
pffiffiffiffi
differs from f 0 by O(h þ εf/h). This difference is minimized if h ¼ O εf : Therefore, the error in
 
pffiffiffiffi
the gradient is εg ¼ O εf : If a forward-difference Hessian is computed by using Dh as an
approximation to the gradient, then the error in the Hessian will be
pffiffiffiffiffi
 
1=4
Δ ¼ O εg ¼ O εf ð4:88Þ

and the accuracy in ∇2f will be much smaller than that of a Jacobian in the nonlinear system of
equations.
If εf is significantly larger than the machine roundoff, then (4.88) indicates that using the numerical
Hessians based on a second numerical differentiation of the minimizing function will not be very
accurate. Even in the best possible cases, where εf is the same size as the machine roundoff, the finite-
difference Hessians will not be very accurate and will be quite expensive to compute if the Hessian is
dense. As known, if the machine roundoff is 1016 for most computers, then (4.88) indicates that the
forward difference Hessian will be accurate to roughly four decimal digits.
Better results can be obtained with central-differences, but at a cost of twice the number of
function evaluations. A central-difference approximation to ∇f is
4.8 Errors in Functions, Gradients, and Hessians 155

fbðx þ hÞ  fbðx  hÞ
Dh f ðxÞ ¼
2h
 
1=3
and the error is O(h2 þ εf/h), which is minimized if h ¼ O εf leading to an error in the gradient of
 
2=3
εg ¼ O εf : Therefore, a central-difference Hessian will have an error of

  
2=3 4=9
Δ¼O εg ¼ O εf ,

which is substantially better.

Therefore, accurate gradients are much more important than the accurate Hessian and one option
is to compute the gradients with central differences and the Hessians with forward differences. In this
 
2=3
case, the central-difference gradient error is O εf , and therefore the forward-difference Hessian
qffiffiffiffiffiffiffiffi  
1=2 1=3
error will be Δ ¼ O εg ¼ O εf :
However, in many optimization problems and applications, the accurate gradients are available. In
these cases, the numerical differentiation by forward-difference for computing the Hessians is, like
the numerical computation of the Jacobians for solving nonlinear algebraic systems, a reasonable idea
for many problems and the less expensive forward-differences work well in practice (Kelley, 1999).
The local convergence and the evolution of the error for the Newton method are proved in the
following theorem.

Theorem 4.15 Suppose that f is twice continuously differentiable, ∇f(x) ¼ 0, ∇2f(x) is positive
definite and k∇2f(x)  ∇2f( y)k  γkx  yk. Then there are K > 0, δ > 0 such that if xk 2 B(δ), then the
Newton iterate from xk given by xkþ1 ¼ xk  (∇2f(xk))1 ∇ f(xk) satisfies

kekþ1 k  K kek k2 ,

where ek ¼ xk  x and B(δ) is the ball of radius δ around x.

Proof Let δ be small enough so that the conclusions of Theorem A2.5 hold. By Theorem 2.1 it
follows that

ekþ1 ¼ ek  ∇2 f ðxk Þ1 ∇f ðxk Þ

ð1
1 2

¼ ∇ f ðxk Þ
2
∇ f ðxk Þ  ∇2 f ðx þ tek Þ ek dt:
0

By Theorem A2.5 and the Lipschitz continuity of ∇2f(x), it follows that

  
kekþ1 k  2∇2 f ðx Þ1  γ kek k2 =2,

which completes the proof with K ¼ γk∇2f(x)1k. ♦

If Δ(xk) and εg(xk) are the errors in the Hessian and in the gradient, respectively, then the following
estimation of the error ek at the iteration k in the perturbed Newton method can be obtained (Kelley,
1999).
156 4 The Newton Method

Theorem 4.16 Suppose that f is twice continuously differentiable, ∇f(x) ¼ 0, ∇2f(x) is positive
definite and k∇2f(x)  ∇2f( y)k  γkx  yk. Then there are K > 0, δ > 0 and δ1 > 0 such that if xk 2 B(δ)
and kΔ(xk)k < δ1, then

1

xkþ1 ¼ xk  ∇2 f ðxk Þ þ Δðxk Þ ∇f ðxk Þ þ εg ðxk Þ

is well defined, that is, ∇2f(xk) þ Δ(xk) is nonsingular and satisfies


1  
1 
 2     
kekþ1 k  K kek k2 þ 16 ∇2 f ðx Þ  ∇2 f ðx ÞkΔðxk Þkkek k þ 4 ∇2 f ðx Þ εg ðxk Þ,
ð4:89Þ

where ek ¼ xk  x and B(δ) is the ball of radius δ around x.

Proof Let δ be small enough such that the conclusions of Theorem A2.5 hold. Define

1
xNkþ1 ¼ xk  ∇2 f ðxk Þ ∇f ðxk Þ: Observe that

1 2
1 
1
xkþ1 ¼ xNkþ1 þ ∇2 f ðxk Þ  ∇ f ðxk Þ þ Δðxk Þ ∇f ðxk Þ  ∇2 f ðxk Þ þ Δðxk Þ εg ðxk Þ:

Theorem A2.5 implies that


1 

1 2  
kekþ1 k  K kek k2 þ 2 ∇2 f ðxk Þ  ∇ f ðxk Þ þ Δðxk Þ ∇2 f ðx Þkek k

1 
  
þ  ∇2 f ðxk Þ þ Δðxk Þ εg ðxk Þ: ð4:90Þ

If

1 
1  1
kΔðxk Þk   ∇2 f ðx Þ  ,
4
then from Theorem A2.5 we get

1 
1  1
kΔðxk Þk   ∇2 f ðxk Þ  :
2
The Banach lemma states that ∇2f(xk) þ Δ(xk) is nonsingular and

1  
1  
1 
 2     
 ∇ f ðxk Þ þ Δðxk Þ   2 ∇2 f ðxk Þ   4 ∇2 f ðx Þ :

Hence,

1 2
1  
1 
 2   2
 ∇ f ðx k Þ  ∇ f ðxk Þ þ Δðxk Þ   8 ∇2 f ðx Þ  kΔðxk Þk:

Therefore, using these estimates, from (4.90) the estimate of the error (4.89) is obtained, that is, the
conclusion of the theorem. ♦

The interpretation of (4.89) is as follows. Firstly, we can see that kekþ1k is proportional to kekk2
and the factor of proportionality is independent of kΔ(xk)k and εg(xk). Observe that in the norm of the
error ekþ1, given by (4.89) the inaccuracy evaluation of the Hessian given by kΔ(xk)k is multiplied by
4.9 Negative Curvature Direction Methods 157

the norm of the previous error. On the other hand, the inaccuracy evaluation of the gradient given by
εg(xk) is not multiplied by the previous error and has a direct influence on kekþ1k. In other words, in
the norm of the error the inaccuracy in the Hessian has smaller influence than the inaccuracy of the
gradient. Therefore, in this context, from (4.89), the following remarks may be emphasized:

• If both kΔ(xk)k and εg(xk) are zero, then we get the quadratic convergence of the Newton method.
• If εg(xk) 6¼ 0 and kεg(xk)k is not convergent to zero, then there is no guarantee that the error for the
Newton method will converge to zero.
• If kΔ(xk)k 6¼ 0, then the convergence of the Newton method is slowed down from quadratic to
linear or to superlinear if kΔ(xk)k ! 0.

Therefore, we can see that the inaccuracy evaluation of the Hessian of the minimizing function is
not so important. The accuracy of the evaluation of the gradient is more important. This is the main
motivation why the diagonal quasi-Newton updating method may be used for unconstrained optimi-
zation (Andrei, 2019a, 2019b, 2020c, 2020d). (See Chap. 6, Sect. 6.3.)

4.9 Negative Curvature Direction Methods

As we know, the Newton method is quadratic convergent when the current point is in a neighborhood
of the minimum point, and the Hessian matrix in the current point is positive definite. If the Hessian in
the current point is an indefinite matrix and the current point is near enough to a saddle point, then the
Newton method is not defined and has to be modified. If ∇f(xk) ¼ 0, then there is no direction of
descent, and if ∇2f(xk)is indefinite, then xk is a saddle point. A vector d is termed a direction of
negative curvature if dT∇2f(xk)d < 0.
In this section, the modified Newton method is based on the concept of negative curvature. At the
beginning the stable Newton method of Gill and Murray (1972) is described, and after that two
methods of negative curvature developed by Fiacco and McCormick (1968) and Fletcher and
Freeman (1977), respectively, are presented. A definition is now given to clarify the concepts (Sun,
& Yuan, 2006).

Definition 4.1 Let f : ℝn ! ℝ be a twice continuously differentiable function on the open set
D ⊂ ℝ n.

(i) If ∇2f(x) has at least a negative eigenvalue, then we say that x is an indefinite point.
(ii) If x is an indefinite point and d satisfies the condition dT∇2f(x)d < 0, then we say that d is a
negative curvature direction of f(x) in x.
(iii) If sT ∇ f(x)  0, dT ∇ f(x)  0, dT∇2f(x)d < 0, then we say that the pair (s, d ) is a descent pair in
the indefinite point x.
If x is not an indefinite point and satisfies sT ∇ f(x) < 0, dT ∇ f(x)  0, and dT∇2f(x)d ¼ 0, then we
say that the pair (s, d ) is a descent pair in x. ♦
An example of descent pair is as follows:
s ¼ ∇f ðxÞ,
(
0, ∇2 f ðxÞ  0,
d¼
signðu ∇f ðxÞÞu,
T
otherwise,

where u is the unitary eigenvector corresponding to a negative eigenvalue of ∇2f(x).

158 4 The Newton Method

From the above definition we can see that in a stationary point, a negative curvature direction has
to be a descent direction. In an arbitrary general point, if the negative curvature direction satisfies
dT ∇ f(x) ¼ 0, then both d and d are descent directions. On the other hand, if dT ∇ f(x)  0, then d is a
descent direction. If dT ∇ f(x)  0, then d is a descent direction.

The Stable Newton Negative Curvature Method of Gill and Murray

If the Hessian Gk is not defined, then the modified Cholesky factorization is used in order to force Gk
to be positive definite. If xk tends to a stationary point, then a negative curvature direction is used in
order to reduce the values of the minimizing function.

Now, consider the modified Cholesky factorization

Gk ¼ Gk þ Ek ¼ Lk Dk LTk , ð4:91Þ

where

Dk ¼ diagðd11 , ⋯, d nn Þ and Ek ¼ diagðe11 , ⋯, enn Þ: ð4:92Þ

With these, if kgkk  ε and ∇2f(xk) is not positive semidefinite, then the following algorithm of
negative curvature direction is used.

Algorithm 4.6 Negative curvature direction algorithm

1. Compute φj ¼ djj  ejj, j ¼ 1, . . ., n

2. Determine an index t such that φt ¼ min {φj : j ¼ 1, . . ., n}
3. If φt  0, stop; otherwise, solve the linear system
LTk dk ¼ et (4.93)
subject to dk, where et is the t -th column of the unity matrix ♦

Theorem 4.17 Let Gk be the Hessian of function f in point xk and

Gk ¼ Gk þ Ek ¼ Lk Dk LTk ,

the modified Cholesky factorization. If the direction dk is generated by Algorithm 4.6, then dk is a
negative curvature direction in xk and at least one of the directions dk and dk is a descent direction
in xk.

Proof Since Lk is a unitary inferior triangular matrix, then the solution of the system (4.93) has the
following form

dk ¼ ½ρ1 , ⋯, ρt1 , 1, 0, ⋯, 0T :

Then,

d Tk Gk d k ¼ dTk Gk dk  d Tk Ek dk ¼ dTk Lk!

Dk LTk dk  dTk Ek dk
P 2
t1 P 2
t1
¼ eTt Dk et  ρp epp þ ett ¼ d tt  ett  ρp epp
p¼1 p¼1
tP
1
¼ φt  ρ2p epp :
p¼1
4.9 Negative Curvature Direction Methods 159

From the modified Cholesky factorization, we get

X
j1
ejj ¼ gjj  gjj ¼ d jj þ l2jp dp  gjj ¼ djj  cjj  0,
p¼1

that is,

X
t1
ρ2p epp  0:
p¼1

Since φt < 0, it follows that dTk Gk dk < 0, that is, dk is a negative curvature direction and dk is
also a negative curvature direction. If gTk d k  0, then dk is a descent direction; otherwise dk is a
descent direction. ♦

Algorithm 4.7 Stable Newton algorithm – Gill and Murray

1. Consider an initial point x0 as well as the convergence tolerance ε > 0 sufficiently small. Set k ¼ 1
2. Compute gkand Gk
3. Compute the modified Cholesky factorization by using Algorithm 4.4 Gk þ Ek ¼ Lk Dk LTk
4. If kgkk > ε, then solve the system Lk Dk LTk dk ¼ gk subject to dk and continue with step 6; otherwise, go
to step 5
5. Perform the steps of Algorithm 4.6. If the direction dk may not be generated, that is, φt  0, then stop;
otherwise, determine the direction dk and set (
dk , gTk dk > 0,
dk ¼
dk , gTk dk  0
6. Compute the stepsize αk and set xkþ1 ¼ xk þ αkdk
7. If f(xkþ1)  f(xk), stop; otherwise, set k ¼ k þ 1 and go to step 2 ♦

The following theorem proves the convergence of the algorithm (Gill, & Murray, 1972).

Theorem 4.18 Let f : ℝn ! ℝ be a twice continuously differentiable function on the open set
D ⊂ ℝn. Suppose there is x 2 D such that the level set SðxÞ ¼ fx : f ðxÞ  f ðxÞg is convex, bounded,
and closed. Suppose that x0 2 SðxÞ: Then the sequence {xk} generated by Algorithm 4.7 in which
ε ¼ 0 satisfies:

(i) If {xk} is a finite sequence, then the final point of this sequence is a stationary point of function
f(x).
(ii) If {xk} is an infinite sequence, then this sequence has accumulation points and all of them are
stationary points of function f(x). ♦

The Negative Curvature Method of Fiacco and McCormick

Fiacco and McCormick (1968) presented a modification of the Newton method for the case in which
the Hessian Gk has negative eigenvalues. Their idea was to move along the negative curvature
direction. Indeed, if

d Tk gk  0 and dTk Gk dk < 0, ð4:94Þ

then
160 4 The Newton Method

1
f ðxk þ dk Þ ffi f ðxk Þ þ d Tk gk þ dTk Gk dk
2
will be reduced. Since Gk is indefinite, the Fiacco-McCormick method uses the factorization

Gk ¼ LDLT , ð4:95Þ

where L is an inferior, unitary triangular matrix and D is a diagonal matrix. If Gk is positive definite,
then the direction dk generated by the factorization (4.95) is descent. However, if there is a diagonal
element dii negative, then the following system is solved:

LT t ¼ a,

where the component ai of the vector a is defined as


1, dii  0,
ai ¼
0, d ii > 0:

Obviously, the search direction

(
t, gTk t  0,
dk ¼
t, gTk t > 0

is a negative curvature direction which satisfies (4.94).

The criticism of this very simple method lies in its numerical instability. The factorization (4.95)
can amplify the errors or it may not exist. This is the reason why Fletcher and Freeman (1977)
introduced a stable symmetric factorization, which we are going to present.

The Negative Curvature Method of Fletcher and Freeman

Fletcher and Freeman used a symmetric, stable factorization due to Bunch and Parlett (1971). Indeed,
for any symmetric matrix Gk, there is a permutation matrix P such that

PT Gk P ¼ LDLT , ð4:96Þ

where L is an inferior, unitary triangular matrix and D is a block-diagonal matrix with 1  1or 2  2-
dimensional blocks. The purpose of introducing the permutation matrix P is to conserve the
symmetry and to maintain the numerical stability of the factorization. It is worth mentioning that
the factorization (4.96) always exists and is numerically stable. This is the difference between this
method and the method of Fiacco and McCormick, which uses the numerically unstable factorization
(4.95).

In the case in which the pivot is an element 1  1-dimensional, then the matrix A, n  n
-dimensional can be expressed as
 
ð0Þ a11 aT21
A¼A ¼ ,
a21 A22

where a21 is a vector (n  1) -dimensional and A22 is a (n  1)  (n  1) matrix. Now, by pivoting

for eliminating the first row and the first column, the following matrix A(1) is obtained:
4.9 Negative Curvature Direction Methods 161

" #
ð1Þ ð0Þ 0 0T
A ¼A  d11 l1 lT1 ¼ , ð4:97Þ
0 A22  a21 aT21 =d 11

where
   
1 a11 1
d11 ¼ a11 , l1 ¼ ¼ :
d11 a21 a21 =d11

In the case in which the pivot is a 2  2 -dimensional matrix, then

" #
T
A A
Að0Þ ¼
11 21
,
A21 A22

where this time A11 is a 2  2 dimensional matrix, A21 is (n  2)  2 -dimensional, and A22 is a
(n  2)  (n  2) sub-matrix. By pivoting, the following A(2) matrix is obtained:
 
I  
Að2Þ ¼ Að0Þ  L1 D1 LT1 ¼ Að0Þ  D1 I LT21
  L21 ð4:98Þ
0 0
¼ T ,
0 A22  A21 D1 1 A21

where
     
A11 1 I I
D1 ¼ A11 , L1 ¼ D ¼ ≜ :
A21 1 A21 A1
11 L21

At the next step, the same procedure is performed on the (n  1)  (n  1) -dimensional

sub-matrix A22  a21 aT21 =d 11 from (4.97) or on the (n  2)  (n  2) -dimensional sub-matrix
A22  A21 D1 T
1 A21 from (4.98). Finally, the factorization (4.96) is obtained.
The problem we face now is that of selecting the pivots, which could be a scalar or a 2  2
-dimensional submatrix. For this, Bunch and Parlett suggested the following procedure. At first
determine the maximum diagonal element in the absolute value ξd and the maximum non-diagonal
element in the absolute value ξn, respectively. If the ratio ξd/ξn is acceptable, then select the maximum
diagonal element in the absolute value ξd as pivot and perform a symmetric permutation of the rows
and columns such that this element is placed on the position of a11. Otherwise, select the maximum
non-diagonal element in the absolute value ξn, let us say the element aij, and select as pivot the
submatrix
 
aii aij
,
aji ajj

followed by a permutation of the rows and columns in such a way that A11 is exactly this sub-matrix.
This factorization generates the matriceal factors L and D as an inferior, unitary triangular matrix
and a block-diagonal matrix, respectively. For example, for the case in which there are 2 blocks 1  1
and 2  2, then the structure of these matriceal factors is
162 4 The Newton Method

2 3 2 3
1
6 7 6   7
6 1 7 6 7
6 7 6 7
6 0 1 7 6   7
L¼6
6  
7, D ¼ 6
7 6  
7
7
6 1 7 6 7
6 7 6 7
4  
0 1 5 4   5
    
1

A similar, more economical factorization was given by Bunch and Kaufman (1977).
Therefore, let the following factorization of the symmetric indefinite matrix Gk be as Gk ¼ LDLT.
With this, solve the triangular system

LT t ¼ a, ð4:99Þ

where, in the case of a 1  1 dimensional pivot, the components of a are


1, dii  0,
ai ¼
0, dii > 0;

and in the case of a 2  2 dimensional pivot, ½ ai aiþ1 T is the unitary eigenvector corresponding to
the negative eigenvalue of the sub-matrix
 
dii di,iþ1
:
diþ1,i diþ1,iþ1

With the solution of the system (4.99), consider

(
t, gTk t  0,
dk ¼ ð4:100Þ
t, gTk t > 0,

which is a negative curvature direction satisfying (4.94).

Observe that
X
d Tk Gk dk ¼ d Tk LDLT dk ¼ aT Da ¼ λi ,
i:λi <0

and

gTk dk  0:

If D has negative eigenvalue, then the direction dk is computed as

þ
e L1 gk ,
dk ¼ LT D ð4:101Þ

e is the positive part of D, that is,

where D

d ii , dii > 0,
ei ¼
D
0, otherwise,
þ
e is the generalized inverse of D:
and D e
4.10 The Composite Newton Method 163

If D has at least a zero eigenvalue, then the direction dk is computed such that

Gk dk ¼ LDLT dk ¼ 0 and gTk dk < 0: ð4:102Þ

If all the eigenvalues of D are positive, then all the blocks are 1  1-dimensional elements and in
this case the generated direction is exactly the usual Newton direction

dk ¼ LT D1 L1 gk :

Note that the negative curvature descent direction (4.100) is limited to a certain subspace. On the
other hand, the direction given by (4.101) is exactly the Newton direction limited to the subspace of
the positive curvature directions.
Even if the negative curvature directions have some attractive properties, they are not efficient,
particularly in situations when they are used at successive iterations. Although no solid theoretical
justification is known for an alternate usage of the directions given by (4.100) or by (4.101),
respectively, in practice this alternation gives better results. In the same experimental manner, if
D has eigenvalues zero, then the results obtained are better if the direction is alternately computed by
(4.101) or by (4.102), respectively.
What is important to notice is the Bunch-Parlett factorization of the symmetric indefinite matrices.
This factorization is not only found in the context of the Newton method, but in many interior point
methods from linear programming (Andrei, 2011d) or in the constrained optimization. Even if these
modifications of the Newton method have theoretical justification, they are not used in solving real
application. Some other unconstrained optimization methods are more efficient and more robust
(conjugate gradient, quasi-Newton, truncated Newton, limited memory BFGS, etc.).

4.10 The Composite Newton Method

Consider the nonlinear algebraic system F(x) ¼ 0, where F : ℝn ! ℝn is continuous differentiable.

Then the Newton method with line-search for solving this system is defined as
J ðxk Þdk ¼ Fðxk Þ, ð4:103Þ

xkþ1 ¼ xk þ αk dk , k ¼ 0, 1, . . . ,

where αk is the stepsize. Choosing αk less than one is important for the global convergence. As we
know, under standard assumptions the Newton method is quadratic convergent. Since the dominant
work of this method is the factorization of the n  n -dimensional Jacobian matrix, then without
taking into account the work needed to evaluate the function F and its Jacobian J, the algebra required
per iteration is O(n3). If n is large, the factorization of the Jacobian matrix can be a very serious
provocation.
In order to reduce the computational effort at each iteration, the simplified Newton method can be
defined as
J ðx0 Þdk ¼ Fðxk Þ, ð4:104Þ

xkþ1 ¼ xk þ αk dk , k ¼ 0, 1, . . .

The simplified Newton method requires the factorization of J(x0) and its usage along all the
iterations. In this case, the computational effort is of order O(n2). Obviously, the convergence is only
linear, thus forcing the method to perform a larger number of iterations.
164 4 The Newton Method

Between these two possibilities given by the Newton method (4.103) and the simplified Newton
method (4.104), we can imagine another method, in which a number of m simplified Newton
iterations is performed between any Newton iterations. In other words, for these m iterations, the
same factorization of the Jacobian is used. This is known as the composite Newton method of order m,
defined as

Algorithm 4.8 Composite Newton of order m

1. Solve the systems

J(xk)d1 ¼  F(xk),
J(xk)d2 ¼  F(xk þ d1),
⋮
J(xk)dm ¼  F(xk þ d1 þ ⋯ þ dm  1)
2. Compute
xkþ1 ¼ xk þ αk(d1 þ d2 þ ⋯ þ dm) ♦

We can see at once that exactly m linear algebraic systems with the same matrix J(xk) have to be
solved at the k-th iteration. The average computational effort on iteration is of order O((n3 þ mn2)/m),
which for m large is of order O(n2). Therefore, for m large, the behavior of the composite Newton
method of order m is like the simplified Newton method. Consequently, advanced implementations of
the composite Newton method of order m not only modify the parameter m at every Newton iteration,
but would also keep m relatively as small as possible. Under the standard Newton method
assumptions, the composite Newton method of order m has a convergence rate of m þ 2. A proof
can be found in Ortega and Rheinboldt (2000).
A composite Newton method of order 1 is of particular interest, where one Newton iteration is
composed with one simplified Newton iteration (Traub, 1964). This is defined as

Algorithm 4.9 Composite Newton of order 1

1. Solve the system

J(xk)dN ¼  F(xk),
subject to the Newton direction dN
2. Solve the system
J(xk)dS ¼  F(xk þ dN),
subject to the simplified Newton direction dS
3. Compute
xkþ1 ¼ xk þ αk(dN þ dS) ♦

The following theorem shows the cubic convergence of the composite Newton method of order
1 (Ortega, & Rheinboldt, 2000).

Theorem 4.19 Let F : ℝn ! ℝn be a continuously differentiable function on B(x, δ) ⊂ ℝn,

for which the Jacobian is Lipschitz continuous with the constant L, that is, for any x 2 B(x, δ),
kJ(x)  J(x)k  Lkx  xk. Suppose that J(x) is nonsingular. Then the composite Newton method of
order 1 is cubic convergent.

Proof From the Theorem 4.1 observe that the function N(x) ¼ x  J(x)1F(x), which defines the
Newton method, is well defined and for any x 2 B(x, δ1) ⊂ B(x, δ) satisfies
4.10 The Composite Newton Method 165

kN ðxÞ  x k  ηkx  x k2 , ð4:105Þ

where δ1 < δ and η is a positive constant. Therefore, the function

GðxÞ ¼ N ðxÞ  J ðxÞ1 FðN ðxÞÞ ð4:106Þ

is also well defined on B(x, δ2), where δ2  (δ1/η)1/2. Hence, if

 
J ðxÞ1   β ð4:107Þ

for all x 2 B(x, δ2), then by using the above relations (4.105), (4.106), and (4.107), it follows that
 
kGðxÞ  x k ¼ N ðxÞ  J ðxÞ1 FðN ðxÞÞ  x 
 
¼ J ðxÞ1 ½J ðxÞN ðxÞ  FðN ðxÞÞ  J ðxÞx 
 
 J ðxÞ1  kJ ðxÞN ðxÞ  FðN ðxÞÞ  J ðxÞx k
 βkJ ðxÞN ðxÞ  FðN ðxÞÞ  J ðxÞx þ Fðx Þ
þJ ðx ÞN ðxÞ  J ðx Þx  J ðx ÞN ðxÞ þ J ðx Þx k
 βkFðN ðxÞÞ  Fðx Þ  J ðx ÞðN ðxÞ  x Þk
þβkJ ðx Þ  J ðxÞk kN ðxÞ  x k
1
 βLkN ðxÞ  x k2 þ βLkx  x k kN ðxÞ  x k
2
1
 βLη2 kx  x k4 þ βLηkx  x k3
2
h i
1
¼ βLη ηδ2 þ 1 kx  x k3
2
for all x 2 B(x, δ2), thus proving the cubic convergence of the composite Newton method of
order 1. ♦

In current practice it is credited that the composite Newton method is efficient when n is large and
the function F is easy to evaluate. This is the case of the interior point methods for solving constrained
optimization problems (including the linear programming) in which the advanced implementations
use the composite Newton method (Mehrotra, 1989), (Zhang, Tapia, & Dennis, 1992), (Andrei,
1999a, 1999b).
In the following, let us present the performances of the composite Newton method of order 1 for
solving some applications from the SMUNO collection (Stationary solution of a chemical reactor,
Robot kinematics problem, Propan combustion in air – reduced variant and Circuit design). These
applications were solved by the Newton method (see Tables 4.3, 4.4, 4.5, and 4.6)

Applications Solved by the Composite Newton Method of Order 1

We now describe the running and the performances of the composite Newton method of order 1 for
solving some applications from the SMUNO collection (see Appendix B). All these applications
considered in this numerical study are expressed as nonlinear algebraic systems.

Application S5 (Stationary solution of a chemical reactor) Application S5 presented in the SMUNO

collection was solved by the Newton method in Sect. 4.1 (see Table 4.3). Table 4.14 contains
the initial point x0, the functions values in the initial point F(x0), the solution x of this application
given by the composite Newton method of order 1 with ε ¼ 106 in 4 iterations, and the functions
values in x.
166 4 The Newton Method

Table 4.14 Stationary solution of a chemical reactor. Initial point, solution, functions values in these points.
Composite Newton method of order 1. 4 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 1.09 –35.164864 0.97424361 0.0
2 1.05 – 0.2403040 0.98282907 – 0.622e–17
3 3.05 3626.94637 0.05151276 0.0
4 0.99 –1779.9233 0.93567106 – 0.693e–17
5 6.05 –1814.7127 0.00009083 0.681e–12
6 1.09 –7.13 0.06423809 – 0.416e–16

Table 4.15 Robot kinematics problem. Initial point, solution, functions values in these points. Composite Newton
method of order 1. 2 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 0.164 0.0026823 0.16443166 0.111e–15
2 –0,98 – 0.0098180 –0.98638847 0.0
3 –0,94 0.1050028 –0.94706369 0.260e–17
4 –0,32 0.0017588 –0.32104573 0.555e–16
5 –0,99 – 0.0127040 –0.99823316 0.0
6 –0.05 – 0.014 0.05941842 0.0
7 0.41 – 0.016764 0.41103315 0.110e–011
8 –0.91 – 0.0038 –0.91162039 0.0

Application S6 (Robot kinematics problem) Application S6 described in the SMUNO collection was
solved by the Newton method (see Table 4.4). Table 4.15 presents the performances of the composite
Newton method of order 1 with ε ¼ 106 for solving this application.

Application S9 (Propan combustion in air – reduced variant) This application from the SMUNO
collection was solved by the Newton method, and the performances of this method were presented in
Table 4.5. Table 4.16 shows the performances of the composite Newton method of order 1 with
ε ¼ 106 for solving this application.

Application S14 (Circuit design) The performances of the Newton method for this application taken
from the SMUNO collection were presented in Table 4.6. Table 4.17 shows the performances of the
composite Newton method of order 1 with ε ¼ 106.

By comparing Tables 4.3, 4.4, 4.5, and 4.6 and Tables 4.10, 4.11, 4.12, and 4.13, we can see that
the composite Newton method of order 1 is more efficient than the Newton method.

Remark 4.1 Observe that from the local convergence point of view, the Newton method is exactly
the same as the one for nonlinear equations applied to the problem of finding a root of ∇f. Therefore,
the positive definiteness of ∇2f can be used in an implementation of the Newton method based on the
4.10 The Composite Newton Method 167

Table 4.16 Propan combustion in air – reduced variant. Initial point, solution, functions values in these points.
Composite Newton method of order 1. 7 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 10 109.85000 0.002757177 0.664e–10
2 10 209.54267 39.24228904 0.221e–09
3 0,05 – 0.3487418 – 0.061387604 0.177e–09
4 50,5 5098.5172 0.859724425 0.885e–09
5 0,05 2659.2930 0.036985043 – 0.111e–15

Table 4.17 Circuit design. Initial point, solution, functions values in these points. Composite Newton method of order
1. 3 iterations. e ¼ 106

Nr. x0 F ( x0 ) x* F ( x* )
1 0.7 0.95525047 0.89999995 – 0.285e– 09
2 0.5 3.04155543 0.44998747 – 0.169e– 08
3 0.9 – 4.16621316 1.00000648 – 0.143e– 08
4 1.9 – 2.11846214 2.00006854 – 0.313e– 08
5 8.1 5.636125856 7.99997144 – 0.840e– 09
6 8.1 22.31397917 7.99969268 –0.365e– 08
7 5.9 24.95943321 5.00003127 – 0.401e– 08
8 1 42.18784331 0.99998772 – 0.687e– 08
9 1.9 – 0.32000000 2.00005248 0.110e–09

Cholesky factorization ∇2f ¼ LLT where L is lower triangular and has positive diagonal. The iterations
are terminated when ∇f is sufficiently small. A natural criterion for a relative decrease in k∇fk and the
termination of the iterations is

k∇f ðxk Þk  τr k∇f ðx0 Þk, ð4:108Þ

where τr is the desired reduction in the gradient norm. However, if k∇f(x0)k is very small, it may not
satisfy the above criterion (4.108) in the floating point arithmetic and an algorithm based only on
(4.108) might never terminate. A standard remedy is to augment the relative error criterion and
terminate the iteration by using a combination of the relative and absolute measures of ∇f i.e., when

k∇f ðxk Þk  τr k∇f ðx0 Þk þ τa , ð4:109Þ

where τa is an absolute error tolerance. Another practical procedure often implemented in Newton’s
method is the modified Cholesky factorization. A modified Cholesky factorization of a symmetric
matrix A is a factorization P(A + E)PT ¼ LDLT where P is a permutation matrix, L is unit lower
triangular, and D is diagonal or block diagonal and positive definite matrix. It follows that A + E is a
positive definite matrix. This corresponds to making a diagonal perturbation E to A and to computing
a Cholesky factorization of the modified matrix A+E (see Appendix A).
168 4 The Newton Method

Notes and References

Regarding the history of the Newton method, we can first mention Heron of Alexandria (10–70 AD).
He described a method (called the Babylonian method) to iteratively approximate a square root.
François Viète (1540–1603) developed a method to approximate roots of polynomials. In 1669 Sir
Isaac Newton (1643–1727) improved the Viète’s method in a paper published in 1711. A simplified
version of Newton’s method was published by Joseph Raphson (1648–1715) in 1690. Further
information about Raphson can be found in a paper of Bicanic and Johnson (1979). The modern
treatment and presentation of the Newton method is due to Thomas Simpson (1710–1761). (See
(Kollerstrom, 1992).) A more detailed history of this method can be found in a paper by Ypma (1995).
The first sufficient conditions for the monotone convergence of the Newton method were given by
Joseph Fourier (1768–1830) in 1818. His result was reconsidered in modern terms by Gaston
Darboux (1842–1917) in 1869. In 1952 Baluev extended the theorem of Fourier-Darboux to nonlinear
operators in partial ordered topological spaces. The conditions of Baluev’s theorem are relatively
powerful and difficult to verify in practice. Moreover, there are simple examples for which Baluev’s
result does not apply, even if the Newton method converges monotonically for these examples.
Starting with 1937, Ovstrowski (1893–1986) published a number of papers in which the local
convergence of the Newton method is proved for functions of a real variable. The proof that the
Newton method is indeed monotonously convergent in more general conditions than those of Baluev
was given by Potra and Rheinboldt (1986). However, the proof of the quadratic convergence of the
Newton method in general Banach spaces was established by Kantorovich in 1948. The result of
Kantorovich is fundamental in the sense that, in addition to the fact that he established a very
ingenious and elegant technique of proving, he introduced the fundamental problems associated to
any iterative method: convergence, ordin of convergence, and estimation of error, problems which
have become classical and continued to dominate the numerical analysis up to the present.
Newton’s method is only locally convergent. Moreover, as in the most cases of algorithms that are
only locally convergent, it is usually not known apriori whether an initial estimate is indeed close
enough to the solution. Ideal algorithms should be globally convergent (converge from an arbitrary
initial point). However, the proofs of global convergence do need assumptions about the properties of
the problem that may not be verifiable for specific problems.
The theoretical developments of this chapter are based on the books of Sun and Yuan (2006) and
Kelley (1999). The most important is Theorem 4.1 which shows that in some specific conditions (the
Jacobian in the minimum point is nonsingular and in a neighborhood of the minimum point the
Jacobian is Lipschitz continuous), the Newton method for solving nonlinear algebraic systems is
quadratic convergent. For the functions minimization, Theorem 4.3 (based on the Lipschitz continu-
ity of the Hessian) and Theorem 4.4 (based on the bounds of the Hessian) are the most important ones,
showing the quadratic convergence of the Newton method. All these results are based on the
assumption that the initial point is near enough to the minimum point.
The analysis of the complexity of the Newton method with backtracking gives only a conceptual
result on the bound of the number of iterations. Initialized in a point near enough to the minimum
point, the Newton method with backtracking accepts the stepsize αk ¼1. The difficulty is given by the
requirement to evaluate the Hessian matrix of the minimizing function and by the solving, using the
factorization of the Newton system.
The modifications of the Newton method may be classified in two classes: computation of the
entries of the Hessian by finite-difference and modifications of the Hessian when this matrix is
indefinite and the current iterations are near a saddle point. All these modifications try to ameliorate
the behavior of the method. The composite Newton method of order 1 is a proper modification which
substantially improves the rate of convergence of the Newton method.
 Conjugate Gradient Methods
5

These methods are characterized by very strong convergence properties and modest storage
requirements. They are dedicated to solving large-scale unconstrained optimization problems and
applications. The history of these methods starts with the researches of Cornelius Lanczos (1950,
1952), Magnus Hestenes (1951, 1955, 1956a, 1956b), Rosser (1953), Forsythe, Hestenes and Rosser
(1951), (Stein), and others from the Institute for Numerical Analysis – National Bureau of Standards,
Los Angeles, as well as with the researches of Eduard Stiefel (1958) from Eidgen€ ossische Technische
Hochschule Z€ urich. During over 70 years of researches in this area, an impressive number of
developments, variants, and algorithms of these methods have appeared. A thorough and detailed
presentation of these methods was given by Andrei (2020a). The search direction of these methods is
a linear combination of the negative gradient and the previous search direction. The conjugate
gradient methods require only the first order derivatives. As we will see, these methods may include
the second order information given by an approximation of the Hessian of the minimizing function,
thus increasing their convergence properties.
In this chapter we will discuss the linear and nonlinear conjugate gradient methods by presenting
their derivation, the convergence properties, and their performances for solving unconstrained
optimization problems and real optimization applications.

5.1 The Concept of Nonlinear Conjugate Gradient

Let us consider the quadratic function

1
f ðxÞ ¼ xT Ax  bT x, ð5:1Þ
2
where A 2 ℝnn is a symmetric and positive definite matrix and b 2 ℝn is a known vector. From (5.1)
we get

∇f ðxÞ ¼ Ax  b, ∇2 f ðxÞ ¼ A: ð5:2Þ

Observe that the Hessian of function (5.1) is independent of x. Since the Hessian A is symmetric
and positive definite, from the optimality conditions for a minimum of a differentiable function, it
follows that there is a unique minimizer x of (5.1). From (5.2) observe that x is the solution of the
linear system Ax ¼ b.

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 169
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_5
170 5 Conjugate Gradient Methods

The idea behind the conjugate gradient approach for minimizing quadratic functions (5.1) is to use
search directions which do not interfere with one another. Given a symmetric and positive definite
matrix A, the set of vectors {d0, . . ., dk  1} is a set of conjugate directions (or “A conjugate,” or even
“A orthogonal”) if and only if d Ti Ad j ¼ 0 for all i 6¼ j, where i ¼ 0, . . ., k  1 and j ¼ 0, . . ., k  1. Note
that dTi Ad i > 0 for all i, since A is positive definite.
In the economy of a conjugate gradient algorithm, the conjugacy condition is an important
ingredient. The main characteristic of conjugate directions is the minimization of a convex quadratic
function in a subspace spanned by a set of mutually conjugate directions is equivalent to the
minimization of the function along each conjugate direction in turn. However, the performances of
algorithms satisfying the conjugacy condition are strongly dependent on the accuracy of the line-
search. If the line-search procedure for the stepsize computation is highly accurate, then the
convergence of the corresponding optimization algorithm is fast.
In the conjugate gradient algorithm, the search direction dkþ1 is computed as a linear combination

dkþ1 ¼ gkþ1 þ βk d k ð5:3Þ

of the current negative gradient and the last search direction in such a way that dkþ1 is A conjugate to
all the previous search directions.
For quadratic functions, the conjugate gradient parameter βk is chosen so that d Tkþ1 Adk ¼ 0, which
determines

gTkþ1 Adk
βk ¼ : ð5:4Þ
d Tk Ad k

The next approximation to the minimum is xkþ1. This is the unique minimum of f along the line
l(α) ¼ xk þ αdk, which is given by
xkþ1 ¼ xk þ αk dk , ð5:5Þ

where

d Tk gk
αk ¼  : ð5:6Þ
dTk Adk

This is the traditional conjugate gradient algorithm which works for quadratic functions, since it
explicitly uses the matrix A, both in (5.4) and (5.6).
Now, the problem is how the algorithm could be modified so that the quadratic nature of the
minimizing function f should not explicitly appear in the algorithm, i.e., in (5.4) and (5.6), and
remains unchanged if f is a quadratic function. As we know, the initial search direction is d0 ¼  g0.
Observe that the initial search direction does not involve the matrix A. Therefore, for nonlinear
functions, the initial search direction could be very well considered as the gradient of the minimizing
function at the initial point. As we can see, in this context two crucial elements have to be computed:
the stepsize αk and the parameter βk. To determine αk, a line-search from xk in the direction dk is
executed. For the convex quadratic functions, this line-search determines an explicit stepsize (5.6),
which is the unique minimum of function f in the direction dk. This is the advantage of quadratic
functions: there is an explicit formula for the stepsize computation. For nonlinear functions, this
formula for the stepsize computation can be replaced by a general line-search procedure (Armijo,
Goldstein, Wolfe, etc.). This will change nothing in the quadratic case, but will generalize the
algorithm to nonlinear functions. For the computation of the conjugate gradient parameter βk, from
5.1 The Concept of Nonlinear Conjugate Gradient 171

(5.4) we need to compute Adk. There are several possibilities to modify the algorithm in order to
eliminate the explicit mention of matrix A in (5.4).
One way is to see from (5.5) that xkþ1  xk ¼ αkdk. Therefore, for quadratic functions,
gkþ1  gk ¼ ðAxkþ1  bÞ  ðAxk  bÞ ¼ αk Ad k :

With this, since Adk ¼ (gkþ1  gk)/αk, from (5.4) it follows that

gTkþ1 yk
βk ¼ , ð5:7Þ
dTk yk

where yk ¼ gkþ1  gk.

If f is a quadratic function, then the definitions (5.4) and (5.7) are equivalent, but the new definition
(5.7) can be used for any differential function. Observe that (5.7) is exactly the Hestenes and Stiefel
formula (Hestenes, & Stiefel, 1952) for the conjugate gradient parameter computation.
Another way to eliminate A from (5.4) is as follows. Suppose that f is quadratic. From (5.7) it
follows that dTk yk ¼ dTk gkþ1  dTk gk : But, dTk gkþ1 ¼ 0: Therefore, dTk yk ¼ dTk gk : Now, observe that
dk ¼  gk þ βk  1dk  1, so that, since d Tk1 gk ¼ 0, it results that


d Tk yk ¼  gTk þ βk1 dTk1 gk ¼ gTk gk :

Therefore, from (5.7) another formula for βk computation is

gTkþ1 yk
βk ¼ : ð5:8Þ
gTk gk

Note that if f is quadratic, (5.8) is equivalent to (5.4). Hence, (5.8) generalizes the computation of
the conjugate gradient parameter to the non-quadratic case. Formula (5.8) is exactly the Polak-
Ribière-Polyak formula (Polak, & Ribière, 1969; Polyak, 1969) for the conjugate gradient parameter
computation.
Furthermore, it is very easy to see that if f is a quadratic function, then gTkþ1 gk ¼ 0, and in this
case, from (5.8),

gTkþ1 gkþ1
βk ¼ , ð5:9Þ
gTk gk

which generalizes the computation of the conjugate gradient parameter to the non-quadratic case.
Observe that (5.9) is exactly the Fletcher-Reeves formula (Fletcher, & Reeves, 1964) for the
conjugate gradient parameter computation.
Formulae (5.7), (5.8), and (5.9) generalize the conjugate gradient algorithm to any nonlinear
differential function. They require only the gradient evaluations. For quadratic functions they are
equivalent, but they are not equivalent for nonlinear functions.
For general nonlinear functions, the matrix A should be replaced by the Hessian. Since
yk ¼ gkþ1  gk, from the mean value theorem it follows that

dTkþ1 yk ¼ αk dTkþ1 ∇2 f ðxk þ ταk dk Þd k , ð5:10Þ

where τ 2 (0, 1). Therefore, from (5.10), for nonlinear functions it is reasonable to consider the
conjugacy condition as
172 5 Conjugate Gradient Methods

dTkþ1 yk ¼ 0, ð5:11Þ


or as dTkþ1 yk ¼ t gTkþ1 sk , where t > 0 is a parameter and sk ¼ xkþ1  xk, (Dai, & Liao, 2001). For
example, the Hestenes-Stiefel method has the property that the conjugacy condition (5.11) always
holds, independent of the line-search.
In general, not all the nonlinear conjugate gradient algorithms satisfy the conjugacy condition
(5.11). However, what is characteristic of nonlinear conjugate gradient algorithms is that the search
direction is computed as a linear combination between the negative gradient and the last search
direction, where the conjugate gradient parameter βk is determined in such a way that the descent
condition or the sufficient descent condition holds. Of course, as it is to be seen, to improve the
convergence of the algorithms, this linear combination between the negative gradient and the last
search direction can be modified in such a way as to take into account, for example, the second order
information of the minimizing function, or some other ingredients which accelerate the convergence.
The descent or the sufficient descent conditions are crucial in the conjugate gradient methods. The
descent condition d Tk gk < 0, (if gk 6¼ 0) is more general and leads to general convergence results for
line-search algorithms under the exact or inexact line-search (and not only for conjugate gradient
algorithms). The sufficient descent condition d Tk gk  ckgk k2 , for some positive constant c, is
fundamental in conjugate gradient methods.

5.2 The Linear Conjugate Gradient Method

The linear conjugate gradient algorithm is dedicated to minimizing the convex quadratic functions
(or to solving linear algebraic systems of equations with positive definite matrices). This algorithm
was introduced by Hestenes and Stiefel (1952). Let us consider the quadratic function (5.1) where
A 2 ℝnn is a symmetric and positive definite matrix and b 2 ℝn is a known vector. From (5.1) we get
∇f(x) ¼ Ax  b and ∇2f(x) ¼ A. Observe that the Hessian of function f is independent of x. Since the
Hessian A is symmetric and positive definite, from the optimality conditions for a minimum of a
differentiable function, it follows that there is a unique minimizer x of f. From (5.2) observe that x is
the solution of the linear system Ax ¼ b. Having in view that function f is quadratic, from the Taylor
theorem, for all t 2 ℝ and all y, z 2 ℝn the following identity is obtained:

t2 T
f ðy þ tzÞ ¼ f ðyÞ þ t∇f ðyÞT z þ z Az: ð5:12Þ
2
The linear conjugate gradient algorithm is exactly a line-search with a special choice of directions.
Given the current approximation xj to the minimum x as well as a direction vector dj, a line-search
algorithm computes the next approximation xjþ1 by using the following two steps:

Algorithm 5.1 Line-search

1. Find the stepsize αj ¼ arg minα > 0f(xj þ αdj)

2. Set xjþ1 ¼ xj þ αjdj ♦

Assuming that an initial point x0 is given, then, applying k steps of the above line-search method,
k iterates are obtained: {x0, x1, . . ., xk  1}. From (5.12) the stepsize αj is computed as
5.2 The Linear Conjugate Gradient Method 173

dTj r j
αj ¼ , ð5:13Þ
dTj Ad j

where rj ¼ Axj  b is the residual at xj.

Definition 5.1 The set of directions {d0, . . ., dk  1} is a set of conjugate directions if and only if
dTj Adi ¼ 0 for all i ¼ 0, . . ., k  1, j ¼ 0, . . ., k  1 and i 6¼ j. ♦

Now, for all k ¼ 1, . . ., the following vector space and affine space are introduced:
W k ¼ spanfd 0 , . . . , dk1 g, ð5:14Þ

U k ¼ x0 þ W k ¼ fz 2 ℝn : z ¼ x0 þ wk , wk 2 W k g: ð5:15Þ

Denote W0 ¼ {0} and U0 ¼ {x0}.

Proposition 5.1 Assume that dTi Ad j ¼ 0 for all 0  j < i, where i is a fixed integer, and that
{x0, . . ., xi} are computed by the above line-search algorithm. Then

d Ti r i ¼ dTi ∇f ðyÞ, ð5:16Þ

for all y 2 Ui.

Proof Firstly, observe that, since {x0, . . ., xi} are computed by the line-search algorithm, it follows
that xi 2 Ui. If y 2 Ui, then, from the definition of Ui, it follows that xi  y 2 Wi and hence
dTi Aðxi  yÞ ¼ 0: Therefore,

d Ti ðr i  ∇f ðyÞÞ ¼ dTi ðAxi  b  Ay þ bÞ ¼ dTi Aðxi  yÞ ¼ 0,

which proves (5.16). ♦

In the following, the fundamental property of the line-search method with conjugate directions is
presented. Obviously, at every step, the line-search algorithm minimizes f(x) only in a fixed direction.
However, if the directions are conjugate according to Definition 5.1, then a stronger result can be
proved, as in Theorem 5.1 below: a choice of conjugate directions in the line-search method leads to
obtaining a minimizer xk for the whole space Uk.

Theorem 5.1 If the directions in the line-search algorithm are conjugate and {x0, . . ., xk} are the
iterates generated after k steps of the line-search algorithm, then

xj ¼ arg min f ðxÞ,

x2Uj

for all 1  j  k.

Proof The theorem is proved by induction. For k ¼ 1 the result is obtained from the definition of
x1 as a minimizer on U1. Assume that for k ¼ i,
xj ¼ arg min f ðyÞ,
y2Uj
174 5 Conjugate Gradient Methods

for all 1  j  i. We need to show that if xiþ1 ¼ xi þ αidi, then

xiþ1 ¼ arg min f ðxÞ:
x2U iþ1

By the definition of Uiþ1, any x 2 Uiþ1 can be written as x ¼ y þ αdi, where α 2 ℝ and y 2 Ui. Now,
using (5.12) and Proposition 5.1, it results

α2
f ðxÞ ¼ f ðy þ αdi Þ ¼ f ðyÞ þ αdTi ∇f ðyÞ þ dTi Adi
  2 ð5:17Þ
α2 T
¼ f ðyÞ þ αdi ∇f ðxi Þ þ di Adi :
T
2

Observe that (5.17) is a decoupled function. The first term of the right hand side of (5.17) does not
depend on α, and the second term does not depend on y. Therefore,
 
α2 T
min f ðxÞ ¼ min f ðyÞ þ min αd i r i þ di Adi :
T
ð5:18Þ
x2Uiþ1 y2Ui α2ℝ 2

But, the right hand side of (5.18) is minimized when y ¼ xi and

dTi r i
α ¼ αi ¼ ,
dTi Adi

that is, the left hand side of (5.18) is minimized exactly for xiþ1 ¼ xi þ αidi. In other words, xiþ1 is the
minimizer of f over the set {x : x ¼ x0 þ span{d0, . . . . di}}. ♦

To show the significance of making the search directions mutually conjugate with respect to A, let
us first state and prove a technical result involving only the first two iterations of the conjugate
gradient algorithm.

Proposition 5.2 After two iterations of the conjugate gradient method, the gradient g2 ¼ Ax2  b
satisfies

dT1 g2 ¼ dT0 g2 ¼ 0:

Proof After the first iteration, the new point is x1. Therefore, g1 ¼ Ax1  b. Since the line-search is
exact, we also have d T0 g1 ¼ 0:
Now, consider the second iteration. At this iteration, the algorithm will generate a point
x2 ¼ x1 þ αd1 where g2 ¼ Ax2  b and d T1 g2 ¼ 0: But

dT0 g2 ¼ dT0 ðAx1 þ αAd 1  bÞ ¼ dT0 g1 þ αdT0 Ad 1 :

The first term, dT0 g1 , in the right hand side of the above equality, is zero because of the line-search
on the first iteration. The second term, αdT0 Ad1 , is zero because d0 and d1 are conjugate with respect
to A. ♦

This result shows that after two iterations the gradient is orthogonal to both search directions d0
and d1. Similarly, the above result can be generalized to prove the following proposition.
5.2 The Linear Conjugate Gradient Method 175

Proposition 5.3 After k iterations of the conjugate gradient method, the gradient gk ¼ Axk  b
satisfies
d Tj gk ¼ 0 for j ¼ 0, 1, . . ., k  1. ♦

This proposition implies that after k iterations, the gradient gk is restricted to the (n  k)-
dimensional subspace orthogonal to the vectors d0, . . ., dk  1. From this, the important finite
termination property of the conjugate gradient method can be obtained.

Proposition 5.4 The conjugate gradient method solves an n  n linear algebraic system Ax ¼ b in n
iterations at the most.

Proof Proposition 5.3 implies that, after n iterations, gn is orthogonal to the n vectors d0, . . ., dn  1.
But this means that gn must lie in a subspace of dimension zero and so gn ¼ 0, which proves that
Axn ¼ b. ♦

The finite termination property is only guaranteed when the calculations are exact. In practice, the
conclusion of Proposition 5.4 may not be exactly satisfied when the iterations are performed in real
arithmetic, being subject to rounding errors. Hence, for solving some n  n linear algebraic systems,
the conjugate gradient method needs a few more than n iterations.

The Linear Conjugate Gradient Algorithm

The result of Theorem 5.1 may now be used to present the linear conjugate gradient algorithm for
generating conjugate directions. To start with, a general recurrence relation that generates a set of
conjugate directions is presented. Next, it is shown that this recurrence relation can be reduced to a
simple expression. Finally, the linear conjugate gradient algorithm is described.

Proposition 5.5 Consider d0 ¼  r0 and for k ¼ 1, 2, . . . set

X
k1 d T Ar
j k
dk ¼ r k þ dj : ð5:19Þ
j¼0 d Tj Ad j

Then, d Tj Ad m ¼ 0 for all 0  m < j  k.

Proof By induction, it is shown that (5.19) generates conjugate directions. For k ¼ 1, dT1 Ad0 ¼ 0:
Assume that for k ¼ i the vectors {d0, . . ., di} are pairwise conjugate. We must show that
dTiþ1 Ad m ¼ 0 for all m  i. Consider m  i. Then

P
i d T Ar iþ1
j
dTiþ1 Ad m ¼ r Tiþ1 Ad m þ T
dTj Adm
j¼0 d j Ad j
T
d Ar iþ1
¼ r Tiþ1 Ad m þ Tm
d T Adm ¼ 0,
dm Adm m

which proves the proposition. ♦

176 5 Conjugate Gradient Methods

Proposition 5.6 Let {d0, . . ., dk} be the directions generated by (5.19). Then:

(i) W k ¼ spanfr 0 , . . . , r k1 g,

(ii) r Tm r j ¼ 0, for all 0  j < m  k,
(iii) dTk r j ¼ r Tk r k , for all 0  j  k,
(iv) The direction dk satisfies

dk ¼ r k þ βk1 dk1 , ð5:20Þ

where

r Tk r k
βk1 ¼ : ð5:21Þ
r Tk1 r k1

Proof Since d0 ¼  r0, (i) follows directly from (5.19).

To prove (ii) observe that, for 0  j < m  k and any t 2 ℝ, rj 2 Wjþ1 ⊂ Wm and hence
xm þ trj 2 Um. Now, from Theorem 5.1, since xm is the unique minimizer of function f over Um, it
follows that t ¼ 0 is the unique minimizer of f(xm þ trj). Therefore,


df xm þ tr j  T
0¼  ¼ ∇f ðxm Þ r j ¼ r m r j
T
dt t¼0

for 0  j < m  k, which proves (ii).

To prove (iii), the identity in (iii) is first proved for j ¼ k. Now, from (i), it follows that rk is
orthogonal to each dl for l < k. Therefore, if we take the inner product with rk, the second term in the
right side of (5.19) will vanish. But this is exactly the identity in (iii) for j ¼ k. If j < k, then


(xk  xj) 2 Wk, and hence d Tk A xk  xj ¼ 0: Therefore,



d Tk r k  r j ¼ d Tk A xk  xj ¼ 0:

To prove (iv), note first that {r0, . . ., rk} form an orthogonal basis of Wkþ1. Hence, we can write
dk 2 Wkþ1 as a linear combination of {r0, . . ., rk} and then apply (iii). Indeed

Pk dT r Xk
r Tk r k r Tk r k Xk1 T
r k1 r k1
k j
dk ¼ T
r j ¼  Tr j
r ¼ r k  T r
rj
j¼0 r j r j r
j¼0 j j r k1 k1 j¼0
r Tj r j
kP1 d T r
k1 j
¼ r k þ βk1 T r rj ¼ r k þ βk1 dk1 : ♦
j¼0 r j j

Remark 5.1 From (5.13) and from Proposition 5.6 (iii), it follows that

dTk r k rT rk
αk ¼ T
¼ Tk : ð5:22Þ
dk Ad k d k Ad k

On the other hand, the residual vector rkþ1 can be written as

r kþ1 ¼ Axkþ1  b ¼ Axk  b þ αk Adk ¼ r k þ αk Ad k : ♦ ð5:23Þ

5.2 The Linear Conjugate Gradient Method 177

With these, using Proposition 5.3 and Remark 5.1, the following linear conjugate gradient
algorithm can be presented.

Algorithm 5.2 Linear Conjugate Gradient

1. Select an initial point x0 and ε > 0 sufficiently small

2. Set r0 ¼ Ax0  b, d0 ¼  r0 and k ¼ 0
3. If krkk  ε, then stop. Otherwise, continue with step 4
4. Compute:
rT r rTkþ1 rkþ1
αk ¼ dTkAdk , xkþ1 ¼ xk þ αkdk, rkþ1 ¼ rk þ αkAdk, βk ¼ r Tk r k
, dkþ1 ¼  rkþ1 þ βkdk
k k

5. Set k ¼ k þ 1 and continue with step 3 ♦

The algorithm is simple and very easy to implement. The initial residual is the same as the first
gradient search direction. If the initial solution x0 is zero, then r0 ¼  b and d0 ¼ b. The major
computational efforts at each iteration are the computation of the matrix-vector product Adk and the
computation of the inner products dTk ðAdk Þ and r Tkþ1 r kþ1 : Observe that if A is not symmetric and
positive definite, then the denominator in αk may vanish, which leads to the breakdown of the
iterations.
The linear conjugate gradient algorithm is recommended only for solving large-scale systems of
equations with a symmetric and positive definite matrix. Otherwise, the Gaussian elimination, the
factorization methods (the product form of the inverse or the elimination form of the inverse), and the
singular value decomposition are preferred, since they are less sensitive to rounding errors.

Remark 5.2 Recall that the basic Newton step dNk is obtained by solving the Newton system
∇2 f ðxk ÞdNk ¼ ∇f ðxk Þ. The Newton-CG method computes the search direction dNk by applying the
linear conjugate gradient Algorithm 5.2 to the Newton system. ♦

Convergence Rate of the Linear Conjugate Gradient Algorithm

In the following, an estimate of the convergence rate of the linear conjugate gradient algorithm is
presented. For this, the error reduction in the linear conjugate gradient algorithm is discussed,
followed by a convergence rate estimate based on the Chebyshev polynomials.

Proposition 5.7 The following relation holds

 
W k ¼ span r 0 , . . . , Ak1 r 0 : ð5:24Þ

Proof For k ¼ 1, (5.24) is true. Assume that (5.24) holds for k ¼ i, and let us show that it holds for
k ¼ i þ 1. From Proposition 5.6 (i), this would be equivalent to showing that ri 2 span{r0, . . ., Air0}.
Observe that ri1 2 Wi and di1 2 Wi. By inductive assumption, we can write ri1 ¼ Ri1(A)r0 and
di1 ¼ Pi1(A)r0, where Ri1(.) and Pi1(.) are polynomials depending on the matrix A of degree less
than or equal to i  1. Hence,

r i ¼ r i1 þ αi1 Ad i1  

¼ Ri1 ðAÞr 0 þ αi1 APi1 ðAÞr 0 2 span r 0 , . . . , Ai r 0 ,

thus proving the proposition. ♦

178 5 Conjugate Gradient Methods

In the following, let us present a general error estimate which connects kx  xkkA and kx  x0kA,
where, for any y 2 ℝn, kyk2A ¼ yT Ay: For this, denote Pk as the set of polynomials of degree less than
or equal to k.

Proposition 5.8 The following estimate holds

kx  xk kA ¼ inf kPð AÞ ð x   x 0 Þ kA : ð5:25Þ
P2Pk , Pð0Þ¼1

Proof Since rk is orthogonal to Wk, it follows that for all y 2 Wk,

ðx  xk ÞT Ay ¼ r Tk y ¼ 0: ð5:26Þ

Denoting wk ¼ xk  x0 2 Wk and e0 ¼ x  x0, from (5.26) we get

0 ¼ ðx  xk ÞT Ay ¼ ðe0  wk ÞT Ay

for all y 2 Wk. Therefore, wk ¼ xk  x0 is an A-orthogonal projection of e0 on Wk. Thus,

ke0  wk kA ¼ min ke0  wkA :

w2W k

But, from Proposition 5.7 it is known that w ¼ Qk1(A)r0, for a polynomial Qk1 2 Pk1 , where
Pk1 is the set of polynomials of degree less than or equal to k  1. Also, Ae0 ¼  r0 and
e0  w ¼ (I þ Qk1(A)A)e0 and hence

kx   x k kA ¼ ke 0  w k kA ¼ min kPk ðAÞe0 kA , ð5:27Þ

Pk 2Pk , Pk ð0Þ¼1

which completes the proof. ♦

This convergence rate is rather general and does not take into account knowledge of the distribu-
tion of the eigenvalues of A. In order to refine the above results and to obtain a qualitative estimate on
the right hand side of (5.27), observe that for A symmetric and positive definite the following spectral
decomposition can be written (see Appendix A):

A ¼ UΛUT ,

where U is an orthogonal matrix whose columns are the eigenvectors of A and Λ is a diagonal matrix
with the positive eigenvalues of A, λ1  λ2  . . .  λn on the diagonal. Since UUT ¼ UTU ¼ I, by the
orthogonality of U it follows that for any j,

Aj ¼ UΛj UT :

Therefore,

Pk ðAÞ ¼ UPk ðΛÞU T :

2
Define A1/2 ¼ UΛ1/2UT. Observe that kxk2A ¼ xT Ax ¼ A1=2 x 2 : Hence, for any x 2 ℝn,

kPk ðAÞxkA ¼ A1=2 Pk ðAÞx 2  kPk ðAÞk2 A1=2 x 2  kPk ðAÞk2 kxkA :
5.2 The Linear Conjugate Gradient Method 179

This, together with (5.27), implies that for any polynomial Pk(λ),

kx   x k kA ¼ min kPk ðAÞe0 kA  min ρð P k ð A Þ Þ k e 0 k A ,

Pk 2Pk , Pk ð0Þ¼1 Pk 2Pk , Pk ð0Þ¼1

where ρ(Pk(A)) is the spectral radius of Pk(A). Since both A and Pk(A) have the same eigenvectors, it
follows that


kx   x k kA  min max Pk λj ke0 kA , ð5:28Þ
Pk 2Pk , Pk ð0Þ¼1 1jn

where λ1  λ2  . . .  λn are the eigenvalues of A. The above inequality shows that the minimization
of the error in the linear conjugate gradient corresponds to the minimization of the polynomial Pk(λ)
over the entire range of eigenvalues [λ1, λn]. This can be accomplished via the Chebyshev
polynomials.
The Chebyshev polynomials of the first kind on the interval [1, 1] are defined as
T k ðξÞ ¼ cos ðk arccos ðξÞÞ, k ¼ 0, 1, . . . :

It is easy to see that Tk(ξ) is a polynomial if the following trigonometric identities are used:

cos ðα þ βÞ ¼ cos α cos β  sin α sin β,

cos ðα þ βÞ þ cos ðα  βÞ ¼ 2 cos α cos β:

Let us denote θ ¼ arccos (ξ), then

T 0 ðξÞ ¼ cos ð0θÞ ¼ 1,

T 1 ðξÞ ¼ cos ð1θÞ ¼ ξ,
T 2 ðξÞ ¼ cos ð2θÞ ¼ cos 2 θ  sin 2 θ ¼ 2 cos 2 θ  1 ¼ 2ξ2  1,
T kþ1 ðξÞ þ T k1 ðξÞ ¼ cos ððk þ 1ÞθÞ þ cos ððk  1ÞθÞ
¼ 2 cos ðkθÞ cos ðθÞ ¼ 2ξT k ðξÞ:

Therefore,
T 0 ðξÞ ¼ 1, T 1 ðξÞ ¼ ξ, ð5:29Þ

T kþ1 ðξÞ ¼ 2ξT k ðξÞ  T k1 ðξÞ ð5:30Þ

for any ξ 2 ℝ.
From (5.30) for fixed ξ it follows that

T k ðξÞ ¼ c1 ðη1 ðξÞÞk þ c2 ðη2 ðξÞÞk , k ¼ 0, 1, . . . ,

where η1(ξ) and η2(ξ) are the roots of the characteristic equation η2  2ξη þ 1 ¼ 0. The constants
c1 and c2 are determined from the initial conditions (5.29). Therefore,
" qffiffiffiffiffiffiffiffiffiffiffiffiffik  qffiffiffiffiffiffiffiffiffiffiffiffiffik #
1
T k ðξÞ ¼ ξ þ ξ  1 þ ξ  ξ2  1
2
: ð5:31Þ
2

Observe that |Tk(ξ)|  1 for all ξ 2 [1, 1]. The polynomial that minimizes (5.28) over the interval
[λ1, λn] is
180 5 Conjugate Gradient Methods

  1   
λ þ λ1 λ þ λ1  2λ
Sk ð λ Þ ¼ T k n Tk n : ð5:32Þ
λn  λ1 λn  λ1

To prove this, assume that there exists another polynomial of degree k, Qk, which is better at
minimizing (5.28) on the appropriate interval [λ1, λn], so that Qk(0) ¼ 1,
  1
λn þ λ1
Q k ðλÞ < T k :
λn  λ1

The polynomial Pk  Qk must have a zero at λ ¼ 0 and at k zeros of the polynomials. This means
that this polynomial must have k þ 1 zeros, which is a contradiction. Therefore, Sk from (5.32) must
be the minimizing polynomial on the interval [λ1, λn]. Therefore, from (5.28) it results that
  1
λ þ λ1
kx   x k kA  T k n kx   x 0 k A : ð5:33Þ
λn  λ1

Theorem 5.2 The error after k iterations of the linear conjugate gradient algorithm can be bounded
as follows:
pffiffiffi k
 2  κ1
kx  xk kA  pffiffi k pffiffi k kx  x0 kA  2 pffiffiffi kx   x 0 kA , ð5:34Þ
pκffiffiþ1 κffiffi1 κþ1
κ 1
þ p
κ þ1

where κ ¼ κ(A) ¼ λn/λ1 is the condition number of A.

Proof The purpose is to calculate

  1
λ þ λ1
Tk n :
λn  λ1

From (5.31), for ξ ¼ (λn þ λ1)/(λn  λ1) ¼ (κ þ 1)/(κ  1), we obtain

qffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffi pffiffiffi pffiffiffi 2 pffiffiffi
κþ1 2 κ κþ12 κ ð κ  1Þ κ1
ξ ξ 1¼
2
 ¼ ¼ p ffiffi
ffi p ffiffi
ffi ¼ pffiffiffi :
κ1 κ1 κ1 ð κ  1Þ ð κ þ 1Þ κ1

Therefore,
  "pffiffiffi k pffiffiffi k #
λn þ λ1 1 κþ1 κ1
Tk ¼ pffiffiffi þ pffiffiffi :
λn  λ1 2 κ1 κþ1

Hence,
  1 pffiffiffi k
λ þ λ1 2 κ1
Tk n ¼ pffiffi k pffiffi k  2 pffiffiffi : ð5:35Þ
λn  λ1 κþ1
pκffiffiþ1 þ pκffiffi1
κ 1 κ þ1

The proof is completed by substituting (5.35) in (5.33). ♦

Knowing only the largest and the smallest eigenvalues of A, the bound (5.34) is the best possible.
Theorem 5.2 shows that the error kx  xkkA is upper bounded by a sequence which is convergent to
5.2 The Linear Conjugate Gradient Method 181

zero. Besides, the convergence is monotone, and this explains why the linear conjugate gradient
algorithm is viewed as an iterative method. Its performance depends on both b and the spectrum of
A (see Kelley (1995), Greenbaum (1997)). The linear conjugate gradient will perform well if κ is near
1, and it may perform very poorly if κ is large. Geometrically, κ is large if the ellipsoidal level
surfaces of the quadratic function f are far from spherical.
If additional information about the eigenvalues of A in the interval [λ1, λn] is available, then the
estimate (5.34) can be improved. Suppose, for example, that A has one eigenvalue much larger than
the others, i.e., λ1  ⋯  λn1  λn, that is, λn/λn1 1. Consider a polynomial Pk that is the product
of a linear factor which is zero at λn and of the (k  1) degree scaled and shifted Chebyshev
polynomial on the interval [λ1, λn  1]:

T k1 λn1 þλ1 2λ
λn1 λ1 λ λ
Pk ð λ Þ ¼  n : ð5:36Þ
λn1 þλ1
T k1 λn1 λ1 λn

Since the second factor in (5.36) is zero at λn and less than one in absolute value at each of the other
eigenvalues, then the maximum absolute value of this polynomial in the entire spectrum {λ1, . . ., λn}
of A is less than the maximum absolute value of the first factor on {λ1, . . ., λn  1}. Therefore, using
similar arguments as those in Theorem 5.2, it follows that
pffiffiffiffiffiffiffiffiffi 
 κn1  1 k1 
kx  xk kA  2 pffiffiffiffiffiffiffiffiffi kx  x 0 kA , ð5:37Þ
κn1 þ 1

where, κn1 ¼ λn1 λ1 : A detailed study of the case of isolated eigenvalues is given by Axelsson and
Lindskog (1986).
Similarly, if the matrix A has just a few large eigenvalues, say, λ1  ⋯λn  m  λn  mþ1  ⋯  λn,
i.e., λn  mþ1/λn  m 1, then we can consider a polynomial Pk which is the product of an m-th degree
factor that is zero at each large eigenvalue and of a scaled and shifted Chebyshev polynomial of
degree k  m on the interval [λ1, λn  m]. Bounding the size of this polynomial, it results that
pffiffiffiffiffiffiffiffiffiffi 
κnm  1 km 
kx  xk kA  2 pffiffiffiffiffiffiffiffiffiffi kx  x 0 kA , ð5:38Þ
κnm þ 1

where, this time, κnm ¼ λnmλ1 : (Greenbaum (1997), Van der Vorst (1993))
It is generally true that, if the eigenvalues of the matrix A occur in m distinct clusters, then the
iterates of the linear conjugate gradient algorithm will approximately solve the problem in about
m steps. This is illustrated in the following example.

Example 5.1 In order to see the influence of the distribution of the eigenvalues on the convergence
of the linear conjugate gradient algorithm, let us consider a linear system Ax ¼ b, where A is a
diagonal matrix and b is selected in such a way that the solution of the system is always [1, 1, . . ., 1],
(Andrei, 2009f). In Algorithm 5.2, let us consider n ¼ 1000 and ε ¼ 106. In the first set of numerical
experiments, the matrix A has four distributions of the eigenvalues: (a) A ¼ diag (1, 2, . . ., 1000) for
which κ(A) ¼ 1000, (b) the diagonal elements are uniformly distributed in [0,1) with
κ(A) ¼ 997.4945, (c) the eigenvalues of A are distributed in 10 intervals with κ(A) ¼ 19.0198, and
(d) the eigenvalues of A are distributed in 5 intervals with κ(A) ¼ 9.0099. Figure 5.1 presents the norm
of the residuals corresponding to these 4 distributions of the eigenvalues.
182 5 Conjugate Gradient Methods

Fig. 5.1 Performance of the linear conjugate gradient algorithm for solving the linear system Ax ¼ b,
where (a) A ¼ diag (1, 2, . . ., 1000), (b) the diagonal elements of A are uniformly distributed in [0,1), (c) the
eigenvalues of A are distributed in 10 intervals, (d) the eigenvalues of A are distributed in 5 intervals

Fig. 5.2 Performance of

the linear conjugate
gradient algorithm for
solving the linear system
Ax ¼ b, where the matrix
A has a large eigenvalue
separated from the others,
which are uniformly
distributed in [0,1)

In the second set of numerical experiments, the matrix A has 999 eigenvalues uniformly distributed
in [0, 1] and one large eigenvalue equal to 100. In this case, the condition number is 4448807.0435,
and the linear conjugate gradient gives a solution in 185 iterations. Figure 5.2 presents the evolution
of the norm of the residuals for this distribution of the eigenvalues.
5.2 The Linear Conjugate Gradient Method 183

Suppose that the eigenvalues of A consist of m large values and of n  m small eigenvalues
clustered around 1. Defining τ ¼ λnm  λ1, then (5.38) tells us that after m þ 1 steps of the linear
conjugate gradient algorithm,

kx  xmþ1 kA
τkx  x0 kA :

Therefore, for a small value of τ, i.e., for small well-clustered eigenvalues, it follows that the
iterates of the algorithm provide a good estimate of the solution after only m þ 1 steps.

Example 5.2 Let us consider the linear algebraic system Ax ¼ b, where

2 3 2 3
2 1 1
6 7 6 7
6 1 2 ⋱ 7 6 0 7
6 7 6 7
A¼6 6 ⋱ ⋱ ⋱ 7, and b ¼ 6 ⋮ 7,
7 6 7
6 7 6 7
4 ⋱ 2 1 5 4 0 5
1 2 1

obtained from the finite difference numerical method of the one-dimensional Poisson equation
(Andrei, 2000, 2009f).
For n ¼ 1000, the linear conjugate gradient algorithm gives a solution in 500 iterations. Figure 5.3
shows the evolution of the error kb  Axkk along the iterations for obtaining a solution with accuracy
less than or equal to 108.

The eigenvalues of A are


πi
λi ¼ 2 1  cos , i ¼ 1, . . . , n:
nþ1

πn
Observe that the maximum eigenvalue is λn ¼ 2 1  cos nþ1
4: On the other hand, the

π
minimum eigenvalue is λ1 ¼ 2 1  cos nþ1 : For i sufficiently small,

Fig. 5.3 Evolution of the

error kb  Axkk
184 5 Conjugate Gradient Methods

!!
 
πi π 2 i2 πi 2
λi ¼ 2 1  cos
2 1 1 ¼ :
nþ1 2ð n þ 1Þ 2 nþ1

Therefore, A is positive definite and for n large, the condition number of A is

2
λ n 4ð n þ 1Þ
κ ð AÞ ¼
:
λ1 π2

The behavior of the linear conjugate gradient algorithm illustrated in Fig. 5.3 has a plateau where,
for a large number of iterations, the error kb  Axkk decreases very slowly. This behavior is typical of
the linear conjugate gradient algorithm. Greenbaum and Strakoš (1992) demonstrated that finite
precision conjugate gradient computations for solving a symmetric positive definite linear system
Ax ¼ b behave very similarly to the exact algorithms applied to any of a certain class of larger
matrices. This class consists of matrices Ae that have lots of eigenvalues spread throughout tiny
intervals around the eigenvalues of A. The width of these intervals is a modest multiple of the
machine precision multiplied by the norm of A. This analogy appears to hold unless the algorithms are
run for huge numbers of steps. See also Naiman, Babuska, and Elman (1997).

Example 5.3 In the following, let us consider the linear algebraic system Ax ¼ b, where
2 3 2 3
B I 4 1
6 7 6 7
6 I B I 7 6 1 4 ⋱ 7
6 7 6 7
A¼6 6 ⋱ ⋱ ⋱ 7 6 7,
7, and B ¼ 6 ⋱ ⋱ ⋱ 7
6 7 6 7
4 I B I 5 4 ⋱ 4 1 5
I B 1 4

obtained from the finite difference numerical method of the two-dimensional Poisson equation
(Andrei, 2000, 2009f). The matrix A has n2 blocks on the main diagonal, where each block B 2
ℝn1 n1 . Hence, A 2 ℝn  n, where n ¼ n1n2. The right hand side b is chosen so that the solution of the
system Ax ¼ b is x ¼ [1, 1, . . ., 1]. Considering n ¼ 10000, the evolution of the error kb  Axkk
computed by the linear conjugate gradient algorithm for five different values of n1 and n2 is presented
in Fig. 5.4.
The eigenvalues of A are as follows:
   
iπ jπ
λij ¼ 4 sin 2 þ 4 sin 2 , i ¼ 1, . . . , n1 , j ¼ 1, . . . , n2 :
2ð n1 þ 1Þ 2ð n2 þ 1Þ

The maximum eigenvalue is λmax ¼ 8, while the minimum eigenvalue is λmin ¼ 8sin2(π/2).
Therefore, A is positive definite and its condition number is
λ max 4

2:
λ min π

From Fig. 5.4, for n1 ¼ 5000 and n2 ¼ 2, that is when there are only two blocks on the main
diagonal of A, the linear conjugate gradient algorithm needs only 31 iterations. Therefore, the
convergence is faster. On the other hand, when n2 ¼ 100, i.e., there are 100 blocks on the main
diagonal of the matrix A, then the algorithm needs 304 iterations. So, the smaller the number of blocks
on the main diagonal of the matrix A is, the faster the convergence is.
5.2 The Linear Conjugate Gradient Method 185

Fig. 5.4 Evolution of the

error kb  Axkk of the
linear conjugate gradient
algorithm for different
numbers (n2) of blocks
on the main diagonal
of matrix A

Preconditioning

Having in view the above discussion, it follows that the conjugate gradient method can be accelerated
by transforming the linear system Ax ¼ b to improve the eigenvalue distribution of A. This is achieved
by a change of the variables from x to xb via a nonsingular matrix C, that is,

xb ¼ Cx: ð5:39Þ

The quadratic (5.1) is transformed as

1
T
fbðxbÞ ¼ xbT CT AC1 xb  CT b xb: ð5:40Þ
2

If Algorithm 5.2 is used to minimize fb or, equivalently, to solve the linear system

T 1 
C AC xb ¼ CT b,

then the rate of the convergence will depend on the eigenvalues of the matrix CTAC1 rather than on
those of A. Therefore, our purpose is to choose C such that the eigenvalues of CTAC1 are more
favorable for the theory presented above. One approach is to choose C such that the condition number
of CTAC1 is much smaller than the condition number of A, so that the constant in (5.34) is smaller.
Another approach is to select C for the eigenvalues of CTAC1 to be clustered, which, as we have
already mentioned, ensures that the number of iterates needed to find a good approximate solution is
not much larger that the number of clusters.
It follows that the transformation (5.39) does not need to be carried out explicitly. Instead, we can
apply Algorithm 5.2 directly to the problem (5.40) in terms of the variables xb and then invert the
transformations to re-express all the equations in terms of x. Define M ¼ CTC.
186 5 Conjugate Gradient Methods

Algorithm 5.3 Preconditioned conjugate gradient

1. Select an initial point x0, the preconditioner matrix M and ε > 0 sufficiently small
2. Set r0 ¼ Ax0  b
3. Solve the system My0 ¼ r0
4. Set d0 ¼  y0 and k ¼ 0
5. If krkk  ε, then stop. Otherwise, continue with step 6
6. Compute:
rT y
αk ¼ dTkAdk , xkþ1 ¼ xk þ αkdk, rkþ1 ¼ rk þ αkAdk, solve the system Mykþ1 ¼ rkþ1,
k k
r Tkþ1 ykþ1
βk ¼ rTk yk
, dkþ1 ¼  ykþ1 þ βkdk
7. Set k ¼ k þ 1 and continue with step 5 ♦

The main difference between the unpreconditioned conjugate gradient Algorithm 5.2 and the
preconditioned conjugate gradient 5.3 is the need to solve the linear system My ¼ r in step 6 of
Algorithm 5.3.
The selection of the preconditioner matrix M is problem dependent. Knowing the structure and the
origin of a problem is the key to devising an effective preconditioner. The preconditioner is often
defined in such a way so that the system My ¼ r should represent a simplified version of the original
system Ax ¼ b. The most important strategies for selecting the preconditioner matrix include
symmetric successive overrelaxation, incomplete Cholesky factorization, banded preconditioners,
etc. (Golub, & Van Loan, 1996). The incomplete Cholesky factorization might be the most effective.

Incomplete Cholesky Factorization

The idea of this factorization is as follows. Perform the Cholesky procedure, but, instead of
computing the exact Cholesky factor L that satisfies A ¼ LLT, an approximation factor L which is
T T T
sparser than L is computed. Therefore, we have A
LL and by choosing C ¼ L we obtain M ¼ LL
1 T
and CT AC1 ¼ L AL
I, so the eigenvalues distribution of CTAC1 is favorable. Obviously,
the matrix M is not explicitly computed. Instead, the factor L is stored, and the system My ¼ r is
solved by performing two triangular substitutions with L:

Remark 5.3 Two methods are known for the acceleration of conjugate gradient algorithms. The first
one consists of modifying the stepsize in a multiplicative manner in such a way as to reduce the values
of the minimizing function, as described in Sect. 3.3 (see Remark 3.1). The second one is the
preconditioning. No comparisons between these acceleration methods are known. ♦

Comparison of the Convergence Rate of the Linear Conjugate Gradient

and of the Steepest Descent

As we have already seen in Proposition 5.4, the linear conjugate gradient algorithm has a quadratic
(finite) termination property, i.e., for convex quadratic functions, the linear conjugate gradient
algorithm with exact line-search terminates after n iterations. In (5.33), (5.34), (5.37) and (5.38),
some formulae for the convergence rates of the linear conjugate gradient algorithm have been
5.2 The Linear Conjugate Gradient Method 187

presented, showing that the convergence rate is not worse than the one of the steepest descent
algorithm, that is, it is not worse than linear.
In the following, let us have a comparison between the linear conjugate gradient algorithm and the
steepest descent algorithm subject to the reduction of the function values along the iterations (Sun, &
Yuan, 2006). Consider the quadratic function
1
f ðxÞ ¼ xT Ax, ð5:41Þ
2
where A 2 ℝnn is symmetric and positive definite. In this case, the explicit expression for the
stepsize is

dTk Axk dT g
αk ¼  T ¼  Tk k : ð5:42Þ
dk Ad k dk Adk

Therefore,
1
f ðxkþ1 Þ ¼ xTkþ1 Axkþ1
2
1
¼ ðxk þ αk d k ÞT Aðxk þ αk d k Þ ð5:43Þ
2
2
1 T 1 gTk dk
¼ xk Axk  :
2 2 dTk Adk

Now, for the steepest descent algorithm dk ¼  gk and from (5.43),

 1 T 1 kgk k
4
kþ1 ¼ 2 xk Axk  2 T
f xSD : ð5:44Þ
gk Agk

On the other hand, for the linear conjugate gradient algorithm dk ¼  gk þ βk1dk1 and from
(5.43),

 1 T 1 kgk k
4
f xCG
kþ1 ¼ x Ax k  : ð5:45Þ
2 k 2 d Tk Ad k

Since

d Tk Ad k ¼ ðgk þ βk1 dk1 ÞT Aðgk þ βk1 dk1 Þ

¼ gTk Agk þ β2k1 d Tk1 Adk1
 gTk Agk ,

it follows that


SD 
kþ1  f xkþ1 :
f xCG

Therefore, the linear conjugate gradient algorithm reduces the value of the minimizing function
f at least as much as the steepest descent algorithm. Since the steepest descent algorithm has a linear
rate of convergence, it follows that the linear conjugate gradient algorithm has a convergence rate that
is not worse than the linear rate. From (5.45) it results that, for the linear conjugate gradient algorithm,
the objective function is strictly decreased along the iterations.
188 5 Conjugate Gradient Methods

5.3 General Convergence Results for Nonlinear Conjugate Gradient

Methods

For solving the nonlinear unconstrained optimization problem

min f ðxÞ, ð5:46Þ

where f : ℝn ! ℝ is a continuously differentiable function, any nonlinear conjugate gradient

algorithm generates the sequence {xk} of the form

xkþ1 ¼ xk þ αk dk , ð5:47Þ

where αk is the stepsize obtained by line-search and dk is the search direction computed by
dkþ1 ¼ gkþ1 þ βk dk , ð5:48Þ

for k  0, where βk is the conjugate gradient parameter and gk ¼ ∇ f(xk). In conjugate gradient
methods, d0 ¼  g0.
A popular strategy for the stepsize determination, which plays a key role in the efficiency of the
unconstrained optimization algorithms, consists in accepting a positive stepsize αk that satisfies the
Wolfe line-search conditions

f ðxk þ αk dk Þ  f ðxk Þ þ ραk dTk gk , ð5:49Þ

∇f ðxk þ αk dk ÞT dk  σd Tk gk , ð5:50Þ

where 0 < ρ < σ < 1. Often, the strong Wolfe line-search is used in the implementation of conjugate
gradient methods. These are given by (5.49) and
 
∇f ðxk þ αk dk ÞT dk   σdT g , ð5:51Þ
k k

where again 0 < ρ < σ < 1. Observe that if σ ¼ 0, then the strong Wolfe line-search reduces to the exact
line-search. Dai and Yuan (1999, 2001) proved that the Wolfe line-search (5.49) and (5.50) ensures
the convergence and therefore can be successfully used in the current implementations in computing
programs of the conjugate gradient methods.
A conjugate gradient algorithm given by (5.47) and (5.48) generates a sequence {xk}. The interest
is to see the conditions under which this sequence converges to the solution x of the problem (5.46).
Since the algorithm given by (5.47) and (5.48) depends only on the parameter βk, it follows that the
interest is to see the values of this parameter for which the algorithm is convergent.
An important requirement for the optimization methods based on line-search is that the search
direction must be a descent one. The search direction dk satisfies the descent property, i.e., it is a
descent direction if for all k ¼ 1, 2, . . .

gTk dk < 0: ð5:52Þ

For conjugate gradient methods, from (5.48) it follows that

2
gTkþ1 dkþ1 ¼  gkþ1 þ βk gTkþ1 dk : ð5:53Þ
2
Now, if the line-search is exact, i.e., if gTkþ1 dk ¼ 0, then gTkþ1 dkþ1 ¼  gkþ1 : Therefore, dkþ1 is
a descent direction if gkþ1 6¼ 0. However, for the inexact line-search this may not be true. Using the
5.3 General Convergence Results for Nonlinear Conjugate Gradient Methods 189

restart with dkþ1 ¼  gkþ1 this situation can be corrected. The search direction dk satisfies the
sufficient descent property, i.e., it is a sufficient descent direction if

gTk dk  ckgk k2 ð5:54Þ

for all k ¼ 1, 2, . . ., where c > 0 is a constant.

The convergence properties of a line-search method, such as the nonlinear conjugate gradient, can
be studied by measuring the effectiveness of the search direction and of the length of the step. The
quality of a search direction dk can be determined by studying the angle between the steepest descent
direction gk and the search direction dk defined by

gTk d k
cos θk ¼ : ð5:55Þ
kgk kkd k k

To establish the general convergence results of any method of the form (5.47) and (5.48), the
following basic assumptions on the objective function are introduced.

Assumption CG
(i) The level set S ¼ {x 2 ℝn : f(x)  f(x0)} is bounded, i.e., there exists a constant B > 0 so that
kxk  B for all x in the level set.
(ii) In some neighborhood N of the level set, f is continuously differentiable, and its gradient is
Lipschitz continuous, i.e., there exists a constant L > 0 so that

kgðxÞ  gðyÞk  Lkx  yk, for all x, y 2 N: ð5:56Þ

Note that these assumptions imply that there is a constant Γ so that kg(x)k  Γ for all x from the
level set S. The boundedness assumption of the level set is not necessary in all the situations. Only the
assumption that f is bounded below on the level set can be used for the global convergence analysis.♦

Under the Assumption CG, the following theorem, due to Zoutendijk (1970) and Wolfe (1969,
1971), is essential in proving the global convergence results of the unconstrained optimization
algorithms, including the conjugate gradient or the Newton one.

Theorem 5.3 Suppose that f is bounded below in ℝn and that f is continuously differentiable in a
neighborhood N of the level set S ¼ {x 2 ℝn : f(x)  f(x0)}. Also assume that the gradient is Lipschitz
continuous, i.e., there exists a constant L > 0 so that (5.56) is satisfied for all x, y 2 N. Consider any
iteration of the form (5.47), where dk is a descent direction and αk satisfies the Wolfe line-search
conditions (5.49) and (5.50). Then,
X
1
cos 2 θk kgk k2 < 1: ð5:57Þ
k¼1

Proof From (5.50) it follows that

T
gkþ1  gk dk  ðσ  1ÞgTk d k :
190 5 Conjugate Gradient Methods

On the other hand, the Lipschitz continuity (5.56) results in

T
gkþ1  gk dk  αk Lkdk k2 :

Therefore, the combination of these two relations gives

ðσ  1Þ gTk dk
αk  : ð5:58Þ
L kd k k2

Now, using the first Wolfe condition (5.49) and (5.58), it results that

2
ðσ  1Þ gTk dk
f kþ1  f k þ ρ : ð5:59Þ
L kd k k2

From the definition (5.55) of cosθk, it follows that (5.59) can be written as

f kþ1  f k þ c cos 2 θk kgk k2 , ð5:60Þ

where c ¼ ρ(σ  1)/L. Summing (5.60) for k  1 and having in view that f is bounded below, (5.57) is
obtained. ♦

The relation (5.57) is called the Zoutendijk condition, and from (5.55) it can be rewritten as

T 2
X1
gk d k
2
< 1: ð5:61Þ
k¼1 kd k k

It is interesting to see how the Zoutendijk condition is used to obtain global convergence results.
Suppose that the iteration (5.47) is in such a way, so that

cos θk  δ > 0, ð5:62Þ

for all k. Then, from (5.57) it follows that

lim kgk k ¼ 0: ð5:63Þ
k!1

In other words, if the search direction generated by any unconstrained optimization method does
not tend to be orthogonal to the gradient, then the corresponding sequence of gradients converges to
zero. For line-search methods (5.47), the limit (5.63) is the best type of the global convergence result
that can be obtained. We cannot guarantee if the method converges to the minimizers of function f,
but only that it converges to stationary points. The implications of the Zoutendijk condition are as
follows.

1. For the steepest descent with Wolfe line-search, cosθk ¼ 1 for all k. Thus, the steepest descent
method is globally convergent only if the stepsizes are adequately computed.
2. Consider the Newton-type methods, where the search direction is computed as d k ¼ B1 k gk ,
where Bk is a nonsingular symmetric matrix (Bk ¼ I, Bk ¼ ∇2f(xk) or Bk is a symmetric and positive
definite approximation of the Hessian ∇2f(xk)). Assuming that the condition number of the matrices

Bk is uniformly bounded, i.e., for any k, kBk k B1
k
 Δ, where Δ > 0 is a constant, then from
(5.55) it follows that
5.3 General Convergence Results for Nonlinear Conjugate Gradient Methods 191

gTk dk gTk B1 g

cos θk ¼  ¼ k k
kgk kkdk k kgk k B1
k gk

2
1 kgk k 1 1 1
 kg k ¼ kBk k B1  Δ :
kgk k kBk k B1
k k k

Hence, cosθk  1/Δ, i.e., it is bounded away from 0. Therefore, the Newton method or the quasi-
Newton methods are globally convergent if the matrices Bk are positive definite (descent condition), if
their condition number is bounded and if the line-search satisfies the Wolfe conditions. Observe that
the condition (5.62) is crucial for obtaining these results.

3. For the conjugate gradient methods, it is not possible to show the limit (5.63), but only a weaker
result, that is,

lim inf kgk k ¼ 0: ð5:64Þ

k!1

This type of result is also obtained from the Zoutendijk condition. Indeed, suppose that (5.64) does
not hold, i.e., the gradients remain bounded away from zero. In other words, suppose that there exists
a constant γ > 0 so that for any k,

kgk k  γ: ð5:65Þ

In this case, from the Zoutendijk condition (5.57) again, it follows that
cos θk ! 0: ð5:66Þ

Therefore, the algorithm can only fail in the sense of (5.65) if the sequence {cosθk} converges to
 
zero. Hence, to establish (5.64) suffice it to show that a subsequence cos θkj of the sequence
{cosθk} is bounded away from zero.
Let us now present some conditions on βk which determine the convergence of the conjugate
gradient algorithms. Suppose that βk  0 and the search direction dk is a descent direction, i.e.,
gTk dk < 0: At this moment, we are interested in finding a βk which produces a descent direction dkþ1,
i.e., a direction which satisfies
2
gTkþ1 dkþ1 ¼  gkþ1 þ βk gTkþ1 dk < 0: ð5:67Þ

Proposition 5.9 Suppose that βk  0. If

g 2
kþ1
βk  , ð5:68Þ
dTk yk

then dkþ1 is a descent direction for function f.

Proof Since gTk dk < 0 and βk  0 for all k  1, we can write

192 5 Conjugate Gradient Methods

2 2
 gkþ1 þ βk gTkþ1 dk ¼  gkþ1 þ βk gTkþ1 dk  βk gTk dk þ βk gTk dk
2
¼  gkþ1 þ βk dTk yk þ βk gTk dk ð5:69Þ
2
  gkþ1 þ βk dTk yk ,

where yk ¼ gk  gk1. Therefore, the non-positivity of (5.69) is sufficient to show that the condition
(5.67) holds. Hence,

g 2  βk d T y , ð5:70Þ
kþ1 k k

proving the proposition. ♦

The following theorem proved by Dai and Yuan (2001) shows how βk selected to satisfy (5.68)
determines the convergence of the conjugate gradient algorithm.

Theorem 5.4 Suppose that the Assumption CG holds. Let {xk} be the sequence generated by the
algorithm (5.47) and (5.48), where βk satisfies (5.68). Then, the algorithm either determines a
stationary point or it converges in the sense that

lim inf kgk k ¼ 0:

k!1

Proof From (5.48) we get dkþ1 þ gkþ1 ¼ βkdk. Squaring both sides of this relation results in
2
kd kþ1 k2 ¼ β2k kd k k2  2gTkþ1 dkþ1  gkþ1 : ð5:71Þ

2
From (5.71), dividing both sides by gTkþ1 dkþ1 , it follows that

kdkþ1 k2 ð β k Þ 2 kd k k2 gTkþ1 dkþ1 g 2
kþ1

2 ¼
2  2
T 2 
T 2
gTkþ1 d kþ1 gTkþ1 dkþ1 gkþ1 dkþ1 gkþ1 d kþ1

ð β Þ 2 kd k k2 2 g 2
kþ1
¼
k 2  
T 2
gTkþ1 dkþ1 gTkþ1 dkþ1 gkþ1 dkþ1
!
ð β k Þ 2 kd k k2 1 g 2 1
¼
2  þ T kþ1
þ 2
T
gkþ1 dkþ1 gkþ1 g d gkþ1
kþ1 kþ1
ðβ Þ2 kdk k2 1

k 2 þ
gTkþ1 dkþ1 g 2

2 ! kþ1
ðβk Þ2 gTk d k kd k k2 1
¼

2 þ :
gT d kþ1
2
gT d k g 2
kþ1 k kþ1

Now, using (5.48) again,

0 1

2
kdkþ1 k 2
B ðβk Þ2 gTk d k C kd k k
2
1

  @ 2  A
 þ : ð5:72Þ
2 2 2 g 2
 gkþ1 þ βk gTkþ1 d k
T
gkþ1 d kþ1 T
gk d k kþ1
5.3 General Convergence Results for Nonlinear Conjugate Gradient Methods 193

Therefore, from the above relations and from (5.68),

0 10 1
B 2 CB 2 C
@ gkþ1 þ βk gTkþ1 d k þ βk gTk d k A@ gkþ1 þ βk gTkþ1 dk  βk gTk dk A  0:
|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl} |fflffl
ffl {zfflffl
ffl } |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}
<0 <0 0

Hence,

2
ðβk Þ2 gTk d k
2 2  1: ð5:73Þ
 gkþ1 þ βk gTkþ1 d k

From (5.72) and (5.73) the following inequality is obtained:

kdkþ1 k2 kd k k2 1

 
2 þ : ð5:74Þ
gTkþ1 dkþ1
2
gTk dk g 2
kþ1

Observe that

kd 0 k2 1

T 2 ¼ :
g0 d 0 kg0 k2

Therefore, from (5.74) it follows that

k d k k2 X 1 k

T 2  2
ð5:75Þ
gk d k i¼0 kgi k

for all k. Now, if the theorem is not true, then there exists a constant γ > 0 so that kgkk  γ for all k,
i.e., from (5.75),

T 2
gk d k γ2
 ,
kd k k2 kþ1

which implies

T 2
X
1
gk d k
¼ 1,
k¼0 kd k k2

thus contradicting the Zoutendijk condition. ♦

Convergence Under the Strong Wolfe Line-Search

The following theorem shows that if βk is chosen to satisfy the condition (5.68) for all k, then, under
the strong Wolfe line-search conditions (5.49) and (5.51), the direction (5.48) satisfies the sufficient
descent condition (5.54).

Theorem 5.5 Suppose that x0 is an initial point and the Assumption CG holds. Let {xk} be the
sequence generated by the conjugate gradient algorithm (5.47) and (5.48). If βk is so that
194 5 Conjugate Gradient Methods

g 2  βk dT y and βk  0 and the stepsize αk satisfies the strong Wolfe conditions (5.49) and
kþ1 k k
(5.51), then the conjugate gradient algorithm satisfies the sufficient descent condition (5.54) with
c ¼ 1/(1 þ σ).

Proof The proof is given by induction as follows. Observe that σ > 0 implies that 1 <  1/(1 þ σ).
Therefore, for k ¼ 1 the conclusion holds, since
1
gT0 d0 ¼ kg0 k2   kg k2 ¼ ckg0 k2 ,
1þσ 0
where c ¼ 1/(1 þ σ). Suppose that (5.54) holds for some k  0. Then, from the second strong Wolfe
condition (5.51), it follows that

gTkþ1 dk
lk  2 ½σ, σ and dTk yk > 0:
gTk dk

Hence,
gTkþ1 dk gTkþ1 dk  gTk dk d Tk yk
lk  1 ¼  1 ¼ ¼ T 6¼ 0: ð5:76Þ
gTk d k gTk d k gk d k

By (5.48) it results that

 2
gTkþ1 dkþ1 ¼ gTkþ1 gkþ1 þ βk dk ¼  gkþ1 þ βk gTkþ1 dk :

Following the sign of gTkþ1 dk , the following two cases arise.

(i) The case gTkþ1 dk  0: Hence,

2 2 1
gTkþ1 dkþ1 ¼  gkþ1 þ βk gTkþ1 dk   gkþ1   g 2 : ð5:77Þ
1 þ σ kþ1

(ii) The case gTkþ1 dk > 0: Using the conditions on βk and (5.76), it follows that

2
gTkþ1 dkþ1 ¼  gkþ1 þ βk gTkþ1 dk
 
2 gkþ1 2 T gTkþ1 dk

  gkþ1 þ T gkþ1 dk ¼ 1 þ T g 2
kþ1
 T  d k yk d k yk
gk d k 2
g ¼ 1 g 2 :

¼
dTk yk
kþ1 lk  1 kþ1

But
1 1 1
  : ð5:78Þ
1 þ σ 1  lk 1  σ

From (5.78),
1
g 2   1 g 2 :

gTkþ1 dkþ1  ð5:79Þ
lk  1 kþ1 1 þ σ kþ1

From (5.77) and (5.79) it follows that at the iteration k þ 1 the sufficient descent condition is
satisfied with c ¼ 1/(1 þ σ). ♦
5.3 General Convergence Results for Nonlinear Conjugate Gradient Methods 195

The following theorem, which introduces the Nocedal condition, presents a general convergence
result for any conjugate gradient method (5.47) and (5.48) under the strong Wolfe line-search (5.49)
and (5.51). The theorem mainly says that, if kdkk2 is at most linearly increasing, i.e., if kdkk2  c1k þ c2
for all k, where c1 and c2 are some constants, then a conjugate gradient method with strong Wolfe
line-search is globally convergent. The theorem is proved by Dai (2011). Also see (Nocedal, 1996).

Theorem 5.6 Suppose that the Assumption CG holds. Consider any conjugate gradient method
(5.47) and (5.48) with dk satisfying gTk d k < 0 and with strong Wolfe line-search (5.49) and (5.51).
Then the method is globally convergent if
X
1
1
¼ 1: ð5:80Þ
k¼1 kd k k2

Proof From (5.53) and (5.51) it follows that

 T   
g dkþ1  þ σ jβk jgT dk   g 2 : ð5:81Þ
kþ1 k kþ1

Applying the following inequality (a þ σb)2  (1 þ σ 2)(a2 þ b2) valid for all a, b, σ  0, with
   
a ¼ gTkþ1 d kþ1  and b ¼ jβk jgTk dk , (5.81) yields to

2
2 4
gTkþ1 dkþ1 þ β2k gTk d k  c gkþ1 , ð5:82Þ

where c ¼ (1 þ σ 2)1 is a positive constant.

Now, from (5.48) it follows that dkþ1 þ gkþ1 ¼ βkdk, for all k  1. Squaring both sides of this
equality, the following is obtained:
2
kdkþ1 k2 ¼  gkþ1  2gTkþ1 d kþ1 þ β2k kd k k2 :

Since gTkþ1 dkþ1 < 0, it follows that

2
kd kþ1 k2   gkþ1 þ β2k kdk k2 : ð5:83Þ

Hence, from (5.82) and (5.83),

T 2
T 2 " #
gkþ1 dkþ1 gk d k 1
T 2 kd kþ1 k2
T 2
þ ¼ gkþ1 dkþ1 þ gk d k
kdkþ1 k2 kd k k2 kdkþ1 k2 kd k k2
"
T 2 #
1
T 2
 g d
 gkþ1 dkþ1 þ β2k gTk dk 
2 k k g 2
ð5:84Þ
kþ1
kdkþ1 k2 kd k k2
"
T 2 #
1 4 g dk 2
 2
c gkþ1  k 2 gkþ1 :
kdkþ1 k kd k k

Assume that (5.64) is not true and there exists a constant γ > 0 so that for any k,

kgk k  γ: ð5:85Þ

Observe that the Zoutendijk condition (5.61) implies that gTk dk =kdk k tends to zero. Therefore, by
(5.84) and (5.85), for sufficiently large k it results that
196 5 Conjugate Gradient Methods

2
2
gTkþ1 dkþ1 gTk dk g 4
kþ1
þ c : ð5:86Þ
kd kþ1 k2 kd k k2 kdkþ1 k2

Thus, by the Zoutendijk condition and (5.85), we must have

X
1
1 1 X kgk k
1 2
 < 1,
k¼1 kd k k2 γ 2 k¼1 kdk k2

which is a contradiction to the assumption (5.80). Therefore, the convergence relation (5.64) holds.♦

The theorem says that the iterations of the conjugate gradient method can fail in the sense of (5.85)
only if kdkk ! 1 rapidly enough. More exactly, the sequence of the gradient norms kgkk can be
bounded away from zero only if
X
1
1
< 1: ð5:87Þ
k¼1 kd k k2

In the following, we shall provide a condition on βk which is sufficient for the global convergence
of the general conjugate gradient method with strong Wolfe line-search (Dai, 2010). By Theorem 5.6
we know that, if (5.80) holds, then the conjugate gradient method is convergent in the sense of (5.64).
Otherwise, we have (5.87), which gives

lim kdk k ¼ 1: ð5:88Þ

k!1

Therefore, from the Assumption CG, it follows that

kgk k  Γ, ð5:89Þ

for some Γ > 0 and for all k  0. Now, from (5.48) and the above relations (5.88) and (5.89), it follows
that

kdkþ1 k
jβk jkdk k: ð5:90Þ

Therefore, if the scalars βk are so that

X
1 Y
k
β2
j ¼ 1, ð5:91Þ
k¼1 j¼0

it is possible to establish the Nocedal condition (5.80) and then, by Theorem 5.6 a contradiction to
(5.87) is obtained. The following theorem details these discussions.

Theorem 5.7 Suppose that the Assumption CG holds. Consider a conjugate gradient method given
by (5.47) and (5.48), where the search direction dk is descent, i.e., dTk gk < 0: Consider that the
stepsize is determined by the strong Wolfe line-search conditions (5.49) and (5.51). If βk satisfies
(5.91), then lim inf kgk k ¼ 0:
k!1

Proof Observe that (5.48) can be written as

5.3 General Convergence Results for Nonlinear Conjugate Gradient Methods 197

dkþ1 þ gkþ1 ¼ βk dk : ð5:92Þ

Squaring both sides of (5.92) leads to

2
kdkþ1 k2 ¼ 2gTkþ1 dkþ1  gkþ1 þ β2k kd k k2 : ð5:93Þ

But,

T 2
2 g d
 gkþ1 
kþ1
2gTkþ1 d kþ1 kþ1
: ð5:94Þ
g 2
kþ1

Therefore, from (5.93) and (5.94), it results that

T 2
gkþ1 d kþ1
kd kþ1 k 
2
þ β k kd k k :
2 2
ð5:95Þ
g 2
kþ1

Having in view the definition of cosθk in (5.55), from (5.95) it follows that

1
kd kþ1 k2  1  cos 2 θkþ1 β2k kdk k2  ⋯
! !
Y
kþ1
1 Yk
 1  cos θj
2
β j kd 0 k2 :
2
ð5:96Þ
j¼1 j¼0

Suppose that liminfk!1kgkk 6¼ 0. Therefore, there exists a constant γ > 0 so that

kgk k  γ, for all k  0: ð5:97Þ

Hence, from the Zoutendijk condition (5.61), from the definition of cosθk and from (5.97), it
follows that
X
1
cos 2 θk < 1: ð5:98Þ
k¼1

With this, (5.98) implies that

Y
kþ1

1  cos 2 θj  c, ð5:99Þ
j¼1

where c > 0 is a constant. Hence

!
1 Y
k

2
c β2
j kd 0 k2 : ð5:100Þ
kdkþ1 k j¼0

From (5.100) and (5.91) it follows that (5.80) holds. Thus, by Theorem 5.6, lim inf kgk k ¼ 0: But
k!1
this, together with (5.97), gives a contradiction, thus proving the theorem. ♦

Theorem 5.7 shows that the global convergence of any conjugate gradient algorithm is obtained if
the conjugate gradient parameters βk satisfy the condition (5.91) and the stepsize is determined by the
strong Wolfe line-search. Mainly, it is based on the Zoutendijk condition. It should be mentioned that
198 5 Conjugate Gradient Methods

in Theorem 5.7, it is the descent condition (5.52) which is used and not the sufficient descent
condition (5.54).

Convergence Under the Wolfe Line-Search

Dai (2010) proved that the conclusion of Theorem 5.7 for the global convergence of any conjugate
gradient method also holds under the Wolfe line-search. This result is based on the following
proposition proved by Dai and Yuan (2003).

Proposition 5.10 Consider any conjugate gradient method (5.47) and (5.48). Define ψk and tk as
follows
(
kgk k2 , for k ¼ 0,
ψk ¼
2
ð5:101Þ
β0 β1 ⋯βk1 ,
2 2 2
for k  1

and

kd k k2
tk ¼ : ð5:102Þ
ψ 2k

Then, for all k  0,

Xk
gTi di X k
kgi k2
tk ¼ 2  : ð5:103Þ
i¼0
ψ 2i i¼0
ψ 2i

Proof Since d0 ¼  g0 and ψ 20 ¼ kg0 k2 , (5.103) holds for k ¼ 0. For k  1, dividing (5.93) by ψ 2kþ1
and using the definitions of ψ k and tk, it follows that
2
gTkþ1 dkþ1 gkþ1
tkþ1 ¼ tk  2  : ð5:104Þ
ψ 2kþ1 ψ 2kþ1

Summing (5.104) results in

X gi d i X
kþ1 T kþ1
kgi k2
tkþ1 ¼ t0  2  : ð5:105Þ
i¼1
ψ 2i i¼1
ψ 2i

Since t0 ¼ kg0 k2 =ψ 20 , it follows that (5.105) is equivalent to (5.103). Therefore (5.103) holds for
any k  0. ♦

To show the sufficiency of the condition (5.91) on βk, the following proposition is needed.
5.3 General Convergence Results for Nonlinear Conjugate Gradient Methods 199

Proposition 5.11 Suppose that {ai} and {bi} are two real positive number sequences satisfying:

X
k
bk  c 1 þ c 2 ai , for all k,
i¼1
P P
where c1 and c2 are positive constants. If ak is divergent, then ak =bk is also divergent.
k1 k1

P
Proof Let Sk ¼ ki¼1 ai , for any k  1. Now, ∑i1ai ¼ 1 implies that limk!1Sk ¼ 1 . Observe that
the sequence {Sk} is increasing. Let c ¼ c1/c2. There exists k0  1 so that Sk  c for k  k0. Note that
X1
ak X
1
ak 1 X ak
1
1 X ak
1
1 X ak
1
 ¼  ¼ :
k¼k
bk k¼k c1 þ c2 Sk c2 k¼k c þ Sk c2 k¼k Sk þ Sk 2c2 k¼k Sk
0 0 0 0 0

Let
Xr
akþi
Aðk, r Þ ¼ , for r  1:
S
i¼1 kþi

It is easy to see that for every k  1 there exists r  1 so that Aðk, r Þ  12 : Indeed,
Xr
akþi Skþr  Sk S
Aðk, r Þ  ¼ ¼1 k :
S
i¼1 kþr
S kþr S kþr

But,
 
S
lim Aðk, r Þ  lim 1  k ¼ 1:
k!1 k!1 Skþr

Hence, there exists r  1 so that A(k, r)  1/2.

Therefore, there is a sequence {rk} with rk  1 so that k1 ¼ k0 þ r1, k2 ¼ k1 þ r2, . . . and
Aðk0 , r 1 Þ  12 , Aðk1 , r 2 Þ  12 , . . . Hence,
X1
ak X
1 X1
1
¼ Aðki , r iþ1 Þ  ¼ 1: ♦
k¼k
S k i¼0 i¼0
2
0

The following theorem proved by Dai (2010) shows that the condition (5.91) on βk is sufficient for
the global convergence of any conjugate gradient method (5.47) and (5.48).

Theorem 5.8 Suppose that the Assumption CG holds. Consider a conjugate gradient method given
by (5.47) and (5.48), where the search direction dk is descent, i.e., d Tk gk < 0 and the stepsize is
determined by the Wolfe line-search conditions (5.49) and (5.50). If βk satisfies (5.91), then
lim inf kgk k ¼ 0:
k!1
200 5 Conjugate Gradient Methods

Proof Define ψ 2k as in (5.101). Then, from (5.91) it follows that

X 1
¼ 1: ð5:106Þ
ψ2
k1 k

Now, using (5.94) in (5.103) we can write

T 2
X
k
gi d i
tk  : ð5:107Þ
i¼0 kgi k2 ψ 2i

Since tkþ1  0, from (5.103) it follows that

Xk
gTi d i X k
kgi k2
2  : ð5:108Þ
i¼0
ψ 2i i¼0
ψ 2i
2
2
But gkþ1 þ 2 gTkþ1 d kþ1  0: Hence, for any k,

T 2
2 gkþ1 d kþ1
4gTkþ1 dkþ1
 gkþ1  4 : ð5:109Þ
g 2
kþ1

Therefore, from (5.108) and (5.109),

T 2
Xk
gi d i Xk
gTi d i X k
kgi k2 X k gi k2
k
4  4   : ð5:110Þ
i¼0 kgi k ψ i
2 2
i¼0
ψ 2i i¼0
ψ 2i i¼0
ψ 2i

Now, let us proceed by contradiction and assume that (5.97) holds. Then, by (5.110) and (5.106) it
follows that


X gTk dk 2
2 2
¼ 1: ð5:111Þ
k1 kgk k ψ k

From (5.111), (5.102) and from Proposition 5.11,

T 2
T 2
X 1
gk d k 1 X 1
gk d k X1

2 2 t
¼ ¼ cos 2 θk ¼ 1, ð5:112Þ
k¼1 k g k k ψ k
k k¼1 k g k k 2
k d k k 2
k¼1

which contradicts (5.98). This contradiction shows that (5.91) is true. ♦

Observe that the proof of this theorem is based on the Zoutendijk condition. Theorem 5.8 provides
the condition (5.91) on βk which is sufficient for the global convergence of the general conjugate
gradient method with Wolfe line-search. Besides, notice that only the descent condition gTk dk < 0 for
any k is used in Theorem 5.8. This is an important contribution to the general theory of conjugate
gradient methods, since very often, in the implementation of the conjugate gradient algorithms, only
the Wolfe line-search conditions are used.
The above theorems present the necessary and sufficient condition on the conjugate gradient
parameter βk, namely, (5.91), for the global convergence of any general conjugate gradient method
under the Wolfe line-search.
5.4 Standard Conjugate Gradient Methods 201

5.4 Standard Conjugate Gradient Methods

In the following, let us present the standard conjugate gradient algorithms as well as their conver-
gence results for solving unconstrained optimization problems. The standard conjugate gradient
methods are listed in Table 5.1. These methods have simple algebraic expressions, and their
convergence results are particularizations or specializations of the general results on the convergence
of conjugate gradient algorithms. In general, the convergence results of these algorithms are based on
the Assumption CG and on some other natural hypotheses on the line-search.
A rapid inspection of Table 5.1 shows that, except for Daniel’s method (1967) which requires the
evaluation of the Hessian at each iteration, the numerator of the update parameter βk is either kgkþ1k2
or gTkþ1 yk and the denominator is either kgkk2 or dTk yk or d Tk gk : Here, yk ¼ gkþ1  gk. Mainly, these
two choices for the numerator and the three choices for the denumerator lead to six different choices
for βk
If function f is strongly convex quadratic and the line-search is exact, then, in theory all the above
choices for the update parameter βk presented in Table 5.1 are equivalent. For non-quadratic objective
functions, each choice for βk leads to algorithms with different numerical performances (number of
iterations, number of function and its gradient evaluations, or CPU time). Therefore, in the following,
the global convergence properties of the standard conjugate gradient methods with the numerator
kgkþ1k2 for the update parameter βk (FR, CD, and DY) and with gTkþ1 yk in the numerator of βk (HS,
PRP, and LS) will be separately presented. As a general remark, the convergence theory for the
methods with numerator kgkþ1k2 is better developed than the theory for the methods with numerator
gTkþ1 yk of βk. However, the methods with gTkþ1 yk in the numerator of βk perform better in practice than

Table 5.1 Choices of bk in standard conjugate gradient methods

g kT1 yk Original linear conjugate gradient method

β kHS of Hestenes and Stiefel (1952)
d kT yk
g kT1 g k 1 First nonlinear conjugate gradient method
β kFR by Fletcher and Reeves (1964)
g kT g k
g kT1∇2 f ( xk )d k Proposed by Daniel (1967). This conjugate
β kD gradient method requires evaluation of the
d kT ∇2 f ( xk )d k Hessian
g kT1 yk Proposed by Polak and Ribière (1969) and
β kPRP by Polyak (1969)
g kT g k
Proposed by Powell (1984) and analyzed
g kT1 yk
β kPRP  max 0, by Gilbert and Nocedal (1992)
g kT g k
g kT1 g k 1 Proposed by Fletcher (1987), known as
βCD
k CD (conjugate descent) method
 d kT g k
g kT1 yk Proposed by Liu and Storey (1991)
β kLS
 d kT g k
g kT1 g k 1 Proposed by Dai and Yuan (1999)
β kDY
d kT yk
202 5 Conjugate Gradient Methods

the methods with kgkþ1k2 in the numerator of βk. The general algorithm for the standard conjugate
gradient methods is as follows.

Algorithm 5.4 General nonlinear conjugate gradient

1. Choose an initial point x0 and a convergence tolerance ε > 0 sufficiently small. Set d0 ¼  g0 and k ¼ 0
2. Test a criterion for stopping the iterations. If this test is satisfied, then stop; otherwise, continue with step
3
3. Using the Wolfe line-search conditions, determine the stepsize αk
4. Compute xkþ1 ¼ xk þ αkdk, fkþ1, gkþ1 and yk ¼ gkþ1  gk
5. Compute the conjugate gradient parameter βk
6. Compute the search direction dkþ1 ¼  gkþ1 þ βkdk
  2
7. Restarting criterion. If gT g  > 0:2 g then set dkþ1 ¼  gkþ1
kþ1 k kþ1
8. Set k ¼ k þ 1 and continue with step 2 ♦

Observe that a restarting criterion is used in step 6 of Algorithm 5.4. This is known as the Powell
restart criterion, introduced by Powell (1977). Crowder and Wolfe (1969) proved that the standard
conjugate gradient method without restart is at most linearly convergent. Yuan (1993) showed that
the convergence rate of the conjugate gradient method without restart is exactly linear for uniformly
convex quadratics. Cohen (1972) and McCormick and Ritter (1974) proved that the convergence rate
of the conjugate gradient method may be improved from linear to n-step quadratic if the method is
restarted with the negative gradient. Powell (1977) reported some numerical results with conjugate
gradient methods, showing that the immediate reduction in the objective function with restart is
usually smaller than the reduction without restart. Therefore, the current implementations of the
conjugate gradient methods use this restart criterion. Some restart procedures for conjugate gradient
methods can be found in Dai, Liao, and Li (2004).

Conjugate Gradient Methods with kgk11k2 in the Numerator of bk

In this section, let us discuss the methods Fletcher-Reeves (FR), Conjugate Descent by Fletcher (CD),
and Dai-Yuan (DY). A general characterization of these methods versus some other choices for the
update parameter βk is that the global convergence theorems require only the Lipschitz assumption,
not the boundedness assumption.

The Fletcher-Reeves Method

This conjugate gradient method is defined by
xkþ1 ¼ xk þ αk dk , ð5:113Þ

dkþ1 ¼ gkþ1 þ βk dk , ð5:114Þ

where the conjugate gradient parameter is computed as

g 2
kþ1
βk ¼
FR
: ð5:115Þ
kgk k2
5.4 Standard Conjugate Gradient Methods 203

The first global convergence result for the FR method under the exact line-search was given by
Zoutendijk (1970). In other words, when the stepsize αk is the exact solution of the problem
minα  0f(xk þ αdk), then the FR method is global convergent. As usual, let S ¼ {x 2 ℝn : f(x)  f(x0)}
be the level set.

Theorem 5.9 Suppose that f : ℝn ! ℝ is continuously differentiable on the level set S and consider
that the FR method is implemented with the exact line-search. Then, the sequence {xk} generated by
the algorithm has at least an accumulation point which is a stationary point of function f, that is:

(i) If {xk} is a finite sequence, then x is a stationary point of f.

(ii) If {xk} is an infinite sequence, then this sequence has a limit point, and any limit point is a
stationary point.
Proof
(i) If {xk} is a finite sequence, then, from the condition of the termination of iterations, it follows that
the final point x satisfies ∇f(x) ¼ 0, i.e., x is a stationary point of f.

(ii) If {xk} is an infinite sequence, then for any k, ∇f(xk) 6¼ 0. Since dkþ1 ¼  gkþ1 þ βkdk and the
line-search is exact, that is, gTkþ1 d k ¼ 0, it follows that

2 2
gTkþ1 dkþ1 ¼  gkþ1 þ βk gTkþ1 dk ¼  gkþ1 < 0, ð5:116Þ

i.e., dkþ1 is a descent direction, {f(xk)} is a monotone decreasing sequence, and consequently,
{xk} ⊂ S. Hence, {xk} is bounded and has a limit. Let x be the limit of {xk}. Then there exists a
subsequence fxk gK1 convergent to x, where K1 is the set of indices corresponding to the subsequence
from {xk}. Since fxk gK1 ⊂ fxk g, it follows that ff ðxk ÞgK1 ⊂ ff ðxk Þg: From the continuity of f we
know that for k 2 K1 (see Appendix A),
 
f ðx Þ ¼ f lim xk ¼ lim f ðxk Þ ¼ f  :

ð5:117Þ
k!1 k!1

Similarly, {xkþ1} is a bounded sequence. Therefore, there exists a subsequence fxkþ1 gK2 conver-
gent to x, where K2 is the set of indices corresponding to the subsequence from {xkþ1}. In this case,
 
f ðx Þ ¼ f lim xkþ1 ¼ lim f ðxkþ1 Þ ¼ f  :

ð5:118Þ
k!1 k!1

Hence,

f ðx  Þ ¼ f ðx  Þ ¼ f  : ð5:119Þ

Now, let us prove by contradiction that ∇f(x) ¼ 0. Suppose that ∇f(x) 6¼ 0. Then, for any α small
enough it follows that

f ðx þ αd  Þ < f ðx Þ: ð5:120Þ

Since for any α > 0,

f ðxkþ1 Þ ¼ f ðxk þ αk dk Þ  f ðxk þ αdk Þ,

then, for k 2 K2, at limit for k ! 1 from (5.120), it results that

204 5 Conjugate Gradient Methods

f ðx Þ  f ðx þ αd  Þ < f ðx Þ, ð5:121Þ

which contradicts (5.119). Therefore, this shows that ∇f(x) ¼ 0, that is, x is a stationary point of f.♦

Powell (1977) showed that the FR method with the exact line-search is susceptible of jamming,
i.e., along the iterations the algorithm could take many short steps without any significant progress to
the minimum. The modest performances of the FR method can be explained by this jamming
phenomenon, as it is detailed by Nocedal and Wright (2006).
The first global convergence result of the FR method under the inexact line-search was given by
Al-Baali (1985). Using the strong Wolfe line-search

f ðxk þ αk dk Þ  f ðxk Þ þ ραk dTk gk , ð5:122Þ

 
∇f ðxk þ αk dk ÞT dk   σdT g , ð5:123Þ
k k

with σ < 1/2, Al-Baali proved that the FR method generates sufficient descent directions.

Theorem 5.10 Suppose that the conjugate gradient FR is implemented with the strong Wolfe line-
search (5.122) and (5.123), where 0 < σ < 1/2. Then the FR method generates descent directions dk
satisfying the following inequalities:

1 gT d k 2σ  1
  k 2 , ð5:124Þ
1  σ kgk k 1σ

for all k ¼ 0, 1, . . .

Proof At first, notice that the function u(t) ¼ (2t  1)/(1  t) on the interval [0,1/2] is monotonically
increasing and u(0) ¼  1 and u(1/2) ¼ 0. Therefore, since σ 2 (0, 1/2), it follows that
2σ  1
1 < < 0: ð5:125Þ
1σ
The proof is given by induction as follows. For k ¼ 0 the middle term in (5.124) is 1, so, by using
(5.125) both inequalities in (5.124) are satisfied. Now, assume that (5.124) holds for some k  1.
From the definition of βFR
k given in (5.115) and from (5.114), it follows that

gTkþ1 dkþ1 gTkþ1 d k

2 ¼ 1 þ : ð5:126Þ
g kgk k2
kþ1

From the second Wolfe condition (5.123) and from (5.126),

gTk d kgT dkþ1 gT d k

1 þ σ  kþ1 2  1  σ k 2 :
kgk k 2 g kgk k
kþ1

Taking into account the left hand side of the induction hypothesis (5.124) and substituting it in the
above inequality, it follows that
5.4 Standard Conjugate Gradient Methods 205

σ gT d kþ1 σ
1   kþ1 2  1 þ ,
1σ g 1  σ
kþ1

which shows that (5.124) holds for k þ 1 as well. ♦

The theorem shows that for the FR conjugate gradient method, the sufficient descent condition
gTk dk ckgk k2 is satisfied. Therefore, from the Zoutendijk condition, it follows the global conver-
gence of the FR method. Besides, the bounds on gTk dk in (5.124) impose a limit on how fast kdkk can
grow along the iterations when the gradients are not small. For σ ¼ 1/2, dk is a descent direction;
however the analysis did not establish the sufficient descent.
The main difficulty in proving the global convergence of a conjugate gradient algorithm is to show
that the search direction dk is descent under the mild assumption on the line-search conditions. If the
line-search is exact, i.e., gTkþ1 d k ¼ 0, then dkþ1 is exactly the steepest descent, since gTkþ1 dkþ1 ¼
2
 g : Using this property and supposing that the Assumption CG holds, Powell showed that the
kþ1
FR algorithm is globally convergent, i.e., liminfk!1kgkk ¼ 0. Al-Baali extended this result, showing
that the descent property holds for all k if αk is determined by the strong Wolfe line-search.
The following theorem shows that under the inexact line-search, the search directions dk generated
by the FR method satisfy the descent condition gTk dk < 0:

Theorem 5.11 If for all k the stepsize αk is determined by the strong Wolfe line-search (5.122) and
(5.123), where σ 2 (0, 1/2), then for the FR method the following inequalities hold:

X
k
gTk dk X
k
 σj   2 þ σj ð5:127Þ
j¼0 kgk k2 j¼0

for any k. As soon as gk 6¼ 0 for all k, the descent property of dk is satisfied, i.e.,

gTk dk < 0: ð5:128Þ

Proof The theorem is proved by induction. For k ¼ 0, d0 ¼  g0, σ 0 ¼ 1 and therefore (5.127) and
(5.128) are true. Now, suppose that (5.127) and (5.128) are true for any k  0. From
dkþ1 ¼  gkþ1 þ βkdk and βk ¼ gTkþ1 gkþ1 =gTk gk (Fletcher-Reeves updating formula), it follows that

gTkþ1 dkþ1 gTkþ1 d k

2 ¼ 1 þ : ð5:129Þ
g kgk k2
kþ1
 
From the second strong Wolfe condition gTkþ1 d k   σgTk dk and from (5.128),

gTk d k gTkþ1 dkþ1 gTk dk

1 þ σ   1  σ : ð5:130Þ
kgk k2 g 2 kgk k2
kþ1

From (5.127) it follows that

206 5 Conjugate Gradient Methods

X
kþ1 X
k
gT dkþ1 X
kþ1
 σ j ¼ 1  σ σ j  kþ1 2  2 þ σj ,
j¼0 j¼0
g j¼0
kþ1

showing that (5.127) holds for k þ 1. On the other hand, since

gTkþ1 dkþ1 X
kþ1
2  2 þ σj ð5:131Þ
g j¼0
kþ1

and

X
kþ1 X
1
1
σj < σj ¼ , ð5:132Þ
j¼0 j¼0
1σ

where σ 2 [0, 1/2), form 1  σ > 1/2 it results that

X
kþ1
2 þ σ j < 0:
j¼0

Therefore, from (5.128), gTkþ1 dkþ1 < 0 is obtained, proving the theorem. ♦

With this result, the global convergence of the FR method with the inexact line-search can be
proved as in the following theorem (Al-Baali, 1985).

Theorem 5.12 Assume that the function f : ℝn ! ℝ is twice continuously differentiable and the level
set S is bounded. Suppose that αk is determined by the strong Wolfe conditions (5.122) and (5.123),
where ρ < σ < 1/2. Then the sequence {xk} generated by the Fletcher-Reeves method is global
convergent, i.e.,
lim inf kgk k ¼ 0: ð5:133Þ
k!1

Proof From the second strong Wolfe condition (5.123) and from the relations (5.127) and (5.132), it
follows that

 T  X
k
σ
g d k   σgT dk  σ kg k2 σj  kg k2 :
kþ1 k k
j¼0
1σ k

From the relations dkþ1 ¼  gkþ1 þ βkdk and βk ¼ gTkþ1 gkþ1 =gTk gk (Fletcher-Reeves formula), it
results that
2
FR 2
kdkþ1 k2 ¼ gkþ1  2βFRk gkþ1 d k þ βk
T
kd k k2
2 2σ

 gkþ1 þ g 2 þ βFR 2 kdk k2
 1  σ kþ1 k
1þσ 2
FR 2
g þ β
¼ kd k k2 :
1σ kþ1 k

But, from the Fletcher-Reeves updating formula (5.115), it is clear that

5.4 Standard Conjugate Gradient Methods 207

4

2
FR 2
FR 2 g
kþ1
βFR
k βk1 ⋯ βki ¼ :
kgki k4

Therefore, by recurrence, the following is obtained:

!
 kþ1 2
4 X
1þσ g
kdkþ1 k  2
gj : ð5:134Þ
1σ kþ1
j¼0

Now, let us prove (5.133) by contradiction. Suppose that (5.133) is not true. Then there exists a
positive constant ε > 0 so that for all k sufficiently large,

kgk k  ε > 0: ð5:135Þ

Since on the level set S the gradient gk is upper bounded, from (5.134) it follows that

kdk k2  c1 k, ð5:136Þ

where c1 is a positive constant. Hence, from (5.127) and (5.132),

!
Xk
gTk dk j kgk k
cos θk ¼   2 σ
kgk kkd k k kd k k
 kg k
j¼0 ð5:137Þ
1  2σ
 k
:
1  σ kd k k

Since σ < 1/2, introducing (5.135) and (5.136) in (5.137), it results that
X  2 X kg k2 X1
1  2σ
cos 2 θk  k
 c2 , ð5:138Þ
k
1σ k kd k k
2
k
k

where c2 is a positive constant. Therefore, the series ∑kcos2θk is divergent. Let M be an upper bound
of k∇2f(x)k on the level set S. Then,

T
gTkþ1 dk ¼ gk þ αk ∇2 f ðxk Þd k dk  gTk d k þ αk Mkdk k2 :

From the second strong Wolfe condition (5.123) written as

σgTk dk  gTkþ1 dk  σgTk dk

it follows that
1σ T
αk   gk d k : ð5:139Þ
M kd k k2

Introducing this value of αk from (5.139) in the first Wolfe condition (5.122), it results that
 2
ð1  σ Þρ gTk dk
f ðxkþ1 Þ  f ðxk Þ  ¼ f ðxk Þ  c3 kgk k2 cos 2 θk ,
M kd k k

where c3 ¼ ð1σ
M
Þρ
> 0. Since f is lower bounded, it follows that the series ∑kkgkk2cos2θ2 is conver-
gent. Hence, from (5.135), the series ∑kcos2θ2 is convergent. But this contradicts (5.138), thus
proving the theorem. ♦
208 5 Conjugate Gradient Methods

The CD Method
The CD (Conjugate Descent) method elaborated by Fletcher (1987) is very close to the FR method.
The conjugate gradient parameter βk is computed as

g 2
kþ1
βk ¼
CD
: ð5:140Þ
d Tk gk

Under the exact line-search, βCD

k ¼ βk : The difference between CD and FR is that in CD with
FR

strong Wolfe line-search, the sufficient descent condition gTk dk  ckgk k2 holds. In this case, the
constraint σ  1/2 that arose in the FR method is not necessary for the CD method. For a line-search
that satisfies the generalized Wolfe conditions σ 1 d Tk gk  dTk gkþ1  σ 2 dTk gk , with σ 1 < 1 and σ 2 ¼ 0,
it can be shown that 0  βCDk  βk : Therefore, from the analysis given by Al-Baali (1985) the global
FR

convergence is achieved. On the other hand, if σ 1  1 or σ 2 > 0, then Dai and Yuan (1996) constructed
numerical examples where kdkk2 increases exponentially, and therefore the CD method converges to
a point where the gradient does not vanish. In particular, the CD method may not converge to a
stationary point under the strong Wolfe line-search. Details are given by Hager and Zhang (2006b).

The Dai-Yuan Method

In the DY method elaborated by Dai and Yuan (1999), the conjugate gradient parameter βk is
computed as

g 2
kþ1
βk ¼
DY
: ð5:141Þ
d Tk yk

The DY method always generates descent directions under the Wolfe line-search. Besides, when
the Lipschitz assumption holds, the DY is globally convergent.
To get (5.141), let us suppose that the current search direction dk is descent, i.e., d Tk gk < 0 and
we are interested in finding a βk for which the new search direction dkþ1 is also descent. This requires
that
2
 gkþ1 þ βk gTkþ1 dk < 0:

Assume that βk > 0 and denote τk ¼ kgkþ1k2/βk. Then the above inequality is equivalent to τk >
gTkþ1 dk : Therefore, we can consider τk ¼ dTk yk , which gives the DY updating formula (5.141).
A new representation of βDY
k can immediately be obtained. From (5.114) and (5.141), the following
can be written:

g 2
kþ1
gkþ1 dkþ1 ¼
T
gT dk ¼ βDY
k gk d k :
T
ð5:142Þ
dTk yk k

Therefore, βDY
k is

gTkþ1 dkþ1
βDY
k ¼ : ð5:143Þ
gTk dk

Observe that (5.141) is well defined because the Wolfe line-search implies that d Tk yk > 0: If the
line-search in this method is exact, then the DY method is the same as the FR method. The
convergence of this method is given by the following theorem.
5.4 Standard Conjugate Gradient Methods 209

Theorem 5.13 Suppose that the initial point x0 satisfies the Assumption CG and let {xk} be the
sequence generated by the general algorithm (5.113) and (5.114), where βk is computed as in (5.143).
Then the algorithm either terminates at a stationary point of f or converges in the sense

lim inf kgk k ¼ 0: ð5:144Þ

k!1

Proof If the algorithm does not terminate after a finite number of iterations, then kgkk > 0 for all k.
Firstly, let us show that the search directions are descent, i.e.,

gTk dk < 0 ð5:145Þ

for all k. For k ¼ 1 the above inequality (5.145) is satisfied. Now, let us prove it for all k > 1 by
induction. Assume that (5.145) holds for k. From the second Wolfe line-search condition
∇f ðxk þ αk d k ÞT d k  σdTk gk , it follows that

dTk yk  ðσ  1ÞdTk gk > 0: ð5:146Þ

From (5.142) it follows that (5.145) holds for k þ 1. Hence, the search direction given by βDY
k is
descent for all the values of k.
Now, (5.114) can be written as

dkþ1 þ gkþ1 ¼ βk dk : ð5:147Þ

Squaring both sides of (5.147), it follows that

2
kdkþ1 k2 ¼ β2k kdk k2  2gTkþ1 dkþ1  gkþ1 : ð5:148Þ

2
Dividing both sides of (5.148) by gTkþ1 d kþ1 and applying (5.143),

kdkþ1 k2 kd k k2 2 g 2
kþ1

T 2 ¼
T 2  T 
2
gkþ1 d kþ1 gk d k gkþ1 d kþ1 gTkþ1 d kþ1
!
kd k k2 1 g 2 1
¼
2  þ T kþ1
þ 2 ð5:149Þ
T
g dk g g d kþ1 g
k kþ1 kþ1 kþ1
kd k2 1

k 2 þ 2 :
T
gk d k gkþ1


But, kd0 k2 = gT0 d0 ¼ 1=kg0 k2 , then (5.149) shows that

kd k k2 X 1 k

T 2  2
, ð5:150Þ
gk d k i¼0 kgi k

for all k. If the theorem is not true, then there exists a positive constant c > 0 so that

kgk k  c ð5:151Þ

for all k. Therefore, from (5.150) and (5.151), it follows that

210 5 Conjugate Gradient Methods

kd k k2 k

T 2  2 ,
gk d k c

which implies that


X gTk dk 2
¼ 1: ð5:152Þ
k0 kd k k2

Observe that the relation (5.152) contradicts the condition (5.61). This contradiction proves the
theorem. ♦

Conjugate Gradient Methods with gTkþ1 y k in the Numerator of bk

In the following, let us consider the conjugate gradient methods Hestenes-Stiefel (HS), Polak-
Ribière-Polyak (PRP), and Liu-Storey (LS). Even if for the methods with kgkþ1k2 in the numerator
of βk a strong convergence theory has been developed, these methods have modest performances,
mainly because of the jamming phenomenon. Namely, these methods begin to take small steps
without making significant progress to the minimum. On the other hand, the HS, PRP, and LS
methods, having gTkþ1 yk in the numerator of βk, possess a built-in restart feature that addresses the
jamming. When the step sk ¼ xkþ1  xk is small (i.e., close to the minimum), then the factor
yk ¼ gkþ1  gk in the numerator of βk tends to zero. Hence, βk becomes small, and the new search
direction dkþ1 is essentially the steepest descent direction gkþ1. This property of the PRP method is
important for its analysis. Such a method with this property is said to have Property (*). These
methods automatically adjust βk in order to avoid jamming. Therefore, the numerical performance of
these methods is better than the numerical performance of the methods with kgkþ1k2 in the numerator
of βk.

The Polak-Ribière-Polyak Method

The Polak-Ribière-Polyak conjugate gradient method is defined by (5.113) and (5.114), where the
conjugate gradient parameter is computed as

gTkþ1 yk
βPRP
k ¼ : ð5:153Þ
gTk gk

If f is strongly convex and the line-search is exact, then Polak and Ribière (1969) and Polyak
(1969) established the global convergence of the PRP method. Powell (1977) proved that for a
general nonlinear function f, if (i) the stepsize sk ¼ xkþ1  xk tends to zero, (ii) the line-search is exact
and (iii) the Lipschitz continuity holds, then the PRP method is globally convergent. On the other
hand, in a laborious paper Powell (1984) constructed a counter-example with three variables and
showed that the PRP method may cycle infinitely without approaching any solution. Therefore, the
assumption that the stepsize tends to zero is needed for convergence.
Later on, under the assumption that the search direction is a descent direction, Yuan (1993)
established the global convergence of the PRP method for strongly convex functions and the Wolfe
line-search. However, Dai (1997), in his Ph.D. Thesis, presented an example which showed that even
when the objective function is strongly convex and σ 2 (0, 1) is sufficiently small, the PRP method
may still fail by generating an ascent search direction. Dai, Han, Liu, Sun, Yin, and Yuan (1999)
constructed an example showing that the boundedness of the level set is necessary for the
5.4 Standard Conjugate Gradient Methods 211

convergence of the PRP method even if the line-search is exact. Therefore, the convergence of the
PRP method is not certain. However, this method proved to be one of the most efficient for solving
large-scale unconstrained optimization problems.
In order to prove the convergence of the PRP method, we need a technical result presented in the
following proposition.

Proposition 5.12 Let ∇f(x) be uniformly continuous on the level set S. Consider the angle θk between
dk and  ∇ f(xk), where dk is a descent direction. If
π
θk   μ, for such μ > 0, ð5:154Þ
2
then ∇f(xk) ¼ 0 for a certain k, or f(xk) !  1, or ∇f(xk) ! 0.

Proof Suppose that for all k, ∇f(xk) 6¼ 0 and that f(xk) is lower bounded. Since {f(xk)} is monotone
decreasing, this sequence has a limit. Hence
f ðxk Þ  f ðxkþ1 Þ ! 0: ð5:155Þ

Suppose, by contradiction, that ∇f(xk) ! 0 is not true. Then, there exists an ε > 0 so that
k∇f(xk)k  ε. Therefore,

∇f ðxk ÞT dk
 ¼ k∇f ðxk Þk cos θk  ε sin μ  ε1 ð5:156Þ
kd k k

Observe that

f ðxk þ αdk Þ ¼ f ðxk Þ þ α∇f ðξk ÞT dk

¼ f ðxk Þ þ α∇f ðxk ÞT dk þ α½∇f ðξk Þ  ∇f ðxk Þ T d k
  ð5:157Þ
∇f ðxk ÞT dk
 f ðxk Þ þ αkdk k þ k∇f ðξk Þ  ∇f ðxk Þk ,
kd k k

where ξk is on the line segment connecting xk and xk þ αdk. Since ∇f(x) is uniformly continuous on S,
there exists α so that, when 0  αkd k k  α we have
1
k∇f ðξk Þ  ∇f ðxk Þk  ε1 : ð5:158Þ
2
From (5.155)–(5.158) it follows that
   
dk ∇f ðxk ÞT d k 1 1
f xk þ α  f ðxk Þ þ α þ ε1  f ðxk Þ  αε1 :
k kk
d k kk
d 2 2

Hence,
 
d 1
f ðxkþ1 Þ  f xk þ α k  f ðxk Þ  αε1 ,
kd k k 2

which contradicts (5.155). Therefore, ∇f(xk) ! 0, thus completing the proof. ♦

With this, for strongly convex functions and under the exact line-search, the following result on the
convergence of the PRP method can be proved.
212 5 Conjugate Gradient Methods

Theorem 5.14 Consider f : ℝn ! ℝ twice continuously differentiable and suppose that the level set
S is bounded. Assume that there exists a positive constant m > 0 so that for x 2 S and any y 2 ℝn,

mkyk2  yT ∇2 f ðxÞy: ð5:159Þ

Then, the sequence {xk} generated by the PRP method with exact line-search converges to the
unique minimum x of function f.

Proof From Proposition 5.12 it follows that it is enough to prove (5.154), that is, there exists a
positive constant ω > 0, so that

gTkþ1 dkþ1  ω gkþ1 kdkþ1 k, ð5:160Þ

i.e., cosθk  ω > 0. Observe that gk ! 0 and g(x) ¼ 0. From (5.159) it follows that {xk} ! x, which
is the unique minimum of function f. Since the line-search is exact, from dkþ1 ¼  gkþ1 þ βkdk and
2
from the fact that gT dk ¼ 0, it follows that gT d kþ1 ¼  g : Then (5.160) is equivalent to
kþ1 kþ1 kþ1

g
kþ1
 ω: ð5:161Þ
kdkþ1 k

Having in view that dkþ1 ¼  gkþ1 þ βkdk and the line-search is exact, from (5.6) the following
value for αk is obtained

gTk dk kg k2
αk ¼  T ¼ Tk , ð5:162Þ
d k Ak d k d k Ak d k

where

Z1
Ak ¼ ∇2 f ðxk þ tαk d k Þdt: ð5:163Þ
0

Using the mean value theorem (see Appendix A), from (5.163),
gkþ1  gk ¼ ∇f ðxk þ αk dk Þ  ∇f ðxk Þ ¼ αk Ak dk : ð5:164Þ

Therefore, βPRP
k can be expressed as


gTkþ1 gkþ1  gk g T Ak d k
βPRP
k ¼ T ¼ αk kþ1 2
gk gk kgk k
T ð5:165Þ
gkþ1 Ak dk
¼ T :
d k Ak d k

Since the level set S is bounded, there exists a positive constant M > 0, so that for x 2 S and any
y 2 ℝ n,

yT AðxÞy  Mkyk2 ð5:166Þ

Using the above relations, the following bound for βPRP

k can be established:
5.4 Standard Conjugate Gradient Methods 213

 PRP  gkþ1 kAk dk k M gkþ1
β    : ð5:167Þ
k
m kd k k2 m kd k k

Hence,
 
kdkþ1 k  gkþ1 þ βPRP  kd k k

k
M M
 gkþ1 þ gkþ1 ¼ 1 þ g ,
kþ1
m m
that is,

g 
M 1
kþ1
 1þ , ð5:168Þ
kdkþ1 k m

showing that (5.161) holds, where ω ¼ m/(m þ M ). ♦

Powell (1984, 1986b) introduced the PRP+ method as

 
βPRPþ
k ¼ max 0, βPRP
k : ð5:169Þ

Later on, Gilbert and Nocedal (1992) proved the global convergence of the conjugate gradient
methods with nonnegative βk under inexact line-search. The PRP+ method was introduced to rectify
the convergence failure of the PRP method when implemented with Wolfe line-search. A modified
Polak-Ribiére-Polyak conjugate gradient algorithm for unconstrained optimization was given by
Andrei (2011a).

The Hestenes-Stiefel Method

The HS conjugate gradient method is defined by (5.113) and (5.114), where the conjugate gradient
parameter is computed as

gTkþ1 yk
βHS
k ¼ : ð5:170Þ
dTk yk

The HS method has the property that the conjugacy condition

d Tkþ1 yk ¼ 0 ð5:171Þ

is always satisfied, independent of the line-search used. For the exact line-search, βHS k ¼ βk .
PRP

Therefore, the global convergence properties of the HS method are similar to the global convergence
properties of the PRP method. In particular, by Powell’s example (1984), the HS method with exact
line-search may not converge for general nonlinear functions.

The Liu-Storey Method

This conjugate gradient method is defined by (5.113) and (5.114), where the conjugate gradient
parameter is computed as

gTkþ1 yk
βLS
k ¼ : ð5:172Þ
dTk gk
214 5 Conjugate Gradient Methods

For the exact line-search, the LS method is identical to the PRP method. Liu and Storey (1991)
studied this method, proving its global convergence. The techniques developed for the analysis of the
PRP method may be applied to the LS method.

Numerical Study: Standard Conjugate Gradient Methods

In this section, some numerical results with the standard conjugate gradient methods are presented.
For this, consider 80 unconstrained optimization problems from the UOP collection (Andrei, 2020a),
each with the number of variables n ¼ 1000, 2000, . . ., 10000. The comparisons of algorithms are
given in the context of Remark 1.1. The maximum number of iterations was limited to 2000.
Figures 5.5 and 5.6 present the Dolan and Moré’s performance profiles of the standard conjugate
gradient methods. The tables inside the plots show the performances of the algorithms subject to the
number of iterations (#iter), the number of function and its gradient evaluations (#fg), and subject to
the CPU time metric (cpu) in seconds. When comparing HS versus FR (see Fig. 5.5) subject to the
number of iterations, we can see that HS was better in 406 problems (i.e., it achieved the minimum
number of iterations in 406 problems). FR was better in 157 problems. Both methods achieved the
same number of iterations in 157 problems, etc. Out of 800 problems considered in this numerical
experiment, only for 720 problems does the criterion (1.3) hold. From Fig. 5.5 we can see that, subject
to the CPU time metric, HS is the fastest. Close to HS are PRP+ and LS. Observe that HS is much
better than FR, CD, and DY. One explanation is that the HS method satisfies the conjugacy condition
(5.171), independent of the line-search. From Figs. 5.5 and 5.6, we notice that PRP+ is more efficient
than FR, PRP, CD, and DY. Close to PRP+ is LS, but LS is slightly more robust. From Fig. 5.6 we can
see that LS is better than FR, CD, and DY. Close to CD is DY. At least, for this set of 800 uncon-
strained optimization problems, HS and PRP+, are the best methods. HS, PRP+ and LS automatically
adjust βk to avoid jamming. Figure 5.7 presents a global comparison of the standard conjugate
gradient methods. Note that there is an experimental confirmation of the classification of conjugate
gradient methods in two classes according to the formula for βk computation: with kgkþ1k2 or with
gTkþ1 yk at the numerator of βk.
We can see that HS, PRP, PRP+, and LS (all with gTkþ1 yk at the numerator of βk) are more efficient
and more robust subject to the CPU time metric than FR, CD, and DY (all with kgkþ1k2 at the
numerator of βk). Despite the strong convergence theory that has been developed for methods with
kgkþ1k2 in the numerator of βk, these methods have modest numerical performances in comparison
with the methods with gTkþ1 yk at the numerator of βk. Some more details on the performance profiles of
conjugate gradient algorithms for unconstrained optimization are presented by Andrei (2008e).
In the second set of numerical experiments, let us present the performances of the standard
conjugate gradient methods for solving five applications from the MINPACK-2 collection described
in Appendix D, where nx ¼ 200 and ny ¼ 200. The conjugate gradient algorithms implement the same
stopping criterion kgkk1  εg, where εg ¼ 106 and the stepsize is computed by the Wolfe line-search
(5.49) and (5.50) with ρ ¼ 0.0001 and σ ¼ 0.9.
Tables 5.2, 5.3, and 5.4 present the performances of the standard conjugate gradient methods for
solving these applications. In these tables, n is the number of variables of the application, #iter is the
number of iterations, #fg the number of function and its gradient evaluations, and cpu is the CPU time
(in seconds) for solving the application. The last line in these tables contains the total number of
iterations, the total number of function and its gradient evaluations, and the total CPU time for solving
these five applications for each conjugate gradient method. Observe that for solving them, CD, HS,
and PRP require the minimum time. For example, CD needs 137.09 seconds, HS 139.65, and PRP
5.4 Standard Conjugate Gradient Methods 215

Fig. 5.5 Performance profiles of the standard conjugate gradient methods

216 5 Conjugate Gradient Methods

Fig. 5.6 Performance profiles of the standard conjugate gradient methods

5.4 Standard Conjugate Gradient Methods 217

Fig. 5.7 Performance

profiles of seven standard
conjugate gradient methods

Table 5.2 Performances of HS, FR, and PRP for solving five applications from the MINPACK-2 collection

HS FR PRP
n #iter #fg cpu #iter #fg cpu #iter #fg cpu
A1 40,000 359 457 8.89 1082 1164 24.05 601 857 5.73
A2 40,000 1286 1650 34.65 2698 2885 35.40 1036 1477 12.84
A3 40,000 1510 1730 45.55 20001 25625 411.04 3001 4035 65.31
A4 40,000 841 1069 41.32 2070 2167 83.06 809 1148 43.91
A5 40,000 583 714 9.24 1830 1931 24.65 668 927 11.63
Total ─ 4579 5620 139.65 27681 33772 578.20 6115 8444 139.42

Table 5.3 Performances of PRP+ and CD for solving five applications from the MINPACK-2 collection

PRP+ CD
n #iter #fg cpu #iter #fg cpu
A1 40,000 604 862 6.67 944 1097 12.80
A2 40,000 2335 1901 21.05 1887 2138 32.17
A3 40,000 3219 4306 69.73 2148 2287 37.52
A4 40,000 915 1314 50.17 1032 1148 43.99
A5 40,000 810 1125 14.09 759 836 10.61
Total ─ 7883 9508 161.71 6770 7506 137.09

139.42 seconds. PRPþ and LS have comparable performances. However, FR and DY need the
longest time for solving all these five applications. Details on the performances of conjugate gradient
algorithms for solving the applications from the MINPACK-2 collection are found in (Andrei,
2006b).
218 5 Conjugate Gradient Methods

Table 5.4 Performances of LS and DY for solving five applications from the MINPACK-2 collection

LS DY
n #iter #fg cpu #iter #fg cpu
A1 40,000 642 886 10.51 464 488 6.08
A2 40,000 1085 1526 19.01 1031 1062 14.72
A3 40,000 2398 3120 50.76 8393 8423 141.57
A4 40,000 1455 2033 77.50 886 909 35.22
A5 40,000 591 821 10.25 2382 2410 31.61
Total ─ 6171 8386 168.03 13156 13292 229.20

5.5 Hybrid Conjugate Gradient Methods

The idea behind the hybrid conjugate gradient methods is to combine the standard conjugate gradient
methods in order to exploit the attractive features of each of them. The standard conjugate gradient
methods may be combined in two distinct ways, thus obtaining two classes of hybrid conjugate
gradient methods. The first class is based on the projection concept. They consider a pair of standard
conjugate gradient methods and use one of them as soon as a certain criterion has been satisfied.
When the criterion is violated, then the other conjugate gradient method from the pair is used. The
hybrid conjugate gradient methods based on the projection concept have a simple algebraic expres-
sion. As soon as a conjugate gradient method has entered a jamming phase, the hybrid scheme
triggers another conjugate gradient method in the pair, thus trying to improve the numerical
performances of the hybrid one. The second class is based on the convex combination of the standard
methods. In this case, some standard conjugate gradient methods are combined in a convex manner,
thus obtaining a hybrid conjugate gradient method. In general, the hybrid methods are more efficient
and more robust than the standard ones (Andrei, 2020a).

Hybrid Conjugate Gradient Methods Based on the Projection Concept

Table 5.5 presents some hybrid selections of the parameter βk based on the projection of the standard
conjugate gradient methods presented in Table 5.1.
As it can be seen, there is a large variety of hybrid conjugate gradient methods. Their purpose is to
combine the properties of the standard ones in order to get new ones, rapidly convergent to the
solution. The idea is to avoid jamming. As we know, the FR method has strong convergence
properties, but it may not perform well in computational experiments. On the other hand, although
the PRP method and the HS method may not generally converge, they often perform better than
FR. Therefore, the combination of these methods tries to exploit the attractive features of each one.
Thus, in the hybrid conjugate gradient method TAS proposed by Touati-Ahmed and Storey (1990), if
the iterations are affected by jamming, the method commutes from FR to PRP.
The same motivation is for the hybrid computational scheme PRP-FR proposed by Hu and Storey
(1991). Indeed, the PRP method possesses a built-in restart feature that directly addresses the
jamming. When the step sk is small, the factor yk in the numerator of βPRP k tends to zero. Hence, in
this case, the search direction dkþ1 computed as in (5.3) with βk ¼ βPRP k is essentially the steepest
descent direction gkþ1. The DY method has even better global convergence properties than the FR
method. Consequently, Dai and Yuan (2001) combined their algorithm with the HS algorithm and
5.5 Hybrid Conjugate Gradient Methods 219

Table 5.5 Hybrid selection of bk based on the projection concept

β kPRP , 0 β kPRP β kFR , Proposed by Touati-Ahmed and

βTAS
k
Storey (1990)
β kFR , otherwise.
β kPRP  FR ^ `
max 0, min ^β kPRP ,β kFR ` Proposed by Hu and Storey
(1991)
βGN
k max ^β , min ^β ,β ``
FR
k
PRP
k
FR
k
Proposed by Gilbert and
Nocedal (1992)
β kHS  DY max ^0, min ^β ,β `` HS
k
DY
k
Proposed by Dai and Yuan
(2001) and Dai and Ni (2003)
1  σ DY Proposed by Dai and Yuan
β hDY max  β k , min ^β kHS ,β kDY ` (2001). σ is the parameter from
1 σ
k
the second Wolfe line-search
condition
2 Proposed by Dai (2002a)
g k 1
β kDDF .
max{ d kT g k , d kT yk }
2 Proposed by Wei, Li, and Qi
μ 1 g k 1
βVFR
k 2 (2006). μ 1 ! 0, μ 3 ! 0, and
μ 2 g kT1d k  μ 3 g k
μ 2 ! μ 1 are parameters
2 Proposed by Wei, Yao and Liu
g k 1  g k 1 ( g kT1 g k ) / g k
βVPRP
k 2
(2006)
gk
2 Proposed by Yao, Wei, and
g k 1  g k 1 ( g kT1 g k ) / g k
βYWH
k
Huang (2007)
d kT yk
2 Proposed by Jiao, Chen, and Pan
β kDY , if g kT1d k g k 1 ,
β kJCP (2007)
β kFR , else.
β kLS CD ^
max 0, min ^β kLS ,βCD
k ` ` Proposed by Andrei (2008g)

2
g k 1  g k 1 ( g kT1 g k ) / g k Proposed by Dai and Wen
β kDPRP 2
(2012). μ ! 1 is a parameter
μ g kT1d k  g k
2 Proposed by Jian, Han, and Jiang
g k 1  g k 1 max{0,( g kT1 g k )} / g k
β kJHJ 2
(2015)
max{ g k , d kT yk }

proposed the hybrid scheme with βHSDY k or βhDY

k in (5.3). The HS-DY hybrid conjugate gradient
method was also discussed by Andrei (2008f). Gilbert and Nocedal (1992) showed that even for
strongly convex functions, it is quite possible for βPRP
k to be negative. Therefore, in order to extend the
number of iterations in which βPRPk is used, they suggested the hybrid method with βGNk in (5.3) to
maintain the global convergence.
The hybrid conjugate gradient method DDF, which employs either the DY or the FR standard
conjugate gradient algorithms, was proposed by Dai (2002a). He established that this hybrid scheme
generates descent directions, independent of the line-search.
220 5 Conjugate Gradient Methods

The VFR hybrid conjugate gradient, which is a variation of the FR method proposed by Wei, Li,
and Qi (2006a), satisfies the sufficient descent condition gTk d k  ð1  μ1 =μ2 Þkgk k2 independent of
the line-search.
The VPRP hybrid conjugate gradient method proposed by Wei, Yao, and Liu (2006b) is a variant
of the PRP method. This hybrid conjugate gradient method inherits the properties of PRP. Under the
strong Wolfe line-search with 0 < σ < 1/4, Huang, Wei, and Yao (2007) showed that the search
direction of the VPRP method satisfies the sufficient descent condition and the algorithm is globally
convergent. An interesting property of VPRP is that it has the Property (*), first introduced by Gilbert
and Nocedal (1992).
The YWH hybrid method proposed by Yao, Wei, and Huang (2007) is a variant of the HS method.
They established that under the strong Wolfe line-search with σ 2 (0, 1/3), the YWH method
generates sufficient descent directions.
Jiao, Chen, and Pan (2007) proposed a hybrid conjugate gradient method JCP and established the
global convergence under the Goldstein line-search (2.56).
The hybrid conjugate gradient method LS-CD was proposed by Andrei (2008g). Its global
convergence under the Wolfe line-search was proved by Yang, Luo, and Dai (2013). Numerical
experiments showed that the hybrid LS-CD method outperforms both the LS and the CD methods.
Dai and Wen (2012) proposed the hybrid DPRP method, where μ > 1. They proved that the
corresponding search direction satisfies the sufficient descent condition for any line-search. They also
proved that the DPRP method is globally convergent under the Wolfe line-search.
A hybridization of the HS and DY conjugate gradient methods was proposed by Babaie-Kafaki
and Ghanbari (2015). In their algorithm, the hybridization parameter is computed by solving the least-
squares problem of minimizing the distance between the search direction of the hybrid method and
the search direction corresponding to the three-term conjugate gradient method of Zhang, Zhou, and
Li (2007), which possesses the sufficient descent property.
The hybrid method JHJ proposed by Jian, Han, and Jiang (2015) generates sufficient descent
directions for any line-search and is globally convergent under the Wolfe line-search.
Other hybrid conjugate gradient methods were proposed by Zhang (2009a, 2009b), Han, Zhang,
and Chen (2017). Two descent hybrid conjugate gradient algorithms as a projection of PRP and HS,
and also of HS and DY, respectively, were developed by Zhang and Zhou (2008).
The following general result proved by Dai, Han, Liu, Sun, Yin, and Yuan (1999) shows that there
are a lot of possibilities to generate hybrid conjugate gradient methods by taking into consideration
the FR method.

Theorem 5.15 Suppose that the Assumption CG holds and consider any general method of the form
(5.47)–(5.48) where the stepsize is determined by the strong Wolfe line-search (5.49) and (5.51) with
0 < ρ < σ < 1 and βk satisfying

σ jβk j  σβFR
k , ð5:173Þ

with σ 2 ð0, 1=2 and

Xk Y k  2
g 2 βi
kþ1  c2 k, ð5:174Þ
j¼0 i¼j βFR
i

for some constant c2 > 0. Then,

lim inf kgk k ¼ 0: ð5:175Þ
k!1
5.5 Hybrid Conjugate Gradient Methods 221

Proof As known, βFR k ¼ gkþ1 gkþ1 =gk gk : Then, from (5.48), from the strong Wolfe line-search
T T
 
gðxk þ αk dk ÞT dk   σgT d k and from (5.173), it results that
k
  T
gTkþ1 d kþ1 gT d k βk gkþ1 dk
¼ 1  βk kþ1 2 ¼ 1  FR
g 2 g βk kgk k2
kþ1
 kþ1  !
 βk  σgTk dk gTk dk

 1 þ  FR   1þσ ð5:176Þ
βk kgk k2 kg k2
! k
Pk gT0 d 0 1  σ kþ1 1
⋯ σ j þ σ kþ1 ¼ < :
j¼0 kg 0 k 2 1  σ 1  σ

Similarly, since σ  1=2,

gTkþ1 dkþ1 1  σk
2  1  σ > 0: ð5:177Þ
g 1σ
kþ1

Therefore, dk is a descent direction. Since dkþ1 þ gkþ1 ¼ βkdk, it follows that

2
kdkþ1 k2 ¼  gkþ1  2gTkþ1 d kþ1 þ β2k kd k k2 :

Thus,

kdkþ1 k2  2gTkþ1 dkþ1 þ β2k kdk k2 : ð5:178Þ

Using (5.178) recursively and noting that d0 ¼  g0, it follows that

P
k Q
k 
kdkþ1 k2  2gTkþ1 dkþ1  2 β2i gTj dj
j¼0 i¼j
0 1
k 2
ð5:179Þ
Pk Q g T
B j jCd
¼ 2gTkþ1 dkþ1  2 gkþ1 βi
4
@ 4 A :
βFR
i
j¼0 i¼j
gj

If the theorem is not true, then (5.174) holds and there exists a positive constant γ so that kgkk  γ
for all k. Thus, from (5.176) and (5.179), it results that
k k  2 !
kdkþ1 k2 2 g 2 X Y β
kþ1
2  1þ i
: ð5:180Þ
g 1σ γ 2
j¼0 i¼j βFR
i
kþ1

From (5.180) and (5.174), it follows that

X1
kgk k2
2
 þ1:
k¼0 kd k k

This implies that lim inf kgk k ¼ 0: ♦

k!1

The theorem shows that for jβk j  ðσ=σ ÞβFR

k the algorithm (5.113)–(5.114) with d0 ¼  g0 is
globally convergent. In other words, to use βk in the hybrid schemes presented in Table 5.5 is quite
FR

natural. The theorem extends the result of Gilbert and Nocedal (1992) and Hu and Storey (1991) to
222 5 Conjugate Gradient Methods

the case when σ ¼ 1=2, i.e., when 2σ jβk j < βFR : If σ 2 ð0, 1=2Þ, then from (5.177) it follows that the
2 k
sufficient descent g d kþ1  c g
T holds for any positive constant c. However, if σ ¼ 1=2,
kþ1 kþ1
then only

gTk d k 1

kgk k2 2k

holds, which does not imply the sufficient descent condition.

Numerical Study: Hybrid Conjugate Gradient Methods

In the following, let us present the performances of some hybrid conjugate gradient methods based on
the projection concept: TAS, PRP-FR, GN, hDY, HS-DY, and LS-CD. For this, the set of 800 uncon-
strained optimization problems from the UOP collection is used (Andrei, 2020a).
Figure 5.8 presents the Dolan and Moré performance profiles of some hybrid conjugate gradient
methods subject to the CPU time metric. The tables inside the plots show the performances of the
algorithms subject to the number of iterations (#iter), the number of function and its gradient
evaluations (#fg), and the CPU time in seconds (cpu).
For example, when comparing TAS versus PRP-FR in Fig. 5.8, we can see that TAS was better in
150 problems (i.e., TAS achieved the minimum number of iterations in 150 problems). PRP-FR was
better in 307 problems, and they achieved the same number of iterations in 291 problems, etc. Out of
800 problems considered in this numerical experiment, only for 748 problems does the criterion (1.3)
hold. From Fig. 5.8, subject to the CPU time metric, it is clear that the TAS method is less efficient
and less robust than PRP-FR, GN, HS-DY, hDY, and LS-CD. On the other hand, the PRP-FR hybrid
conjugate gradient method is faster than hDY. Notice that HS-DY is more efficient than LS-CD, etc.
Figure 5.9 presents a global comparison of the hybrid conjugate gradient methods. Concerning
their robustness, for this set of unconstrained optimization problems, subject to the CPU time metric,
observe that the LS-CD algorithm is top performer, being more robust than the hybrid conjugate
gradient methods considered in this study. On the other hand, HS-DY is the most efficient. TAS and
GN are less efficient. Anyway, the performance profiles of the compared hybrid methods are grouped,
one method being slightly more efficient or more robust than the other one. Out of 800 problems in
this numerical experiment, only for 714 problems does the criterion (1.3) hold. The table inside
Fig. 5.9 shows that, out of 714 problems, HS-DY was faster in 328 problems, followed by hDY,
which was faster in 316 problems, etc.
From all these computational experiments with a large set of nonlinear unconstrained optimization
problems with different structures and complexities, we can see that the hybrid conjugate gradient
methods do not have spectacular performances versus the standard conjugate gradient methods,
subject to the CPU time metric. The first hybrid conjugate gradient method based on the projection
concept was given by Touati-Ahmed and Storey (1990) and by Hu and Storey (1991). The motivation
of introducing the hybrid conjugate gradient methods was to use the PRP update parameter when the
iterations jam. However, the PRP-FR method is less efficient and less robust than LS, HS, and PRP+.
In the following, the performances of the hybrid conjugate gradient methods based on the
projection concept for solving some applications from the MINPACK-2 applications are presented.
Tables 5.6 and 5.7 show the performances of the hybrid conjugate gradient methods for solving five
MINPACK-2 applications, each of them having 40,000 variables.
5.5 Hybrid Conjugate Gradient Methods 223

Fig. 5.8 Performance profiles of some hybrid conjugate gradient methods based on the projection concept
224 5 Conjugate Gradient Methods

Fig. 5.9 Global

performance profiles of six
hybrid conjugate gradient
methods

Table 5.6 Performances of TAS, PRP-FR, and GN for solving five applications from the MINPACK-2 collection

TAS PRP-FR GN
n #iter #fg cpu #iter #fg cpu #iter #fg cpu
A1 40,000 402 576 6.80 385 540 6.65 460 650 7.97
A2 40,000 602 1179 15.46 602 1179 15.46 602 1179 15.45
A3 40,000 806 1615 36.76 806 1615 34.86 806 1615 34.55
A4 40,000 294 595 30.21 294 595 30.29 294 595 30.21
A5 40,000 368 701 11.45 368 701 11.42 368 701 11.44
Total ─ 2472 4666 100.68 2455 4630 98.68 2530 4740 99.62

Table 5.7 Performances of HS-DY, hDY, and LS-CD for solving five applications from the MINPACK-2 collection

HS-DY hDY LS-CD

n #iter #fg cpu #iter #fg cpu #iter #fg cpu
A1 40,000 533 684 6.97 591 753 6.89 402 563 6.44
A2 40,000 935 1771 23.67 935 1771 23.73 649 1242 16.36
A3 40,000 1034 1939 42.53 1034 1939 41.90 1159 2073 45.28
A4 40,000 577 1148 58.39 577 1148 58.32 294 594 30.30
A5 40,000 400 760 12.62 400 760 12.64 375 702 11.53
Total ─ 3479 6302 144.18 3537 6371 143.48 2879 5174 109.91

It appears that the hybrid PRP-FR is the fastest, followed by GN, etc. The most time consuming is
HS-DY. Observe that Table 5.3 shows that for solving all five applications, CD needs only
137.09 seconds, the minimum time among all the methods considered in the numerical study. Clearly,
the hybrid methods are top performers versus the standard ones.
5.5 Hybrid Conjugate Gradient Methods 225

Hybrid Conjugate Gradient Methods as Convex Combinations

of the Standard Conjugate Gradient Methods

The hybrid conjugate gradient methods based on the convex combination of the standard conjugate
gradient methods are defined by (5.113) and (5.114), where the conjugate gradient parameter βk is
computed as

βk ¼ ð1  θk ÞβM1
k þ θk βk ,
M2
ð5:181Þ

where βM1
k and βk are the conjugate gradient parameters of the standard conjugate gradient methods
M2

which we want to hybridize (see Table 5.1) and 0  θk  1 is a parameter. The idea is to combine the
standard algorithms in a convex way in order to get algorithms with better performances. The
parameter θk in the convex combination (5.181) can be determined by means of two procedures.
The first one is given by the conjugacy condition yTk dkþ1 ¼ 0, where


d kþ1 ¼ gkþ1 þ ð1  θk ÞβM1 k þ θk βk
M2
dk : ð5:182Þ

From the equality yTk d kþ1 ¼ 0, where dkþ1 is given by (5.182), the following value for the
parameter θk is obtained:

T 
yTk gkþ1  βM1 yk d k


θk ¼ M2 k

: ð5:183Þ
β k  βkM1
yTk dk

Obviously, instead of the standard conjugacy condition yTk d kþ1 ¼ 0, it is quite possible to use the
Dai and Liao conjugacy condition

yTk dkþ1 ¼ tsTk gkþ1 , ð5:184Þ

where t  0 is a scalar parameter. In this case, from (5.184) where dkþ1 is given by (5.182), it results
that

T 
yT g  tsT g  βM1 y dk
θk ¼ k kþ1
M2 k kþ1 
k
k : ð5:185Þ
βk  βM1k y Td
k k

This is another value for the parameter θk obtained from the Dai and Liao conjugacy condition
(5.184). However, a value for t must be chosen in this case, which is rather difficult (see Andrei
(2011b)).
On the other hand, if the point xkþ1 is close enough to a local minimizer x, then a good direction to
follow is the one given by the Newton direction, that is, dkþ1 ¼  ∇2f(xkþ1)1gkþ1. Therefore, the
second procedure to determine θk in (5.181) considers the formal equality between the search
direction of the hybrid algorithm and the Newton direction, i.e.,


gkþ1 þ ð1  θk ÞβM1 k þ θk βk
M2
d k ¼ ∇2 f ðxkþ1 Þ1 gkþ1 : ð5:186Þ

Observe that (5.186) is only a technical argument to get a value for θk. With some simple algebraic
manipulations, from (5.186),
226 5 Conjugate Gradient Methods

T 2 
sTk ∇2 f ðxkþ1 Þgkþ1  sTk gkþ1  βM1 sk ∇ f ðxkþ1 Þdk
θk ¼
M2 
T 2 k
 : ð5:187Þ
βk  βM1k sk ∇ f ðxkþ1 Þdk

Both these procedures for the θk computation are operational and can be used to generate hybrid
conjugate gradient methods. The salient point in (5.187) for the θk computation is the presence of the
Hessian ∇2f(xkþ1). For large-scale problems, in practice, choices for the update parameters that do not
require the evaluation of the Hessian are preferred over the methods that require the Hessian at each
iteration. In order to avoid the exact computation of ∇2f(xkþ1), a solution is to use the secant equation.
In quasi-Newton methods, the secant equation plays an essential role for the approximation of the
Hessian of the objective function at each iteration. Therefore, in the second procedure for the θk
computation, the search direction dk can be calculated as solution of the following algebraic linear
system Bkdk ¼  gk, where Bk is an approximation of ∇2f(xk). In the quasi-Newton methods, the
matrix Bk, which is symmetric and positive definite, is effectively updated to obtain a new matrix,
Bkþ1, also symmetric and positive definite, as an approximation of ∇2f(xkþ1). The matrix Bkþ1 needs
to satisfy some suitable equations, namely, the secant equations, which include the second order
information. The most popular is the standard secant equation

Bkþ1 sk ¼ yk : ð5:188Þ

By using the standard secant equation in (5.187), the following value for θk could be obtained:

T 
yTk gkþ1  sTk gkþ1  βM1 y dk
θk ¼

k
k : ð5:189Þ
β k  βk
M2 M1 T
yk d k

The parameter θk, computed as in (5.183), (5.185), (5.187), or (5.189), may be outside the interval
[0,1]. To get a convex combination in (5.181), a simple procedure is followed: if θk  0, then in
(5.181) θk ¼ 0, that is, βk ¼ βM1
k ; if θ k  1, then θ k ¼ 1 in (5.181), that is, βk ¼ β k :
M2

Hence, the general hybrid conjugate gradient method based on the convex combination of the
standard conjugate gradient methods is as follows:

Algorithm 5.5 General hybrid conjugate gradient algorithm by using the convex combination of
standard schemes

1. Choose an initial point x0 2 ℝn, ε  0 sufficiently small and εb  0. Compute f(x0) and g0. Set d0 ¼  g0
and the initial guess α0 ¼ 1/kg0k. Set k ¼ 0
2. Test a criterion for stopping the iterations. For example, if kgkk1  ε, then stop; otherwise, continue
with step 3
3. Compute the stepsize αk satisfying the Wolfe line-search conditions
4. Set xkþ1 ¼ xk þ αkdk. Compute f(xkþ1) and gkþ1. Compute sk ¼ xkþ1  xk and yk ¼ gkþ1  gk


T 
5. If  βM2
k  βk
M1
yk dk   εb , then set θk ¼ 0. Otherwise, compute θk by (5.183), (5.185) or by (5.189)
according to the selected procedure
6. If 0 < θk < 1, then compute βk by (5.181). If θk  0, then set βk ¼ βM1k : If θk  1, then set β k ¼ β k
M2

7. Compute d ¼  gkþ1 þ βkdk

  2
8. If the Powell restart criterion gT g   0:2 g is satisfied, then set dkþ1 ¼  gkþ1, otherwise set
kþ1 k kþ1
dkþ1 ¼ d.
9. Compute the initial guess αk ¼ αk  1kdk  1k/kdkk
10. Set k ¼ k þ 1 and go to step 2 ♦
5.5 Hybrid Conjugate Gradient Methods 227

Algorithm 5.5 is general. In step 5 it can be particularized by combining in a convex manner

different standard conjugate gradient methods, where the parameter θk in the convex combination
(5.181) may be selected by means of the conjugacy or the Newton direction procedures.
As it can be seen, there is a large variety of possibilities to combine the standard conjugate gradient
methods in a convex way by using the conjugacy conditions or the standard or the modified secant
equations. In the following, let us present a hybrid conjugate gradient method based on the convex
combination of the standard conjugate gradient algorithms.

The Hybrid Convex Combination of LS and DY

The method is based on a convex combination of LS and DY (see Table 5.1) (Liu and Li, 2014). In it,

yTk gkþ1 gTkþ1 gkþ1

k ¼ βk ¼ 
βM1 LS
and β M2
k ¼ β DY
k ¼ : ð5:190Þ
d Tk gk dTk yk

Using the Dai and Liao conjugacy condition (5.184), from (5.185), where βM1
k and βk are given as
M2

in (5.190), the following value for the parameter θk is obtained:

T 



yk gkþ1 dTk gkþ1  t sTk gkþ1 d Tk gk
θk ¼
DL



T  : ð5:191Þ
g 2 d T g þ y T g d y
kþ1 k k k kþ1 k k

It is interesting to see the value of θk obtained from the second procedure based on the equality of
the Newton direction and the direction corresponding to the hybrid conjugate gradient algorithm.
Indeed, from (5.189), where βM1 k and βk are given as in (5.190), after some algebraic manipulations,
M2

it results that

T 



yk gkþ1 dTk gkþ1  sTk gkþ1 dTk gk
θk ¼
NT



T  : ð5:192Þ
g 2 d T g þ y T g d y
kþ1 k k k kþ1 k k

Observe that θDLk ¼ θ k when t ¼ 1. Therefore, an interesting property of this hybrid conjugate
NT

gradient algorithm is that the search direction dkþ1 not only satisfies the Dai and Liao conjugacy
condition, but it is also the Newton direction when t ¼ 1. Hence, in this hybrid algorithm t ¼ 1 is
considered.
The algorithm corresponding to this convex combination of LS and DY, which we call
NDLSDY, is a particularization of Algorithm 5.5, where step 5 is modified as follows: “If
 2


T 

 g d T g þ yT g
kþ1 k k k kþ1 d y   εb , then set θk ¼ 0. Otherwise, compute θk by (5.192)” and
k k

set βM1
k ¼ βLS
k andβM2
k ¼βDY
k in step 6.
The following theorem shows that the search direction dk generated by the algorithm NDLSDY
satisfies the sufficient descent condition.

Theorem 5.16 Let {gk} and {dk} be the sequences generated by the algorithm NDLSDY with strong
Wolfe line-search. Then, the search direction dk satisfies the sufficient descent condition

d Tk gk  ckgk k2 ð5:193Þ

for any k  0, where c ¼ (1  1.2σ)/(1  σ), σ < 0.5.

228 5 Conjugate Gradient Methods

Proof Assume that the Powell restart criterion does not hold, i.e.,
 T 
g g  < 0:2 g 2 : ð5:194Þ
kþ1 k kþ1

The proof is given by induction. For k ¼ 0, gT0 d0 ¼ kg0 k2 : Since c < 1, it follows that (5.193) is
satisfied. Now, assume that (5.193) holds for some k  1.
 
From the second strong Wolfe condition ∇f ðxk þ αk dk ÞT dk   σdTk gk , it results that

dTk yk ¼ dTk gkþ1  dTk gk  ð1  σ ÞdTk gk  0: ð5:195Þ

From (5.114), we get

2
dTkþ1 gkþ1 ¼  gkþ1 þ βk dTk gkþ1 : ð5:196Þ

Now, when θk  1, as seen in step 6 of the algorithm NDLSDY, it follows that βk ¼ βDYk :
Therefore, from the above relations (5.195) and (5.196), it results that

2 gkþ1 2  T 

dkþ1 gkþ1   gkþ1 þ T
T d g    1  2σ g 2 : ð5:197Þ
d k yk k kþ1 1  σ kþ1

From step 6 of the algorithm NDLSDY, when θk  0, βk ¼ βLS

k : Therefore, from the second strong
Wolfe line-search and from (5.194), it results that
 
2 gTkþ1 yk   T  2
d kþ1 gkþ1   gkþ1 þ  T  dk gkþ1   ð1  1:2σ Þ gkþ1 :
T   ð5:198Þ
d k gk

Finally, when θk 2 (0, 1), as seen in step 6 of the algorithm NDLSDY, βk is computed as in (5.181).
Observe that βDY
k can be written as β k ¼ d kþ1 gkþ1 =d k gk : Now, by using (5.194), the above relations
DY T T

and the definition of βk , it follows that

LS

2   T    
dTkþ1 gkþ1   gkþ1 þ βLS d g  þ βDY dT g 
2   T 
k k kþ1
 
k k kþ1

  gkþ1 þ σ βLS d g  þ σ βDY dT g 
2  Tk 
k k
 T k k k
¼  gkþ1 þ σ gkþ1 yk  þ σ dkþ1 gkþ1  ð5:199Þ
2 2  T   T 
  gkþ1 þ σ gkþ1 þ σ gkþ1 gk  þ σ d kþ1 gkþ1 
2 2  
  gkþ1 þ 1:2σ gkþ1 þ σ dTkþ1 gkþ1 :

But from (5.199),

  2
d Tkþ1 gkþ1  σ dTkþ1 gkþ1   ð1  1:2σ Þ gkþ1 :

Since σ < 0.5, it follows that there always is a constant v > 0 so that
 

dTkþ1 gkþ1  σ dTkþ1 gkþ1  ¼ v d Tkþ1 gkþ1 :

Therefore,
2
d Tkþ1 gkþ1  q gkþ1 , ð5:200Þ

where q ¼ (1  1.2σ)/v, v ¼ 1 þ σ or 1  σ.
In conclusion, (5.197), (5.198), and (5.200) show that (5.193) holds for k þ 1. ♦
5.5 Hybrid Conjugate Gradient Methods 229

Suppose that the Assumption CG holds. Then the following theorem proves the global conver-
gence of the algorithm NDLSDY.

Theorem 5.17 Suppose that the Assumption CG holds. Let {gk} and {dk} be the sequences generated
by the algorithm NDLSDY with strong Wolfe line-search. Then, lim inf kgk k ¼ 0:
k!1

Proof From the Assumption CG it follows that there exists a positive constant Γ > 0 so that kg(x)k  Γ
for all x 2 S. Observe that from Proposition 2.4, αk  λ, where λ is a positive constant.
Suppose that lim inf kgk k 6¼ 0: Then there exists a positive constant r > 0 so that for all
k!1
k sufficiently large,

kgk k  r: ð5:201Þ

By the second strong Wolfe condition and from (5.193),

d Tk yk ¼ dTk gkþ1  d Tk gk  ð1  σ ÞdTk gk  cð1  σ Þkgk k2 : ð5:202Þ

From the Lipschitz continuity of the gradient, it results that

kyk k ¼ gkþ1  gk  Lkxkþ1  xk k  LD, ð5:203Þ

where D ¼ max {kx  yk : x, y 2 S} is the diameter of the level set S.

Now, having in view the above inequalities, it follows that
 
jβk j ¼ ð1  θk ÞβLS k þ θk βk
DY 
 
 LS   DY  gTkþ1 yk  gkþ1 2
   
 βk þ βk ¼  T  þ  T 
  
d g d k yk
k k 2
g ky k g ΓLD Γ2
k
 kþ1 2 þ kþ1
 2 þ  M:
c kgk k c ð 1  σ Þ kgk k 2 cr cð1  σ Þr 2

According to the selection of the parameter βk, in step 6 of the algorithm NDLSDY, when
θk 2
= (0, 1), it is obvious that the above inequality also holds. Therefore,
j β j ks k MD
kdkþ1 k  gkþ1 þ jβk jkd k k ¼ gkþ1 þ k k  Γ þ  E,
αk λ

which implies that

X 1
2
¼ 1: ð5:204Þ
k0 k d kk

On the other hand, from (5.193), (5.201) and from the Zoutendijk condition, it results that


X 1 X c2 kg k4 X gTk dk 2
2 4
c r 2
 k
2
 2
< 1,
k0 kd k k k0 kd k k k0 kd k k

which contradicts (5.204). Hence, (5.201) does not hold and hence lim inf kgk k ¼ 0 is proved. ♦
k!1

New hybrid conjugate gradient algorithms for the unconstrained optimization were designed by
Andrei (2008b, 2008d, 2008g, 2009a, 2010a).
230 5 Conjugate Gradient Methods

Numerical Study: NDLSDY

In the following, let us present some numerical results with NDLSDY for solving unconstrained
optimization problems from the UOP collection, as well as some comparisons with standard conju-
gate gradient methods and with hybrid methods based on the projection concept.
Figure 5.10 presents Dolan and Moré’s performance profiles of NDLSDY versus LS, DY, PRP,
CD, FR, and HS conjugate gradient methods for solving the problems from the UOP collection. For
this set of unconstrained optimization test problems, Fig. 5.10 shows that NDLSDY, as a convex
combination of LS and DY, is more efficient than DY. However, LS is slightly more efficient and
more robust than NDLSDY. Figure 5.6 points out the computational evidence that LS is more
efficient and more robust than DY. Therefore, in this convex combination of LS and DY, the main
role is played by LS. Comparisons with PRP show that NDLSDY is more efficient, but PRP is slightly

Fig. 5.10 Performance profiles of NDLSDY versus the standard conjugate gradient methods LS, DY, PRP, CD, FR,
and HS
5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 231

Table 5.8 Performances of NDLSDY for solving five applications from the MINPACK-2 collection

NDLSDY
n #iter #fg cpu
A1 40,000 451 521 13.39
A2 40,000 1411 1740 23.55
A3 40,000 1166 1272 38.44
A4 40,000 1018 1172 65.59
A5 40,000 274 301 4.04
Total ─ 4320 5006 145.01

more robust. NDLSDY is seen as being more efficient and more robust than CD and than FR. Even if
HS is slightly more robust than NDLSDY, NDLSDY is close to HS as regards its efficiency.
In the following, Table 5.8 presents the performances of the NDLSDY hybrid conjugate gradient
method for solving five applications from the MINPACK-2 collection.
Table 5.8 shows that, subject to the CPU time metric, the hybrid convex combination of HS and
DY based on the Newton direction is more efficient. Tables 5.2 and 5.4 show that HS with
139.65 seconds is top performer versus DY with 229.20 seconds. Table 5.4 reveals that LS with
168.03 seconds is again better than DY. In another realm of numerical experiments, in the above
hybrid algorithms, DY with kgkþ1k2 in the numerator of βk, is combined in a convex way with HS and
LS, which have gTkþ1 yk in the numerator of βk. Notice that the hybrid convex combination with the
Newton direction using HS is top performer versus the hybrid combination with LS.

5.6 Conjugate Gradient Methods as Modifications of the Standard

Schemes

Due to their simplicity and low memory requirements, conjugate gradient methods represent an
important contribution to the class of methods for solving unconstrained optimization problems.
These methods have good convergence properties and their iterations do not involve any matrices,
making them extremely attractive for solving large-scale problems.
In this section, some conjugate gradient methods obtained as modifications of the standard scheme
are developed. The idea is to modify the formula of the standard conjugate gradient methods for
computing the conjugate gradient parameter βk in order to improve the numerical performances of the
algorithm. Any standard conjugate gradient method (see Table 5.1) may be modified, but some
modifications of the HS method are to be presented in the following.
A modification of the HS method is CG-DESCENT of Hager and Zhang (2005). CG-DESCENT is
interpreted as a particular value of the parameter in the Dai-Liao conjugate gradient algorithm.
Another interpretation of CG-DESCENT is a particularization of the Perry-Shanno self-scaling
memoryless BFGS algorithm. A deeper modification of the HS method is to determine the search
direction satisfying both the sufficient descent and the conjugacy conditions. Thus, the DESCON
conjugate gradient algorithm is obtained, in which the stepsize is determined by a modification of the
Wolfe line-search (Andrei, 2013c). Before presenting them, let us discuss the Dai-Liao conjugate
gradient method.
232 5 Conjugate Gradient Methods

The Dai-Liao Conjugate Gradient Method

For general nonlinear functions, by the mean value theorem (see Appendix A), there exists ξ 2 (0, 1)
so that d Tkþ1 gkþ1 ¼ dTkþ1 gk þ αk d Tkþ1 ∇2 f ðxk þ ξαk d k Þdk : Defining yk ¼ gkþ1  gk, the following can
be written dTkþ1 yk ¼ αk dTkþ1 ∇2 f ðxk þ ξαk dk Þd k : Therefore, for the nonlinear optimization, it is
reasonable to replace the conjugacy condition from the linear case with d Tkþ1 yk ¼ 0:
But, for unconstrained optimization methods, the search direction dkþ1 can be written as
dkþ1 ¼  Hkþ1gkþ1, where Hkþ1 is an approximation to the inverse of the Hessian ∇2f(xkþ1),
symmetric and positive definite, which satisfies the secant equation Hkþ1yk ¼ sk, where sk ¼ xkþ1  xk.
Therefore,

T
dTkþ1 yk ¼  H kþ1 gkþ1 yk ¼ gTkþ1 ðH kþ1 yk Þ ¼ gTkþ1 sk :

Hence, the conjugacy condition dTkþ1 yk ¼ 0 is satisfied if the line-search is exact, since, in this case
gTkþ1 sk ¼ 0: However, in practical situations, the exact line-search is not used. Therefore, it is quite
natural to replace the conjugacy condition d Tkþ1 yk ¼ 0 with

d Tkþ1 yk ¼  tg Tkþ1 sk , ð5:205Þ

where t  0 is a scalar. To determine the parameter βk in the search direction (5.48) that satisfies the
conjugacy condition (5.205), let us multiply (5.48) by yk and use (5.205), thus obtaining

gTkþ1 yk  tg Tkþ1 sk
k ¼
βDL , ð5:206Þ
yTk dk

known as the Dai-Liao conjugate gradient parameter. The method where the search direction is
computed as in (5.48) with βk computed as in (5.206) is called the Dai- Liao conjugate gradient
method (Dai, & Liao, 2001). It is obvious that

gTkþ1 sk
βDL
k ¼ βk  t
HS
, ð5:207Þ
yTk d k

showing that the DL method is a modification of the HS method. If the line-search is exact
(gTkþ1 sk ¼ 0), then βDL
k ¼ β k : Dai and Liao (2001) proved that for strongly convex functions, the
HS

norm of the search directions generated by the DL method is bounded. Therefore, the DL method with
strong Wolfe line-search is globally convergent.
In order to ensure the global convergence for general nonlinear functions, similar to the PRP+
method introduced by Powell (1984), Dai and Liao restrict βDL k to be positive, thus suggesting β k
DLþ
,
where
 T 
gkþ1 yk gTkþ1 sk
βDLþ
k ¼ max , 0  t : ð5:208Þ
dTk yk dTk yk

If the Lipschitz and boundedness assumptions hold and if dk satisfies the sufficient descent
condition gTk dk  ckgk k2 where c > 0 is a constant, then Dai and Liao (2001) showed that DL+
with strong Wolfe line-search is globally convergent.
5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 233

The Conjugate Gradient with Guaranteed Descent (CG-DESCENT)

For solving the problem (5.46), Hager and Zhang (2005, 2006a) proposed one of the most respected
conjugate gradient algorithms:
xkþ1 ¼ xk þ αk dk , ð5:209Þ
N
dkþ1 ¼ gkþ1 þ βk dk , d0 ¼ g0 , ð5:210Þ
N  
βk ¼ max βNk , ηk , ð5:211Þ

1
ηk ¼ , ð5:212Þ
kd k k min fη, kgk kg
 T
1 ky k k2
βNk ¼ T y  2 T d k gkþ1 , ð5:213Þ
yk d k k yk d k

where η > 0 is a constant (η ¼ 0.01). In order to achieve the global convergence for general nonlinear
functions, similar to the PRP+ method, the truncation (5.211) is introduced. Observe that in the
N
restricted scheme (5.211), the lower bound on βk is dynamically adjusted in order to make the lower
bound smaller as the iterates converge. Observe that (5.213) is a modification of the HS method.
Hager and Zhang obtained this scheme by deleting a term from the direction of the memoryless
quasi-Newton method of Perry (1976, 1977) and Shanno (1978b). Indeed, the search direction of
Hager and Zhang may be written as
 
yTk yk d Tk gkþ1 yTk gkþ1
dHZ
kþ1 ¼ g kþ1  2  dk : ð5:214Þ
dTk yk d Tk yk yTk yk

On the other hand, the search direction of Perry/Shanno d PS

k is
 
yTk sk sTk gkþ1 yTk gkþ1 sT g
dkþ1 ¼  T gkþ1  2 T
PS
 T sk þ k T kþ1 yk : ð5:215Þ
yk yk yk sk yk yk yk yk

(The Perry-Shanno direction is to be presented in the next section.) Observe that the relationship
between the search direction of Perry/Shanno and that of Hager/Zhang is as follows:
 
sTk yk dTk gkþ1
d PS ¼ d HZ
þ y , ð5:216Þ
kþ1
yTk yk kþ1 yTk dk k

or

yTk yk PS d Tk gkþ1
kþ1 ¼
d HZ d  y: ð5:217Þ
yTk sk kþ1 yTk dk k

Obviously, d HZ
kþ1 may be written as
" #
dk yTk yTk yk
dkþ1 ¼  I  T þ 2

HZ
2 dk dk gkþ1  Qkþ1 gkþ1 :
T HZ
ð5:218Þ
yk d k T
y dk k
234 5 Conjugate Gradient Methods

Observe that QHZ kþ1 is not symmetric and does not satisfy the quasi-Newton equation, properties
satisfied by dkþ1 : If in a canonical manner QHZ
PS
kþ1 is symmetrized and imposed to satisfy the quasi-
Newton equation, the Perry/Shanno direction (5.215) is obtained. From (5.216), when the angle
between dk and gkþ1 is sufficiently small and f is strongly convex, we can see that the term dHZ kþ1
dominates the yk term. In this case, the directions d HZ
kþ1 are approximately multiples of d PS
kþ1 : The
Perry/Shanno scheme, analyzed by Shanno and Phua (1980) and by Shanno (1985), has global
convergence for convex functions and an inexact line-search (Shanno, 1978b), but in general it
does not necessarily converge, even when the line-search is exact (Powell, 1984). However, the
Perry/Shanno scheme is convergent if the restarts are employed, but in this case the speed of
convergence can decrease. Han, Liu, and Yin (1997) proved that if the Wolfe line-search is employed,
then the convergence to a stationary point of the minimizing function f is achieved when
limk ! 1kykk ¼ 0 and the gradient of f is Lipschitz continuous.
The following theorem shows that if yTk dk 6¼ 0, a condition which is satisfied when f is strongly
convex or the line-search satisfies the Wolfe conditions, then the computational method given by
(5.209)–(5.213) always generates descent directions (Hager, & Zhang, 2005).

Theorem 5.18 If yTk dk 6¼ 0 and

dkþ1 ¼ gkþ1 þ τdk , ðd 0 ¼ g0 Þ ð5:219Þ

  
for any τ 2 βNk , max 0, βNk , then

7
gTk d k   kgk k2 : ð5:220Þ
8

Proof Since d0 ¼  g0, it follows that gT0 d0 ¼ kg0 k2, which satisfies (5.220). Suppose that τ ¼ βNk.
Multiplying (5.219) by gTkþ1 it results that
2
gTkþ1 d kþ1 ¼  gkþ1 þ βNk gTkþ1 d k !
2 y T
g k y k 2 T
d g
¼  gkþ1 þ k T kþ1  2
k k 2kþ1 gTkþ1 d k
yk d k yT dk

T 
T 
T  k 2
T  2
2
yk gkþ1 yk dk gkþ1 d k  gkþ1 yk dk  2kyk k2 gTkþ1 dk
¼
T 2
yk d k




Now, for the first term yTk gkþ1 yTk dk gTkþ1 dk from the above equality, let us apply the classical



inequality juT vj  12 kuk2 þ kvk2 , where u ¼ 12 yTk d k gkþ1 and v ¼ 2 gTkþ1 dk yk, to get (5.220). On
the other hand, if τ 6¼ βNk , then βNk  τ  0: After multiplying (5.219) by gTkþ1 , it follows that
2
gTkþ1 d kþ1 ¼  gkþ1 þ τgTkþ1 d k :

If gTkþ1 dk  0, then (5.220) immediately follows, since τ  0. If gTkþ1 d k < 0, then

2 2
gTkþ1 dkþ1 ¼  gkþ1 þ τgTkþ1 dk   gkþ1 þ βNk gTkþ1 dk

since βNk  τ  0: Therefore, (5.220) is true by the above analysis. ♦

5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 235

Numerical Study: CG-DESCENT

In the following, let us see the performances of CG-DESCENT (version 1.4) of Hager and Zhang for
solving 80 unconstrained optimization problems from the UOP collection (Andrei, 2020a), where the
number of variables is n ¼ 1000, 2000, . . ., 10000. The numerical experiments are given in the
context of Remark 1.1. The maximum number of iterations was limited to 2000.
Figure 5.11 presents the Dolan and Moré performance profiles of CG-DESCENT (version 1.4)
with Wolfe line-search versus the standard conjugate gradient methods HS, PRP, DY, and LS. On the
other hand, Fig. 5.12 presents the performance profiles of CG-DESCENT with approximate Wolfe
line-search (CG-DESCENTaw) versus the standard conjugate gradient methods HS, PRP, DY, and
LS.
From Figs. 5.11 and 5.12, it is clear that CG-DESCENT and CG-DESCENTaw are more efficient
and more robust than the standard conjugate gradient methods considered in this numerical study.
CG-DESCENT is a modification of the self-scaling BFGS method of Perry and Shanno. Therefore, it
is able to capture the curvature of the minimizing function better. Besides, CG-DESCENTaw
implements the approximate Wolfe line-search (2.67). The difference between these two versions
of CG-DESCENT is important. In contrast to the Wolfe conditions, the approximate Wolfe
conditions (2.67) are satisfied at a minimizer of φk(α) ¼ f(xk þ αdk). Therefore, when trying to satisfy
the approximate Wolfe conditions, we focus on minimizing φk; when trying to satisfy the usual Wolfe
conditions, we focus on minimizing ψ k ðαÞ ¼ φk ðαÞ  φk ð0Þ  αρφ_ k ð0Þ:
Since ψ k(0) ¼ 0, it is required that the local minimizer α satisfy ψ k(α) < 0 and ψ 0k ðα Þ ¼ 0: But,
these two relations together imply that the Wolfe conditions hold in a neighborhood of α when ρ < σ.
Although there is no theory to guarantee the convergence of the algorithm with approximate Wolfe
conditions, however, Hager and Zhang pointed out that there is a numerical advantage in using the

Fig. 5.11 Performance profiles of CG-DESCENT versus HS, PRP, DY, and LS
236 5 Conjugate Gradient Methods

Fig. 5.12 Performance profiles of CG-DESCENTaw (CG-DESCENT with approximate Wolfe conditions) versus HS,
PRP, DY, and LS

approximate Wolfe conditions: with approximate Wolfe conditions the local minimizers are
computed with the accuracy on the order of the machine epsilon rather than with the accuracy on
the order of the square root of the machine epsilon.
We must emphasize that the conjugate gradient method has an n-step quadratic convergence
property when αk is the minimum of φk(α) ¼ f(xk þ αdk).
Figure 5.13 shows the performance profiles of CG-DESCENT and CG-DESCENTaw versus DL
(t ¼ 1) and DLþ (t ¼ 1). Both CG-DESCENT and CG-DESCENTaw are more efficient and more
robust than DL or DLþ.
The performances of CG-DESCENT with Wolfe line-search (CG-DESCENT) and of
CG-DESCENT with approximate Wolfe line-search (CG-DESCENTaw) for solving the
MINPACK-2 applications are presented in Table 5.9. (#f represents the number of function calls.)
For solving the applications from the MINPACK-2, the performances of CG-DESCENTaw are
similar to those of CG-DESCENT.

The Conjugate Gradient with Guaranteed Descent and Conjugacy Conditions

and a Modified Wolfe Line-Search (DESCON)

For solving the unconstrained optimization problem (5.46), Andrei (2013c) developed the following
conjugate gradient algorithm:
5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 237

Fig. 5.13 Performance profiles of CG-DESCENT and CG-DESCENTaw (CG-DESCENT with approximate Wolfe
conditions) versus DL (t ¼ 1) and DLþ (t ¼ 1)

Table 5.9 Performances of CG-DESCENT and CG-DESCENTaw for solving five applications from the MINPACK-
2 collection
CG-DESCENT CG-DESCENTaw
n #iter #f cpu #iter #f cpu
A1 40,000 323 647 9.67 323 647 7.77
A2 40,000 788 1577 31.35 788 1577 27.05
A3 40,000 1043 2088 64.96 1043 2088 66.03
A4 40,000 435 871 81.40 435 871 72.24
A5 40,000 286 573 9.89 286 573 13.25
Total ─ 2875 5756 197.27 2875 5756 186.34

xkþ1 ¼ xk þ αk dk , ð5:221Þ

where αk > 0 is obtained by a variant of the Wolfe line-search discussed below and the directions dk
are generated as
d kþ1 ¼ θk gkþ1 þ βk sk , ð5:222Þ

yTk gkþ1  tk sTk gkþ1

βk ¼ , ð5:223Þ
yTk sk

d0 ¼  g0, where θk and tk are scalar parameters which are to be determined. Observe that in dkþ1
given by (5.222), gkþ1 is scaled by parameter θk while parameter tk in (5.223) is changed at every
iteration. Algorithms of this form or their variations were studied in many papers. For example,
Andrei (2007a, 2007b, 2007c, 2008c) considered a preconditioned conjugate gradient algorithm
238 5 Conjugate Gradient Methods

where the preconditioner is a scaled memoryless BFGS matrix and the parameter scaling the gradient
is selected as the spectral gradient. Andrei (2012) developed another conjugate gradient algorithm, in
which the search direction satisfies both the descent and the conjugacy condition at every iteration.
Stoer and Yuan (1995) studied the conjugate gradient algorithm on a subspace, where the search
direction dkþ1 is taken from the subspace span{gkþ1, dk}. Observe that, if for every k  1, θk ¼ 1
and tk ¼ t, then (5.222) reduces to the Dai and Liao search direction.
Suppose that the Assumption CG holds. Therefore, there exists a constant Γ  0 so that k∇f(x)k  Γ
for all x 2 S ¼ {x 2 ℝn : f(x)  f(x0)}. Besides, it is easy to see that
kskk ¼ kxkþ1  xkk  kxkþ1k þ kxkk  2B, where B  0 is a scalar.
In our algorithm, for all k  0 the scalar parameters θk and tk in (5.222) and (5.223), respectively,
are determined in such a way so that both the sufficient descent and the conjugacy conditions are
satisfied. Therefore, from the sufficient descent condition
2
gTkþ1 d kþ1  w gkþ1 ð5:224Þ

it follows that

T 

T 2
2 yk gkþ1 sTk gkþ1 sk gkþ1 2

θk gkþ1 þ  tk ¼ w gkþ1 ð5:225Þ
yTk sk yTk sk

and from the Dai-Liao conjugacy condition


dTkþ1 yk ¼ v gTkþ1 sk ð5:226Þ

it results that


θk yTk gkþ1 þ yTk gkþ1  tk sTk gkþ1 ¼ v sTk gkþ1 , ð5:227Þ

where v > 0 and w > 0 are known scalar parameters. Observe that in (5.225) the classical sufficient
descent condition (5.224) is modified with equality. As known, the main condition in any conjugate
gradient algorithm is the descent condition gTk dk < 0 or the sufficient descent condition (5.224). In
(5.224), w is selected close to 1. This is quite a reasonable value. For example, Hager and Zhang
(2005, 2006a) showed that in their CG-DESCENT algorithm, w ¼ 7/8. On the other hand, the
conjugacy condition is not so strict. In fact, very few conjugate gradient algorithms satisfy this
condition. For example, the Hestenes and Stiefel algorithm has the property that the pure conjugacy
condition always holds, independent of the line-search.
If v ¼ 0, then (5.227) is the “pure” conjugacy condition. However, in order to improve the
algorithm and to incorporate the second order information, v > 0 is taken.
Now, let us define


 2

Δk  yTk gkþ1 sTk gkþ1  gkþ1 yTk sk , ð5:228Þ


Δk  sTk gkþ1 Δk , ð5:229Þ


ak  v sTk gkþ1 þ yTk gkþ1 , ð5:230Þ
2



bk  w gkþ1 yTk sk þ yTk gkþ1 sTk gkþ1 : ð5:231Þ

Supposing that Δk 6¼ 0 and yTk gkþ1 6¼ 0, then, from the linear algebraic system given by (5.225)
and (5.227), tk and θk can be determined as
5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 239


 2
bk yTk gkþ1  ak yTk sk gkþ1
tk ¼ , ð5:232Þ
Δk


ak  tk sTk gkþ1
θk ¼ , ð5:233Þ
yTk gkþ1

with which the parameter βk and the direction dkþ1 can immediately be computed. Observe that by
using (5.232) in (5.233), θk can be expressed as
"
T  2 #
ak yk sk gkþ1 b
θk ¼ T 1þ  k: ð5:234Þ
yk gkþ1 Δk Δk

Again, by using (5.232) in (5.223), βk can be determined as

 
yTk gkþ1 bk g 2
kþ1
βk ¼ T 1 þ ak : ð5:235Þ
yk sk Δk Δk

Note that the crucial element in our algorithm is Δk : Therefore, the proposed conjugate gradient
algorithm with guaranteed descent and conjugacy conditions is defined by (5.221) and (5.222), where
the scalar parameters θk and βk are given by (5.234) and (5.235), respectively, and αk is computed by a
variant of the Wolfe line-search, which is discussed in the following.
Modified Wolfe line-search conditions. In order to define the algorithm, a small modification of the
second Wolfe line-search condition (5.50) is considered as

gðxk þ αk d k ÞT d k  σ k gTk dk , ð5:236Þ

where σ k is a sequence of the parameters satisfying the condition 0 < ρ < σ k < 1 for all k. The
interpretation of (5.236) is that the rate of decrease of f in the direction dk at xkþ1 is larger than a
fraction σ k of the rate of decrease of f in the direction dk at xk. Note that σ k is modified at every
iteration. The condition ρ < σ k for all k  0 guarantees that the Wolfe line-search (5.49) and (5.236)
can be satisfied simultaneously. Relations (5.49) and (5.236) are called the modified Wolfe line-search
conditions.

Proposition 5.13 If

1 g 2
kþ1
< σk    , ð5:237Þ
2 yT g  þ g 2
k kþ1 kþ1

then, for all k  1, Δk < 0.

Proof Observe that

sTk gkþ1 ¼ sTk yk þ sTk gk < sTk yk : ð5:238Þ

The modified Wolfe condition (5.236) gives

gTkþ1 sk  σ k gTk sk ¼ σ k yTk sk þ σ k gTkþ1 sk : ð5:239Þ

Since σ k < 1, (5.239) can be rearranged to obtain

240 5 Conjugate Gradient Methods

σ k T
gTkþ1 sk  y s: ð5:240Þ
1  σk k k

Since yTk sk > 0 (if kgkk 6¼ 0), the combination of this lower bound for gTkþ1 sk with the upper bound
(5.238) gives
 
 T   
g sk   yT sk  max 1, σ k : ð5:241Þ
kþ1 k 1  σk

But, since σ k > 1/2, from (5.241) it follows that

 T   
g sk  < σ k y T sk : ð5:242Þ
kþ1 1  σk k

If (5.237) is true, then

σ k  T  2
yk gkþ1   gkþ1 : ð5:243Þ
1  σk

From (5.243), since yTk sk > 0, it follows that

σ k  T   T    2
yk sk gkþ1 yk   yTk sk  gkþ1 : ð5:244Þ
1  σk

Now, from (5.242) and (5.244) it results that

T       
s g yT g  < σ k yT sk yT g   yT sk  g 2 , ð5:245Þ
k kþ1 k kþ1 1  σk k k kþ1 k kþ1

i.e., Δk < 0 for all k  1. ♦

Therefore, in our algorithm, σ k is computed as

g 2
kþ1
σk ¼   2 : ð5:246Þ
yT g  þ g
k kþ1 kþ1

If gk 6¼ 0 for all k  0, then 0 < σ k < 1 for all k  0.

Taking into account the acceleration Scheme (3.32), see Remark 3.1, where the acceleration factor
ηk is computed as in (3.33), according to the value of the parameter “acceleration” (true or false), the
following algorithms DESCON and DESCONa can be presented. DESCONa is the accelerated
version of DESCON.

Algorithm 5.6 Guaranteed descent and conjugacy conditions with a modified Wolfe line-search:
DESCON / DESCONa

1. Select a starting point x0 2 dom f and compute: f0 ¼ f(x0) and g0 ¼ ∇ f(x0). Select some positive values
for ρ and σ 0 as well as for v and w. Set d0 ¼  g0 and k ¼ 0. Select the small positive constants εA and εm
sufficiently small
2. Test a criterion for stopping the iterations. If the test is satisfied, then stop; otherwise, continue with step
3
3. Determine the stepsize αk by the modified Wolfe line-search conditions (5.49) and (5.246). Update the
variables as xkþ1 ¼ xk þ αkdk. Compute fkþ1 and gkþ1. Compute yk ¼ gkþ1  gk and sk ¼ xkþ1  xk
5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 241

4. If acceleration is true, then:

(a) Compute: z ¼ xk þ αkdk, gz ¼ ∇ f(z) and yk ¼ gk  gz
(b) Compute:
  ak ¼ αk gTk d k , and bk ¼ αk yTk dk
 
(c) If bk  εA , then compute ηk ¼ ak =bk and update the variables as xkþ1 ¼ xk þ ηkαkdk. Compute
fkþ1 and gkþ1. Compute yk ¼ gkþ1  gk and sk ¼ xkþ1  xk
5. Compute Δk as in (5.228)
 
6. If Δk   εm , then determine θk and βk as in (5.234) and (5.235), respectively; else set θk ¼ 1 and βk ¼ 0
7. Compute the search direction as: dkþ1 ¼  θkgkþ1 þ βksk
2   2 
8. Compute σ k ¼ gkþ1 = yTk gkþ1  þ gkþ1
  2
9. Restart criterion. If gTkþ1 gk  > 0:2 gkþ1 then set dkþ1 ¼  gkþ1
10. Consider k ¼ k þ 1 and go to step 2 ♦

Under reasonable assumptions, the modified Wolfe line-search conditions and the Powell restart
criterion are sufficient to prove the global convergence of the algorithm. The first trial of the stepsize
crucially affects the practical behavior of the algorithm. At every iteration k  1, the starting guess for
the step αk in the line-search is computed as αk  1kdk  1k/kdkk. Observe that in the line-search
procedure (step 3 of the algorithm), the stepsize αk is computed by using the updated value of the
parameter σ k computed as in step 8. For strongly convex functions, the linear convergence of the
acceleration scheme is proved as in Andrei (2009c).
The convergence for general nonlinear functions. In order to prove the global convergence of
DESCON, assume that the stepsize αk satisfies the strong Wolfe line-search conditions (5.49) and
 
gðxk þ αk d k ÞT dk   σ k gT dk , ð5:247Þ
k

where ρ and σ k are arbitrary positive constants so that 0 < ρ < σ k < 1. Observe that, since ρ is small
enough, the parameter σ k in (5.247) can be selected at each iteration as in (5.246), thus satisfying the
above condition 0 < ρ < σ k < 1.
In the following, let us prove that in very mild conditions the direction dk generated by (5.222),
where θk and βk are given by (5.234) and (5.235), respectively, is bounded.

Theorem 5.19 Suppose that the Assumption CG holds and kgkk  γ > 0 for all k  0. Consider the
conjugate gradient algorithm (5.221), where the direction dkþ1 given by (5.222) and (5.223) satisfies
the descent condition gTk d k ¼ wkgk k2 , where w > 1 and the stepsize αk is obtained by the strong
Wolfe line-search (5.49) and (5.247), where 1/2  σ k < 1. Then, lim inf kgk k ¼ 0:
k!1

Proof From (5.223), by using (5.232), after some algebraic manipulations, we have
 
yTk gkþ1 bk g 2
kþ1
βk ¼ T 1 þ ak : ð5:248Þ
yk sk Δk Δk

From: the definition of ωk, the modified Wolfe condition (5.236) and from the descent condition
¼ wkgk k2 , since kgkk  γ > 0 and σ k < 1 for all k  0, it follows that
gTk dk

yTk sk  wωk ð1  σ k Þγ 2 > wωð1  σ k Þγ 2 > 0:

However, from the Assumption CG,

242 5 Conjugate Gradient Methods

 T 
y g ksk k  ky k g ksk k  Lksk k2 Γ  LΓð2BÞ2 :
k kþ1 k kþ1

Therefore,
 T 
y g  LΓð2BÞ2 1 c
k kþ1
  ¼ , ð5:249Þ
y T sk  wω ð 1  σ k Þγ 2 ks k
k k skk
k

where

LΓð2BÞ2
c¼ :
wωð1  σ k Þγ 2

Now, observe that, since for all k  0, Δk < 0 and bk > 0, it follows that bk =Δk > 0: Besides,
from (5.228) and (5.231), it follows that

T 

bk yk gkþ1 sTk gkþ1
 ¼ w þ ð1 þ wÞ : ð5:250Þ
Δk Δk

Since Δk > 0 and sTk gkþ1 tends to zero along the iterations, it follows that bk =Δk tends to
w > 0. Hence 1  bk =Δk tends to 1 þ w. Therefore, there exists a positive constant c4 > 1 so that
1 < 1  bk =Δk  c4 :
Again, from the Assumption CG,
 T 
y sk ksk k  ky kksk k2  Lksk k3  Lð2BÞ3 :
k k
 
Therefore, yTk sk   Lð2BÞ3 =ksk k: Now, from (5.230) and (5.241), the following estimation is
obtained


    
jak j ¼ v sTk gkþ1 þ yTk gkþ1  vsTk gkþ1  þ yTk gkþ1 
  σk  
 vyTk sk  max 1, þ yTk gkþ1 
1  σk ð5:251Þ
 
Lð2BÞ3 σk LΓð2BÞ2
v max 1, þ :
ks k k 1  σk ks k k

Since 1/2  σ k < 1, there exists a positive constant c5 > 0 so that max{1, σ k/(1  σ k)}  c5. Hence,

1 cb
jak j  vLc5 ð2BÞ3 þ LΓð2BÞ2 ¼ , ð5:252Þ
ksk k ksk k

where cb ¼ vLc5 ð2BÞ3 þ LΓð2BÞ2 : With this, from (5.248) the following estimation is obtained:
 T  
yk gkþ1   b  g 2 cc cbΓ2 1
jβk j   T  1  þ jak j    4 þ
k kþ1
ks k k c 3 ks k k
 Δk Δk
y s ð5:253Þ
 k k
cbΓ2 1
¼ cc4 þ :
c 3 ks k k

Therefore, from (5.222) it follows that

5.6 Conjugate Gradient Methods as Modifications of the Standard Schemes 243

kdkþ1 k  jθk j gkþ1
 þ jβk j2ksk k
cbΓ 1 ð5:254Þ
 c2 Γ þ cc4 þ ks k  E,
c3 ksk k k

where E is a positive constant. Hence, for all k  0, kdkk  E implies ∑k  11/kdkk ¼ 1. Since dk is a
descent direction, it follows that lim inf kgk k ¼ 0: ♦
k!1

Numerical Study: DESCON

Let us solve the problems from the UOP collection with the number of variables n ¼ 1000, 2000, . . .,
10000. The DESCON algorithm implements the Wolfe line-search conditions with ρ ¼ 0.0001,
2   2 
σ ¼ g = yT g  þ g : In DESCON we take w ¼ 7/8 and v ¼ 0.05. The numerical
kþ1 k kþ1 kþ1
experiments are given in the context of Remark 1.1. Figure 5.14 shows the performance profiles of the
accelerated variant of DESCON (DESCONa) versus the standard conjugate gradient algorithms HS
and PRP.
Figure 5.15 shows the performance profiles of DESCONa versus DL (t ¼ 1) and versus
CG-DESCENT (version 1.4) with Wolfe line-search.
The performance profiles of accelerated DESCON (DESCONa) versus CG-DESCENT with
approximate Wolfe line-search (CG-DESCENTaw) are illustrated in Fig. 5.16.

Fig. 5.14 Performance profile of DESCONa versus HS and versus PRP

Fig. 5.15 Performance profile of DESCONa versus DL (t ¼ 1) and versus CG-DESCENT

244 5 Conjugate Gradient Methods

Fig. 5.16 Performances of

DESCONa versus
CG-DESCENTaw

Table 5.10 Performances of DESCONa for solving five applications from the MINPACK-2 collection

DESCONa
n #iter #fg cpu
A1 40,000 241 510 4.12
A2 40,000 631 1297 12.11
A3 40,000 1036 2100 32.25
A4 40,000 299 632 23.80
A5 40,000 278 576 6.71
Total ─ 2485 5115 78.99

Figure 5.16 points out that DESCONa is top performer in comparison with CG-DESCENTaw.
However, CG-DESCENT with approximate Wolfe line-search is more efficient than DESCONa.
Figure 5.15 shows that both DESCONa and CG-DESCENT with Wolfe line-search practically have
the same efficiency. From the table inside Fig. 5.15 notice that DESCONa was faster in 241 problems,
while CG-DESCENT in 252 problems. Once again, this illustrates the importance of the line-search
in conjugate gradient algorithms. An accurate line-search increases the performances of the
algorithm.
Table 5.10 presents the performances of DESCONa for solving the applications from the
MINPACK-2 collection, where nx ¼ 200 and ny ¼ 200.

5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned

The purpose of this section is to see how the second order information of the minimizing function
may be used in the formula for the search direction computation. Thus, new conjugate gradient
algorithms with better convergence properties are obtained. In fact, these methods include a special
preconditioning of conjugate gradient algorithms by using the BFGS updating. Memoryless quasi-
Newton methods were first introduced by Perry (1977) and Shanno (1978a). These methods can be
5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned 245

considered as the quasi-Newton methods, for which the approximation to the inverse of the Hessian is
taken as the identity matrix at every iteration.
Four such BFGS preconditioned conjugate gradient methods are known. The first preconditioned
conjugate gradient method called CONMIN, developed by Shanno (1983), is based on the idea that
the conjugate gradient methods are exactly the BFGS quasi-Newton method, where the approxima-
tion to the inverse Hessian of the minimizing function is restarted as the identity matrix at every
iteration. The second one, developed by Andrei (2007a, 2007b, 2007c, 2010b), is using scaling in the
frame of the memoryless BFGS method. In the third method, called DK/CGOPT, the search direction
is chosen in one-dimensional manifold closest to the Perry-Shanno self-scaling memoryless BFGS
method, developed by Dai and Kou (2013). Finally, using the trace and the determinant or a
combination of these operators of the self-scaling memoryless BFGS iteration matrix, a new
conjugate gradient method is developed (Andrei, 2019c). Only CONMIN and DK/CGOPT are
presented in this section (Andrei, 2020a).

The Memoryless BFGS Preconditioned Conjugate Gradient (CONMIN)

Perry (1976) noted that the search direction dkþ1 ¼  gkþ1 þ βkdk, where βk is given by
βk ¼ gTkþ1 yk =dTk yk , can be written as
 
d k yT
dkþ1 ¼  I  T k gkþ1 : ð5:255Þ
yk d k

Defining sk ¼ αkdk ¼ xkþ1  xk and since d k yTk =yTk dk ¼ sk yTk =yTk sk , (5.255) can be rewritten as
 
sk y T
d kþ1 ¼  I  T k gkþ1 : ð5:256Þ
y k sk

Perry relaxed the assumption that αk is chosen to minimize f(xk þ αdk) and added the correction
term sk sTk =yTk sk to the matrix from (5.256) and proposed the modified search direction
 
sk yTk sk sTk
d kþ1 ¼  I  T þ T gkþ1  Qkþ1 gkþ1 , ð5:257Þ
yk sk yk sk

where Qkþ1 defined in (5.257) satisfies the equation

yTk Qkþ1 ¼ sTk , ð5:258Þ

which is similar, but not identical to the secant equation

Hkþ1 yk ¼ sk , ð5:259Þ

where Hkþ1 is an approximation to the inverse Hessian. If the line-search is exact, then sTk gkþ1 ¼ 0
and (5.257) is identical to the HS method.
The major difficulty with this approach, not corrected by Perry, is that the matrix Qkþ1 is not
symmetric and therefore it is not positive definite. Thus, the search directions dkþ1 given by (5.257)
are not necessarily descent directions and hence numerical instabilities can result, ruining the
convergence of the algorithm. To overcome this difficulty, Shanno (1978a, 1978b) investigated the
relationship between the conjugate gradient and the BFGS methods.
246 5 Conjugate Gradient Methods

As known, the best quasi-Newton method is BFGS, where the updating to the inverse Hessian of
the minimizing function is given by
 
sk yTk H k þ H k yk sTk yTk Hk yk sk sTk
Hkþ1 ¼ Hk  þ 1þ T : ð5:260Þ
yTk sk y k sk yTk sk

The major difference between the conjugate gradient and the quasi-Newton methods is the
presence of the matrix Hk in the updating formula for the search direction dkþ1. For problems with
a large number of variables, it is impossible to store an approximation to the inverse Hessian, and
therefore the conjugate gradient methods are preferred.
As in Shanno (1978a), to see the relationship between the conjugate gradient and the BFGS
methods, the matrix Qkþ1 defined by (5.257) is first symmetrized to get

sk yTk yk sTk sk sTk

Qkþ1 ¼ I   þ : ð5:261Þ
yTk sk yTk sk yTk sk

Afterwards, let us force Qkþ1 defined by (5.261) to satisfy the quasi-Newton eq. (5.259), yielding
the symmetric update
 
sk yT þ y sT yT y sk sT
Qkþ1 ¼ I  k T k k þ 1 þ kT k T k : ð5:262Þ
y k sk y k sk y k sk

Observe that, if in (5.260) Hk ¼ I then it results precisely Qkþ1 from (5.262). Therefore, the
important result noticed for the first time by Shanno was that the conjugate gradient methods are
precisely the BFGS quasi-Newton methods, where the approximation to the inverse Hessian is
restarted as the identity matrix at every iteration. Hence, the conjugate gradient methods are often
called memoryless quasi-Newton methods.

Proposition 5.14 If x, y, z 2 ℝn, then

T 



x y z ¼ zyT x and xT yzT ¼ xT y zT :

Proof The above equalities are obtained by direct calculation. ♦

By using Proposition 5.14, a conjugate gradient method in which the search directions are
computed as
dkþ1 ¼ Qkþ1 gkþ1 ð5:263Þ

does not explicitly require the construction of Qkþ1. Indeed, from (5.263) and (5.262), it is easy to get
  
y T y sT g yT g sT g
d kþ1 ¼ gkþ1  1 þ kT k k Tkþ1  k T kþ1 sk þ k Tkþ1 yk , ð5:264Þ
yk sk yk sk yk sk yk sk

which shows that no additional information is needed more than that required by the known standard
conjugate gradient methods.
To get a superlinear convergence of a conjugate gradient method in which the search direction is
computed as dkþ1 ¼  gkþ1 þ βkdk with βk given by the HS method, Beale (1972) suggested a
modification of the search direction as
5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned 247

dtþ1 ¼ gtþ1 þ βt dt , ð5:265Þ

d kþ1 ¼ gkþ1 þ βk d k þ μk d t , ð5:266Þ

where

yTt gkþ1
μk ¼ , with t < k < t þ n ð5:267Þ
yTt d t

In particular, starting with t ¼ 0, at every n iterations a new pair of vectors dt and yt are stored,
known as the restart vectors of Beale, and the generated search directions are conjugate subject to
these restart vectors as well as to the previous ones. After n iterations, the restart direction is replaced
by the current one.
Powell (1977) made a modification of the restarting of the conjugate gradient methods. Powell’s
criterion was to use (5.266) whenever

(a) k (or k  t) is a multiple of n, or if

 T 
g g   0:2 g 2 :
kþ1 k kþ1

Obviously, (5.266) may be written as

 
dk yTk dt yTt
dkþ1 ¼  I  T  T gkþ1  Pk gkþ1 , ð5:268Þ
yk d k yt d t

where Pk, which modifies the gradient gkþ1, is an update using the information from two prior points.
It is easy to show that if f(x) is quadratic and the line-search is exact, then Pk is a projection matrix of
rank n  2 rather than rank n  1 of the matrix from (5.255).
Using this double update with information in two points and keeping the philosophy behind the
matrix Qkþ1 from (5.262), based on the fact that the conjugate gradient method is exactly the
memoryless BFGS method, Shanno (1978a) suggested the following search direction:

dkþ1 ¼ H kþ1 gkþ1 , ð5:269Þ

where
!
b Tb bk y
bk  H k yk sk þ sk yk H k þ
T
yT H sk sTk
H kþ1 ¼H T
1þ kT k , ð5:270Þ
yk sk y k sk yTk sk

with
 
b yt sTt þ st yTt yTt yt st sTt
Hk ¼ I  þ 1þ T : ð5:271Þ
yTt st yt st yTt st

bk ¼ I is set, then exactly the matrix Q is obtained. Besides, observe

Observe that, if in (5.270) H kþ1
that the conjugate gradient method (5.269)–(5.271) does not imply additional memory requirements.
From Proposition 5.14 it is clear that the search direction may be written as
248 5 Conjugate Gradient Methods

T
bk gkþ1 þ sk gkþ1 H
dkþ1 ¼ H bk yk
y Ts
" k !
k # ð5:272Þ
yTk Hbk yk sTk gkþ1 yTk Hbk gkþ1
 1þ T  sk ,
yk sk yTk sk yTk sk

where the vectors Hbk gkþ1 and Hbk yk are computed as

  
sT g y T y sT g yT g
Hbk gkþ1 ¼ gkþ1  t Tkþ1 yt þ 1 þ tT t t Tkþ1  t Tkþ1 st , ð5:273Þ
yt st yt st y t st yt st
  
sT y yTt yt sTt yk yTt yk
Hbk yk ¼ yk  tT k yt þ 1þ  s: ð5:274Þ
y t st yTt st yTt st yTt st t

It should be emphasized that the implementation of this method requires only seven vectors to
store: xk, xkþ1, gk, gkþ1, dk, dt and yt. By the time the update has been accomplished, the information in
xk is no longer required and so, H bk yk from (5.274) may be stored in xk. Once H bk yk and the scalars yT sk
 k
and yT H bk yk have been computed, gk is no longer needed, so H bk gkþ1 from (5.273) can be explicitly
k
computed and stored in gk. This double update scheme was first proposed by Perry (1976, 1977), but
with yt and st replaced by yk  1 and sk  1, respectively.
Another modified conjugate gradient method can be obtained as
  
yT y sT g yT g sT g
dkþ1 ¼ γgkþ1  1 þ γ kT k k Tkþ1  γ k T kþ1 sk þ γ k Tkþ1 yk , ð5:275Þ
yk sk yk sk yk sk y k sk

where

yTk sk
γ¼ , ð5:276Þ
yTk yk

where γ is a scaling parameter used only at the initial step. Now, substituting (5.276) in (5.275), after
some simple algebraic manipulations, it results that
 T 
y T sk s g yT g sT g
d kþ1 ¼  Tk gkþ1  2 k Tkþ1  k T kþ1 sk þ k T kþ1 yk , ð5:277Þ
yk yk y k sk yk yk yk yk

kþ1 ) search direction. Again, observe that if sk gkþ1 ¼ 0, that is, if the
which is the Perry/Shanno (d PS T

line-search is exact, then

 
yTk sk yTk gkþ1 yTk gkþ1
dkþ1 ¼  T gkþ1 þ T sk ¼ γ gkþ1 þ T sk , ð5:278Þ
yk yk yk yk y k sk

so the effect is one of multiplying the search direction (5.256) with a scalar, thus the n- step
convergence to the minimum of a quadratic function being maintained.
Therefore, using this scaling technique in (5.271), it follows that
 
b st yTt þ yt sTt yTt yt st sTt st sTt
Hk ¼ γ t I  þ þ , ð5:279Þ
yTt st yTt st yTt st yTt st

where
5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned 249

yTt st
γt ¼ :
yTt yt

With these developments, the conjugate gradient memoryless BFGS preconditioned algorithm is
defined by (5.272), where, this time, the vectors H bk gkþ1 and H bk yk are computed as
 T 
b yTt st sTt gkþ1 st gkþ1 yTt gkþ1
H k gkþ1 ¼ T gkþ1  T yt þ 2 T  T s, ð5:280Þ
yt yt yt yt y t st yt yt t
T T
 T T

Hbk yk ¼ yt st yk  st yk yt þ 2 st yk  yt yk st : ð5:281Þ
yTt yt yTt yt yTt st yTt yt

As a final remark of these developments, the fact that, under the exact line-search the conjugate
gradient methods are simple projections of the gradient, they make their initial length a poor
approximation to the desired stepsize. Fletcher (1987) proposed scaling dkþ1 by
2ðf ðxkþ1 Þ  f ðxk ÞÞ
dbkþ1 ¼ dkþ1 :
d Tkþ1 gkþ1

Observe that this scaling of the search direction includes the function values in two successive
steps.

Algorithm 5.7 Conjugate gradient memoryless BFGS preconditioned: CONMIN

1. Choose an initial point x0 2 ℝn and ε > 0 sufficiently small. Set k ¼ 0. Compute f(xk), gk ¼ ∇ f(xk) and set
dk ¼  gk
2. Compute the stepsize αk > 0 satisfying the Wolfe line-search conditions (5.49) and (5.50)
3. Compute xkþ1 ¼ xk þ αkdk, f(xkþ1), gkþ1, yk ¼ gkþ1  gk and sk ¼ xkþ1  xk
4. Test a criterion for stopping the iterations. If this criterion is satisfied, then stop; otherwise continue with
step 5
5. Test the criterion for restarting the iterations. If:
(a) Iteration k is a multiple of n, or
  2
(b) gTkþ1 gk   0:2 gkþ1 ,
then compute the search direction dkþ1 as in (5.275). Set st ¼ dk, yt ¼ yk, k ¼ k þ 1 and continue with
step 2. Otherwise, continue with step 6
6. Compute the search direction dkþ1 as in (5.272), where the vectors H bk gkþ1 and H
bk yk are computed as in
(5.280) and (5.281), respectively
7. Scale the search direction dkþ1 as  
d kþ1 ¼ 2ðf ðxkþ1 Þ  f ðxk ÞÞ=d Tkþ1 gkþ1 d kþ1 ,
set k ¼ k þ 1 and continue with step 2 ♦

For convex functions, using the exact line-search, Shanno (1978b) proved the convergence of
Algorithm 5.7 when the Hessian of the minimizing function is strictly bounded. However, since the
search direction is computed using the BFGS updating strategy and since the line-search is based on
the Wolfe conditions and the Beale restart is implemented, the convergence of the algorithm is
ensured. For general nonlinear functions bounded below with the level set bounded, Shanno showed
that the algorithm is not possible to converge to a point where the gradient is bounded away from
zero. Cycling in the form defined by Powell is the only way in which the conjugate gradient
memoryless BFGS preconditioned can possibly fail to converge on general nonlinear functions.
250 5 Conjugate Gradient Methods

Fig. 5.17 Performance profiles of CONMIN versus HS, PRP, DY, and LS

Numerical Study: CONMIN

CONMIN is one of the oldest and most respectable conjugate gradient methods, implemented by
Shanno (1983). In the following, let us present some results with this method and its comparisons
versus some other methods for solving the unconstrained optimization test problems from the UOP
collection (Andrei, 2020a).
In the first set of numerical experiments, CONMIN is compared versus the standard conjugate
gradient methods HS, PRP, DY, and LS. Figure 5.17 shows the Dolan and Moré performance profiles
of these methods. We can see that CONMIN is top performer, being more robust versus all these
methods.
Comparisons of CONMIN versus the modified conjugate gradient methods DL (t ¼ 1), DLþ
(t ¼ 1), CG-DESCENT (version 1.4), and DESCONa are presented in Fig. 5.18. Observe that
CONMIN is more robust than DL and DL+. However, both CG-DESCENT and DESCONa are
more efficient and slightly more robust than CONMIN. CG-DESCENT and CONMIN are based on
the memoryless BFGS quasi-Newton method. The search direction in CG-DESCENT is obtained
from the Perry/Shanno search direction by deleting a term. On the other hand, DESCON is a
modification of the HS method with guaranteed sufficient descent and conjugacy conditions and a
modified Wolfe line-search.
In the second set of numerical experiments, let us present the performances of CONMIN for
solving the applications from the MINPACK-2 collection, each of them with 40,000 variables.
Table 5.11 shows the performances of CONMIN for solving these applications.
5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned 251

Fig. 5.18 Performance profiles of CONMIN versus DL (t ¼ 1), DLþ (t ¼ 1), CG-DESCENT and DESCONa

Table 5.11 Performances of CONMIN for solving five applications from the MINPACK-2 collection

CONMIN
n #iter #fg cpu
A1 40,000 241 484 5.67
A2 40,000 827 1674 21.25
A3 40,000 1094 2217 41.65
A4 40,000 486 985 40.21
A5 40,000 374 757 11.39
Total ─ 3022 6117 120.17

The Conjugate Gradient Method Closest to the Scaled Memoryless BFGS Search
Direction (DK / CGOPT)

The search directions in the quasi-Newton methods are computed as d k ¼ H k gk , where H k 2 ℝnn
is an approximation to the inverse Hessian of the minimizing function (see Chap. 6). At the iteration k,
the approximation Hk to the inverse Hessian is updated to achieve H kþ1 as a new approximation to the
inverse Hessian in such a way, so that H kþ1 satisfies a particular equation, namely, the secant
equation, which includes the second order information. The most used is the standard secant equation
H kþ1 yk ¼ sk , where sk ¼ xkþ1  xk and yk ¼ gkþ1  gk.
Given the initial approximation H0 to the inverse Hessian as an arbitrary symmetric and positive
definite matrix, the most known quasi-Newton updating formula is the BFGS update
252 5 Conjugate Gradient Methods

 
sk yTk H k þ H k yk sTk yTk Hk yk sk sTk
Hkþ1 ¼ Hk  þ 1þ T : ð5:282Þ
yTk sk y k sk yTk sk

The self-scaling memoryless BFGS method of Perry (1977) and Shanno (1978a) is obtained by
updating the scaled identity matrix (1/τk)I by the BFGS updating formula (5.282), i.e., by considering
H k ¼ ð1=τk ÞI in (5.282), where I is the n  n identity matrix and τk is the scaling parameter.
Therefore, the search direction in the self-scaling memoryless BFGS method is computed as

dkþ1 ¼ H kþ1 gkþ1 , ð5:283Þ

where
   2
1 sk yT þ y sT 1 kyk k sk sTk
Hkþ1 ¼ I k T k k þ 1þ , ð5:284Þ
τk yk sk τk yTk sk yTk sk

and τk is the scaling parameter, known as the SSML-BFGS updating. Now, substituting (5.284) in
(5.283), the SSML-BFGS search direction, i.e., the self-scaling memoryless BFGS search direction of
Perry and Shanno (with a multiplier difference) is obtained as
 T   
gkþ1 yk kyk k2 gTkþ1 sk gT sk
dPS ¼ g þ  τ k þ sk þ kþ1 y: ð5:285Þ
kþ1 kþ1 T
y k sk T
y k sk T
y k sk yTk sk k

(Observe that the Perry/Shanno search direction (5.277) is a scaling of (5.285), with a particular
value for the scaling parameter τk.) Thus, subject to the parameter τk, a family of the Perry-Shanno
self-scaling memoryless BFGS quasi-Newton methods is obtained. Now, the following particular-
izations of the search direction dPS
kþ1 may be considered.

1. Having in view that sk ¼ αkdk, by deletion of the last term in (5.285) the following search direction
is obtained:

   
yTk gkþ1 kyk k2 sTk gkþ1
dkþ1 ¼ gkþ1 þ  τk þ T dk : ð5:286Þ
yTk dk y k sk yTk dk

As suggested by Oren and Spedicato (1976), if τk in (5.286) is chosen as

ky k k2
τOS
k ¼ , ð5:287Þ
yTk sk

then (5.286) reduces to the well-known conjugate gradient algorithm CG-DESCENT, proposed by
Hager and Zhang (2005)

d kþ1 ¼ gkþ1 þ βCGDESCENT

k dk , ð5:288Þ

where

gTkþ1 yk kyk k2 gTkþ1 sk

βCGDESCENT
k ¼  2 : ð5:289Þ
yTk d k yTk sk yTk d k
5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned 253

To establish the global convergence for general nonlinear functions, the conjugate gradient
parameter is truncated as
 
1
βk
CGDESCENTþ
¼ max βk CGDESCENT
, , ð5:290Þ
kdk k min fη, kgk kg

where η > 0 is a constant (η ¼ 0.01). The numerical experiments showed that CG-DESCENT is
more efficient and more robust than the self-scaling memoryless BFGS method given by (5.285)
(Dai and Kou, 2013).
Other values for the scaling parameter τk in (5.286) were proposed as follows. Oren (1974) and


Oren and Luenberger (1984) proposed for τk the value yTk sk = sTk Bk sk with Bk ¼ H 1
k : If Hk is the
identity matrix, then this value reduces to

yTk sk
τOL
k ¼ : ð5:291Þ
ks k k2

Al-Baali (1998) suggested the following two choices:

  ( )
ky k2 yTk sk
τ ¼ min 1, Tk
H
and τ ¼ min 1,
B
:
yk sk ks k k2

For a general nonlinear convex objective function, Nocedal and Yuan (1993) proved the global
convergence of the self-scaling BFGS method with τk given by (5.291) and with Wolfe line-
search. They also presented numerical results indicating that the unscaled BFGS method is in
general superior to the self-scaling BFGS with τk given by (5.291).

2. Observe that the self-scaling memoryless BFGS search direction of Perry and Shanno (5.285) is a
three-term conjugate gradient algorithm. A more reasonable way to deal with the last term in
(5.285) was suggested by Dai and Kou (2013), who proposed to seek the search direction closest to
kþ1 as a vector on the manifold Skþ1 ¼ {gkþ1 þ βdk : β 2 ℝ}. This approach is not new. For
dPS
example, Andrei (2017b) presented accelerated adaptive Perry conjugate gradient algorithms
based on the minimization of the Frobenius norm of the difference between the symmetrical
scaled Perry conjugate gradient direction and the self-scaling memoryless BFGS update. Also,
Livieris, Tampakas, and Pintelas (2018) developed a convex hybridization of the conjugate
gradient algorithms DY and HS, in which the hybridization parameter is computed by minimizing
the distance between the hybrid direction and the self-scaling memoryless BFGS direction.

The search direction in Skþ1 closest to dPS

kþ1 is obtained as solution of the following least-squares
problem
n o
dkþ1 ¼ arg min d  dPS : d 2 Skþ1 ,
kþ1 2 ð5:292Þ

which is

kþ1 ¼ gkþ1 þ β k ðτk Þsk ,

dDK ð5:293Þ
DK

where
254 5 Conjugate Gradient Methods

!
gT y k ky k k2 sTk yk gTkþ1 sk
βDK
k ðτ k Þ ¼ kþ1  τk þ T  : ð5:294Þ
yTk sk y k sk ksk k2 yTk sk

If the line-search is exact, sTk gkþ1 ¼ 0, then the second term in (5.294) is missing and the search
direction reduces to the HS formula.
In order to avoid the non-convergence of the algorithm, similar to Gilbert and Nocedal (1992),
who proved the global convergence of the PRP methods for general nonlinear functions by restricting
βk  0, (5.294) is truncated, being replaced by
( )
gTkþ1 d k
βk ðτk Þ ¼ max βk ðτk Þ, η
DKþ DK
, ð5:295Þ
kd k k2

where η 2 [0, 1) is a parameter. (η ¼ 0.5).

Hence, the Dai-Kou conjugate gradient algorithm belongs to the same family of conjugate gradient
methods obtained from the memoryless BFGS method by Perry and Shanno. Under the Assumption
CG, the family of conjugate gradient methods (5.293) and (5.294) generate sufficient descent
directions. Numerical experiments with this family of conjugate gradient algorithms showed that
the most efficient is the one where the parameter τk is given by (5.291). Therefore, substituting this
choice of τk in (5.294) and (5.295), it results that

yTk gkþ1 kyk k2 dTk gkþ1

βDK
k ¼  T ð5:296Þ
d Tk yk dk yk dTk yk

and
( )
dTk gkþ1
βDKþ
k ¼ max βDK
k ,η , ð5:297Þ
kd k k2

where η 2 [0, 1). Observe that (5.296) is exactly the Dai-Liao conjugate gradient algorithm with


t ¼ kyk k2 = sTk yk : Again, note that (5.296) differs from the Hager and Zhang algorithm only in a
constant coefficient in the second term. Thus, the family of Dai-Kou self-scaling memoryless BFGS
quasi-Newton methods is obtained.
If τk in (5.294) is selected as τOL
k , then the CGOPT conjugate gradient algorithm of Dai and Kou
(2013) is obtained, where the search direction is computed as

dCGOPT
kþ1 ¼ gkþ1 þ βCGOPT
k dk , ð5:298Þ

where

gTkþ1 yk kyk k2 gTkþ1 sk

βCGOPT
k ¼  T , ð5:299Þ
yTk d k yk sk yTk dk

which is identical to βDK

k given by (5.296). Observe that the difference between the conjugate gradient
parameters of CG-DESCENT given by (5.289) and of the CGOPT given by (5.299) is the absence of
the constant factor 2 in the second term of the parameter from (5.299). Again, the numerical
experiments showed that CGOPT performs more efficiently than the self-scaling memoryless
BFGS method given by (5.285) (Dai and Kou, 2013). If the line-search is exact, i.e., gTkþ1 sk ¼ 0,
5.7 Conjugate Gradient Methods Memoryless BFGS Preconditioned 255

then the second term in (5.294) or in (5.289) or in (5.299) is missing and the search direction reduces
to that of the HS algorithm.
Dai and Kou (2013) (see Lemma 2.1) proved that if yTk sk > 0, then the search direction given by
(5.293) and (5.294) satisfies
 
ks k2 3
g 2 :
gTkþ1 d kþ1   min τk Tk , kþ1 ð5:300Þ
y k sk 4

More generally, if function f is continuously differentiable and bounded below and its gradient g is
Lipschitz continuous, then Dai and Kou (2013) (see Lemma 2.2) proved that the search direction
(5.293), where τk in (5.294) is chosen to be any of τOS k , τk , τk or τ k and yk sk > 0 , satisfies
OL H B T

gT dkþ1  c g for some positive constant c > 0.
2
kþ1 kþ1
Dai and Kou (2013) implemented the algorithm (5.2) with (5.298) and (5.299) endowed with two
ingredients which improve its performances. The first ingredient is an improved Wolfe line-search,
which avoids the numerical drawback of the first Wolfe line-search condition and guarantees the
global convergence of the algorithm. The second one is an adaptive restart of the algorithm along the
negative gradient based on how the minimizing function is close to some quadratic function.
The improved Wolfe line-search consists in the modified first Wolfe condition as
   
ψ k ðαÞ  ψ k ð0Þ þ min εψ 0k ð0Þ, ραψ 0k ð0Þ þ ηk , ð5:301Þ

where ψ k(α) ¼ f(xk þ αdk), ε > 0 is a given constant and {ηk} is a positive sequence satisfying
∑k  1ηk < 1 , as well as in the second Wolfe line-search

ψ 0k ðαÞ  σψ 0k ð0Þ, ð5:302Þ

where 0 < ρ < σ < 1.

The dynamic restart strategy is based on measuring how the minimizing function is close to some
quadratic function. The deviation from quadratic can be measured by using an approximation of the
size of the third derivative, as given by Nash and Nocedal (1991). Instead, similar to the ratio used for
adjusting the trust-region radius, Dai and Kou (2013) evaluated the following quantity:


2 f kþ1  f k
rk ¼
, ð5:303Þ
αk gTk d k þ gTkþ1 dk

k ¼ 0, . . . . If rk is close to 1, then it can be supposed that ψ k(α) ¼ f(xk þ αdk) is close to some
quadratic function. If there are continuously a number of iterations so that rk is close to 1, then the
algorithm is restarted with the steepest descent direction.

Numerical Study: DK/CGOPT

In the following, let us present the performances of the conjugate gradient algorithm denoted as
DKþw, given by (5.47), where the search direction is computed as

d kþ1 ¼ gkþ1 þ βDKþ

k sk ð5:304Þ

and βDKþ
k is given by (5.297) with Wolfe line-search (5.49) and (5.50), for solving all the
problems from the UOP collection. Even if βDKþ
k  βCGOPTþ
k , where βCGOPTþ
k ¼
256 5 Conjugate Gradient Methods

Fig. 5.19 Performance profiles of DKþw versus CG-DESCENT and DESCONa

Fig. 5.20 Performance profiles of DKþaw versus CONMIN and CG-DESCENTaw

n
 o
max βCGOPT
k , η sTk gkþ1 =ksk k2 , the algorithm given by (5.304) and (5.297) with Wolfe line-search
is called DKþw because we did not implement in it the dynamic restart strategy used in CGOPT or
any other ingredients specific to CGOPT. For each test problem from the UOP collection (Andrei,
2020a), 10 numerical experiments were run, with the number of variables n ¼ 1000, . . ., 10000. The
initial value for the stepsize is computed as α0k ¼ αk1 kdk1 k=kd k k: All the numerical experiments
were run in the context of Remark 1.1.
Figure 5.19 presents the performance profiles of DKþw with Wolfe line-search versus
CG-DESCENT (version 1.4) and versus DESCONa. CG-DESCENT is more efficient than DKþw,
but DKþw is slightly more robust. DESCONa is clearly more efficient and more robust than DKþw.
Figure 5.20 shows the performance profiles of DKþaw (DKþ with approximate Wolfe line-
search) versus CONMIN and versus CG-DESCENTaw (CG-DESCENT with approximate Wolfe
line-search.
To see the performances of the DKþ algorithm with the improved Wolfe line-search, in (Andrei,
2019c), we manufactured a new code implementing (5.304) with (5.297), where this time the stepsize
is determined by the improved Wolfe line-search (5.301) and (5.302) (without the dynamic restart
strategy).
The algorithm using (5.304), (5.297), with improved Wolfe line-search (5.301) and (5.302) is
called DKþiw (DKþ with improved Wolfe line-search). The improved Wolfe line-search is
implemented with ε ¼ 106 and ηk ¼ 1/(k2), where k is the iteration number.
5.8 Solving Large-Scale Applications 257

Fig. 5.21 Performance profiles of DKþiw versus DKþw and DKþaw

Table 5.12 Performances of DKþw and DKþaw for solving five applications from the MINPACK-2 collection

DK+w DK+aw
n #iter #fg cpu #iter #fg cpu
A1 40,000 406 637 7.92 323 647 7.17
A2 40,000 940 1508 20.54 791 1583 32.64
A3 40,000 4001 6260 182.63 987 1976 65.87
A4 40,000 670 1065 84.22 435 871 80.42
A5 40,000 417 654 8.80 289 580 9.54
Total ─ 6434 10124 304.11 2825 5657 195.64

Figure 5.21 presents the performance profiles of DKþiw (DKþ with improved Wolfe line-search)
versus DKþw (DKþ with Wolfe line-search) and DKþaw (DKþ with approximate Wolfe line-
search). Observe that DKþiw is more efficient and more robust than DKþw.
In the following, let us present the performances of DKþw and DKþaw for solving the
applications from the MINPACK-2 collection as in Table 5.12.
Table 5.12 points out that DKþ with the approximate Wolfe line-search (DKþaw) is top
performer versus DKþ with the Wolfe line-search (DKþw).

5.8 Solving Large-Scale Applications

Nonlinear conjugate gradient algorithms are used in solving large-scale unconstrained optimization
problems. They have the significant advantage of a low storage requirement over most of the
other unconstrained optimization algorithms discussed in this book, only the steepest descent
being an exception. A practical conjugate gradient algorithm uses restart procedures. The most
important theoretical result about restarting is the n-step quadratic convergence, that is,
kxkþn  xk ¼ O(kxk  xk2). The critical point with these algorithms is the procedure for stepsize
computation. The conjugate gradient algorithms implement the weak Wolfe line-search (5.49) and
(5.50) or the strong Wolfe line-search (5.49) and (5.51). In the advanced conjugate gradient
algorithms, the stepsize is computed by means of the approximate Wolfe line-search (2.67) of
Hager and Zhang, the modified Wolfe line-search (5.49) and (5.236) of Andrei, or the improved
Wolfe line-search (2.72) and (5.50) of Dai and Kou. Tables 5.13, 5.14, and 5.15 show the
performances of CG-DESCENT with Wolfe and with approximate Wolfe line-searches
(CG-DESCENTaw), the performances of DESCON with modified Wolfe line-search and the
258 5 Conjugate Gradient Methods

Table 5.13 Performances of CG-DESCENT and of CG-DESCENTaw for solving five large-scale applications from
the MINPACK-2 collection

CG-DESCENT CG-DESCENTaw
n #iter #f cpu #iter #f cpu
A1 250,000 610 1221 92.42 610 1221 136.47
A2 250,000 1752 3505 382.78 1752 3505 448.94
A3 250,000 2370 4742 878.06 2370 4742 943.39
A4 250,000 925 1851 902.03 925 1851 961.31
A5 250,000 635 1271 145.72 635 1271 196.56
Total ─ 6292 12590 2401.01 6292 12590 2686.67

Table 5.14 Performances of DESCON and of DESCONa for solving five large-scale applications from the
MINPACK-2 collection

DESCON DESCONa
n #iter #fg cpu #iter #fg cpu
A1 250,000 602 950 53.01 591 1209 69.58
A2 250,000 2578 4056 309.55 1495 3021 261.97
A3 250,000 5001 7626 1134.73 2342 4727 607.68
A4 250,000 1644 2577 868.62 727 1489 468.01
A5 250,000 1070 1674 216.98 655 1334 130.31
Total ─ 10895 16883 2582.89 5810 11780 1537.55

Table 5.15 Performances of DKþw and of DKþiw for solving five large-scale applications from the MINPACK-2
collection

DK+w DK+iw
n #iter #fg cpu #iter #fg cpu
A1 250,000 693 1093 107.58 613 1227 145.21
A2 250,000 2299 3650 413.70 1762 3525 457.22
A3 250,000 4001 6257 1048.29 2354 4710 1007.74
A4 250,000 1396 2211 846.27 923 1847 937.49
A5 250,000 931 1455 227.40 622 1245 202.24
Total ─ 9320 14666 2643.24 6274 12554 2749.90

accelerated DESCON with modified Wolfe line-search (DESCONa), and the performances of
DK/CGOPT with Wolfe line-search (DKþw) and DK/CGOPT with improved Wolfe line-search
(DKþiw), respectively, for solving 5 applications from the MINPACK2 collection described in
Appendix D, each of them with 250,000 variables.
Firstly, observe that the conjugate gradient algorithms can solve large-scale unconstrained opti-
mization applications. Besides, from the above tables, we can see that the accelerated DESCON with
modified Wolfe line-search is the fastest. From the practical point of view, both the acceleration and
the modified Wolfe line-search play a crucial role in establishing the performances of DESCON.
Even if the acceleration at each iteration involves one additional evaluation of the gradient of the
minimizing function, the efficiency of the algorithm endowed with the acceleration scheme is
significantly improved.
5.8 Solving Large-Scale Applications 259

Notes and References

The idea of the methods described in this chapter is to include the approximations to the Hessian of
the minimizing function into the formula for computing the conjugate gradient parameter βk. This was
first considered by Perry (1976, 1977). Actually, the foundation of the self-scaling memoryless BFGS
algorithm was first presented by Perry as a technique for developing a nonlinear conjugate gradient
algorithm with memory, i.e., with stored information from the previous iterations, as an alternative to
the quasi-Newton methods for large-scale problems, where to store and handle the Hessian matrix is
impractical. This method was the first effort for solving large-scale problems, preceding the intro-
duction by Nocedal (1980) of the limited memory BFGS method. Shanno (1978a) reinterpreted
Perry’s algorithm and showed that the conjugate gradient methods are exactly the BFGS quasi-
Newton method, where the approximation to the inverse Hessian is restarted as the identity matrix at
every step. He introduced a scaling term, thus improving the final form of the self-scaling memoryless
BFGS method, i.e., the SSML-BFGS method. A modification of the self-scaling memoryless BFGS
method was given by Kou and Dai (2015). They multiplied the third term in (5.285) by some
nonnegative parameter, thus obtaining a new self-scaling BFGS algorithm with better convergence
properties.
The SSML-BFGS method provided a very good understanding of the relationship between
nonlinear conjugate gradient methods and quasi-Newton methods. For convex quadratic functions,
if the line-search is exact and the identity matrix is used as the initial approximation to the Hessian,
then both BFGS and SSML-BFGS methods generate the same iterations as the conjugate gradient
method. This was the starting point for the conjugate gradient methods memoryless BFGS
preconditioned.
Using this approach, Shanno and Phua (1976, 1978, 1980) and Shanno (1983) developed the
CONMIN algorithm, one of the best conjugate gradient algorithms and codes. Comparisons among
different conjugate gradient methods (ZXCGR (Powell, 1977), E04DBF (NAG – The Numerical
Algorithms Group), CONMIN) were given by Navon and Legler (1987).
Hager and Zhang (2005) presented the CG-DESCENT, one of the most reliable conjugate gradient
algorithms. In CG-DESCENT, Hager and Zhang introduced an approximate Wolfe line-search. Later
on, Dai and Kou (2013) proposed the CGOPT algorithm, where the search direction is closest to the
direction of the scaled memoryless BFGS method. Similar to Hager and Zhang, Dai and Kou
developed an improved Wolfe line-search. In this way, a family of conjugate gradient algorithms
was obtained, where the stepsize was computed by an improved Wolfe line-search. Andrei (2013c)
introduced DESCON, a new conjugate gradient algorithm with guaranteed sufficient descent and
conjugacy conditions, where the stepsize is computed by a modified weak Wolfe line-search.
Intensive numerical experiments proved that, for solving large-scale unconstrained optimization
problems and applications, DESCON is top performer versus CONMIN, CG-DESCENT, and
CGOPT. Further on, Andrei (2018d, 2020a), by using the determinant, the trace or a combination
of these operators known as the measure function of Byrd and Nocedal, developed new efficient self-
scaling memoryless BFGS conjugate gradient methods.
An adaptive conjugate gradient algorithm for the large-scale unconstrained optimization was
presented by Andrei (2016). In this algorithm, the search direction is computed as the sum of the
negative gradient and a vector determined by minimizing the quadratic approximation of the
objective function at the current point. Using a special approximation to the inverse Hessian of the
objective function, which depends on a positive parameter, a new search direction satisfying both the
sufficient descent condition and the Dai–Liao conjugacy is obtained.
Two different approaches based on the eigenvalues and the singular values of the matrix
representing the search direction in conjugate gradient algorithms were considered by Andrei
260 5 Conjugate Gradient Methods

(2017a). Using a special approximation of the inverse Hessian of the objective function, which
depends on a positive parameter, a search direction is obtained, satisfying both the sufficient descent
condition and the Dai–Liao conjugacy condition. In the first approach, the parameter in the search
direction is determined by clustering the eigenvalues of the matrix defining it. The second approach
uses the minimization of the condition number of the matrix representing the search direction. The
conclusion of these two approaches is that basically they represent two different ways to pursue
similar ideas based on eigenvalues or on singular values of the iteration matrix, respectively. See also
Andrei (2018b), where a Dai-Liao conjugate gradient algorithm with the clustering of the eigenvalues
is presented. A modification of the Dai-Yuan conjugate computational scheme with sufficient descent
condition is presented in (Andrei, 2009b, 2010c). An accelerated conjugate gradient algorithm with
modified secant condition for unconstrained optimization is developed by Andrei (2009g). In this
algorithm, the modified secant equation of Zhang, Deng, and Chen (1999) is used.
Another class is represented by the three-term conjugate gradient algorithms introduced by Beale
(1972) and developed by Powell (1984), Nazareth (1977, 1986), Dai and Yuan (1999), Narushima,
Yabe, and Ford (2011), and Andrei (2013a, 2013b, 2013d, 2020a). A family of three-term conjugate
gradient methods with sufficient descent property was presented by Al-Baali, Narushima, and Yabe
(2015). Even if these three-term conjugate gradient algorithms are based on different concepts which
include satisfying the descent and conjugacy conditions (Andrei, 2013b), the subspace minimization
(Andrei, 2014), and the minimization of one-parameter quadratic model of the minimizing function
(Andrei, 2015b), they have similar performances. A numerical comparison of conjugate gradient
algorithms for the unconstrained optimization is given in (Andrei, 2007d).
The most important modern packages implementing different variants of the conjugate gradient
algorithms for solving large-scale unconstrained optimization problems are CONMIN (Shanno,
1983), CG-DESCENT (Hager, & Zhang 2005, 2006a), CGOPT (Dai, & Kou, 2013), and DESCON
(Andrei, 2013c).
In Andrei (2011b), some open questions regarding the conjugate gradient algorithms are
disscused: Why is the initial search direction d0 ¼  g0 critical? Can we take advantage of the
problem structure to design more effective nonlinear conjugate gradient algorithms? Which is the
best conjugacy condition? Which is the best hybrid conjugate gradient algorithm? What is the most
convenient restart procedure of the conjugate gradient algorithms? etc.
 Quasi-Newton Methods
6

The idea of these methods is not to use the Hessian ∇2f(xk) of the minimizing function in the current
point at every iteration, but instead to use an approximation of it. In this chapter, consider Bk as an
approximation to ∇2f(xk) and Hk an approximation to the inverse Hessian ∇2f(xk)
1. Obviously, both
these approximations Bk or Hk may be used, but the use of Bk involves the solving of a linear algebraic
system. The quasi-Newton method requires only the gradient of the minimizing function, gradient
which must be supplied by the user at every iteration. The purpose of this chapter is to present these
methods together with the theoretical aspects concerning their convergence to solution, as well as
their performances for solving large-scale complex optimization problems and applications. In quasi-
Newton methods, the approximations to the Hessian (or to the inverse Hessian) may be achieved
through matrices of rank one or through matrices of rank two. From the multitude of quasi-Newton
methods, this chapter will present only the rank-two updating methods, that is, Davidon-Fletcher-
Powell (DFP) and Broyden-Fletcher-Goldfarb-Shanno (BFGS), and the rank one, that is, Symmetric
Rank 1 (SR1), together with some modifications of them.
Since the quasi-Newton methods use only the gradient of the minimizing function, they are very
attractive for solving large classes of complex, large-scale unconstrained optimization problems and
applications. Moreover, since the second derivatives are not required, sometimes quasi-Newton
methods are more efficient and more robust than the Newton method. It is worth mentioning that
the development of the automatic differentiation techniques gives the user the possibility not to
supply second derivatives in the Newton method. The automatic differentiation is not applicable in
many situations, and it is more difficult to work with second derivatives in it than with the gradient.
This is the reason why quasi-Newton methods represent an important class of methods for the
unconstrained optimization.

6.1 DFP and BFGS Methods

Consider the minimizing problem

min f ðxÞ, ð6:1Þ
x2ℝn

where f : ℝn ! ℝ is twice continuously differentiable and bounded below. In the current point xk, let
us consider the following quadratic model of the minimizing function f

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 261
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_6
262 6 Quasi-Newton Methods

1
mk ðdÞ ¼ f ðxk Þ þ ∇f ðxk ÞT d þ dT Bk d, ð6:2Þ
2
where Bk 2 ℝnn is a symmetric and positive definite matrix which, as we said, will be updated at
every iteration. Observe that the minimizer dk of (6.2) can be explicitly written as

dk ¼
B
1
k gk , ð6:3Þ

where gk ¼ ∇ f(xk). The next approximation to the minimum point is computed as

xkþ1 ¼ xk þ αk dk , ð6:4Þ

where αk is the stepsize computed by line-search, for example, the Wolfe line-search:

f ðxk þ αk dk Þ  f ðxk Þ þ ραk ∇f ðxk ÞT d k ð6:5Þ

∇f ðxk þ αk dk ÞT dk  σ∇f ðxk ÞT dk , ð6:6Þ

If in (6.3) Bk ¼ ∇2f(xk), then the Newton method is obtained.

In the quasi-Newton methods, the matrix Bk is updated in a simple manner in order to take into
account the curvature of the minimizing function during the most recent iterates. Suppose that we
generated a new iteration xkþ1 as in (6.4) and then construct a new quadratic model:
1
mkþ1 ðdÞ ¼ f ðxkþ1 Þ þ ∇f ðxkþ1 ÞT d þ dT Bkþ1 d, ð6:7Þ
2
where Bkþ1 is an update of Bk. At this moment, we are facing the following problems. How to
generate the approximations Bkþ1 symmetric and positive definite, for which dkþ1 ¼
B
1 kþ1 gkþ1 is a
descent direction? Computationally, how to generate Bkþ1 as easily as possible? Finally, what are the
conditions the approximation Bkþ1 has to satisfy for the convergence of the corresponding
algorithms?
Since we have access to the gradient, it is quite reasonable to require that the gradient of mkþ1
should match the gradient of the minimizing function at the latest two iterates xk and xkþ1. However,
as we can see, ∇mkþ1(0) ¼ ∇ f(xkþ1). Therefore, the second requirement is automatically satisfied.
The first requirement can be written as
∇f ðxk Þ ¼ ∇mkþ1 ð
αk dk Þ ¼ ∇f ðxkþ1 Þ
αk Bkþ1 dk :

Therefore,
Bkþ1 αk d k ¼ ∇f ðxkþ1 Þ
∇f ðxk Þ: ð6:8Þ

Let us define
sk ¼ xkþ1
xk ¼ αk dk , yk ¼ ∇f ðxkþ1 Þ
∇f ðxk Þ: ð6:9Þ

With this, (6.8) becomes

Bkþ1 sk ¼ yk , ð6:10Þ

known as the secant equation.

Observe that the secant equation requires that the symmetric and positive definite matrix Bkþ1 map
the displacement sk into the change of gradients yk. Since sTk Bkþ1 sk > 0 (Bkþ1 is positive definite), it
follows that the secant equation is satisfied only if sk and yk satisfy the curvature condition
6.1 DFP and BFGS Methods 263

sTk yk > 0: ð6:11Þ

If f is strongly convex, the curvature condition (6.11) will be satisfied for any two points xk and
xkþ1. However, this condition will not always hold for nonconvex functions. In these cases, the
curvature condition (6.11) has to be enforced by imposing restrictions on the line-search procedure
that computes the stepsize αk. In fact, the curvature condition (6.11) is guaranteed to hold if the
stepsize is computed by the weak Wolfe conditions (6.5) and (6.6) or by the strong Wolfe conditions
(6.5) and (6.12)




∇f ðxk þ αk dk ÞT dk

σ∇f ðxk ÞT dk : ð6:12Þ

Indeed, from (6.9) and (6.12), we get

yTk sk  ðσ
1Þαk ∇f ðxk ÞT dk  0,

since σ < 1 and dkis a descent direction.

If the curvature condition (6.11) is satisfied, then the secant equation (6.10) always has a solution
Bkþ1. Actually, the secant equation admits an infinite number of solutions. This is because the
symmetric matrix Bkþ1 has a number of n(n þ 1)/2 elements which is greater than n conditions
imposed by the secant equation plus n additional conditions (inequalities) imposed by the positive
definiteness of Bkþ1 (all the principal minors must be positive). In order to determine Bkþ1 uniquely, a
reasonable additional condition is imposed: among all the symmetric matrices satisfying the secant
equation, Bkþ1 is in a certain sense closest to the current matrix Bk.In other words, Bkþ1 is the
solution of the following problem:
min kB
Bk k
B
subject to ð6:13Þ
B ¼ BT , Bsk ¼ yk ,

where sk and yk satisfy the curvature condition (6.11) and Bk is symmetric and positive definite.
Obviously, different norms in (6.13) may be used, each of them determining different quasi-Newton
methods. A norm that allows an easy solution of the minimization problem (6.13) is the weighted
P P
Frobenius norm defined as kAkW  kW1/2AW1/2kF, where kCkF ¼ ni¼1 nj¼1 c2ij (see Appendix A).

1
Now, selecting W ¼ Gk , where Gk is the average Hessian

ð1
Gk ¼ ∇2 f ðxk þ tαk d k Þdt
0

observe that yk ¼ Gk sk : This choice of the weighting matrix W has an important property that is
non-dimensional, i.e., the solution of (6.13) is not dependent on the units of the problem.
With the selection of the weighting matrix and of the norm, the unique solution of (6.13) is
   
ðDFP directÞ Bkþ1 ¼ I
ρk yk sTk Bk I
ρk sk yTk þ ρk yk yTk , ð6:14Þ

where
264 6 Quasi-Newton Methods

1
ρk ¼ : ð6:15Þ
yTk sk

The update (6.14) is called the DFP update, since it was proposed by Davidon (1959, 1980) and
studied and popularized by Fletcher and Powell (1963). After simple algebraic manipulations, from
(6.14) and (6.15), another expression of Bkþ1 is obtained as
 
Bk sk yTk þ yk sTk Bk sTk Bk sk yk yTk
ðDFP directÞ Bkþ1 ¼ Bk
þ 1 þ : ð6:16Þ
yTk sk yTk sk yTk sk

As we said, the inverse of Bk is Hk, i.e., Hk ¼ B
1

k : Using the matrix Hk in the implementations of
the DFP method is quite appealing, since it allows to compute the search direction dk ¼
Hk ∇ f(xk)
by a simple matrix-vector multiplication. Using the Sherman-Morrison-Woodbury formula twice,
from (6.16) (see Appendix A), the following expression for the update of the inverse Hessian
approximation Hk that corresponds to the DFP update is obtained

Hk yk yTk H k sk sTk
ðDFP inverseÞ H kþ1 ¼ Hk
þ T : ð6:17Þ
yTk H k yk y k sk

Observe that the Hk matrix in (6.17) is modified by two terms, each of them of rank one. Also,
(6.16) is a rank-two modification of Bk.
Even if the DFP updating is efficient and robust for solving unconstrained optimization problems,
it was superseded by the BFGS updating. The BFGS introduced by Broyden (1970), Fletcher (1970),
Goldfarb (1970), and Shanno (1970) follows the same methodology as above, but on the inverse
Hessian Hk. Obviously, the updated Hessian Hkþ1 must be a symmetric and positive definite matrix
which satisfies the secant equation
H kþ1 yk ¼ sk : ð6:18Þ

As above, the updated Hessian Hkþ1 closests to Hk is determined as the unique solution of the
minimizing problem
min kH
Hk k
H
subject to ð6:19Þ
H ¼ H , Hyk ¼ sk :
T

Using the same weighted Frobenius norm, where this time the weight matrix W satisfies Wsk ¼ yk,
the unique solution Hkþ1 of the problem (6.19) is
   
ðBFGS inverseÞ H kþ1 ¼ I
ρk sk yTk Hk I
ρk yk sTk þ ρk sk sTk , ð6:20Þ

where ρk is defined in (6.15). After some simple algebraic manipulation, from (6.20) and (6.15), we
get
 
sk yTk H k þ H k yk sTk yTk H k yk sk sTk
ðBFGS inverseÞ H kþ1 ¼ H k
þ 1þ T : ð6:21Þ
yTk sk y k sk yTk sk

Now, applying twice the Sherman-Morrison-Woodbury formula (see Appendix A) to (6.21), we

get a version of the BFGS algorithm that works with the Hessian approximation Bk rather than with
the inverse Hk. Therefore, from (6.21), we get
6.1 DFP and BFGS Methods 265

Bk sk sTk Bk yk yTk
ðBFGS directÞ Bkþ1 ¼ Bk
þ T : ð6:22Þ
sTk Bk sk y k sk

Note that the DFP and the BFGS updating formulae are duals of each other in the sense that one
can be obtained from the other one by making the interchanges sk $ yk and Bk $ Hk.
In the current implementations of the BFGS algorithm, the inverse update Hkþ1 is used for the
search direction computation dkþ1 ¼
Hkþ1 ∇ f(xkþ1). This is very appealing because it can be
performed at a cost of O(n2) arithmetic operations.
The implementation of the BFGS algorithm requires the initial matrix H0. This is often taken as the
identity matrix or the identity matrix multiplied by a parameter τ. If τ is too large, then the first step
d0 ¼
τg0 is too long, and many function evaluations may be required for determining a suitable
stepsize α0. Thus, to obtain H1, Nocedal and Wright recommend selecting H0 as

yT0 s0
H0 ¼ I
yT0 y0

before applying (6.17) and (6.21). The BFGS algorithm can be presented as follows.

Algorithm 6.1 DFP and BFGS methods

1. Initialization. Consider an initial point x0 2 ℝn, the inital approximation H0 to the inverse Hessian as well
as the convergence toleance ε > 0 sufficiently small. Set k ¼ 0
2. Test a criterion for stopping the iterations. For example, if k∇f(xk)k  ε, then stop, otherwise set k ¼ k þ 1
and go to step 3
3. Compute the descent direction dk ¼
Hk ∇ f(xk)
4. Compute the stepsize αk which satisfies the Wolfe line-search conditions
5. Compute xkþ1 ¼ xk þ αkdk
6. Compute sk ¼ xkþ1
xk and yk ¼ ∇ f(xkþ1)
∇ f(xk)
7. For DFP compute Hkþ1 by (6.17). For BFGS compute Hkþ1 by (6.21)
8. Set k ¼ k þ 1 and go to step 2 ♦

In step 4 of the algorithm, the stepsize is determined by the weak Wolfe conditions (6.5) and (6.6)
(or by the strong Wolfe conditions (6.5) and (6.12)), which should always try the initial stepsize
αk ¼ 1. Numerical experiments showed that, eventually, this stepsize will always be accepted, thus
producing the superlinear convergence of the algorithm.
The following numerical example illustrates the importance of the initial approximation H0 on the
performances of the algorithms DFP and BFGS.

Example 6.1 Let us minimize the function

f ðxÞ ¼ ðx1
2Þ4 þ ðx1
2Þ2 x22 þ ðx2 þ 1Þ2 ,

which has the minimum x ¼ [2,
1]T with f(x) ¼ 0. In the minimum point, we have
   
0 2 0
∇f ðx Þ ¼ , ∇ 2 f ðx  Þ ¼ :
0 0 2
266 6 Quasi-Newton Methods

Consider x0 ¼ [1.1, 0]T and H0 ¼ I. Then, the DFP (6.17) and the BFGS (6.21) methods with weak
Wolfe line-search (6.5) and (6.6) implemented in subroutine wolfeLS with cubic interpolation (see
Fig. 2.3) give the following results.

DFP BFGS
x15 ¼ [2
0.999999]T x13 ¼ ½ 2
1 T
f(x15) ¼ 0.1022e
24 f(x13) ¼ 0.7674e
24
   
0:500732 0:000125 0:500089 0:000135
H 15 ¼ H 13 ¼
0:000125 0:5002158 0:000135 0:500195

Now, from the initial point x0 ¼ [1.1, 0]T and with H0 ¼ ∇2f(x0), then the DFP and the BFGS
methods give the following results.

DFP BFGS
T
x25 ¼ ½ 2
0:999999  x32 ¼ ½ 2
1 T
f(x25) ¼ 0.2317e
23 f(x32) ¼ 0.141342e
14
   
0:5 0:408111e
6 0:4995756
0:423871e
3
H 25 ¼ H 32 ¼
0:408111e
6 0:5
0:423871e
3 0:4995739

 
From the initial point x0 ¼ [1.1, 0]T and with H0 ¼ sT0 y0 =yT0 y0 I, then the DFP and the BFGS
methods give the following results.

DFP BFGS
x14 ¼ ½ 2
1 T x14 ¼ ½ 1:999999
0:999999 T
f(x14) ¼ 0.6751e
13 f(x14) ¼ 0.6006455e
13
   
0:500222 0:113729e
3 0:5002339 0:115678e
3
H 14 ¼ H 14 ¼
0:113729e
3 0:50005778 0:1156781e
3 0:50005478

Proposition 6.1 Let Bk be a symmetric and positive definite matrix, and assume that Bkþ1 is obtained
from Bk using the BFGS update formula. Then, Bkþ1 is positive definite if and only if yTk sk > 0:

Proof If Bk is positive definite, then it can be factored as Bk ¼ Lk LTk , where Lk is nonsingular. (This
is exactly the Cholesky factorization of Bk. ) Now, if this factorization is introduced into the BFGS
formula for the Bkþ1 computation, then

Bkþ1 ¼ Lk W k LTk ,

where

b
sk b
sTk ybk ybTk
1
Wk ¼ I
b b
T þ T , sk ¼ Lk sk , yk ¼ Lk yk :
T
b
sk b
sk ybk b sk
6.1 DFP and BFGS Methods 267

Observe that Bkþ1 will be positive definite if and only if Wk is positive definite. To see that Wk is
positive definite, let us test if vTWkv > 0 for all v 6¼ 0. Let θ1 be the angle between v and b
sk , θ2 the
angle between v and ybk , and θ3 the angle between b sk and ybk : Then,
2 2
ðv T b
sk Þ ðvT yb Þ
vT W k v ¼ vT v
þ Tk
bsk b
T
sk ybk b
sk
kvk kb2
sk k cos 2 θ1 kvk2 kybk k2 cos 2 θ2
2
¼ kv k2
þ
kbs k k2 kybk kkbs k cos θ3
  k
k b
y k cos 2
θ 2
¼ kvk2 1
cos 2 θ1 þ k
 kbsk k cos
 θ3
kyb k cos 2 θ2
¼ kvk2 sin 2 θ1 þ k :
kbsk k cos θ3

If yTk sk > 0, then ybTk b

sk > 0 and cosθ3 > 0. Hence, vTWkv > 0, that is, Wk is positive definite. If
yTk sk < 0, then cosθ3 < 0. In this case, v can be chosen so that vTWkv < 0, that is, Wk is not positive
definite, which completes the proof. ♦

The new update Bkþ1 matrix will be positive definite only if yTk sk > 0: This property can be
guaranteed by performing an appropriate line-search, for example, the Wolfe line-search. Therefore,
the quasi-Newton method based on the BFGS updating formula is often implemented with the (weak)
Wolfe line-search (6.5) and (6.6).

Numerical Study: BFGS

In the first set of the numerical experiments that follow, let us see the performances of the BFGS
algorithm versus the conjugate gradient algorithm CONMIN for solving 800 unconstrained optimi-
zation problems from the UOP collection (Andrei, 2020a) with the number of variables in the range
[100, 1000]. In the second set of numerical experiments, the performance profile of BFGS versus the
conjugate gradient algorithm DESCON is presented. The third set of numerical experiments presents
the performances of BFGS versus the conjugate gradient CG-DESCENT with the Wolfe line-search
and versus CG-DESCENTaw with the approximate Wolfe line-search. Figure 6.1 shows the perfor-
mance profiles of these algorithms. The conjugate gradient algorithms CONMIN, DESCON,
CG-DESCENT, and CG-DESCENTaw are way more efficient and more robust than BFGS. In
Fig. 6.1, we can see that CONMIN was the fastest in solving 653 problems from this collection,
while BFGS was the fastest only in 25 problems. Out of 800 problems considered in this numerical
study, only for 785 problems does the criterion (1.3) hold. Similarly, DESCON was the fastest in
solving 666 problems, but BFGS was the fastest only in 0 problems, etc. All these algorithms
implement the weak Wolfe line-search with cubic interpolation and safeguarded stepsizes (see
subroutine wolfeLS in Fig. 2.3). Even if the BFGS algorithm has some very interesting convergence
properties which will be discussed in the following lines, the conjugate gradient methods are way
more efficient and more robust.

The Broyden Class

There are plenty of quasi-Newton updating formulae, but we have described only DFP and BFGS.
The Broyden class of the quasi-Newton methods is a family of updates specified by the following
general formula:
268 6 Quasi-Newton Methods

Fig. 6.1 Performance profiles of BFGS versus CONMIN, DESCON, CG-DESCENT, and CG-DESCENTaw

Bk sk sTk Bk yk yTk  
Bkþ1 ¼ Bk
þ T þ Φk sTk Bk sk vk vTk , ð6:23Þ
sTk Bk sk yk sk

where Φk is a scalar parameter and

yk Bs
vk ¼ T
Tk k : ð6:24Þ
y k sk sk B k sk

The BFGS and DFP algorithms are members of this family. By setting Φk ¼ 0, the BFGS update is
obtained, while by setting Φ ¼ 1, the DFP is obtained.
Observe that the last term in (6.23) is a rank-one correction, which, by the interlacing eigenvalue
theorem, increases the eigenvalues of the matrix Bkþ1 when Φk is positive. Therefore, Bkþ1 is positive
definite for all Φk  0. On the other hand, when Φk is negative, the last term in (6.23) decreases the
eigenvalues of Bkþ1. Obviously, when Φk is decreasing, Bkþ1 becomes singular and then indefinite.
By simple algebraic manipulations, we obtain that Bkþ1 is singular for Φk ¼ Φck , where
 T
1  T 
1 y k Bk y k s k B k s k
Φk ¼
c
, and μk ¼  T 2 : ð6:25Þ
1
μk y sk
k
6.1 DFP and BFGS Methods 269

By the Cauchy-Schwarz inequality, we see that μk  1, and therefore, Φck  0: If the initial Hessian
approximation B0 is symmetric and positive definite and if yTk sk > 0 and Φk > Φck for each k, then all
the matrices Bk generated by the Broyden update (6.23) remain symmetric and positive definite.
Moreover, for general nonlinear functions, when the line-search is exact, all the methods in the
Broyden class with Φk  Φck generate the same sequence of iterates because, when all the line-
searches are exact, the directions generated by the Broyden class of updates differ only in their
lengths.
Applied with exact line-searches to quadratic functions, the Broyden class of methods has some
remarkable properties, as it is stated in the following theorem.

Theorem 6.1 Suppose that a method in the Broyden class is applied to minimize the strongly convex
quadratic function f ðxÞ ¼ 12 xT Ax þ bT x, where x0 is the initial point and B0 is any symmetric and
positive definite matrix. Assume that αk is the exact stepsize and that Φk  Φck for all k, where Φck is
given by (6.25). Then, the following statements are true:

1. The iterates are independent of Φk and converge to the minimum point in at most n iterations.
2. The secant equation is satisfied for all previous search directions, that is, Bksj ¼ yj, j ¼ k
1,
k
2, . . ., 1.
3. If the initial matrix B0 ¼ I, then the iterations are identical to those generated by the conjugate
gradient method. In particular, the search directions are conjugate, that is, sTi Asj ¼ 0, for i 6¼ j.
4. If n iterations are performed, then Bn ¼ A. ♦

Theorem 6.1 can be generalized in the sense that it continues to hold if the Hessian approximations
remain nonsingular, but not necessarily positive definite. More than that, if the initial matrix B0 is not
the identity matrix, then the Broyden class of methods is identical to the preconditioned conjugate
gradient method that uses B0 as preconditioner.
Since BFGS and DFP conserve the positive definiteness of the Hessian approximations when
yTk sk > 0, it follows that the same property will hold for the Broyden family if 0  Φk  1. The
restricted Broyden class is obtained by restricting Φk to the interval [0,1).
The following result shows the convergence of the restricted Broyden class of update formulae
(Byrd, Nocedal, & Yuan, 1987a). The theorem assumes that ∇2f(x) is always positive definite, that is,
the objective function f is strictly convex. It excludes the DFP formula.

Theorem 6.2 Let f : ℝn ! ℝ be a real-valued function. Let x0 be an initial point and {xk} a sequence
where xkþ1 ¼ xk þ αkdk, αk  0 is a scalar, and dk 2 ℝn. Assume that the level set S ¼ {x : f(x)  f(x0)}
is bounded, f(x), ∇f(x), and ∇2f(x) are continuous for all x 2 S, ∇2f(x) is positive definite for all x, the
search directions {dk} are computed as solutions of the linear systems Bkdk ¼
∇ f(xk), where
B0 ¼ I, and the matrices {Bk} are updated using the restricted Broyden class, with 0  Φk < 1, the
stepsizes {αk} satisfy the Wolfe line-search (6.5) and (6.6) with 0 < ρ < σ < 1, and the line-search uses
αk ¼ 1 whenever possible. Then,
lim xk ¼ x ,
k!1

where x is the unique global minimizer of f and the rate of convergence of {xk} is superlinear. ♦
270 6 Quasi-Newton Methods

The Global Convergence of BFGS

In the convergence analysis, a key requirement for a line-search algorithm is that the search direction
dk is a direction of sufficient descent, which is defined as

gTk dk

ε, ð6:26Þ
kgk kkd k k

where ε > 0. This condition bounds the elements of the sequence {dk} of the search directions from
being arbitrarily close to the orthogonality to the gradient. Often, the line-search methods are so that
dk is defined in a way that satisfies the sufficient descent condition (6.26), even though an explicit
value for ε > 0 is not known.

Theorem 6.3 Suppose that {Bk} is a sequence of bounded and positive definite symmetric matrices
whose condition number is also bounded, i.e., the smallest eigenvalue is bounded away from zero. If
dk is defined to be the solution of the system Bkdk ¼
gk, then {dk} is a sequence of sufficient descent
directions.

Proof Let Bk be a symmetric positive definite matrix with eigenvalues 0 < λk1  λk2  ⋯  λkn :
Therefore, from Bkdk ¼
gk, it follows that

kgk k ¼ kBk d k k  kBk kkd k k ¼ λkn kdk k: ð6:27Þ

From the system Bkdk ¼
gk, using (6.27), we have

gTk dk d T Bk d k kd k k2 kd k kd k λk

¼ k  λk1 ¼ λk1 k  λk1 k k ¼ 1k > 0:
kgk kkd k k kgk kkd k k kgk kkd k k kgk k λn kd k k λn

The quality of the search direction dk can be determined by studying the angle θk between the
steepest descent direction
gk and the search direction dk. Hence, by applying this result to each
matrix in the sequence {Bk}, we get

gTk dk λk 1
cos θk ¼
 1k  , ð6:28Þ
kgk kkd k k λ n M

where M is a positive constant. Observe that M is a positive constant and it is well-defined since the
smallest eigenvalue of the matrices Bk in the sequence {Bk} generated by the algorithm is bounded
away from zero. Therefore, the search directions {dk} generated as solutions of Bkdk ¼
gk form a
sequence of sufficient descent directions. ♦

The main consequence of this theorem on how to modify the quasi-Newton system defining the
search direction dk is to ensure that dk is a solution of a system that has the same properties as Bk.
A global convergence result for the BFGS method was given by Powell (1976). Using the trace and
the determinant to measure the effect of the two rank-one corrections on Bk in (6.22), he proved that if
f is convex, then for any starting point x0 and any positive definite starting matrix B0, the BFGS
method gives liminfk!1kgkk ¼ 0. In addition, if the sequence {xk} converges to a solution point at
which the Hessian matrix is positive definite, then the rate of convergence is superlinear. The analysis
of Powell was extended by Byrd, Nocedal, and Yuan (1987a) to the Broyden class of quasi-Newton
methods (see Theorem 6.2).
6.1 DFP and BFGS Methods 271

With the Wolfe line-search, the BFGS approximation is always positive definite, so the line-search
works very well. It behaves “almost” like the Newton method in the limit (the convergence is
superlinear). DFP has the interesting property that, for a quadratic objective, it simultaneously
generates the directions of the conjugate gradient method while constructing the inverse Hessian.
However, DFP is highly sensitive to the inaccuracies in the line-searches.
Consider the BFGS algorithm with the Wolfe line-search for minimizing a twice continuously
differentiable function from an arbitrary initial point x0 and from any initial approximation of the
Hessian B0, that is, a symmetric and positive definite matrix. The following assumptions are supposed
to be satisfied by the minimizing function f:

Assumption A
(i) The minimizing function f is twice continuously differentiable.

(ii) The level set S ¼ {x 2 ℝn : f(x)  f(x0)} is convex, and there exist the positive constants m and M
such that

mkzk2  zT GðxÞz  Mkzk2 , ð6:29Þ

for all z 2 ℝn and x 2 S, where G(x) ¼ ∇2f(x).

Observe that (ii) implies that the Hessian G(x) is positive definite on S and, therefore, f has a unique
minimum.
Now, since yk ¼ Gk sk , where Gk is the average Hessian, from (6.29), it follows that

yTk sk sTk Gk sk
¼ T  m: ð6:30Þ
sTk sk sk sk
1=2
On the other hand, by Assumption A, Gk is positive definite, and defining zk ¼ Gk sk , it follows
that
2
yTk yk sTk Gk sk zTk Gk zk
¼ ¼ T  M: ð6:31Þ
yTk sk sTk Gk sk zk zk

For proving the global convergence of the BFGS method, instead of establishing a bound on the
condition number of the Hessian approximations Bk as it was considered for proving the convergence
of the descent direction algorithms (see Chap. 2), Byrd, Nocedal, and Yuan (1987a) introduced two
new tools based on the trace and determinant in order to estimate the size of the largest and the
smallest eigenvalues of the Hessian approximations.

Theorem 6.4 Let B0 be any symmetric and positive definite initial matrix, and consider x0 as the
starting point for which Assumption A is satisfied. Then, the sequence {xk} generated by the BFGS
Algorithm 6.1 is convergent to the minimizer x of f.

Proof Define

yTk sk yTk yk
mk ¼ , M k ¼ : ð6:32Þ
sTk sk yTk sk

From (6.30) and (6.31), it follows that

272 6 Quasi-Newton Methods

mk  m, Mk  M: ð6:33Þ

Now, from (6.22) (see Appendix A), we have

k Bk s k k 2 k y k k 2
traceðBkþ1 Þ ¼ traceðBk Þ
þ T ð6:34Þ
sTk Bk sk y k sk

and

yTk sk
detðBkþ1 Þ ¼ detðBk Þ T : ð6:35Þ
s k Bk s k

Now, let us define

sTk Bk sk s T Bk s k
cos θk ¼ , qk ¼ k T , ð6:36Þ
ks k kk Bk s k k sk sk

where, as we can see, θk is the angle between sk and Bksk.

With this,
 
kBk sk k2 kBk sk k2 ksk k2 sTk Bk sk qk
¼  2 ¼ 2θ
: ð6:37Þ
sTk Bk sk T
s Bk s k ksk k 2 cos k
k

Now, a combination of the trace and determinant is introduced by the following function of a
positive definite matrix B:
φðBÞ ¼ traceðBÞ
ln ðdetðBÞÞ, ð6:38Þ

where ln(.) is the natural logarithm. Fletcher (1991) observed that both the BFGS and the DFP
formulae can be derived by a variational argument using function φ. This is an elegant and efficient
tool for analyzing the global behavior of quasi-Newton methods. Observe that function φ works
simultaneously with trace and determinant, thus simplifying the analysis of the quasi-Newton
methods. In fact, this function is a measure of the matrices involving all the eigenvalues of B, not
only the smallest and the largest, as it is traditionally used in the analysis of the quasi-Newton
methods based on the condition number of matrices (Andrei, 2015c, 2016, 2017a). Observe that this
function is strictly convex on the set of symmetric and positive definite matrices and is minimized by
B ¼ I. Besides, it becomes unbounded as B becomes singular or infinite, and therefore, it works as a
barrier function that keeps B positive definite.

With this, we can write

qk
φðBkþ1 Þ ¼ traceðBk Þ þ Mk

ln ðdetðBk ÞÞ
ln ðmk Þ þ ln ðqk Þ
cos 2 θk  
qk qk ð6:39Þ
¼ φðBk Þ þ ðMk
ln ðmk Þ
1Þ þ 1
þ ln þ ln cos 2 θk :
cos 2 θk cos 2 θk

Since the function h(t) ¼ 1
t þ ln (t) is nonpositive for all t > 0, it follows that the term inside the
brackets is nonpositive, and thus, from (6.33) and (6.39), it follows that
X
k
0 < φðBkþ1 Þ  φðB0 Þ þ cðk þ 1Þ þ ln cos 2 θj , ð6:40Þ
j¼0
6.1 DFP and BFGS Methods 273

where without the loss of the generality, we can assume that the constant c ¼ M
ln (m)
1 is
positive.
Observe that cosθk defined by (6.36) is the angle between the steepest descent direction and the
search direction, which is crucial in the global convergence of the descent direction algorithms. We
know that the sequence {kgkk} generated by the line-search algorithm is bounded away from zero
only if cosθj ! 0 (see Corollary 2.1).
Now, let us proceed by contradiction, and assume that cosθj ! 0. Then, there exists k1 > 0 such
that for all j > k1 we have ln(cos2θj) <
2c, where c is the constant defined above. Therefore, from
(6.40), it follows that for all k > k1 we have

X
k1 X
k
0 < φð B0 Þ þ c ð k þ 1 Þ þ ln cos 2 θj þ ð
2cÞ
j¼0 j¼k1 þ1

X
k1
¼ φð B0 Þ þ ln cos 2 θj þ 2ck1 þ c
ck:
j¼0

However, the right-hand side is negative for large k, thus giving a contradiction. Therefore, there
exists a subsequence of indices {jk}k¼1, 2, . . . such that cos θjk  δ > 0: By the Zoutendijk condition, it
follows that lim inf k∇f(xk)k ! 0. Since the problem is strongly convex, this limit is sufficient to
prove that xk ! x. ♦

Theorem 6.4 has been generalized to the entire restricted Broyden class, except for the DFP
method. That is, Theorem 6.4 holds for all Φk 2 [0, 1).
The BFGS method has many interesting properties. For example, the BFGS updating formula has
effective self-correcting properties. If the matrix Bk incorrectly estimates the curvature of the
minimizing function, and if this bad estimate slows down the iteration, then the Hessian approxima-
tion will tend to correct itself within a few steps. The DFP method is less effective in correcting bad
Hessian approximations, and it is believed that this is the reason for its poorer practical performances.
The self-correcting property of BFGS holds only when the Wolfe line-search is used for the stepsize
determination.
The BFGS updating formula is known to have the so-called bounded deterioration property
(Broyden, Dennis, & Moré, 1973), namely, the approximate Hessians do not drift too far from the
exact Hessian if the initial data are good. An updating formula for the Hessian approximation Bk
satisfies the bounded deterioration property if there exists a constant c > 0 so that for all xk and Bk the
new xkþ1 and Bkþ1 satisfy

Bkþ1
∇2 f ðx Þ  Bk
∇2 f ðx Þ ð1 þ σÞ þ cσ,

where σ ¼ max {kxk
xk, kxkþ1
xk}. If a quasi-Newton method satisfies the bounded
deterioration property and there exist the positive constants ε and δ so that kx0
xk  ε and
kB0
∇2f(x)k  δ, then the sequence {xk} generated by the algorithm is well-defined and converges
q-linearly to x. In order to have q-superlinearly convergence of the algorithm, it is necessary to have
the consistency of it. A quasi-Newton method is consistent if {xk} converges to x, then {Bk}
converges to ∇2f(x). The consistency condition is sufficient but not necessary.
Assuming that the initial point x0 is sufficiently close to the solution x and that the initial Hessian
approximation is sufficiently close to ∇2f(x), then using the bounded deterioration property we can
prove that the iteration cannot stay away from the solution.
274 6 Quasi-Newton Methods

Another interesting property is the Dennis-More´ condition. If dk ¼
B
1

k ∇f ðxk Þ is a BFGS search
direction or a quasi-Newton search direction, then
 
B k
∇2 f ð x  Þ d k
lim ¼ 0: ð6:41Þ
k!1 kd k k

Therefore, a superlinear convergence rate can be attained even if the sequence of the quasi-Newton
matrices Bk does not converge to ∇2f(x); suffice it to say that the Bk become increasingly accurate
approximations to ∇2f(x) along the search directions dk. Specifically, we have

Theorem 6.5 Suppose that f : ℝn ! ℝ is twice continuously differentiable, and consider the iteration
xkþ1 ¼ xk þ αkdk, where dk is a descent direction and αk satisfies the Wolfe line-search conditions
(6.5) and (6.6) with ρ  1/2. If the sequence {xk} converges to a point x such that ∇f(x) ¼ 0 and
∇2f(x) is positive definite, and if the search direction satisfies

∇f ðxk Þ
∇2 f ðxk Þd k
lim ¼ 0,
k!1 kd k k

then the stepsize αk ¼ 1 is admissible for all k greater than a certain index k0; if αk ¼ 1 for all k > k0,
then {xk} converges to x superlinearly.

The following theorem emphasizes that the Dennis-Moré condition (6.41) is both necessary and
sufficient for the superlinear convergence of the quasi-Newton methods.

Theorem 6.6 Suppose that f : ℝn ! ℝ is twice continuously differentiable. Consider the iteration
xkþ1 ¼ xk þ dk, that is, the stepsize αk is uniformly one and dk ¼
B
1
k ∇f ðxk Þ. Let us assume that the
sequence {xk} converges to a point x such that ∇f(x) ¼ 0 and ∇2f(x) is positive definite. Then, the
sequence {xk} converges superlinearly if and only if the Dennis-More´ condition (6.41) holds.

Proof Let dNk ¼
∇2 f ðxk Þ
1 ∇f ðxk Þ be the Newton step. Let us show that (6.41) is equivalent to

d k
dNk ¼ oðkd k kÞ: ð6:42Þ

Assuming that (6.41) holds and since k∇2f(xk)
1k is bounded above for xk sufficiently close to x
and since ∇2f(x) is positive definite, then it follows that
 
dk
dNk ¼ ∇2 f ðxk Þ
1 ∇2 f ðxk Þdk þ ∇f ðxk Þ
 
¼ ∇2 f ðxk Þ
1 ∇2 f ðxk Þ
Bk dk
 2 
¼ O ∇ f ðxk Þ
Bk ¼ oðkdk kÞ:

The converse follows if (6.42) is multiplied by ∇2f(xk). We have


kxk þ dk
x k  xk þ dNk
x þ dk
d Nk ¼ O kxk
x k2 þ oðkd k kÞ:

Therefore, kdkk ¼ O(kxk
xk), that is, kxk þ dk
xk  o(kxk
xk), proving the superlinear
convergence of {xk}. ♦
6.1 DFP and BFGS Methods 275

An extension of this analysis shows that the rate of convergence of the BFGS algorithm is linear.
More exactly, we can show that the sequence {kxk
xk} converges to zero rapidly enough, so that
X
1
kxk
x k < 1: ð6:43Þ
k¼1

If the minimizing function is twice continuously differentiable and the iterates generated by the
BFGS algorithm converge to a minimizer x at which the Hessian ∇2f(x) is Lipschitz continuous and
(6.43) holds, then {xk} converges to x at a superlinear rate. More exactly, the local superlinear
convergence of the BFGS algorithm can be established by the following arguments, presented for
completeness.
Consider that the Hessian ∇2f(x) is Lipschitz continuous, i.e., k∇2f(x)
∇2f(x)k  Lkx
xk for
all x near enough x and L, a positive constant. Let us introduce the following quantities:

e
sk ¼ G1=2 ek ¼ G
1=2
 sk , y  yk , Bek ¼ G
1=2
 Bk G
1=2


where G ¼ ∇2f(x) and x is a minimizer of function f. Define:

sTk Bek e
e sk sT Bek e
e sk e k ¼ kyek k , m
2
yeTk e
sk
cos e
θk ¼ , qek ¼ k 2 , M e ¼ T :
ke e
sk k B k esk ke
sk k yek e
T
sk
k
e
sk esk

Now, pre- and post-multiplying (6.22) by ∇2F(x)
1/2 and grouping the terms, we obtain

Bek e sT Bek ye yeT

sk e
Bekþ1 ¼ Bek
T k þ kT k :
sk Bek e
e sk yek e
sk

Therefore, as in Theorem 6.4 above, it follows that



  qek qek

e e e
φ Bkþ1 ¼ φ Bk þ Mk
ln ðm ek Þ
1 þ 1
þ ln þ ln cos 2e
θk : ð6:44Þ
cos 2eθk cos 2e θk
   
However, yk
G sk ¼ Gk
G sk : Therefore, yek
e sk ¼ G
1=2
 G
G G
1=2 e
sk : Since the
Hessian matrix G is Lipschitz continuous, and having in view the definition of the average Hessian,
it follows that
2 2
sk k  G
1=2
kyek
e 
ke
sk k Gk
G  G
1=2

ke
sk kLεk ,

where εk ¼ max {kxkþ1
xk, kxk
xk}. Therefore,

kyek
e sk k
 cεk , ð6:45Þ
kesk k

for some positive constant c: This inequality has a crucial role in the local superlinear convergence of
the BFGS method.

Theorem 6.7 Suppose that f is twice continuously differentiable and the sequence {xk} generated by
the BFGS algorithm converges to the minimum point x at which the Hessian is Lipschitz continuous.
Also suppose that (6.43) holds. Then, the sequence {xk} converges to x at a superlinear rate.
276 6 Quasi-Newton Methods

Proof From (6.45) and from the triangle inequality, it follows that kyek k
ke
sk k  cεk ke
sk k,
sk k
kyek k  cεk ke
ke sk k, so that
sk k  kyek k  ð1 þ cεk Þke
ð1
cεk Þke sk k: ð6:46Þ

By squaring (6.45) and using (6.46), we get

ð1
cεk Þ2 ke
sk k2
2e sk þ ke
yTk e sk k2  kyek k2
2e sk þ ke
yTk e sk k2  c2 ε2k ke
s k k2 ,

and therefore,
 
yTk e
2e sk k2 ¼ 2ð1
cεk Þke
sk  1
2cεk þ c2 ε2k þ 1
c2 ε2k ke s k k2 :

Hence,

yeTk e
sk
ek ¼
m  1
cεk : ð6:47Þ
e
sTk e
sk

Now, from (6.46), it follows that

e k ¼ kyek k  1 þ cεk :
2
M ð6:48Þ
yeTk e
sk 1
cεk

Since xk ! x, it follows that εk ! 0. By (6.48), there exists a positive constant c > c such that the
following inequalities hold for all sufficiently large k:

ek  1 þ 2c
M ε  1 þ cεk : ð6:49Þ
1
cεk k

Since the function h(t) ¼ 1
t þ ln (t) is nonpositive, it follows that



x 1

ln ð1
xÞ ¼ h  0:
1
x 1
x
Now, for k large enough, we can assume that cεk < 1=2, and therefore,

cεk
ln ð1
cεk Þ  
2cεk :
1
cεk

This relation and (6.47) imply that for k large enough, we have
ek Þ  ln ð1
cεk Þ 
2cεk >
2cεk :
ln ðm ð6:50Þ

Therefore, from (6.44), (6.49) and (6.50), we obtain



  qek qek

0 < φ Bekþ1  φ Bek þ 3cεk þ ln cos 2e
θk þ 1
þ ln : ð6:51Þ
cos 2eθk cos 2eθk

Now, summing these expressions and using (6.43), we obtain

6.1 DFP and BFGS Methods 277

" #!
X
1
qej qej
 X1
1
ln
1
þ ln  φ Be0 þ 3c εj < þ1
j¼0 cos 2e
θj cos 2eθj cos 2eθj j¼0


Since the term in the square brackets is nonpositive and since ln 1= cos 2e
θj  0 for all j, it
follows that
!
1 qej qej
lim ln ¼ 0, lim 1
þ ln ¼ 0,
j!1 cos e
2 θj j!1 cos e2 θj cos 2eθj

which implies that

lim cos e
θj ¼ 1, lim qej ¼ 1: ð6:52Þ
j!1 j!1

From (6.37), we have

 2

1=2 2 e
G ð B
G Þs
 k
B k
I e
sk
 k
1=2 2 ¼
G sk ke
s k k2

2
Bek e
sk
2e sTk Bek e
sk þ e
sTk sk qe2k
¼ ¼
2e
qk þ 1:
eT
sk sk cos e
2
θk

Since by (6.52) the right-hand side converges to zero, it follows that

kðBk
G Þsk k
lim ¼ 0:
k!1 ks k k

Theorem 6.5 implies that the unit stepsize αk ¼ 1 will satisfy the Wolfe line-search near the
solution, and therefore, the rate of convergence is superlinear. ♦

The Memoryless BFGS Method

By considering Hk ¼ I in (6.21), the memoryless BFGS method is obtained as
 
sk y T þ y sT y T y sk sT
H kþ1 ¼ I
k T k k þ 1 þ kT k T k : ð6:53Þ
y k sk yk sk yk sk

The corresponding search direction is dkþ1 ¼
Hkþ1gkþ1, where Hkþ1 is given by (6.53), i.e.,
 T      T 
y g sk þ sT g y yT y sk gkþ1 sk
dkþ1 ¼
gkþ1 þ k kþ1 T k kþ1 k
1 þ kT k : ð6:54Þ
y k sk y k sk yTk sk

Observe that the numerical computation of dkþ1 from (6.54) involves only four scalar products:
yTk sk , yTk yk , yTk gkþ1 , and sTkþ1 gkþ1 : Therefore, it is very suitable to solve large-scale problems. It is
worth seeing that the search direction corresponding to the memoryless BFGS updating has three
terms. Besides, it is easy to prove that this search direction satisfies the Dai-Liao conjugacy condition,
i.e., yTk dkþ1 ¼
sTk gkþ1 : For the exact line-search, that is, when sTk gkþ1 ¼ 0, the memoryless BFGS
search direction is exactly the search direction of Hestenes and Stiefel. Therefore, there is a close
connection between the quasi-Newton and the conjugate gradient methods. Shanno (1978a, 1978b)
278 6 Quasi-Newton Methods

was the first who observed that the conjugate gradient methods are precisely the quasi-Newton
methods where the approximation to the inverse to the Hessian is restarted as the identity matrix at
every iteration.

6.2 Modifications of the BFGS Method

In the following, some modifications of the BFGS updating method, both subject to its updating
formula and to the line-search conditions, are going to be presented. Intensive numerical experiments
on minimizing functions with different dimensions and complexities showed that the BFGS method
may require a large number of iterations or function and gradient evaluations on certain problems
(Gill, & Leonard, 2003). The sources of the inefficiency of the BFGS method may be caused by a poor
initial approximation to the Hessian or, more importantly, by the ill-conditioning of the Hessian
approximations along the iterations. To improve the efficiency and the robustness of the BFGS
method and to overcome the difficulties, some modified versions of it were given. All these modified
BFGS methods can be classified into three large classes: the scaling of the BFGS update matrix, the
BFGS update with modified secant equation, and the modified BFGS method using different line-
search conditions for the stepsize computation. The scaling of the BFGS update has two
developments: sizing, i.e., multiplying the approximate Hessian matrix by an appropriate scalar
before it is updated in the BFGS method (Contreras and Tapia (1993), Oren and Luenberger
(1974), Oren and Spedicato (1976), Shanno and Phua (1978), Yabe, Martı́nez, and Tapia (2004)),
and the proper scaling of the terms on the right-hand side of the BFGS updating formula with positive
factors (Biggs (1971, 1973), Oren (1972), Liao (1997), Nocedal and Yuan (1993), Andrei (2018a,
2018c, 2018d, 2018f)). The purpose of the BFGS update with modified secant equation is to
approximate the curvature of the objective function along the search direction more accurately than
the standard secant equation does (Yuan (1991), Yuan and Byrd (1995), Al-Baali (1998), Zhang,
Deng, and Chen (1999), Zhang, and Xu (2001), Wei, Yu, Yuan, and Lian (2004), Zhu, and Wen
(2006), Yabe, Ogasawara, and Yoshino (2007), Al-Baali, and Grandinetti (2009), Yuan, and Wei
(2010), Wu, and Liang (2014), Arzam, Babaie-Kafaki, and Ghanbari (2017)). The BFGS methods
with new line-search conditions for the stepsize computation try to ensure the global convergence by
modifying the Wolfe line-search conditions (Wan, Huang, and Zheng (2012), Wan, Teo, Shen, and
Hu (2014), Yuan, Wei, and Lu (2017), Yuan, Sheng, Wang, Hu, and Li (2018), Dehmiry, 2019).

Scaling the Terms on the Right-Hand Side of the BFGS Update

From (6.22), we can see that the BFGS update involves two correction matrices, each of rank one.
Therefore, by the interlocking eigenvalue theorem of Wilkinson (1965), the first rank-one correction
matrix which is subtracted decreases the eigenvalues, i.e., it shifts the eigenvalues to the left. On the
other hand, the second rank-one matrix which is added shifts the eigenvalues to the right. More
exactly, two important tools in the analysis of the properties and of the convergence of the BFGS
method are the trace and the determinant of the standard Bkþ1 given by (6.22). The trace of a matrix is
exactly the sum of its eigenvalues. The determinant of a matrix is the product of its eigenvalues. By
direct computation, from (6.22), we get (see Appendix A)

k Bk s k k 2 k y k k 2
tr ðBkþ1 Þ ¼ tr ðBk Þ
þ T :
sTk Bk sk y k sk

On the other hand,

6.2 Modifications of the BFGS Method 279

yTk sk
detðBkþ1 Þ ¼ detðBk Þ :
sTk Bk sk

As it is known, the efficiency of the BFGS method is dependent on the structure of the eigenvalues
of the approximation to the Hessian matrix (Nocedal, 1992). Powell (1987) and Byrd, Liu, and
Nocedal (1992) emphasized that the BFGS method actually suffers more from the large eigenvalues
than from the small ones. Observe that the second term on the right-hand side of tr(Bkþ1) is negative.
Therefore, it produces a shift of the eigenvalues of Bkþ1 to the left. Thus, the BFGS method is able to
correct large eigenvalues. On the other hand, the third term on the right-hand side of tr(Bkþ1) being
positive produces a shift of the eigenvalues of Bkþ1 to the right. If this term is large, Bkþ1 may have
large eigenvalues, too. Therefore, a correction of the eigenvalues of Bkþ1 can be achieved by scaling
the corresponding terms in (6.22), and this is the main motivation for which the scaled BFGS methods
is used. There must be a balance between these eigenvalue shifts; otherwise, the Hessian approxima-
tion could either approach singularity or become arbitrarily large, thus ruining the convergence of the
method. The scaling procedures of the BFGS update (6.22) with one or two parameters know the
following developments.

1. One parameter scaling the third term on the right-hand side of the BFGS update. In this case, the
general scaling BFGS updating formula is

Bk sk sTk Bk yk yTk
Bkþ1 ¼ Bk
þ γ k , ð6:55Þ
sTk Bk sk yTk sk

where γ k is a positive parameter. For the selection of the scaling factor γ k in (6.55), the following
procedures have been considered in literature.

1.1 Scaling BFGS with Hermite interpolation conditions (Biggs, 1971, 1973). If the objective
function is cubic along the line segment connecting xk
1 and xk if the Hermite interpolation is
used on the same line between xk
1 and xk, Biggs (1971) proposed the following value for the
scaling factor γ k:
6  
γk ¼ f ðxk Þ
f ðxkþ1 Þ þ sTk gkþ1
2: ð6:56Þ
yTk sk

For one-dimensional problems, Wang and Yuan (1992) showed that the scaling BFGS (6.55)
with γ k given by (6.56) and without line-search is r-linearly convergent.

1.2 Scaling BFGS with a simple interpolation condition (Yuan, 1991). By using a simple interpola-
tion condition on the quadratic approximation of the minimizing function f, the value for the
scaling parameter in (6.55) suggested by Yuan (1991) is
2  
γk ¼ f ðxk Þ
f ðxkþ1 Þ þ sTk gkþ1 : ð6:57Þ
yTk sk

Powell (1986a) showed that the scaling BFGS update (6.55) with γ k given by (6.57) is globally
convergent for convex functions with inexact line-search. However, for general nonlinear
functions, the inexact line-search does not involve the positivity of γ k. In these cases, Yuan
280 6 Quasi-Newton Methods

restricted γ k in the interval [0.01, 100] and proved the global convergence of this variant of the
scaling BFGS method.

1.3 Spectral scaling BFGS (Cheng, & Li, 2010). In this update, the scaling parameter γ k in (6.54) is
computed as

yTk sk
γk ¼ , ð6:58Þ
ky k k2

which is obtained as solution of the problem: minksk
γ kykk2.Observe that γ k given by (6.58) is
exactly one of the spectral stepsizes introduced by Barzilai and Borwein (1988). Therefore, the
scaling BFGS method (6.55) with γ k given by (6.58) is viewed as the spectral scaling BFGS
method. It is proved that this spectral scaling BFGS method with the Wolfe line-search is
globally convergent and r-linearly convergent for convex optimization problems. Cheng and Li
(2010) presented the computational evidence that their spectral scaling BFGS algorithm is the top
performer versus the standard BFGS and also versus the scaling BFGS algorithms by Al-Baali
(1998), Yuan (1991), and Zhang and Xu (2001).

1.4 Scaling BFGS with diagonal preconditioning and conjugacy condition (Andrei, 2018a). Andrei
(2018a) introduced another scaling BFGS update given by (6.54), in which the scaling parameter
γ k is computed in an adaptive manner as
( )
yTk sk
γ k ¼ min ,1 , ð6:59Þ
ky k k2 þ β k

where βk > 0 for all k ¼ 0, 1, . . . . Since under the Wolfe line-search conditions (6.5) and (6.6)
yTk sk > 0 for all k ¼ 0, 1, . . ., it follows that γ k given by (6.59) is bounded away from zero, i.e.,
0 < γ k  1. If γ k is selected as in (6.59), where βk > 0 for all k ¼ 0, 1, . . ., then the large eigenvalues
of Bkþ1 given by (6.55) are shifted to the left (Andrei, 2018a). Intensive numerical experiments



showed that this scaling BFGS algorithm with βk ¼
sTk gkþ1
is one of the best, being more
efficient and more robust versus the standard BFGS algorithm as well as versus some other
scaling BFGS algorithms, including the versions of Biggs (1971, 1973), Yuan (1991), and Cheng
and Li (2010).
Andrei (2018a) gives the following theoretical justification for selecting the parameter γ k as in



(6.59) with βk ¼
sTk gkþ1
: To have a good algorithm, we need γ kI to be a diagonal preconditioner
of ∇2f(xkþ1) that reduces the condition number to the inverse of ∇2f(xkþ1). Such matrix γ kI should
be a rough approximation to the inverse of ∇2f(xkþ1). Therefore, γ k can be computed to minimize
ksk
γ kykk2. On the other hand, for nonlinear functions, as known, the classical conjugacy
condition used by Hestenes and Stiefel (1952) for quadratic functions which incorporate the
second-order information is dTkþ1 yk ¼
sTk gkþ1 : Therefore, in this algorithm, γ kI is selected to be
a diagonal preconditioner of ∇2f(xkþ1) and also to minimize the conjugacy condition, i.e., γ k is
selected to minimize a combination of these two conditions:
n

o
min ksk
γ k yk k2 þ γ 2k
sTk gkþ1
:

2. One parameter scaling the first two terms of the BFGS update ((Oren, & Luenberger, 1974) and
(Nocedal, & Yuan 1993)). This scaling BFGS update was introduced by Oren and Luenberger
6.2 Modifications of the BFGS Method 281

(1974) in their study on self-scaling variable metric algorithms for the unconstrained optimization
and was defined as
 
Bk sk sTk Bk y yT
Bkþ1 ¼ δk Bk
T þ kT k , ð6:60Þ
s k Bk s k y k sk

where δk is a positive parameter. Oren and Luenberger (1974) suggested

yTk sk
δk ¼ T
ð6:61Þ
sk Bk sk

as being one of the best factors, since it simplifies the analysis of the eigenvalues structure of the
inverse Hessian approximation. Furthermore, Nocedal and Yuan (1993) presented a deep analysis
of this scaling quasi-Newton method and showed that even if the corresponding algorithm with
inexact line-search is superlinear convergent on general functions, it is computationally expensive
as regards the stepsize computation.

3. Two parameters scaling the terms on the right-hand side of the BFGS update ((Liao, 1997) and
(Andrei, 2018c, 2018d, 2018f)). In these methods, the scaling parameters of the terms on the right-
hand side of the BFGS update are selected to modify the structure of the eigenvalues of the
iteration matrix Bkþ1, mainly to cluster them and to shift the large ones to the left. The following
two approaches are known.
3.1 Scaling the first two terms on the right-hand side of the BFGS update with a positive
parameter and the third one with another positive parameter (Andrei, 2018c). Motivated by
the idea of changing the structure of the eigenvalues of the BFGS approximation to the
Hessian matrix, Andrei (2018c) proposed a double parameter scaling BFGS method in
which the updating of the approximation Hessian matrix Bkþ1 is computed as

 
Bk sk sT Bk y yT
Bkþ1 ¼ δk Bk
T k þ γ k kT k , ð6:62Þ
sk B k sk yk sk

where δk and γ k are positive parameters. In this scaling BFGS method, the parameter δk is
selected to cluster the eigenvalues of Bkþ1. On the other hand, γ k is determined to reduce the
large eigenvalues of Bkþ1, i.e., to shift them to the left, thus obtaining a better distribution of
the eigenvalues:
( )
yTk sk
γ k ¼ min

, 1 ð6:63Þ
ky k2 þ
s T g

k k kþ1

and
2
n
γ k kyyTkskk
δk ¼ k
2 : ð6:64Þ
n
ksBTkBskkskk
k

Theorem 6.8 If the stepsize αk is determined by the Wolfe line-search (6.5) and (6.6), Bk is positive
definite, and γ k > 0, then Bkþ1 given by (6.62) is also positive definite. ♦
282 6 Quasi-Newton Methods

For general nonlinear functions, this scaling BFGS algorithm with inexact line-search is globally
convergent under the very reasonable condition that the scaling parameters are bounded. Intensive
numerical experiments using over 80 unconstrained optimization test problems of different structures
and complexities showed that this double parameter scaling BFGS update is more efficient than the
standard BFGS algorithm and also than some other well-known scaling BFGS algorithms, including
those by Biggs (1971, 1973), Cheng and Li (2010), Liao (1997), Nocedal and Yuan (1993), and
Yuan (1991).

3.2 Scaling the first two terms on the right-hand side of the BFGS update with a positive parameter
and the third one with another positive parameter using the measure function of Byrd and
Nocedal (Andrei, 2018d, 2018f). In this method, the BFGS update is scaled as in (6.62), where
parameters δk and γ k are computed to minimize the measure function φ(.) of Byrd and Nocedal
(1989). Minimizing the function φ(Bkþ1) ¼ tr(Bkþ1)
ln (det(Bkþ1)) with respect to the
parameters δk and γ k, where Bkþ1 is given in (6.62), the following values are obtained:

n
1
δk ¼ 2 : ð6:65Þ
trðBk Þ
ksBTkBskkskk
k

yTk sk
γk ¼ : ð6:66Þ
ky k k2

Theorem 6.9 If the stepsize αk is determined by the Wolfe line-search (6.5) and (6.6), then the
scaling parameters δk and γ k given by (6.65) and (6.66), respectively, are the unique global solution
of the problem min δk >0,γk >0 φðBkþ1 Þ: ♦

Intensive numerical experiments in Andrei (2018d) proved that this scaling procedure of the BFGS
with two parameters is more efficient and more robust than the other scaling procedures, including
those of Biggs (1971, 1973), Cheng and Li (2010), Yuan (1991), Nocedal and Yuan (1993), Liao
(1997), and Andrei (2018c).

3.3 Scaling the last terms on the right-hand side of the BFGS update with two positive parameters
(Liao, 1997). Liao (1997) introduced the two parameter scaling BFGS method as

Bk sk sTk Bk yk yTk
Bkþ1 ¼ Bk
δk þ γ k ð6:67Þ
sTk Bk sk yTk sk

and proved that this method corrects the large eigenvalues better than the standard BFGS method
given by (6.22). In other words, it was proved that this scaling BFGS method has a strong self-
correcting property with respect to the determinant (Liao, 1997). In Liao’s method, the
parameters scaling the terms in the BFGS update are computed in an adaptive way subject to
the values of a positive parameter as
6.2 Modifications of the BFGS Method 283

8 
< sTk Bk sk yTk sk sTk Bk sk
, , if  τk ,
ðδ k , γ k Þ ¼ sT Bk sk þ yTk sk sTk Bk sk þ yTk sk sTk Bk sk þ yTk sk ð6:68Þ
: k
ðτk , 1Þ, otherwise,

where 0 < τk < 1. (Liao proposed τk ¼ exp (
1/k2). ) Liao proved that the scaling BFGS method
given by (6.67) and (6.68) with the Wolfe line-search generates iterations which converge
superlinearly to the optimal solution. Limited numerical experiments with Liao’s scaling
BFGS method proved that this is competitive with the standard BFGS method, and it corrects
large eigenvalues better than the standard BFGS method does. However, subject to other scaling
BFGS updates, the scaling BFGS update by Liao is less efficient and less robust (Andrei, 2018d).

Numerical Study: Scaling the Terms on the Right-Hand Side of the BFGS Update
In the following, let us consider the set of 80 unconstrained optimization test problems from the UOP
collection (Andrei, 2020a) each of them with n ¼ 100 variables. The following algorithms were
compared: BFGS defined by (6.22), BFGSB defined by (6.55) with (6.56), BFGSY defined by (6.55)
with (6.57), BFGSC defined by (6.55) with (6.58), BFGSD defined by (6.55) with (6.59), BFGSE
defined by (6.62) with (6.63) and (6.64), and BFGSFI defined by (6.62) with (6.65) and (6.66). For the
BFGSB and BFGSY, the scaling parameter γ k given by (6.56) and (6.57), respectively, is restricted in
the interval [0.01, 100]. Besides, at the very first iteration of these methods, the scaling is not applied.
The stepsize is computed by the Wolfe line-search conditions (6.5) and (6.6), where σ ¼ 0.8 and
ρ ¼ 0.0001. The iterations are stopped if the inequality kgkk1  10
5 is satisfied, where k.k1 is the
maximum absolute component of a vector, or if the number of iterations exceeds 1000. Figure 6.2
presents the Dolan and Moré performance profiles of these algorithms for this set of unconstrained
optimization test problems based on the CPU time metric.

From Fig. 6.2, we can see that BFGSFI is more efficient versus BFGS, BFGSB, and BFGSY and
more robust versus the same algorithms. Scaling is an important ingredient in increasing the
performances of BFGS.

BFGS with Modified Secant Equation

The standard secant equation (6.10) or its equivalent form (6.18) uses only the gradient information,
without referring to the function values. Therefore, to obtain better approximations for the Hessian,
the idea was to modify the secant equation in order to include more available information about the
minimizing function f. Wei, Li, and Qi (2006a) and Wei, Yu, Yuan, and Lian (2004) proposed the
following modified secant equation. Suppose that the objective function f is smooth enough. From the
Taylor series, we obtain


1 1
f k ¼ f kþ1
sTk gkþ1 þ sTk ∇2 f ðxkþ1 Þsk
sTk ðT kþ1 sk Þsk þ O ksk k4 , ð6:69Þ
2 6
where
X n 3
∂ f ðxkþ1 Þ i j l
sTk ðT kþ1 sk Þsk ¼ sss: ð6:70Þ
i, j, l
∂xi ∂xj ∂xl k k k

After some simple algebraic manipulations, from (6.69), we get

284 6 Quasi-Newton Methods

Fig. 6.2 Performance profiles of BFGSFI versus BFGS, BFGSB, BFGSC, BFGSD, BFGSE, and BFGSY. CPU time
metric

    1

sTk ∇2 f ðxkþ1 Þsk ¼ sTk yk þ 2 f k
f kþ1 þ sTk gk þ gkþ1 þ sTk ðT kþ1 sk Þsk þ O ksk k4 :
3
Therefore, neglecting the terms which include the tensor, the following approximation is obtained:

sTk ∇2 f ðxkþ1 Þsk ¼ sTk yk þ ϑk ,

where
6.2 Modifications of the BFGS Method 285

   
ϑk ¼ 2 f k
f kþ1 þ sTk gk þ gkþ1 : ð6:71Þ

Hence, the following modified secant equation is obtained:

ϑk
Bkþ1 sk ¼ zk , zk ¼ yk þ uk , ð6:72Þ
sTk uk

where uk 2 ℝn is a vector satisfying sTk uk ¼ 6 0 (see: Yuan (1991), Yuan and Byrd (1995), Babaie-
Kafaki (2011)).
Another modified secant equation was suggested by Zhang, Deng, and Chen (1999). From the
Taylor series, we get


1
sTk gk ¼ sTk gkþ1
sTk ∇2 f ðxkþ1 Þsk þ sTk ðT kþ1 sk Þsk þ O ksk k4 : ð6:73Þ
2
Now, combining (6.69) and (6.73) and cancelling the terms including the tensor, it follows that


sTk ∇2 f ðxkþ1 Þsk ¼ sTk yk þ 3ϑk þ O ksk k4 ,

where ϑk is defined in (6.71). Therefore, the following modified secant equation can be obtained:
3ϑk
Bkþ1 sk ¼ wk , wk ¼ yk þ uk , ð6:74Þ
sTk uk

where uk 2 ℝn is a vector satisfying sTk uk 6¼ 0:

The theoretical advantages of the modified secant equations (6.72) and (6.74) can be seen from the
following theorem, which shows their accuracy versus the standard secant equation (6.10) (see
Zhang, Deng, and Chen (1999), Wei, Li, and Qi (2006a)).

Theorem 6.10 If the function f is sufficiently smooth and kskk is small enough, then the following
estimating relations hold:


1
sTk ∇2 f ðxkþ1 Þsk
sTk yk ¼ sTk ðT kþ1 sk Þsk þ O ksk k3 ,
2


1
sTk ∇ f ðxkþ1 Þsk
sTk zk ¼ sTk ðT kþ1 sk Þsk þ O ksk k4 ,
2
3

sk ∇ f ðxkþ1 Þsk
sk wk ¼ O ksk k4 ,
T 2 T

where Tkþ1 is the tensor of f at xkþ1 defined in (6.70). ♦

In this context, we can see that for quadratic objective functions, ϑk ¼ 0, and therefore, the
modified secant equations (6.72) and (6.74) reduce to the standard secant equation. As regards the
vector uk, it can usually be selected as uk ¼ sk or uk ¼ yk provided that the line-search satisfies the
Wolfe conditions (6.5) and (6.6). To get positive definite quasi-Newton approximations for the
Hessian based on the modified secant equations (6.72) or (6.74), we should have sTk zk > 0 and
sTk wk > 0, respectively. To overcome this difficulty, a simple procedure is to replace ϑk in (6.72)
and (6.74) by max{0, ϑk}.
For nonconvex objective functions, Li and Fukushima (2001a, 2001b) proposed a new modified
BFGS (called cautious BFGS), for which the local and the global superlinear convergence were
proved. The method is based on the following modified secant equation:
286 6 Quasi-Newton Methods

Bkþ1 sk ¼ yk , yk ¼ yk þ hk kgk kr sk , ð6:75Þ

where r is a positive constant and hk is defined as

( )
yTk sk
hk ¼ C þ max 0,
kgk k
r
ks k k2

for a positive constant C. In the cautious BFGS method, the update of Bk is defined as
8 T T T
< B
Bk sk sk Bk þ yk yk , if yk sk  εkg kγ ,
k
Bkþ1 ¼ T
s k Bk s k T
y k sk ksk k2
k
ð6:76Þ
:
Bk , otherwise,

where ε and γ are positive constants. For the modified secant equation (6.75), yTk sk > 0 independent of
the line-search conditions and of the objective function convexity. This ensures the heredity of the
positive definiteness of the corresponding BFGS update (Guo, Liu, & Wang, 2008).
Recently, Babaie-Kafaki (2012, 2013, 2014) and Babaie-Kafaki and Ghanbari (2014) proposed
scaled memoryless BFGS methods with modified secant equations (6.72), (6.74), or (6.75) which
satisfy the sufficient descent property dTk gk 
ckgk k2 , where c is a positive constant. A new
approach using the polynomial interpolation of the data from the most recent m steps in modified
secant equations was developed by Ford and Moghrabi (1994, 1996a, 1996b) and by Ford,
Narushima, and Yabe (2008).

BFGS with Modified Line-Search

Usually, the BFGS method is implemented by using the Wolfe line-search (6.5) and (6.6). This is
important since it ensures that yTk sk > 0 for any k ¼ 1, . . ., thus conserving the positive definiteness of
the BFGS approximations along the iterations. Suppose that the gradient of the minimizing function
is Lipschitz continuous with the constant L. Let Lk be an approximation of L. Wan, Huang, and Zheng
(2012) proposed the cautious BFGS method (6.76) with the following modification of the Armijo


line-search procedure. Set βk ¼
gTk dk = Lk kdk k2 : “Find the stepsize αk as the largest component in
the set {βk, βkρ, βkρ2, . . .} so that the inequality


1
f ðxk þ αk dk Þ  f ðxk Þ þ σαk gTk dk
αk μLk kdk k2 ð6:77Þ
2
holds, where σ 2 (0, 1), μ 2 [0, 1) and ρ 2 (0, 1) are given constants.” Under classical assumptions,
if kBkskk  a1kskk and a2 ksk k2  sTk Bk sk , then lim inf kgk k ¼ 0, where a1 and a2 are positive
k!1
constants.

Another interesting modification of the strong Wolfe line-search (6.12) was given by Wan, Teo,
Shen, and Hu (2014). In the Wolfe line-search, the choice of σ can affect the cost of finding a stepsize
satisfying (6.5) and (6.6) or (6.5) and (6.12). For a larger value of σ, the cost of searching a stepsize
decreases. Therefore, it seems reasonable to select a sufficiently large value for σ. On the other hand,
for a large value for σ, the obtained stepsize might be far away from the optimal one, i.e., the one
obtained by the exact line-search. With the value of sTk yk far away from
sTk gk , it is clear that
g(xk þ αkdk)Tdk is far away from zero. In other words, a large value for σ may give rise to a stepsize
which is not a good approximation to the one obtained by the exact line-search. To overcome this
difficulty, Wan, Teo, Shen, and Hu (2014) replaced the strong Wolfe condition (6.12) by
6.3 Quasi-Newton Methods with Diagonal Updating of the Hessian 287

   

1 þ σU sTk gk  sTk yk 
1
σL sTk gk , ð6:78Þ

where σL and σU are two sufficiently small constants satisfying 0  σL  σ < 1 and 0  σU  σ (see
also Al-Baali and Grandinetti, (2009)). As above, it is proved that if kBkskk  b1kskk and
b2 ksk k2  sTk Bk sk , where Bk is the BFGS update, then lim inf kgk k ¼ 0, where b1 and b2 are positive
k!1
constants. Numerical experiments reported by the authors show that this variant of the BFGS with
modified line-search (6.78) is competitive versus the standard BFGS or the cautious BFGS (6.76).
Dai (2003) presented a numerical example showing that the standard BFGS method fails in the
case of nonconvex functions under the Wolfe line-search. Therefore, the question is “are there any
other inexact line-searches that possess the global convergence of the BFGS method for general
functions?” A positive answer was given by Yuan, Wei, and Lu, (2017) and Yuan, Sheng, Wang, Hu,
and Li (2018). They presented the following modified Wolfe line-search:
n o
α
f ðxk þ αk dk Þ  f k þ ραk gTk dk þ αk min
ρ1 gTk dk , ρ k kdk k2 , ð6:79Þ
2
n o
gðxk þ αk dk ÞT dk  σgTk dk þ min
ρ1 gTk d k , ραk kdk k2 , ð6:80Þ

where ρ 2 (0, 1/2), ρ1 2 (ρ/2, ρ), and σ 2 (ρ, 1). Under classical assumptions, if kBkskk  b1kskk and
b2 ksk k2  sTk Bk sk , where Bk is the BFGS update (6.22), then, for the corresponding algorithm with
modified line-search given by (6.79) and (6.80), liminfk!1kgkk ¼ 0, where b1 and b2 are positive
constants.
An improvement of the global convergence of the BFGS method with Yuan-Wei-Lu line-search
(6.79) and (6.80) was presented by Dehmiry (2019)

α2k
f ðxk þ αk dk Þ  f k þ ραk gTk d k
ρ kd k2 , ð6:81Þ
2βk k
ραk
gðxk þ αk dk ÞT dk  σgTk dk
kd k k2 , ð6:82Þ
βk

βk ð1
σÞ gTk d k
αk <
, ð6:83Þ
ε 0 β k þ ρ kd k k2

where ρ 2 (0, 1/2), σ 2 (ρ, 1), ε0 is a small parameter (ε0 ¼ 10
6) and {βk} is an arbitrary increasing
sequence of positive numbers so that limk!1βk/k ¼ þ 1 . Like in Byrd and Nocedal (1989),
Dhemiry (2019) proved that there exist the constants b1 > b2 > 0 such that kBkskk  b1kskk and
b2 ksk k2  sTk Bk sk for at least t/2 values of k 2 {1, . . ., t} with any positive integer t. Therefore, the
sequence {gk} generated by the corresponding algorithm with the modified line-search given by
(6.81), (6.82), and (6.83) satisfies liminfk!1kgkk ¼ 0. Numerical experiments with the BFGS
algorithm, where the stepsize is determined by the modified Yuan-Wei-Lu line-search (6.81),
(6.82), and (6.83), show that it is more efficient and more robust versus the algorithm with Yuan-
Wei-Lu line-search (6.79) and (6.80) and versus the Li and Fukushima (2001b) algorithm.

6.3 Quasi-Newton Methods with Diagonal Updating of the Hessian

A relative recent idea to generate simple minimization algorithms for the unconstrained optimization
in the frame of the quasi-Newton methods is to approximate the Hessian of the minimizing function
by a diagonal matrix with positive diagonal elements. This approach is motivated by Theorem 4.16
288 6 Quasi-Newton Methods

which shows that, for the Newton method, in the norm of the error, the inaccuracy in the Hessian has a
smaller influence than the inaccuracy of the gradient. These methods were introduced by Gill and
Murray (1979) and discussed by Gilbert and Lemaréchal (1989). The search direction is computed as

dkþ1 ¼
B
1
kþ1 gkþ1 , ð6:84Þ
 
where Bkþ1 ¼ diag b1kþ1 , . . . , bnkþ1 is a positive definite diagonal matrix retaining only the diagonal
elements of the BFGS update matrix (6.22)
 i 2  i 2  i 2
bk s k yk
bkþ1 ¼ bk
Pn
i i
  þ T , i ¼ 1, . . . , n: ð6:85Þ
i i 2 y
b s
i¼1 k k k sk

This diagonal-updating approach uses only O(n) storage. If we assume that yTk sk > 0, then Bkþ1 is
positive definite.
Another quasi-Newton algorithm with diagonal approximation to the Hessian is based on the weak
quasi-Newton equation sTk Bkþ1 sk ¼ sTk yk and was introduced and studied by Dennis and Wolkowicz
(1993). The update proposed by Dennis and Wolkowicz is as follows:

sTk yk
sTk Bk sk
Bkþ1 ¼ Bk þ T
T
 2 Bk s k s k B k , ð6:86Þ
sk Bk sk

where Bk is positive definite. The condition sTk yk > 0 implies that Bkþ1 in (6.86) is also positive
 
definite. If Bk is taken to be a positive definite diagonal matrix Bk ¼ diag b1k , . . . , bnk , then (6.86)
can be restricted to update only the diagonal elements of Bkþ1 as
P  2
sTk yk
ni¼1 bik sik  i 2  i 2
bkþ1 ¼ bk þ
P
i i
  2 bk sk , i ¼ 1, . . . , n, ð6:87Þ
n i i 2
i¼1 bk sk

yielding a positive definite diagonal matrix. The search direction in this algorithm is computed as in
(6.84), where the diagonal elements of Bkþ1 are computed as in (6.87).
The diagonal quasi-Newton approximation algorithm presented by Zhu, Nazareth, and Wolkowicz
(1999) is as follows. Suppose that Bk is a positive definite diagonal matrix and Bkþ1, which is also
diagonal, is the updated version of Bk.The algorithm requires that the updated Bkþ1 satisfy the quasi-
Newton secant equation and the deviation between Bk and Bkþ1 be minimized under a variational
principle. The search direction is computed as dkþ1 ¼
B
1 kþ1 gkþ1 , where the diagonal elements of
 
Bkþ1 ¼ diag b1kþ1 , . . . , bnkþ1 are computed as
P  2
sTk yk
ni¼1 bik sik  i 2
bikþ1 ¼ bik þ Pn  i  4 sk , i ¼ 1, . . . , n: ð6:88Þ
i¼1 sk

In the same way of developments, Andrei (2019a) presented a new diagonal quasi-Newton updating
method, where the elements of the diagonal matrix approximating the Hessian are determined by
minimizing both the size of the change from the previous estimate and the trace of the update, with
respect to the weak secant equation. The search direction in this algorithm is computed as
6.3 Quasi-Newton Methods with Diagonal Updating of the Hessian 289

gikþ1
dikþ1 ¼
, i ¼ 1, . . . n, ð6:89Þ
bikþ1

and
P  2
sTk yk þ sTk sk
ni¼1 bik sik  i 2
bikþ1 ¼ bik þ Pn  i  4 sk
1, i ¼ 1, . . . n: ð6:90Þ
i¼1 sk

Another approach that uses the same paradigm of the diagonal quasi-Newton updating method was
given by Andrei (2018e), where the diagonal elements are determined by minimizing the measure
function of Byrd and Nocedal. The components of the search direction in this approach are computed
as

 2 
d ikþ1 ¼
gikþ1 1 þ λ sik , i ¼ 1, . . . , n, ð6:91Þ

where
(  
r þ θ, if λ < r, t sTk gkþ1
yTk gkþ1
λ¼ and λ¼P
 i 2  ,
λ if λ  r, n
y i gi
i¼1 k kþ1 sk

and

2 n  2 o
r ¼
1= sjk ¼ max i¼1,...,n; sik 6¼0
1= sik :

t is a positive parameter and θ is a small perturbation (e.g., θ ¼ 1).

Andrei (2019b) has recently presented a new quasi-Newton method, in which the Hessian of the
function f is approximated as a positive definite diagonal matrix. In this method, the approximation
Hessian Bkþ1 is a diagonal matrix computed as

Bkþ1 ¼ Y k S
1 k , ð6:92Þ

 1   
where Y k ¼ diag yk , . . . , ynk and Sk ¼ diag s1k , . . . , snk , yik , i ¼ 1, . . .n being the components of
the vector yk and sik , i ¼ 1, . . .n being the components of the vector sk. In other words,

yik gikþ1
gik gi ðxk þ αk dk Þ
gi ðxk Þ

bikþ1 ¼ ¼ ¼ , i ¼ 1, . . . n, ð6:93Þ
sik xikþ1
xik αk d ik

where gik is the i-th component of the gradient in xk and d ik is the i-th component of the search direction
dk. Therefore, in this approach, the element bikþ1 may be considered as an approximation of the
second-order derivative of the function f, corresponding to the i-th diagonal element of the Hessian
computed in xkþ1 by a scaled forward finite differences directional derivative scheme.

Numerical Study
Consider the set of 80 unconstrained optimization test functions from the UOP collection (Andrei,
2020a). Figure 6.3 presents the Dolan and Moré performance profiles of YONS defined by (6.92) with
(6.93) versus QND defined by (6.84) with (6.85), WQND defined by (6.84) with (6.87), QCD defined
by (6.84) with (6.88), and DNRTR defined by (6.84) with (6.90), for minimizing this set of
unconstrained optimization test problems with n ¼ 500 variables, subject to the CPU time metric.
290 6 Quasi-Newton Methods

Fig. 6.3 Performance profiles of YONS versus QND, WQND, QCD, and DNRTR. CPU time metric, n ¼ 500

The stepsize is determined by the Wolfe line-search conditions (6.5) and (6.6), implemented with
σ ¼ 0.8 and ρ ¼ 0.0001. The iterations are stopped if the inequality kgkk1  εg is satisfied, where
k.k1 is the maximum absolute component of a vector, or if the number of iterations exceeds a
prespecified limit (50000).

Observe that YONS is significantly more efficient and more robust versus all the algorithms
considered in this numerical study. More numerical results with quasi-Newton methods with diagonal
updating to the Hessian, presented in Andrei (2018e, 2019a, 2019b, 2020b), proved that the algorithm
in which the diagonal elements of the approximation to the Hessian are computed by a scaled forward
finite differences directional derivative scheme is competitive among the algorithms in this class.

6.4 Limited-Memory Quasi-Newton Methods

Using (6.15) in (6.20), the inverse BFGS updating can be written as

   
sk yTk yk sTk sk sT
H kþ1 ¼ I
T Hk I
T þ Tk:
y k sk yk sk y k sk

Therefore, the corresponding search direction based on this inverse BFGS formula is
6.4 Limited-Memory Quasi-Newton Methods 291

dkþ1 ¼
Hkþ1 ∇f ðxkþ1 Þ

   
sk y T y sT sk sT
¼
I
T k H k I
kT k ∇f ðxkþ1 Þ
T k ∇f ðxkþ1 Þ:
y k sk y k sk y k sk

Note that this formula requires no matrix storage except for storage of Hk and, therefore, uses only
vector operations. This is one version of a limited-memory BFGS method. However, more elaborate
formulae can be developed as it is going to be presented in the following.
The limited-memory quasi-Newton methods are dedicated to solving large-scale unconstrained
optimization problems whose Hessian matrix cannot be computed and stored at a reasonable cost
(Nocedal, 1980). Instead of storing an n  n approximation to the Hessian, these methods save only a
few vectors which can be used to represent the approximation implicitly. Among different limited-
memory methods, the best known and used is L-BFGS, which is based on the BFGS update. The main
idea of this method is to use the curvature information only from the most recent iterations in order to
construct the Hessian approximation.
As known, each step of the BFGS method is computed as
xkþ1 ¼ xk
αk H k gk , k ¼ 0, 1, . . . , ð6:94Þ

where αk is the stepsize, gk ¼ ∇ f(xk), and Hk is the approximation to the inverse Hessian updated by
the formula

H kþ1 ¼ V Tk H k V k þ ρk sk sTk , ð6:95Þ

where
1
ρk ¼ , V k ¼ I
ρk yk sTk : ð6:96Þ
yTk sk

Since the inverse Hessian Hk is generally a dense matrix, the limited-memory BFGS method will
implicitly store a modified version of Hk by storing only a certain number (say m) of the vector pairs
{si, yi} used in the updating formulae (6.95) and (6.96). After the new iterate has been computed, the
oldest vector pair in the set of pairs {si, yi} is replaced by the new pair {sk, yk} obtained from the
current iteration. Therefore, the set of vector pairs includes the curvature information from the most
recent m iterations. It is worth mentioning that only a small number of vector pairs, between 3 and
11, needs to be used for solving large-scale optimization problems. The product Hkgk is computed
during the updating process by performing a sequence of inner products and vector summations
involving only gk and the pairs {si, yi}. Nocedal and Wright (2006) presented a two-loop recursion
L-BFGS algorithm to compute the product Hkgk in (6.94). At iteration k, the current iterate is xk, and
the set of vector pairs is given by {si, yi} for i ¼ k
m, . . ., k
1. Choose some initial Hessian
approximation H 0k , and by repeated application of (6.95), find that the L-BFGS approximation Hk
satisfies the following formula:
 
H k ¼ V Tk
1 ⋯V Tk
m H 0 ðV k
m ⋯V k
1 Þ
 
þρk
m V Tk
1 ⋯V Tk
mþ1 sk
m sTk
m ðV k
mþ1 ⋯V k
1 Þ
 
þρk
mþ1 V Tk
1 ⋯V Tk
mþ2 sk
mþ1 sTk
mþ1 ðV k
mþ2 . . . V k
1 Þ
þ⋯
þρk
1 sk
1 sTk
1 :
292 6 Quasi-Newton Methods

This expression of Hk may be used to derive a recursive procedure for computing the product
Hk ∇ f(xk). Given H0k , the two-loop recursion algorithm for the search direction computation is as
follows (Liu, & Nocedal, 1989).

Algorithm 6.2 L-BFGS two-loop recursion

Set q ¼ gk. The first loop is:

For i ¼ k
1, k
2, . . ., k
m compute:
αi ¼ ρi sTi q, q ¼ q
αiyi
End for
Set r ¼ H 0k q:
The second loop is:
For i ¼ k
m, k
m þ 1, . . ., k
1 compute:
β ¼ ρi yTi r, r ¼ r þ si(αi
β)
End for
Set Hkgk ¼ r ♦

 
Usually, H 0k is chosen as a diagonal matrix. Often, H 0k ¼ γ k I, where γ k ¼ sTk
1 yk
1 =kyk
1 k2 : The
parameter γ k is a scaling factor that attempts to estimate the size of the true Hessian along the most
recent search direction. This selection of γ k ensures that the search direction is well scaled, and
therefore, as a by-product, the stepsize αk ¼ 1 is accepted in most iterations (Gilbert & Lemaréchal,
1989). In the implementation of Liu and Nocedal (1989), L-BFGS is one of the best algorithms (and
computing programs) for minimizing large-scale unconstrained optimization functions. Now, the
limited-memory BFGS algorithm can be presented as follows.

Algorithm 6.3 L-BFGS

1. Initialization. Choose the initial point x0, the convergence tolerance ε > 0 sufficiently small, and a value
for integer m, and set k ¼ 0
2. Test a criterion for stopping the iterations. For example, if k∇f(xk)k  ε, then stop; otherwise, set
k ¼ k þ 1 and go to step 3
 
3. Choose H 0k : For example, H 0k ¼ γ k I, where γ k ¼ sTk
1 yk
1 =kyk
1 k2
4. Use Algorithm 6.2 (two-loop recursion) to compute dk ¼
Hk ∇ f(xk)
5. Compute the stepsize αk by using the Wolfe line-search
6. Compute xkþ1 ¼ xk þ αkdk
7. If k > m, then discard the vector pair {sk
m, yk
m} from the storage
8. Compute and save sk ¼ xkþ1
xk and yk ¼ ∇ f(xkþ1)
∇ f(xk)
9. Set k ¼ k þ 1 and go to step 2 ♦

During the first m
1 iterations, if the initial matrix H0 is the same in both algorithms and if
L-BFGS chooses H 0k ¼ H 0 at each iteration, then Algorithm 6.3 is equivalent to Algorithm 6.1 (BFGS
method). Observe that the performance of Algorithm 6.3 is subject to the level of memory m.
Table 6.1 shows the performances of L-BFGS (Liu & Nocedal, 1989) for solving five unconstrained
optimization problems, where ε ¼ 10
6. The steplength is determined at each iteration by means of
the line-search routine MCVSRCH, which is a slight modification of the routine CSRCH written by
Moré and Thuente (1990, 1992, 1994).
6.4 Limited-Memory Quasi-Newton Methods 293

Table 6.1 Performances of L-BFGS for different values of m

m=3 m=5 m=17

Problem n #iter #fg cpu #iter #fg cpu #iter #fg cpu
FreuRoth 10,000 16 18 0.00 17 18 0.01 19 20 0.01
Rosenbrock 10,000 68 93 0.01 67 93 0.01 69 92 0.02
BDQRTIC 10,000 191 252 0.03 168 223 0.04 61 90 0.02
NONDQUAR 10,000 1805 2001 0.29 1796 2001 0.33 1796 2001 0.89
DIXON3DQ 10,000 1892 2001 0.26 1938 2001 0.30 1948 2001 0.94

Fig. 6.4 Performance profiles of L-BFGS versus CONMIN, DESCONa, CG-DESCENT, and CG-DESCENTaw

From Table 6.1, we can see that the optimal choice of m is problem dependent. For large values of
m, the cost of each iteration increases, and therefore, L-BFGS requires more CPU computing time.
The best CPU time is obtained for small values of m.

Numerical Study: L-BFGS

Figure 6.4 shows the performance profiles of L-BFGS versus CONMIN, DESCONa (accelerated
DESCON), CG-DESCENT (CG-DESCENT with Wolfe line-search), and CG-DESCENTaw
(CG-DESCENT with approximate Wolfe line-search) for solving 800 unconstrained optimization
problems from the UOP collection with the number of variables in the range [1000, 10000].
294 6 Quasi-Newton Methods

Table 6.2 Performances of L-BFGS Elastic-plastic torsion. n ¼ 40, 000. e ¼ 10
6

#iter #fg cpu

L-BFGS (m=3) 322 338 20.90
L-BFGS (m=5) 328 677 21.51
L-BFGS (m=7) 340 1027 22.96
L-BFGS (m=9) 318 1353 22.19

Fig. 6.5 Solution of

application A1. Elastic-
plastic torsion. nx ¼ 200, 0.4
ny ¼ 200
0.3

0.2

0.1

0
200
150 200
100 150
100
50 50
0 0

Table 6.3 Performances of L-BFGS Pressure distribution in a journal bearing. n ¼ 40, 000. e ¼ 10
6

#iter #fg cpu

L-BFGS (m=3) 898 935 58.47
L-BFGS (m=5) 788 1750 52.15
L-BFGS (m=7) 813 2584 55.33
L-BFGS (m=9) 782 3386 54.94

Observe that L-BFGS is more robust than all these conjugate gradient algorithms. However,
DESCON, CG-DESCENT, and CG-DESCENTaw are more efficient than L-BFGS.

Applications from the MINPACK-2 Collection

In the following, let us see the performances of L-BFGS in the implementation of Liu and Nocedal
(1989) for solving the applications from the MINPACK-2 collection (see Appendix D) for different
values of m.

Application A1. Elastic-Plastic Torsion

For nx ¼ 200, ny ¼ 200, and c ¼ 5, for different values of m, the performances of L-BFGS for solving
elastic-plastic torsion are as in Table 6.2, where #iter is the number of iterations, #fg is the number of
function and its gradient evaluations, and cpu is the CPU computing time in seconds. Figure 6.5
shows the solution of this application for m ¼ 3.

Application A2. Pressure Distribution in a Journal Bearing

For nx ¼ 200, ny ¼ 200, b ¼ 10, and ε ¼ 0.1, Table 6.3 shows the performances of L-BFGS for
solving this application.
6.4 Limited-Memory Quasi-Newton Methods 295

Fig. 6.6 Solution of

application A2. Pressure
distribution in a journal 0.4
bearing. nx ¼ 200,
ny ¼ 200 0.3

0.2

0.1

0
200
150 200
100 150
100
50 50
0 0

Table 6.4 Performance of L-BFGS. Optimal design with composite materials. n ¼ 40, 000. e ¼ 10
6

#iter #fg cpu

L-BFGS (m=3) 1468 1482 121.10
L-BFGS (m=5) 856 2343 71.83
L-BFGS (m=7) 864 3210 74.30
L-BFGS (m=9) 659 3871 57.93

Fig. 6.7 Solution of

application A3. Optimal
design with composite 0
materials. nx ¼ 200, −0.01
ny ¼ 200
−0.02
−0.03
−0.04
−0.05
−0.06
200
150 200
100 150
100
50 50
0 0

Figure 6.6 illustrates the solution of this application for nx ¼ 200, ny ¼ 200, and m ¼ 3.

Aplication A3. Optimal Design with Composite Materials

For nx ¼ 200 and ny ¼ 200, the performances of L-BFGS for solving optimal design with composite
materials with parameters λ ¼ 0.008, μ1 ¼ 1, and μ2 ¼ 2 are presented in Table 6.4.

Figure 6.7 shows the solution of this application for nx ¼ 200, ny ¼ 200, and m ¼ 3.

Application A4. Steady-State Combustion

For nx ¼ 200 and ny ¼ 200, the performances of L-BFGS for solving steady-state combustion with
the Frank-Kamenetskii parameter λ ¼ 5 are as in Table 6.5.
296 6 Quasi-Newton Methods

Table 6.5 Performances of L-BFGS. Steady-state combustion. n ¼ 40, 000. e ¼ 10
6

#iter #fg cpu

L-BFGS (m=3) 721 763 80.08
L-BFGS (m=5) 523 1308 58.06
L-BFGS (m=7) 524 1846 58.61
L-BFGS (m=9) 412 2272 47.10

Fig. 6.8 Solution of

application A4. Steady-
state combustion. nx ¼ 200, 0.8
ny ¼ 200
0.6

0.4

0.2

0
200
150 200
100 150
100
50 50
0 0

Table 6.6 Performances of L-BFGS. Minimal surfaces with Enneper boundary conditions. n ¼ 40, 000. e ¼ 10
6

#iter #fg cpu

L-BFGS (m=3) 484 507 37.77
L-BFGS (m=5) 459 977 35.93
L-BFGS (m=7) 424 1408 33.94
L-BFGS (m=9) 427 1844 35.10

Figure 6.8 illustrates the solution of this application for nx ¼ 200, ny ¼ 200, and m ¼ 3.

Application A5. Minimal Surfaces with Enneper Boundary Conditions

For nx ¼ 200 and ny ¼ 200, the performances of L-BFGS for minimal surface with Enneper boundary
conditions are as in Table 6.6. The solution of this application is illustrated in Fig. 6.9.

Application A6. Inhomogeneous Superconductors: 1-D Ginzburg-Landau

For d ¼ 3.2 Å and the temperature T ¼ 5, for n ¼ 1000, the performances of L-BFGS for solving 1-D
Ginzburg-Landau application are as in Table 6.7. The solution of this application is presented in
Fig. 6.10.

Compact Representation of the L-BFGS Updating

Limited-memory quasi-Newton approximations are an important component of the unconstrained or
constrained optimization methods. L-BFGS is a line-search algorithm that updates an approximation
to the inverse Hessian Hk. On the other hand, the trust-region methods require an approximation to the
6.4 Limited-Memory Quasi-Newton Methods 297

Fig. 6.9 Solution of

0.4
application A5. Minimal
surfaces with Enneper 0.3
boundary conditions. 0.2
nx ¼ 200, ny ¼ 200 0.1
0
−0.1
−0.2
−0.3
−0.4
200
100 200
100 150
0 0 50

Table 6.7 Performances of L-BFGS. Inhomogeneous superconductors: 1-D Ginzburg-Landau. n ¼ 1000. e ¼ 10
6

#iter #fg cpu

L-BFGS (m=3) 1904 2001 9.45

Fig. 6.10 Solution of × 10−3

application A6. 7.73
Inhomogeneous
superconductors: 1-D 7.725
Ginzburg-Landau.
n ¼ 1000 7.72

7.715

7.71

7.705

7.7

7.695

7.69

7.685

7.68
0 100 200 300 400 500 600 700 800 900 1000

Hessian matrix Bk, not to its inverse. In the following, we show that, by representing quasi-Newton
matrices in a compact form, an efficient implementation of all quasi-Newton formulae and their
inverse can be derived. These compact representations are useful in designing limited-memory
methods for the constrained optimization, where the approximations to the Hessian or to the reduced
Hessian of the Lagrangian are needed.

In the following, let us describe the limited-memory updating that is based on representing quasi-
Newton matrices in compact form. This is illustrated for the case of the BFGS approximation Bk to the
Hessian (Byrd, Nocedal, & Schnabel, 1994b).
298 6 Quasi-Newton Methods

Theorem 6.11 Consider B0 a symmetric and positive definite matrix, and assume that the k vector
pairs {si, yi} satisfy sTi yi > 0, for all i ¼ 0, . . ., k
1. Using (6.22), let Bk be obtained by applying k
BFGS updates with these vector pairs to B0. Then,
" #
1 " #
STk B0 Sk Lk STk B0
Bk ¼ B0
½ B 0 S k Y k  , ð6:97Þ
LTk
Dk Y Tk

where

Sk ¼ ½s0 , . . . , sk
1  2 ℝnk , Y k ¼ ½y0 , . . . , yk
1  2 ℝnk , ð6:98Þ

while Lk 2 ℝk  k and Dk 2 ℝk  k are defined by

 T
si
1 yj
1 if i > j, 
ðLk Þi,j ¼ Dk ¼ diag sT0 y0 , . . . , sTk
1 yk
1 : ð6:99Þ
0, otherwise,

The proof is obtained by induction. ♦

Observe that the conditions sTi yi > 0, i ¼ 0, . . ., k
1, ensure that the middle matrix in (6.97) is
nonsingular, so Bk is well-defined. Now, let us consider this result in the limited-memory updating.
As we know, the L-BFGS algorithm keeps the m most recent pairs {si, yi}, and at every iteration,
this set of vectors is refreshed by removing the oldest pair and by adding a newly generated
pair. Obviously, during the first m iterations, the update procedure described in Theorem 6.11 can
be used without any modification, except that Bk0 ¼ δk I for the initial matrix, where δk ¼ 1/γ k and
   
γ k ¼ sTk
1 yk
1 = yTk
1 yk
1 :
For the iterations k > m, the update procedure has to be slightly modified in order to take into
consideration the vector pairs {si, yi} for i ¼ k
m, k
m þ 1, . . ., k
1. Define
Sk ¼ ½sk
m , . . . , sk
1  2 ℝnm , Y k ¼ ½yk
m , . . . , yk
1  2 ℝnm : ð6:100Þ

Therefore, the matrix Bk obtained after m updates of Bk0 ¼ δk I is given by

" #
1 " #
δk STk Sk Lk δk STk
Bk ¼ δ k I
½ δ k S k Y k  , ð6:101Þ
LTk
Dk Y Tk

where Lk 2 ℝmm and Dk 2 ℝmm are defined by

 T
sk
m
1þi yk
m
1þj if i > j, 
ðLk Þi,j ¼ Dk ¼ diag sTk
m yk
m , . . . , sTk
1 yk
1 : ð6:102Þ
0 otherwise,

As soon as the new iterate xkþ1 has been generated, the matrix Skþ1 is obtained by deleting sk
m
from Sk and by adding the new sk. The matrix Ykþ1 is updated in a similar way. Similarly, the new
matrices Lkþ1 and Dkþ1 are obtained.
Since the middle matrix in (6.101) is of small dimension 2m, its factorization requires a negligible
amount of computation. The idea of the compact representation (6.101) is that the corrections to the
initial matrix can be expressed as the outer product of ½ δk Sk Y k  and its transpose with an
intervening multiplication by a small 2m  2m matrix. The limited-memory updating of Bk requires
approximately 2mn þ O(m3) operations and the matrix-vector products of the form Bkv which can be
6.5 The SR1 Method 299

performed at a cost of (4m þ 1)n þ O(m2) multiplications. Therefore, updating the direct limited-
memory BFGS matrix Bk is economical when m is small (m ¼ 3, or m ¼ 5).
This approximation Bk can be used in methods for the bound constrained and the general
constrained optimization. For example, the program L-BFGS-B (Byrd, Lu, Nocedal, & Zhu,
1995b; Zhu, Byrd, Lu, & Nocedal, 1997) makes extensive use of compact limited-memory
approximations to solve large-scale nonlinear optimization problems with bound constraints (see
Chap. 12). Other codes for the general constrained optimization like KNITRO (Byrd, Hribar, &
Nocedal, 1999) and IPOPT (Wächter, & Biegler, 2001, 2006) also make use of the compact limited-
memory matrix Bk to approximate the Hessian of the Lagrangians (see Chaps. 15, 17, and 19,
respectively).
A formula similar to (6.97), but this time for a compact representation of the inverse BFGS
approximation Hk, was developed by Byrd, Nocedal, and Schnabel (1994b). An implementation of
the unconstrained L-BFGS algorithm based on this expression requires a similar amount of computa-
tion as the algorithm described above.

6.5 The SR1 Method

The symmetric rank-one SR1 update formula can be derived as solution of the following simple
problem. “Given a symmetric matrix Bk and the vectors sk and yk, find a new symmetric matrix Bkþ1
such that Bkþ1
Bk has rank one, such that the secant equation Bkþ1sk ¼ yk is satisfied.” It is easy to
see that if (yk
Bksk)Tsk 6¼ 0, then the unique solution of the above problem is

ð y k
B k s k Þ ð y k
Bk s k Þ T
ðSR1directÞ Bkþ1 ¼ Bk þ : ð6:103Þ
ð y k
Bk s k Þ T s k

If yk ¼ Bksk, then the solution is Bkþ1 ¼ Bk. However, if (yk
Bksk)Tsk ¼ 0 and yk 6¼ Bksk, then there
is no solution to the problem.
Let Hk be the inverse approximation to the Hessian at iteration k. By using the Sherman-Morrison-
Woodbury formula, from (6.103), the following update to the inverse Hessian for SR1 is obtained as

ðsk
H k yk Þðsk
H k yk ÞT
ðSR1inverseÞ H kþ1 ¼ H k þ : ð6:104Þ
ðsk
H k y k ÞT y k

This variant of the algorithm is only applicable in cases in which the inverse Hk exists. The main
drawbacks of the SR1 update are as follows:

1. The denominator (yk
Bksk)Tsk of the SR1 update term in (6.103) may vanish, i.e.,
(yk
Bksk)Tsk ffi 0, cases in which Bkþ1 is not well-defined.
2. The step directions computed by using the SR1 updating formula given by (6.103) may no longer
be uniform linear independent, thus leading to slowing down the convergence or even the stalling.
3. The SR1 Hessian approximation may not be positive definite along the iterations, thus resulting in
a direction that does not produce descent.

To prevent the method from failing because of the first drawback, one simple remedy is to set
Bkþ1 ¼ Bk. However, this may slow down the convergence of the method. Conn, Gould, and Toint
(1991b) and Khalfan, Byrd, and Schnabel (1993) showed that the denominator of (6.103) rarely
300 6 Quasi-Newton Methods

vanishes in practice, and setting Bkþ1 ¼ Bk does not have a significant impact on the performances of
the SR1 method subject to the number of the iterations or runtimes.
The second drawback is more delicate, being in close connection with the uniform linear
independence of the search directions generated by the SR1 algorithm. A more precise definition
of the uniform linear independence was given by Conn, Gould, and Toint (1991b). “A sequence {sk}
is uniformly linearly independent if there exist ξ > 0, k0 and m  n such that, for each k  k0, there are
n distinct indices k  k1  k2  . . .  kn  k þ m for which the minimum singular value of the matrix
 
s
S ¼ sk1 , ⋯, ksskkn k is at least ξ.” Roughly speaking, uniformly linearly independent steps mean
k k1 k n

that the steps do not tend to fall in a subspace of dimension less than n.
Conn, Gould, and Toint (1991b) proved that the sequence of matrices generated by the SR1
formula converges to the exact Hessian when the sequence of iterates converges to a limit point and
the sequence of steps is uniformly linearly independent. Kelley and Sachs (1998) provided similar
convergence results by removing the first of these assumptions. Fiacco and McCormick (1968)
showed that if the search directions are linearly independent and the denominator of (6.103) is always
non-zero, then the SR1 method without line searches minimizes a strongly convex quadratic function
in at most n þ 1 steps. In this case, Bnþ1 is exactly the Hessian of the quadratic function. Observe that
this result is significant since it does not require the exact line-search as in the case of the BFGS
update. Generally, the above condition given by the definition of the uniform linear independency is
not implemented in practice, but it serves only as one of the main assumptions of a proof that the SR1
approximations to the Hessian converge to the true Hessian as the iterates converge to the solution of
(6.1).
Subject to the uniform linear independency of the search directions, Khalfan, Byrd, and Schnabel
(1993) showed that many problems do not satisfy this requirement. Instead, they proved the local
convergence of the SR1 method by using only the positive definiteness and boundedness assumptions
for the approximate Hessian. Moreover, Conn, Gould, and Toint (1991b) proved that if the
minimizing function f is twice continuously differentiable and its Hessian is bounded and Lipschitz
continuous, the iterates generated by the SR1 method converge to a point x, and in addition, if for
all k,




ðy
Bk sk ÞT sk
 ξky
Bk sk kksk k, ð6:105Þ
k k

for some ξ 2 (0, 1), (ξ ¼ 10
8), and the steps sk are uniformly linearly independent, then

lim Bk
∇2 f ðx Þ ¼ 0: ð6:106Þ
k!1

The SR1 methods have the ability to generate good Hessian approximations. Let us prove this
property for the quadratic functions. Observe that for the functions of this type, the choice of the
stepsize does not affect the update. Therefore, to examine the properties of this update, we can assume
a uniform step of length 1, that is, dk ¼
Hk ∇ f(xk) and xkþ1 ¼ xk þ dk, i.e., sk ¼ dk. In fact, the
following results can be proved.

Theorem 6.12 Suppose that f ðxÞ ¼ 12 xT Ax þ bT x is a strongly convex quadratic function, where
x 2 ℝn and A is symmetric positive definite. Then, for any initial point x0 and any initial symmetric
matrix H0, the iterates {xk} generated by the SR1 method (6.104) converge to the minimizer in at most
n steps, provided that (sk
Hkyk)Tyk 6¼ 0 for all k. Assume a uniform step length of 1. Moreover, if n
steps are performed and if the search directions dk ¼
Hk ∇ f(xk) are linearly independent, then
Hn ¼ A
1.
6.5 The SR1 Method 301

Proof Since (sk
Hkyk)Tyk 6¼ 0, it follows that that the SR1 method is well-defined. Let us show that
H k yj ¼ sj , j ¼ 0, 1, . . . , k
1: ð6:107Þ

In other words, let us show that the secant equation is satisfied not only along the most recent
search directions but also along all the previous directions.
Observe that H1y0 ¼ s0. Assume that (6.107) holds for a certain value k > 1, and show that it also
holds for k þ 1. From (6.107), since f is a quadratic function, we have yi ¼ Asi. Therefore,


ðsk
H k yk ÞT yj ¼ sTk yj
yTk H k yj ¼ sTk yj
yTk sj ¼ 0, for all j < k:

Using this in (6.104), we get Hkþ1yj ¼ Hkyj ¼ sj, for all j < k. Since Hkþ1yk ¼ sk by the secant
equation, it follows that (6.107) holds for k þ 1. By induction, this relation holds for all k.
If the algorithm performs n steps and if the steps {sj} are linearly independent, we obtain
sj ¼ Hnyj ¼ HnAsj, for j ¼ 0, 1, . . ., n
1. Hence, HnA ¼ I, that is, Hn ¼ A
1. In other words, the
step taken at xn is the Newton step. Therefore, the next iterate xnþ1 will be the solution and the
algorithm terminates.
Now, consider the case in which the steps become linearly dependent. For this, suppose that sk is a
linear combination of the previous steps, that is, sk ¼ ξ0s0 þ . . . þ ξk
1sk
1, for some scalars ξi,
i ¼ 0, 1, . . ., k
1. From (6.107), we have
Hk yk ¼ H k Ask
¼ ξ0 H k As0 þ . . . þ ξk
1 H k Ask
1
¼ ξ0 H k y0 þ . . . þ ξk
1 Hk yk
1
¼ ξ0 s0 þ . . . þ ξk
1 sk
1 ¼ sk :

However, yk ¼ ∇ f(xkþ1)
∇ f(xk). Since sk ¼ dk ¼
Hk ∇ f(xk), it follows that

H k ð∇f ðxkþ1 Þ
∇f ðxk ÞÞ ¼
H k ∇f ðxk Þ:

By the nonsingularity of Hk, it follows that ∇f(xkþ1) ¼ 0, that is, xkþ1 is the minimum point. ♦

The relation (6.107) proved in Theorem 6.12 for SR1 shows that, for quadratic functions, the
secant equation is satisfied along all the previous search directions, no matter how the line-search is
implemented. Such a result is established for the BFGS updating only under the assumption of the
exact line-search.
It is important to notice that for general nonlinear functions, under certain conditions, the SR1
update continues to generate good Hessian approximations, as shown further:

Theorem 6.13 Suppose that f is twice continuously differentiable and its Hessian is bounded and
Lipschitz continuous in a neighborhood of a point x. Let {xk} be any sequence of iterates such that
xk ! x for some x 2 ℝn. Additionally, suppose that (6.105) holds for all k, where ξ 2 (0, 1), and that
the steps sk are uniformly linearly independent. Then, the matrices Bk generated by the SR1 updating
formula (6.103) satisfy

lim Bk
∇2 f ðx Þ ¼ 0: ♦
k!1
302 6 Quasi-Newton Methods

Often, the condition (6.105) is used in the implementations of SR1 to make sure this update
behaves well. If this condition is not satisfied, then the update is skipped. Conn, Gould, and Toint
(1991b) and Khalfan, Byrd, and Schnabel (1993) provided theoretical and computational results,
respectively, that if the uniform linear independence assumption is satisfied, then the approximations
to the Hessian generated by the SR1 method are more accurate than those generated by BFGS, and
SR1 converges faster than BFGS to the true Hessian. Therefore, if all the above drawbacks are
addressed in a reliable and efficient manner, then SR1 can be used for solving (6.1) instead of the
rank-two updates. More details on the SR1 method concerning the undefined updates, the choice of
the initial approximate B0, and the uniform linear independence of the steps are found in Benson and
Shanno (2018) and Chen, Lam, and Chan (2019).
The convergence properties of the SR1 method are not understood so well as those of the BFGS
method. No global results like in Theorem 6.3 have been established for SR1, apart from some results
for quadratic functions. We emphasize that there is no rank-one update formula that maintains both
the symmetry and the positive definiteness of the Hessian approximations. However, there is an
infinity of rank-two formulae that does it. The most widely used and considered to be the most
effective is the BFGS update formula (6.22).

The Memoryless SR1 Method with Generalized Secant Equation

This quasi-Newton method introduced by Andrei (2021c, 2021d, 2022) is based on the generalized
secant equation

yk ¼ γ k Bkþ1 sk , ð6:108Þ

where γ k is a positive parameter. The generalized secant equation is obtained from the requirement
that the quadratic model of the minimizing function with scaled approximation to the Hessian should
match the gradient of the minimizing function at the latest two iterations xk and xkþ1.

Suppose that in the current point xk we know the approximation Bk to the Hessian to be a
symmetric matrix. To derive the SR1 method with generalized secant equation, we establish that
Bkþ1, which is the approximation to the Hessian in xkþ1, satisfies (6.108) and is obtained after a rank-
one update of Bk, i.e., it has the form

Bkþ1 ¼ Bk þ δk uuT , ð6:109Þ

where δk is a scalar and u 2 ℝn. Substituting this form into the generalized secant equation, we get

yk ¼ γ k Bk sk þ γ k δk uuT sk ,

or, alternatively,
 
yk
γ k Bk sk ¼ γ k δk uT sk u:

Since γ kδk(uTsk) is a scalar, in order to satisfy this equation, we can simply set δk ¼ γ 1
and
k ðu
Ts Þ
k

u ¼ yk
γ kBksk. Therefore, introducing these elements in (6.109), the SR1 method with generalized
secant equation is obtained as

ðyk
γ k Bk sk Þðyk
γ k Bk sk ÞT
Bkþ1 ¼ Bk þ : ð6:110Þ
γ k ðyk
γ k Bk sk ÞT sk
6.5 The SR1 Method 303

Note that the SR1 update formula (6.110) is unique, that is, there is exactly one rank-one update
satisfying the generalized secant equation. Moreover, if γ k ¼ 1 in (6.110), then the SR1 update (6.103)
is obtained.
If yk ¼ γ kBksk, then the solution is Bkþ1 ¼ Bk. However, if (yk
γ kBksk)Tsk ¼ 0 and yk 6¼ γ kBksk,
then there is no solution to the problem.
Let Hk be the inverse approximation to the Hessian at iteration k. By using the Sherman-Morrison-
Woodbury formula in (6.110), the following update to the inverse Hessian for SR1 with generalized
secant equation is obtained as

ðH k yk
γ k sk ÞðH k yk
γ k sk ÞT
H kþ1 ¼ H k
: ð6:111Þ
ðH k yk
γ k sk ÞT yk

This variant of the algorithm is only applicable in cases in which the inverse Hk exists.
From (6.111), the search direction corresponding to the inverse Hessian SR1 updating with
generalized secant equation is dkþ1 ¼
Hkþ1gkþ1, i.e.,

ðH k yk
γ k sk ÞT gkþ1
dkþ1 ¼
H k gkþ1 þ ðHk yk
γ k sk Þ: ð6:112Þ
ðH k y k
γ k sk ÞT y k

Now, the memoryless SR1 method with generalized secant equation is obtained by considering
Bk ¼ I in (6.110), i.e.,

ðyk
γ k sk Þðyk
γ k sk ÞT
Bkþ1 ¼ I þ : ð6:113Þ
γk ðyk
γ k sk ÞT sk

Observe that this is a very simple updating formula, in which the information about the Hessian is
not accumulated from iteration to iteration. Besides, we can see that the memoryless SR1 method
with generalized secant equation has the same drawbacks as the SR1 method, i.e., when the
denominator (yk
γ ksk)Tsk is zero, or is very close to zero, then the method is not defined.
Now, choosing in (6.111) Hk ¼ I, i.e.,

ðyk
γ k sk Þðyk
γ k sk ÞT
H kþ1 ¼ I
, ð6:114Þ
ðy k
γ k sk ÞT y k

the memoryless inverse of SR1 with generalized secant equation is obtained. Therefore, from (6.114),
the memoryless SR1 search direction with generalized secant equation is dkþ1 ¼
Hkþ1gkþ1, that is,

ðyk
γ k sk ÞT gkþ1
d kþ1 ¼
gkþ1 þ ðyk
γ k sk Þ: ð6:115Þ
ðyk
γ k sk ÞT yk

The main advantage of the memoryless SR1 update (6.115) is that, for its implementation in
computer programs, only two scalar products (yk
γ ksk)Tgkþ1 and (yk
γ ksk)Tyk must be computed.
This is advantageous for solving large-scale problems. Observe that in this memoryless SR1 update,
the information from the previous iteration is not accumulated into the current iteration.
304 6 Quasi-Newton Methods

Proposition 6.2 If

yTk yk
γk > , ð6:116Þ
sTk yk
then the memoryless SR1 search direction with generalized secant equation (6.115) is a descent
direction.

Proof From (6.116), it follows that (yk
γ ksk)Tyk < 0. Therefore, by direct computation, from
(6.115), we get

2
T
2 ð y k
γ k s k Þ g kþ1
gTkþ1 d kþ1 ¼
gkþ1 þ T < 0, ♦
ðy k
γ k sk Þ y k

Note that the memoryless SR1 search direction with generalized secant equation (6.115) has three
terms. The first is the negative gradient
gkþ1; the last two terms involve sk and yk, both of them being
multiplied by some scalars. It is obvious that the search direction (6.115) satisfies the conjugacy
condition, i.e., yTk dkþ1 ¼
γ k sTk gkþ1 , where γ k is a positive parameter. Therefore, the memoryless
SR1 method with generalized secant equation is a conjugate gradient method which satisfies the Dai
and Liao (2001) conjugacy condition.
The memoryless algorithms corresponding to SR1 with generalized secant equation
(MM-SR1gen) and to the memoryless BFGS (MM-BFGS) are very simple. In both of them, the
stepsize is computed by the Wolfe line-search. The search direction in the memoryless SR1 method
with generalized secant equation is computed as in (6.115), while the search direction in the
memoryless BFGS method is computed as in (6.54). In the numerical experiments with these
algorithms, an acceleration scheme developed by Andrei (2006b, 2009c) was implemented.

Algorithm 6.4 Accelerated MM-SR1gen and MM-BFGS

MM-SR1gen MM-BFGS
1. Consider an initial point x0. Set k ¼ 0. Select some values for the Wolfe line-search conditions σ and ρ
with 0 < ρ < σ < 1. Compute g0 ¼ ∇ f(x0) and set d0 ¼
g0. Select the sufficiently small parameters: ε > 0
used in the criterion for stopping the iterations, εq > 0 used in the search direction computation, and εA > 0
used in the acceleration scheme
2. Test a criterion for stopping the iterations: if kgkk1  ε, then stop the iterations; otherwise, go to step 3
3. Compute the stepsize αk using the Wolfe line-search conditions
4. Update the variables xkþ1 ¼ xk þ αkdk and compute fkþ1 and gkþ1. Compute sk ¼ xkþ1
xk and
yk ¼ gkþ1
gk
5. Acceleration scheme:
a) Compute: z ¼ xk þ αkdk, gz ¼ ∇ f(z), and yk ¼ gk
gz
b) Compute:


ak ¼ αk gTk dk and bk ¼
αk yTk d k



c) If bk  εA , then compute ηk ¼
ak =bk and update the variables as xkþ1 ¼ xk þ ηkαkdk. Otherwise,
update the variables as xkþ1 ¼ xk þ αkdk. Compute fkþ1 and gkþ1. Compute yk ¼ gkþ1
gk and
sk ¼ xkþ1
xk



6. Select

T a value
for the parameter γ k as in (6.116). If If
yTk sk
 εq , compute the search direction

y y
γ k sT y
 εq , then compute the search dkþ1 as in (6.54). Otherwise, set
k k k k
direction dkþ1 as in (6.115). Otherwise, set dkþ1 ¼
gkþ1
dkþ1 ¼
gkþ1

7. Restart the iterations. If gTkþ1 dkþ1 >
10
3 gkþ1 kd kþ1 k, then set dkþ1 ¼
gkþ1
8. Consider k ¼ k þ 1 and go to step 2 ♦
6.5 The SR1 Method 305

Observe that the algorithm is equipped with an acceleration scheme (see step 5). This scheme
modifies the stepsize determined by the Wolfe line-search conditions in such a way as to improve the
reduction of the minimizing function values along the iterations. It is proved that this acceleration
scheme is linear convergent, but the reduction in the function value is significantly improved (see
Sect. 3.4 in Chap. 3).
If f is bounded along the direction dk, then there exists a stepsize αk satisfying the Wolfe line-
search conditions (6.5) and (6.6). The first trial of the stepsize crucially affects the practical behavior
of the algorithm. At every iteration k  1, the starting guess for the step αk in the line-search is
computed as αk
1kdk
1k/kdkk. Observe that in step 6 of the algorithms, if the search direction for the
memoryless SR1 method with generalized secant equation is not defined, i.e., if |(yk
γ ksk)Tyk| is
close to zero or if the search direction for the memoryless BFGS is not defined, i.e., if yTk sk is close to
zero, then the search direction is commuted to be the steepest descent. A restarting condition has been
introduced in our algorithm. If this condition is satisfied, then the algorithm is restarted with the
negative gradient.

Convergence of the MM-SR1gen Method

The global convergence of the MM-SR1gen algorithm is established under the following
assumptions:

(A1) The level set S ¼ {x 2 ℝn : f(x)  f(x0)} is bounded, i.e., there exists a constant b > 0 such that
for any x 2 S, kxk  b.
(A2) The function f : ℝn ! ℝ is continuously differentiable, and its gradient is Lipschitz continuous
in a neighborhood N of S, i.e., there exists a constant L > 0 such that k∇f(x)
∇ f( y)k 
Lkx
yk, for any x, y 2 N.

Note that under these assumptions, there exists a constant Γ > 0 such that k∇f(x)k  Γ for any x 2 S.

Proposition 6.3 Suppose that (yk
γ ksk)Tyk < 0 and (yk
γ ksk)(yk
γ ksk)T is a matrix bounded
above in norm. Then, the memoryless SR1 with generalized secant equation update matrix Hkþ1 given
by (6.114) is bounded above in norm.

Proof From (6.114), since the denominator (yk
γ ksk)Tyk is strictly negative, the numerator
(yk
γ ksk)(yk
γ ksk)T is bounded above in norm, and kskk ¼ kxkþ1
xkk  kxkþ1k þ kxkk  2b,
it follows that Hkþ1 given by (6.114) is bounded above in norm. ♦

Proposition 6.4 Suppose that (yk
γ ksk)Tyk < 0. Then, the memoryless SR1 with generalized secant
equation search direction (6.115) is a descent direction, that is, gTkþ1 dkþ1 < 0: Besides, kdkþ1k  D
for some D > 0.

Proof Observe that the numerator of the update (6.114) is a rank-one positive semidefinite matrix
and the denominator of (6.114) is strictly negative. Therefore, the update matrix Hkþ1 given by
(6.114) is positive semidefinite. Without loss of generality, it follows that Hkþ1 given by (6.114) is
positive definite. Thus, gTkþ1 d kþ1 ¼
gTkþ1 H kþ1 gkþ1 < 0:

The boundedness of dkþ1 may be established as follows. If the restart condition is satisfied, then
dkþ1 ¼
gkþ1. Therefore, kdkþ1k ¼ kgkþ1k < D, where D ¼ Γ by assumption (A2). On the other
hand, for non-restart iterations, the search direction is computed as dkþ1 ¼
Hkþ1gkþ1, where Hkþ1
is given by (6.114). In this case, by Proposition 6.3, the sequence of the Hessian matrices given by
306 6 Quasi-Newton Methods

(6.114) remains bounded above in norm. Therefore, kdkþ1k ¼ k
Hkþ1gkþ1k  kHkþ1kkgkþ1k, which
is bounded above by some constant D > 0. ♦

Theorem 6.14 Suppose that the assumptions (A1) and (A2) are satisfied, and assume that the
generated matrix Hk is positive definite, and there are the positive constants m and M such that
mI  Hk  MI. Furthermore, assume that the stepsize αk satisfies the weak Wolfe conditions (6.5) and
(6.6) and that the function f is bounded below in ℝn. Then, lim k∇f ðxk Þk ¼ 0:
k!1

Proof From the first Wolfe condition (6.5), it follows that f ðxk þ αk dk Þ < f ðxk Þ þ ραk gTk d k : Taking
the summation of both sides from 0 to k, we have

X
k
f ðxkþ1 Þ < f ðx0 Þ þ ρ αi gTi di :
i¼0

Since from Proposition 6.4 the algorithm generates descent directions and the function f is
P
bounded below, it follows that
ki¼0 αi gTi di is bounded above by a positive scalar. Now, by
Proposition 2.4, for any i, the stepsize αi is bounded below as

T

1
σ
gi d i

αi  :
L kd i k2

Therefore,
 2
ρð1
σÞ X gTi d i
k
f ðx0 Þ
f ðxkþ1 Þ  2
:
i¼0 kd i k
L

In other words,
 T 2
X
1
gi d i Lðf ðx0 Þ
f min Þ
 γ, where γ¼  0,
i¼0 kd i k 2 ρð 1
σ Þ

where fmin is the minimum value of the sequence ff ðxk Þg1k¼1 : Since di ¼
Higi, we have
 T 2  T 2
gi d i gi H i gi m2
2
¼ T  2 kgi k2 :
kd i k gi H i H i gi M

Therefore,
 T 2
m2 X X
1 1
gi d i
2
kgi k 
2
2
< γ,
M i¼1 i¼0 kd i k

that is,
X
1
M2
kgi k2  γ < 1,
i¼0
m2

involving that lim k∇f ðxk Þk ¼ 0: ♦

k!1
6.5 The SR1 Method 307

Fig. 6.11 Accelerated

MM-SR1gen versus
MM-BFGS, range [1000,
10000]

Hence, the algorithm MM-SR1gen is globally convergent to a point in which the first-order
optimality conditions are satisfied.

Numerical Study: MM-SR1gen versus MM-BFGS and versus BFGS

In the following, let us report some numerical results obtained with the accelerated MM-SR1gen and
the MM-BFGS methods for solving 800 unconstrained optimization problems from the UOP collec-
tion. In the first set of numerical experiments, we compare accelerated MM-SR1gen versus
MM-BFGS for solving the unconstrained minimization problems from UOP, with the number of
variables in the range [1000, 10000]. Figure 6.11 shows the Dolan and Moré performance profiles of
these algorithms; see Remark 1.1.

From Fig. 6.11, we can see that MM-SR1gen is more efficient and more robust than MM-BFGS. In
the second set of numerical experiments, we compare the performances of the accelerated
MM-SR1gen versus BFGS which is implemented in CONMIN (Shanno, 1983). The CONMIN
package includes two optimization algorithms: a BFGS preconditioned conjugate gradient and a
variant of the BFGS quasi-Newton algorithm. Figure 6.12 shows the performance profiles of these
algorithms for solving 800 unconstrained optimization problems from our collection, with the number
of variables in the range [100, 1000].
Obviously, MM-SR1gen is more efficient and more robust than BFGS from CONMIN, one of the
best implementations of this quasi-Newton method. Observe that subject to the CPU computing time,
MM-SR1 is faster in 664 problems, while BFGS from CONMIN is faster only in nine problems.
The BFGS from CONMIN is a variable metric method with initial scaling which approximately needs
n2/2 þ 11n/2 double precision words of working storage. By comparison, MM-SR1gen requires
approximately 6n double precision words of working storage.
BFGS requires more memory and involves a greater computational effort. We emphasize that at
every iteration, BFGS from CONMIN updates an approximation to the Hessian by accumulating the
information from the previous iterations. On the other hand, at every iteration, the memoryless
MM-SR1gen algorithm updates the identity matrix (see (6.113), or (6.114) for the inverse Hessian
308 6 Quasi-Newton Methods

Fig. 6.12 Accelerated

MM-SR1gen versus BFGS
from CONMIN, range
[100, 1000]

Table 6.8 Performances of MM-SR1gen versus MM-BFGS (40,000 variables, cpu seconds)

MM-SR1gen MM-BFGS
#iter #fg cpu #ig #iter #fg cpu #ig
A1 40,000 372 772 7.79 0 6711 20133 180.56 6611
A2 40,000 1257 2547 31.97 0 9312 27937 298.50 8883
A3 40,000 4093 10001 202.28 0 861 2584 53.28 15
A4 40,000 609 1260 65.10 0 666 1997 140.00 416
A5 40,000 308 697 9.47 0 2177 6520 148.46 199
Total 6639 15277 316.61 0 19727 59171 820.80 16124

updating). Obviously, MM-SR1gen does not accumulate the information from iteration to iteration
when updating the approximation to the Hessian. However, MM-SR1gen, having a very simple
updating formula, is way more efficient and more robust than BFGS from CONMIN.
Now, let us present comparisons between the MM-SR1gen and MM-BFGS algorithms for solving
five applications from the MINPACK-2 collection, each of them with 40,000 variables (see Appendix
D). The performances of the MM-SR1gen method versus the MM-BFGS method are given in
Table 6.8 where #iter is the number of iterations, #fg is the number of function and its gradient
evaluations, #ig is the number of iterations in which the search direction is the negative gradient, and
cpu is the CPU computing time for solving these applications in seconds.
From Table 6.8, we can see that MM-SR1gen is more efficient for solving these applications from
the MINPACK-2 collection. We can see that MM-SR1gen needs 316.61 seconds, while MM-BFGS
needs 820.80 seconds, i.e., MM-SR1gen is 2.60 times faster than MM-BFGS. It is worth pointing out
that, for solving all these applications, MM-SR1gen does not use the negative gradient in any
iteration. This is in sharp contrast with MM-BFGS, which, out of 19727 iterations for solving all
the five applications, the negative gradient is used in exactly 16124 iterations (i.e., 81.73%).
The performances of MM-SR1gen and of MM-BFGS should be compared with the performances
of the conjugate gradient algorithms in Tables 5.2, 5.3, 5.4, 5.6, 5.7, 5.9, 5.10, 5.11, and 5.12. Observe
that the conjugate gradient algorithms are more efficient than MM-SR1gen or MM-BFGS.
6.6 Sparse Quasi-Newton Updates 309

Both these algorithms, MM-SR1gen and MM-BFGS, are memoryless, i.e., they do not accumulate
the information from iteration to iteration. They use the same initialization and exactly the same
implementation of the Wolfe line-search conditions (6.5) and (6.6). The differences are in the formula
for the search direction computation.

6.6 Sparse Quasi-Newton Updates

These updates are designed for solving large-scale optimization problems. The idea is that the quasi-
Newton approximations Bk have the same sparsity pattern as the true Hessian. Obviously, this
approach would reduce the storage requirement and perhaps would generate a more accurate Hessian
approximations (Toint, 1977, 1981).
Consider
n   o
Ω≜ ði, jÞ : ∇2 f ðxÞ i,j 6¼ 0 for some x in the domain of f :

Suppose that the current Hessian approximation Bk has the same zero/nonzero structure of the
exact Hessian, i.e., (Bk)i, j ¼ 0 for all (i, j) 2
= Ω.. Now, in updating Bk to get Bkþ1, the following
requirements are imposed: Bkþ1 satisfies the secant condition, and it has the same sparsity pattern as
Bk and is as close as possible to Bk. Specifically, Bkþ1 is defined as solution of the following quadratic
minimization problem
X
2
min kB
Bk k2F ¼ Bi,j
ðBk Þi,j ,
B
ði, jÞ2Ω
subject to ð6:117Þ
Bsk ¼ yk , B¼B , T
and Bi,j ¼ 0 for ði, jÞ=
2Ω:

The solution Bkþ1 of this problem can be obtained by solving an n  n linear system whose sparsity
pattern is Ω, the same as the sparsity of the true Hessian. The main drawbacks of this approach are as
follows: the possible Bkþ1 solution of (6.117) is not positive definite and the updating process is not
scale invariant under linear transformations of the variables. However, the fundamental weakness of
this approach is that (6.117) is an inadequate model and can produce poor Hessian approximations.
An alternative approach is to relax the secant equation by requiring that it should be approximately
satisfied along the last few steps rather than on the latest step. For this, define Sk and Yk as in (6.100) so
that they contain the m most recent difference pairs. The new Hessian approximation Bkþ1 is obtained
as solution of the following quadratic problem:

min kBSk
Y k k2F

B
subject to ð6:118Þ
B ¼ BT and Bi,j ¼ 0 for ði, jÞ=
2Ω:

This minimizing problem has a solution which is not easy to compute. Moreover, this approach
can produce singular or poorly conditioned Hessian approximations (Nocedal and Wright, 2006).
Details on sparse quasi-Newton update can be found in Toint (1981); Gill, Murray, Saunders, and
Wright (1982); Lucia (1983); Spedicato and Zhao (1993); Byrd, Nocedal, and Schnabel (1994b);
Fletcher (1995); and Yamashita (2005).
310 6 Quasi-Newton Methods

6.7 Quasi-Newton Methods and Separable Functions

A separable function f(x) : ℝn ! ℝ is a function which can be decomposed into a sum of element
functions, each element function depending on different components of the vector x. For example, the
function
 2  2
f ðxÞ ¼ x3
x21 þ x34
x2 ð6:119Þ

is a separable function because it can be written as a sum of two element functions

f ðxÞ ¼ f 1 ðxÞ þ f 2 ðxÞ,
 2  2
where f 1 ðxÞ ¼ x3
x21 and f 2 ðxÞ ¼ x34
x2 : In general, a separable function can be written as
X
ne
f ðxÞ ¼ f i ðxÞ, ð6:120Þ
i¼1

where each of the element function fi depends on only a few components of x and no component of
x appears in more than one element function. Separable unconstrained minimization consists of the
minimization of each element function in turn. It is easy to see that the gradient and the Hessian of the
function (6.120) are
X
ne X
ne
∇f ðxÞ ¼ ∇f i ðxÞ, ∇2 f ðxÞ ¼ ∇2 f i ðxÞ:
i¼1 i¼1

The quasi-Newton approximations of the Hessian ∇2f(x) of the function f can be computed and
handled by computing the quasi-Newton approximations of the Hessians ∇2fi(x), i ¼ 1, . . ., ne,
corresponding to each element function.
To clarify the concept and explain the quasi-Newton method with separable functions, let us
consider the function from (6.119). Observe that f1 is a function which formally depends on x, but it
actually depends only on x1 and x3, which we call element variables. Assembling the element
variables into a vector x[1] ¼ [x1, x3]T and defining the compactifying matrix
 
1 0 0 0
U1 ¼ ,
0 0 1 0
 2
we get x[1] ¼ U1x. Introducing the function φ1 ðz1 , z2 Þ ¼ z2
z21 , we can write f1(x) ¼ φ1(U1x). By
applying the chain rule, we get

∇f 1 ðxÞ ¼ U T1 ∇φ1 ðU 1 xÞ, ∇2 f 1 ðxÞ ¼ U T1 ∇2 φ1 ðU 1 xÞU1 : ð6:121Þ

In our case, we have

2 3
12x21
4x3 0
4x1 0
  6
12x21
4x3
4x1 6 0 0 0 077
∇2 φ1 ðU 1 xÞ ¼ , ∇2 f 1 ð x Þ ¼ 6 7:

4x1 2 4
4x1 0 2 05
0 0 0 0

The idea of using the quasi-Newton method with separable functions is instead of storing and
handling a quasi-Newton approximation to ∇2f1, it is more profitable to store and handle a 2  2-
6.7 Quasi-Newton Methods and Separable Functions 311

dimensional quasi-Newton approximation B[1] of ∇2φ1 and use (6.121) to transform it into a quasi-
Newton approximation to ∇2f1. Obviously, after a typical iteration from x to xþ, both s½1 ¼ xþ
½1
x½1

  
and y½1 ¼ ∇φ1 xþ½1
∇φ1 x½1 have to be recorded and used in the BFGS or SR1 updating
formulae to obtain a new approximation Bþ
½1 of ∇ φ1(U1x). With this, the approximation of the
2

element Hessian ∇2f1 is obtained from (6.121), that is, ∇2 f 1 ðxÞ

U T1 B½1 U 1 : Obviously, this opera-
tion will map the elements of B[1] to their correct positions in the full n  n Hessian approximation
to f1.
From (6.120), the function f can be written as
X
ne
f ðxÞ ¼ φi ðU i xÞ:
i¼1

Therefore, the n  n-dimensional quasi-Newton approximation to the full Hessian ∇2f is obtained
by summing the quasi-Newton approximations B[i] i ¼ 1, . . ., ne of the element function as follows:
X
ne
B¼ U Ti B½i Ui : ð6:122Þ
i¼1

With this approximation to the Hessian, the search direction can be computed as solution of the
system Bkdk ¼
∇ f(xk). Of course, for solving this system, it is not necessary to assemble Bk
explicitly, but rather use the conjugate gradient algorithm which is based only on matrix-vector
products of the form Bkv by performing operations with the matrices Ui and B[i], i ¼ 1, . . ., ne.
It is important to compare these two approaches. Suppose that we have to minimize a separable
function which depends on 1000 variables and where each element function depends only on two
variables. Of course, ignoring the partially separable structure of the minimizing function, a quasi-
Newton approximation BFGS or SR1 to a 1000  1000-dimensional matrix has to be computed. In
this case, many iterations are needed to get a good quality quasi-Newton approximation of ∇2f(x). By
contrast, in the quasi-Newton with separable functions, the functions φi still depend only on two
variables, so that each quasi-Newton approximation to the Hessian B[i] is a 2  2-dimensional matrix.
After a few iterations, we have enough directions s[i] and differences y[i] to compute each B[i] as an
approximate to ∇2φi accurate enough. Therefore, the full quasi-Newton approximation (6.122) tends
to be a very good one to ∇2f(x).
Obviously, there is no guarantee that the curvature condition sT½i y½i > 0 is satisfied for all i ¼ 1, . . .,
ne. In this case, even if the full Hessian ∇2f(x) is at least positive semidefinite at the solution x, some
of the element Hessians ∇2φi(.) may be indefinite. A solution to overcome this situation is to apply the
SR1 update to each of the element Hessians. This approach is used in the LANCELOT package
(Conn, Gould, & Toint, 1992b), which is designed to take full advantage of the partial separability of
the minimizing function.
The main limitations of this approach based on the separability of the minimizing function are the
difficulty of identifying a good partially separable structure of f and the cost of the step computation
management.
Another possibility to take advantage of the separability of the minimizing function f is to identify
the vectors g[i] with the components of the gradient ∇f(x) corresponding to the components of each x[i]
and solve the systems ∇2φi(Ux)d[i] ¼
g[i], for i ¼ 1, . . ., ne, thus obtaining the components
d[i] of the search direction. In this context, another idea is to compute ∇2φi(.) by finite-differences (see
Chap. 4).
312 6 Quasi-Newton Methods

6.8 Solving Large-Scale Applications

The quasi-Newton methods are more efficient than the Newton methods. Using only the change in
gradients, the quasi-Newton methods construct a model of the minimizing function that produces
superlinear convergence. Optimization software libraries contain a large variety of quasi-Newton
programs for solving unconstrained, constrained, and large-scale optimization problems. From the
practical viewpoint, the most popular quasi-Newton method is BFGS, in which the stepsize is
computed by using the Wolfe line-search. The performances of BFGS can degrade if the line-search
is not based on the Wolfe conditions.
Two difficulties affect the BFGS method: the storage and the strategy for dealing with the
possibility that yTk sk  0: The storage difficulty is solved by using an efficient storage way to compute
the BFGS step by using the history of the iteration rather than the full matrix storage (Byrd, Nocedal,
& Schnabel, 1994b). A strategy based on reducing the storage cost to one vector for each iteration was
described by Kelley (1999). Another strategy is the limited-memory BFGS method (L-BFGS) (see
Nocedal (1980)). Neither of these approaches for controlling the storage, although essential in
practice for large-scale problems, has the superlinear convergence properties like the full-storage
BFGS algorithm. On the other hand, when yTk sk is not sufficiently positive, the BFGS update is
restarted with the identity matrix. One of the best implementations of the limited-memory BFGS,
with a highly practical efficiency, is L-BFGS of Liu and Nocedal (1989).
Another point which should be emphasized is the Hessian initialization of the L-BFGS method. A
 
popular Hessian initialization is H0 ¼ yTk yk =yTk sk I: There are also more sophisticated Hessian
initializations. Gilbert and Lemaréchal (1989) proposed a sparse Hessian initialization.
Table 6.9 shows the performances of L-BFGS (m ¼ 5) of Liu and Nocedal for solving five
applications from the MINPACK-2 collection, each of them with 250,000 variables.
Comparisons of L-BFGS (m ¼ 5) (see Table 6.9) versus the modern conjugate gradient algorithms
CG-DESCENT (Table 5.13), DESCON (Table 5.14), and DK (Table 5.15) show that L-BFGS
(m ¼ 5) is faster. Subject to the number of iterations, DESCONa, with 5810 iterations for solving
all the five applications, is better than L-BFGS (m ¼ 5).
MM-SR1gen is very easy to implement and requires only two scalar products (yk
γ ksk)Tgkþ1 and
(yk
γ ksk)Tyk per iteration, where γ k is given by (6.114). In our numerical experiments, γ k is computed
 
as γ k ¼ 100 yTk yk =sTk yk : Table 6.10 presents the performances of MM-SR1gen for solving five
applications from the MINPACK2 collection, each of them with 250,000 variables.
Observe that the limited-memory BFGS, L-BFGS (m ¼ 5) is more efficient than the memoryless
SR1 method with the generalized secant equation MM-SR1gen. This is because MM-SR1gen does
not accumulate information about the Hessian from iteration to iteration.

Table 6.9 Performances of L-BFGS (m ¼ 5) for solving five applications from the MINPACK-2 collection (250,000
variables, cpu seconds)

n #iter #fg cpu

A1 250,000 672 695 80.00
A2 250,000 1924 1986 222.73
A3 250,000 1999 2001 366.84
A4 250,000 1295 1338 439.87
A5 250,000 921 937 137.65
Total - 6811 6957 1247.09
6.8 Solving Large-Scale Applications 313

Table 6.10 Performances of MM-SR1gen for solving five applications from the MINPACK-2 collection (250,000
variables, cpu seconds)

n #iter #fg cpu

A1 250,000 805 1638 89.96
A2 250,000 3384 6799 580.42
A3 250,000 4114 10001 1258.14
A4 250,000 1375 2775 729.36
A5 250,000 742 1561 49.08
Total - 10420 22774 2706.96

Notes and References

The quasi-Newton methods were introduced by Davidon (1959, 1980). A comprehensive description
of quasi-Newton methods was given by Dennis and Schnabel (1983), Dennis and Moré (1977), and
Fletcher (1987). The convergence of the BFGS matrices was considered by Ge and Powell (1983) and
by Boggs and Tolle (1994). The global convergence of the BFGS method was established by Powell
(1976). This result was extended to the restricted Broyden class, except for the DFP method, by Byrd,
Nocedal, and Yuan (1987a). For the self-correcting property of the quasi-Newton methods, see
Nocedal (1992). The early analysis of the quasi-Newton methods was based on the bounded deterio-
ration principle. This is a tool for the local analysis that treats the case of the worst behavior of the
quasi-Newton updating. In principle, assuming that the initial point is sufficiently close to the
minimum point x and the initial Hessian approximation is sufficiently close to ∇2f(x), then the
bounded deterioration proves that the iteration cannot stay away from the solution. This property can
be used to show the superlinear convergence. For details, see Dennis and Schnabel (1983) and the
landmark paper by Dennis and Moré (1977).

The limited-memory BFGS method was designed by Nocedal (1980). The main weakness of the
L-BFGS method is that it slowly converges on ill-conditioned problems, that is, on problems for
which the Hessian matrix contains a wide distribution of eigenvalues.
Plenty of papers were dedicated to the modifications of the BFGS quasi-Newton method. They are
based on sizing, i.e., multiplying by an appropriate scalar the approximate Hessian matrix before it is
updated in the BFGS method, on the proper scaling of the terms on the right-hand side of the BFGS
updating formula with positive factors, on the modified secant equation in order to approximate the
curvature of the objective function along the search direction more accurately than the standard
secant equation does, and on the new line-search conditions for the stepsize computation to ensure the
global convergence by modifying the Wolfe line-search conditions.
The quasi-Newton methods with diagonal updating of the Hessian are based on Theorem 4.16 (see
4.89), which shows that for the Newton method, in the norm of the error, the inaccuracy in the
Hessian has a smaller influence than the inaccuracy of the gradient. In other words, the inaccuracy
evaluation of the Hessian of the minimizing function is not so important. It is the accuracy of the
evaluation of the gradient which is more important. A diagonal approximation of the Hessian by finite
differences for the unconstrained optimization was developed by Andrei (2020c) (see also Andrei,
2019a, 2019b, 2020d).
The memoryless SR1 method with generalized secant equation was introduced by Andrei (2021c,
2021d).
314 6 Quasi-Newton Methods

Sparse quasi-Newton updates were studied by Toint (1977, 1981), Fletcher (1995), and Fletcher,
Grothey, and Leyffer (1996). An efficient computation of the sparse Jacobian was presented by
Andrei (1983).
The numerical experience with the limited-memory quasi-Newton (and the truncated Newton)
methods is described in Zou, Navon, Berger, Phua, Schlick, and Le Dimet (1993). Limited-memory
BFGS methods are implemented in L-BFGS (Liu, & Nocedal, 1989), M1QN3 (Gilbert, &
Lemaréchal, 1989), E04DGF (Gill, & Murray, 1979), and BBVSCG (Buckley, & Lenir, 1985). Gill
and Leonard (2003) give a variant of the limited-memory BFGS that requires less storage and appears
to be quite efficient. The compact limited-memory representations are used in L-BFGS-B (Zhu, Byrd,
Lu, & Nocedal, 1997), IPOPT (Wächter & Biegler, 2000) (see Chap. 19), and KNITRO (Byrd,
Hribar, & Nocedal, 1999) (see Chaps. 15 and 17).
The quasi-Newton methods for partially separable functions were developed by Griewank and
Toint (1982a, 1982b). The LANCELOT package by Conn, Gould, and Toint (1992b) implements the
concept of partial separability of the minimizing functions.
 Inexact Newton Methods
7

The main idea behind the inexact Newton method, also known as the truncated Newton method,
introduced by Dembo, Eisenstat, and Steihaug (1982) and analyzed by Dembo and Steihaug (1983)
and Deuflhard (1990) is to approximate the solution of the Newton system

∇2 f ðxk Þdk ¼ ∇f ðxk Þ ð7:1Þ

for the search direction dk. Since far away from a local minimum the objective function cannot be
well approximated by a quadratic model, it follows that it is not necessary to spend too much time on
computing an accurate Newton search vector. Therefore, in the following, we describe some
procedures for obtaining approximations of the search direction dk, which are good enough and
inexpensive to calculate. Solving (7.1) in an iterative manner has the advantage of avoiding the
factorization of the Hessian ∇2f(xk) and, therefore, the fill-in during the factorization. Besides, these
methods can be implemented in a Hessian-free manner, so the Hessian ∇2f(xk) need not be explicitly
calculated or stored. Solving (7.1) is based on using the conjugate gradient method with some
modifications to handle the negative curvature in the Hessian.
The criterion for terminating the iterative method for solving (7.1) is based on the residual

r k ¼ ∇2 f ðxk Þd k þ ∇f ðxk Þ, ð7:2Þ

where dk is the inexact Newton step. Usually, the conjugate gradient iterations are terminated when

kr k k  ηk k∇f ðxk Þk, ð7:3Þ

where the sequence {ηk} with 0 < ηk < 1 for all k is called the forcing sequence. Therefore, in the
inexact Newton method, the linear algebraic system (7.1) for the Newton step is solved by an iterative
method and (7.3) is considered the termination criterion. It is standard to refer to the sequence of the
Newton steps {xk} as the outer iteration and to the sequence of iterates for solving (7.1) as the inner
iterations (Nash, 1984b, 1985). In this context, the naming convention is that the Newton-CG, for
example, refers to the Newton iterative method in which the conjugate gradient algorithm is used to
perform the inner iterations (see Remark 5.2). The Newton-CC is particularly appropriate for the
large-scale optimization since we expect positive definite Hessians near a local minimizer.

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 315
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_7
316 7 Inexact Newton Methods

7.1 The Inexact Newton Method for Nonlinear Algebraic Systems

In the following, let us discuss the inexact Newton method for solving nonlinear algebraic systems
FðxÞ ¼ 0, ð7:4Þ

where F : ℝn ! ℝn. Suppose that

H1. There exists a point x such that F(x) ¼ 0.

H2. Fis continuously differentiable in a neighborhood of x.
H3. The Jacobian matrix J(x) ¼ ∇ F(x) is nonsingular.

As we know, for solving (7.4) the Newton method is defined by the Newton system
J ðxk Þsk ¼ Fðxk Þ ð7:5Þ

and

xkþ1 ¼ xk þ sk , ð7:6Þ

where J(xk) is the Jacobian matrix of F in point xk.

The inexact or truncated Newton method is defined by
J ðxk Þsk ¼ Fðxk Þ þ r k , ð7:7Þ

where

kr k k  ηk kFðxk Þk ð7:8Þ

and
xkþ1 ¼ xk þ sk : ð7:9Þ

In this context of truncation,

r k ¼ J ðxk Þsk þ Fðxk Þ ð7:10Þ

represents the residual, while {ηk}, with 0 < ηk < 1 is a forcing sequence which controls the
approximation with which the Newton system is solved. We are interested in the local convergence
of this method.

Proposition 7.1 Let F : D ⊂ ℝn ! ℝn be a continuously differentiable function in a neighborhood

of x 2 D and J(x) ¼ ∇ F(x) nonsingular. Then, there exist δ > 0, ξ > 0, and ε > 0, such that as soon
as ky  xk < δ and y 2 D, J( y)is nonsingular and




J ðyÞ1
 ξ: ð7:11Þ

Also J( y)1 is continuous in x, that is,

J ðyÞ1  J ðx Þ1
< ε: ð7:12Þ
7.1 The Inexact Newton Method for Nonlinear Algebraic Systems 317

Proof Denote α ¼ kJ(x)1k. For a β < α1, choose δ so that, when ky  xk < δ with y 2 D, it follows
that

kJ ðx Þ  J ðyÞk  β:

Using the von-Neumann lemma (see Appendix A), it results that J( y) is nonsingular, and therefore
(7.11) is true with ξ ¼ α/(1  βα). Hence,

 1




J ðx Þ  J ðyÞ1
¼
J ðx Þ1 ðJ ðyÞ  J ðx ÞÞJ ðyÞ1

 αξkJ ðx Þ  J ðyÞk  αβξ ≜ ε,

which proves that the continuity of J guarantees the continuity of J1. ♦

With this, the convergence of the inexact Newton method can be established as follows:

Theorem 7.1 Let F : ℝn ! ℝn be a function which satisfies the assumptions H1–H3. Suppose that
the forcing sequence {ηk} satisfies 0  ηk  η < t < 1. Then, for ε > 0, if the initial point x0 is
sufficiently close to x, the sequence {xk} generated by the inexact Newton method (7.7), (7.8), and
(7.9) converges to x, and the rate of convergence is linear, that is,
kxkþ1  x k  tkxk  x k , ð7:13Þ

where kyk ¼ kJ(x)yk.

Proof Since J(x) is nonsingular, then for y 2 ℝn it follows that

1
kyk  kyk  μkyk, ð7:14Þ
μ

where
n

o
μ ¼ max kJ ðx Þk,
J ðx Þ1
: ð7:15Þ

Since η < t, then there exists γ > 0 sufficiently small, so that

ð1 þ γμÞðηð1 þ μγ Þ þ 2μγ Þ  t: ð7:16Þ

Now, choose ε > 0 sufficiently small, so that if ky  xk  μ2ε, we have

kJ ðyÞ  J ðx Þk  γ, ð7:17Þ

J ðyÞ1  J ðx Þ1
 γ, ð7:18Þ

kFðyÞ  Fðx Þ  J ðx Þðy  x Þk  γ ky  x k: ð7:19Þ

Let kx0  xk  ε. We prove (7.13) by induction. From (7.14), (7.15), and the inductive
hypothesis, it follows that

kxk  x k  μkxk  x k  μtk kx0  x k  μ2 kx0  x k  μ2 ε:

When y ¼ xk, it follows that (7.17), (7.18), and (7.19) are true. Since
318 7 Inexact Newton Methods

J ðx Þðxkþ1  x Þ  
¼ J ðx Þ xk  x  J ðxk Þ1 Fðxk Þ þ J ðxk Þ1 r k
1
¼ hJ ðx ÞJ ðxk Þ ðJ ðxk Þðxk  x Þ Fiðxk Þ þ r k Þ ð7:20Þ
¼ I þ J ðx Þ J ðxk Þ1  J ðx Þ1  ½r k þ ðJ ðxk Þ  J ðx ÞÞðxk  x Þ
ðFðxk Þ  Fðx Þ  J ðx Þðxk  x ÞÞ,

then, taking the norm, we obtain

kxkþ1  x k
h

i
 1 þ kJ ðx Þk
J ðxk Þ1  J ðx Þ1
 ½kr k kþkJ ðxk Þ  J ðx Þkkxk  x k
ð7:21Þ
þkFðxk Þ  Fðx Þ  J ðx Þðxk  x Þk
 ð1 þ μγ Þ½ηk kFðxk Þk þ γ kxk  x k þ γ kxk  x k:

However,

Fðxk Þ ¼ ½J ðx Þðxk  x Þ þ ½Fðxk Þ  Fðx Þ  J ðx Þðxk  x Þ;

therefore,

kFðxk Þk  kxk  x k þ γ kxk  x k: ð7:22Þ

Using (7.22) in (7.21), we obtain

   
kxkþ1  x k  ð1 þ μγ Þ ηk kxk  x k þ γ kxk  x k þ 2γ kxk  x k
 ð1 þ μγ Þ½ηð1 þ μγ Þ þ 2μγ kxk  x k
 tkxk  x k : ♦

To prove the superlinear convergence of the inexact Newton method, some conditions on the
residual have to be introduced.

Proposition 7.2 Let

1
α ¼ max kJ ðx Þk þ , 2β ,
2β

where β ¼ kJ(x)1k. Then, for ky  xk sufficiently small, the following inequality
1
ky  x k  kFðyÞk  αky  x k: ð7:23Þ
α
holds.

Proof Since Fis continuously differentiable, then there exists δ > 0 sufficiently small, so that as soon
as ky  xk < δ,
1
kFðyÞ  Fðx Þ  J ðx Þðy  x Þk  ky  x k:
2β

However,
7.1 The Inexact Newton Method for Nonlinear Algebraic Systems 319

FðyÞ ¼ ½J ðx Þðy  x Þ þ ½FðyÞ  Fðx Þ  J ðx Þðy  x Þ:

Therefore,
kFðyÞk  kJ ðx Þkky  x
k þ kFðyÞ  Fðx Þ  J ðx Þðy  x Þk
1 ð7:24Þ
 kJ ðx Þk þ ky  x k:
2β

On the other hand,

 1
1

J ð x Þ
k y  x  k  k Fð y Þ  Fð x  Þ  J ð x  Þ ð y  x  Þ k
k Fð y Þ k   

 1
1 1

J ðx Þ
 ky  x k
2β ð7:25Þ
1
¼ ky  x k:
2β

From (7.24) and (7.25), we obtain (7.23). ♦

Theorem 7.2 Let F : ℝn ! ℝn be a function which satisfies the assumptions H1–H3. Suppose that
the sequence {xk} generated by the inexact Newton method (7.7), (7.8), and (7.9) converges to x.
Then, the convergence is superlinear if and only if

k r k k ¼ o ð k Fð x k Þ k Þ ð7:26Þ

for k ! 1.

Proof Suppose that the sequence {xk} is superlinear convergent to x. Therefore,
r k ¼ Fðxk Þ þ J ðxk Þðxkþ1  xk Þ
¼ ½Fðxk Þ  Fðx Þ  J ðx Þðxk  x Þ  ½J ðxk Þ  J ðx Þðxk  x Þ
þ½J ðx Þ þ ðJ ðxk Þ  J ðx ÞÞðxkþ1  x Þ:

Having in view the assumptions H1–H3 and that {xk} is superlinear convergent, it follows that
k r k k  kFð x k Þ  Fð x  Þ  J ð x  Þ ð x k  x  Þ k þ kJ ð x k Þ  J ð x  Þ k kx k  x  k
þ½kJ ðx Þk þ kJ ðxk Þ  J ðx Þkkxkþ1  x k
¼ oðkxk  x kÞ þ oð1Þkxk  x k þ ½kJ ðx Þk þ oð1Þoðkxk  x kÞ:

Now, from Proposition 7.2, when k ! 1, we obtain

kr k k ¼ oðkxk  x kÞ ¼ oðkFðxk ÞkÞ: ð7:27Þ

On the contrary, let us suppose that krkk ¼ o(kF(xk)k). Then, from (7.20), it follows that
h



i
kxkþ1  x k 
J ðx Þ1
þ
J ðxk Þ1  J ðx Þ1

    
h
½kr k k þ k
J ðxk Þ  iJ ðx Þkkxk  x k þ kFðxk Þ  Fðx Þ  J ðx Þðxk  x Þk
¼
J ðx Þ1
þ oð1Þ ½oðkFðxk ÞkÞ þ oð1Þkxk  x k þ oðkxk  x kÞ:
320 7 Inexact Newton Methods

Therefore, from Proposition 7.2, we get

kxkþ1  x k ¼ oðkFðxk ÞkÞ þ oðkxk  x kÞ ¼ oðkxk  x kÞ,

which proves the superlinear convergence of {xk} generated by the inexact Newton method (7.7),
(7.8), and (7.9). ♦

Observe that if {ηk} ! 0, then the sequence {xk} generated by the inexact Newton method (7.7),
(7.8), and (7.9) converges to x superlinearly. This result can be proved as the following corollary.

Corollary 7.1 Suppose that the sequence {xk} generated by the inexact Newton method (7.7), (7.8),
and (7.9) converges to x. If {ηk} is convergent to zero, then the sequence {xk} is superlinear
convergent to x.

Proof If limk ! 1ηk ¼ 0, then

kr k k
lim sup ¼ 0,
k!1 kFðxk Þk

that is, krkk ¼ o(kF(xk)k). ♦

Theorem 7.3 Let F : ℝn ! ℝn be a function which satisfies the assumptions H1–H3. Suppose that
the forcing sequence {ηk} satisfies 0  ηk  η < 1. For ε > 0, if the initial point x0 is sufficiently close
to x, then the sequence {xk} generated by the inexact Newton method (7.7), (7.8), and (7.9) converges
to x, and the rate of convergence is linear, that is, for k sufficiently large,
kxkþ1  x k  ckxk  x k, ð7:28Þ

where 0 < c < 1. If ηk ! 0, then the sequence {xk} is superlinear convergent to x. If ηk ¼ O(kF(xk)k),
then the sequence {xk} is quadratic convergent to x.

Proof From (7.7), it follows that

sk ¼ J ðxk Þ1 ½Fðxk Þ þ r k :

Then, from Proposition 7.1, we have

ksk k  ξðkFðxk Þk þ kr k kÞ  ξð1 þ ηÞkFðxk Þk  2ξkFðxk Þk: ð7:29Þ

Therefore, using the Taylor development, we get

   
Fðxkþ1 Þ ¼ Fðxk Þ þ J ðxk Þsk þ O ksk k2 ¼ r k þ O kFðxk Þk2 : ð7:30Þ

Hence,
 
kFðxkþ1 Þk  ηk kFðxk Þk þ O kFðxk Þk2 : ð7:31Þ

Now, dividing both members of (7.31) by kF(xk)k and having in view that ηk < η < 1, then, for
k ! 1 , we obtain
7.1 The Inexact Newton Method for Nonlinear Algebraic Systems 321

kFðxkþ1 Þk
lim sup  η < 1: ð7:32Þ
k!1 k Fð x k Þ k

From Proposition 7.2, it follows that

kxkþ1  x k kFðxkþ1 Þk
lim sup  C lim sup , ð7:33Þ
k!1 kxk  x k k!1 kFð x k Þ k

where C is a constant. Therefore, if xk is sufficiently close to x and Cη < 1, then, locally, the sequence
{xk} is linear convergent to x.
Now, let us suppose that ηk ! 0. Then,
kr k k
lim sup ¼ 0,
k!1 kFðxk Þk

that is, krkk ¼ o(kF(xk)k). Using (7.30) it follows that

kFðxkþ1 Þk
lim sup ¼ 0, ð7:34Þ
k!1 kFðxk Þk

thus proving the superlinear convergence of the values of the function F(x). As above, from
Proposition 7.2, it follows that
kxkþ1  x k
lim sup ¼ 0: ð7:35Þ
k!1 kxk  x k

If ηk ¼ O(kF(xk)k), then there exists a constant c1 so that ηk  c1kF(xk)k. From (7.8), we get

kr k k
lim sup  c1 ,
k!1 k Fð x k Þ k 2

thus proving that

 
r k ¼ O k Fð x k Þ k 2 :

Therefore, from (7.30), we obtain

kFðxkþ1 Þk
lim sup ¼ c, ð7:36Þ
k!1 k Fð x k Þ k 2

where c is an arbitrary constant, thus proving the quadratic convergence of the sequence {F(xk)},
that is,

kxkþ1  x k
lim sup ¼ c: ♦
k!1 kx k  x  k 2
322 7 Inexact Newton Methods

7.2 Inexact Newton Methods for Functions Minimization

As we know, for solving the minimizing problem

min f ðxÞ ð7:37Þ
x2ℝn

where f : ℝn ! ℝ is twice continuously differentiable, the Newton method in the current point xk
minimizes the quadratic model
1
m2 ðd Þ ¼ f ðxk Þ þ dT ∇f ðxk Þ þ d T ∇2 f ðxk Þd ð7:38Þ
2
and considers the new approximation to the minimizer x of the function f as
xkþ1 ¼ xk þ αk dk , ð7:39Þ

where dk is the minimum point of m2(d ) and αk > 0 is the stepsize. Obviously, the minimum of m2(d),
i.e., the search direction dk, is obtained as solution of the following algebraic linear system

∇2 f ðxk Þd k ¼ ∇f ðxk Þ, ð7:40Þ

known as the Newton system.

On the other hand, for minimizing the function f, the inexact Newton method does not consider the
exact solution of the Newton system. Instead, the inexact Newton method considers an approximate
solution of (7.40) based on the residual

r k ¼ ∇2 f ðxk Þdk þ ∇f ðxk Þ: ð7:41Þ

A strategy for implementing this idea is to require that

kr k k  ηk k∇f ðxk Þk, ð7:42Þ

where, as above, the sequence {ηk} with 0 < ηk < 1 for all k is called the forcing sequence. The local
convergence of the truncated Newton method is simply obtained by ensuring that the sequence {ηk} is
bounded away from 1.
The following results proved by Dembo, Eisenstat, and Steihaug (1982) are local and show the
convergence of the inexact Newton method.

Theorem 7.4 Suppose that ∇2f(x) exists and is continuous in a neighborhood of a minimizer x, with
∇2f(x) positive definite. Consider the iteration xkþ1 ¼ xk þ dk, where dk satisfies krkk  ηkkgkk, and
assume that ηk  η for some constant η 2 (0, 1). If the starting point x0 is sufficiently close to x, then
the sequence {xk} generated by the truncated Newton method converges to x and satisfies

2 




∇ f ðx Þðxkþ1  x Þ
 bη
∇2 f ðx Þðxk  x Þ
, ð7:43Þ

for some constant b

η with η < b
η < 1:
7.2 Inexact Newton Methods for Functions Minimization 323

Proof In the following, an informal proof will be presented (Nocedal & Wright, 2006). Since the
Hessian matrix ∇2f(x) is positive definite at x and continuous near x, it follows that there exists a
positive constant L such that k∇2f(xk)1k  L for all xk sufficiently close to x. Therefore, from (7.41)
and since krkk  ηkk∇f(xk)k where ηk < 1, it follows that the inexact Newton step satisfies

kdk k  Lðk∇f ðxk Þk þ kr k kÞ  2Lk∇f ðxk Þk:

From the continuity of ∇2f(x), we have

ð1
∇f ðxkþ1 Þ ¼ ∇f ðxk Þ þ ∇ f ðxk Þd k þ ð∇f ðxk þ td k Þ  ∇f ðxk ÞÞdk dt
2

0
ð7:44Þ
¼ ∇f ðxk Þ þ ∇ f ðxk Þd k þ oðkdk kÞ
2

¼ ∇f ðxk Þ  ð∇f ðxk Þ  r k Þ þ oð∇f ðxk ÞÞ

¼ r k þ oðk∇f ðxk ÞkÞ:

Having in view (7.42), from (7.44), we get

k∇f ðxkþ1 Þk  ηk k∇f ðxk Þk þ oðk∇f ðxk ÞkÞ  ðηk þ oð1ÞÞk∇f ðxk Þk: ð7:45Þ

When xk is close enough to x, it follows that the o(1) term in the last estimate is bounded by
(1  η)/2. Therefore, from (7.45), we obtain
1þη
k∇f ðxkþ1 Þk  ðηk þ ð1  ηÞ=2Þk∇f ðxk Þk  k∇f ðxk Þk, ð7:46Þ
2
that is, the norm of the gradient decreases by a factor of (1 þ η)/2 at this iteration. By choosing the
initial point x0 sufficiently close to x, it follows that this rate of decreases occurs at every iteration.
To prove (7.43), observe that under the smoothness assumptions, we have

∇f ðxk Þ ¼ ∇2 f ðx Þðxk  x Þ þ oðkxk  x kÞ:

Therefore, for xk close to x, it can be shown that the gradient ∇f(xk) differs from the scaled error
∇ f(x)(xk  x) by only a relatively small perturbation. A similar estimate holds at xkþ1, so (7.43)
2

follows from (7.46). ♦

For the superlinear convergence, from (7.45), observe that

k∇f ðxkþ1 Þk
 ηk þ oð1Þ: ð7:47Þ
k∇f ðxk Þk

If limk!1ηk ¼ 0, it follows that

k∇f ðxkþ1 Þk
lim ¼ 0, ð7:48Þ
k!1 k∇f ðxk Þk

showing the superlinear convergence of the gradient norms k∇f(xk)k to zero. As a consequence, we
have the superlinear convergence of the iterates {xk} to x.
324 7 Inexact Newton Methods

By making the additional assumption that the Hessian ∇2f(x) is Lipschitz continuous near x, the
quadratic convergence of {xk} to x is obtained. In this case, the estimate (7.44) can be tightened to
 
∇f ðxkþ1 Þ ¼ r k þ O k∇f ðxk Þk2 :

Therefore, by choosing the forcing sequence so that ηk ¼ O(k∇f(xk)k), it follows that

 
k∇f ðxkþ1 Þk ¼ O k∇f ðxk Þk2 ,

thus indicating the quadratic convergence of the gradient norms to zero and also the quadratic
convergence of the iterates {xk} to x. These remarks can be assembled in the following theorem.

Theorem 7.5 Suppose that the conditions of Theorem 7.4 hold and assume that the iterates {xk}
generated by the inexact Newton method converge to x. Then, the rate of convergence is superlinear
if ηk ! 0. If in addition ∇2f(x) is Lipschitz continuous for x near x and if ηk ¼ O(k∇f(xk)k), then the
convergence is quadratic. ♦

The best selection of the sequence {ηk} is unknown. The only requirement is ηk ! 0. If ηk ! 0,
then the rate of convergence of the inexact Newton method is superlinear. If in addition ∇2f(x) is
Lipschitz continuous for x near x and if ηk ¼ O(k∇f(xk)k), then the convergence is quadratic. To
 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
obtain the superlinear convergence, set ηk ¼ min 0:5, k∇f ðxk Þk : The quadratic convergence is
obtained for ηk ¼ min {0.5, k∇f(xk)k}.

7.3 The Line-Search Newton-CG Method

In this method, the search direction is computed by applying the conjugate gradient method to the
Newton system (7.1) until the termination test (7.3) is satisfied. Recall that the conjugate gradient
method is designed to solve positive definite linear systems. However, it is quite possible for the
Hessian ∇2f(xk) to have negative eigenvalues when xk is far away from x. Therefore, the algorithm
terminates as soon as a direction of negative curvature is generated. This adaptation of Algorithm 5.2
produces a search direction dk which is a descent direction. Moreover, the adaptation guarantees that
the fast convergence rate of the pure Newton method is preserved, provided that the stepsize αk ¼ 1 is
used whenever it satisfies the acceptance criteria.
The following algorithm is a modification of Algorithm 5.2, which implements the inner iterations
to compute the search direction dk. In this algorithm, the linear system (7.1) is written as Bkd ¼  gk,
where Bk represents ∇2f(xk) and gk is ∇f(xk). For the inner conjugate gradient iterations, the search
directions are denoted by dj, and the sequence of iterates generated is denoted by zj. When Bk is
positive definite, the inner iteration sequence {zj} converges to the Newton step dNk , solution of
Bkd ¼  gk. At each outer iteration, a tolerance εk is computed, which specifies the accuracy of the
computed solution. To obtain the superlinear convergence, the forcing sequence is selected to be
 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ηk ¼ min 0:5, k∇f ðxk Þk :
7.4 Comparison of TN Versus Conjugate Gradient Algorithms 325

Algorithm 7.1 Line-search Newton-CG (truncated Newton)

1. Initialization. Select the initial point x0. Set k ¼ 0

 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2. Compute the tolerance εk ¼ min 0:5, k∇f ðxk Þk k∇f ðxk Þk
3. Set z0 ¼ 0, r0 ¼ ∇ f(xk), d0 ¼  r0 ¼  ∇ f(xk)
4. For j ¼ 0, 1, 2, . . .
If dTj Bk dj  0
If j ¼ 0
return dk ¼  ∇ f(xk)
Else
return dk ¼ zj
End if
End if
Set: αj ¼ r Tj r j =dTj Bk d j , zjþ1 ¼ zj þ αjdj, rjþ1 ¼ rj þ αjBkdj
If krjþ1k < εk
return dk ¼ zjþ1
End if
Set: βjþ1 ¼ r Tjþ1 r jþ1 =r Tj r j , djþ1 ¼  rjþ1 þ βjþ1dj
End for
5. Compute the stepsize αk by means of Armijo, Goldstein, or Wolfe line-search
6. Set xkþ1 ¼ xk þ αkdk
7. Go to step 2 ♦

The main differences between the inner loop of Algorithm 7.1 and of Algorithm 5.2 are as follows:
the specific starting point z0 ¼ 0 is used; the positive tolerance εk allows the conjugate gradient
iterations to terminate at an inexact solution; and the negative curvature test dTj Bk dj  0 ensures that
dk is a descent direction for f at xk. If the negative curvature is detected at the very first iteration j ¼ 0,
then the returned direction d ¼  ∇ f(xk) is both a descent direction and a direction of nonpositive
curvature for f at xk (Nocedal & Wright, 2006). Algorithm 7.1 can be modified by introducing the
preconditioning in conjugate gradient iterations.
It is worth saying that the line-search Newton-CG algorithm does not require explicit knowledge
of the Hessian Bk ¼ ∇2f(xk). Instead, it requires only the Hessian-vector products of the form Bkd for a
given vector d. When the second-order derivatives are difficult to compute or the Hessian requires too
much storage, then the finite differencing can be used to calculate the Hessian-vector products. In this
case, the product ∇2f(xk)d can be approximated as
∇f ðxk þ hdÞ  ∇f ðxk Þ
∇2 f ðxk Þd ¼ ð7:49Þ
h
for a certain small differencing interval h. The differencing interval h can be computed as
pffiffiffiffiffi
2 εm ð1 þ kxk kÞ
h¼ , ð7:50Þ
1 þ kd k k
where εm is the epsilon machine.

7.4 Comparison of TN Versus Conjugate Gradient Algorithms

Observe that the truncated Newton method has two imbedded loops. The first outer loop implements
the Newton algorithm. The second inner loop implements the mechanism for an approximate solution
326 7 Inexact Newton Methods

Fig. 7.1 Performance of TN (Nash) versus conjugate gradient algorithms CONMIN, DESCON, CG-DESCENT, and
CG-DESCENTaw

to the Newton system. Two implementations of this method are known. In the first one (TN), given by
Nash (1985), a BFGS approximation of the Hessian is computed and an approximate solution to the
Newton system is determined by the Newton-CG algorithm. In the second one (TNPACK), given by
Schlick and Fogelson (1992a, 1992b), the Hessian matrix is computed by finite differences, and an
approximate solution to the preconditioned Newton system is computed by the Newton-CG
algorithm.
In the following, let us present a comparison between the truncated Newton method TN and the
conjugate gradient methods CONMIN, DESCON, CG-DESCENT with Wolfe line-search
(CG-DESCENT), and CG-DESCENT with the approximate Wolfe line-search (CG-DESCENTaw),
for solving 800 unconstrained optimization problems from the UOP collection, with the number of
variables in the range [1000, 10000]. Figure 7.1 illustrates the performance profile of these
algorithms.
Figure 7.1 illustrates that the conjugate gradient algorithms CONMIN, DESCON, CG-DESCENT,
and CG-DESCENTaw are way more efficient and more robust than TN. For example, subject to the
CPU time metric, observe that CG-DESCENTaw is the fastest in solving 526 problems, while TN is
the fastest in solving only 67 problems, etc. The modern conjugate gradient algorithms are able to
better catch the curvature of the minimizing function, thus reducing the number of the function and its
gradient evaluations and implicitly the CPU computing time. All these conjugate gradient algorithms
implement a lot of ingredients which improve their numerical performances: automatic restarts,
storing some additional information, using the approximate or the modified Wolfe line-search, etc.
7.5 Comparison of TN Versus L-BFGS 327

On the other hand, by taking an approximate solution of the Newton system, TN spends more time in
using the approximate search direction compared to the conjugate gradient algorithms.

7.5 Comparison of TN Versus L-BFGS

A numerical study on the performances of the limited-memory BFGS method L-BFGS (Liu &
Nocedal, 1989) and on the truncated-Newton TN (Nash, 1985) was given by Nash and Nocedal
(1991). Some details on the truncated Newton method in the implementation of TN, given by Nash,
and on the limited-memory BFGS in the implementation of L-BFGS, given by Liu and Nocedal, are
as follows.
At each outer iteration of TN, an approximate solution to the Newton system is computed. This is
done by using an inner iteration based on a preconditioned linear conjugate gradient algorithm, as
described by Concus, Golub, and O’Leary (1976). If indefiniteness in the Hessian is detected, then the
inner iteration is stopped. This approximate solution, that is, the search direction, is then used in a
line-search to get a new point which approximates the minimizer of the minimizing function. TN
implements the strong Wolfe line-search described by Gill and Murray (1979). In TN, the conjugate
gradient used in the inner iteration is preconditioned by a scaled two-step limited memory BFGS
method with Powell’s restarting strategy used to reset the preconditioner periodically. More than that,
the matrix-vector products in TN required by the inner conjugate gradient algorithm are obtained by
finite differencing. The truncated Newton method TN requires storage for 16 vectors of length n.
The limited memory L-BFGS in the implementation of Liu and Nocedal uses a scaling at the first
iteration by a diagonal matrix suggested by Oren and Spedicato (1976). In L-BFGS, the approxima-
tion of the inverse Hessian is obtained by applying m BFGS corrections to the initial diagonal matrix,
using the m previous stored vectors si and yi. The stepsize is computed by using the strong Wolfe line-
search implemented in the subroutine CVSRCH of Moré and Thuente (1990) and by trying the
stepsize αk ¼ 1 first. The L-BFGS method requires 2m(n þ 1) þ 4n storage locations.
The conclusions of Nash and Nocedal (1991) for comparing TN with L-BFGS are as follows.
L-BFGS and TN use different principles to compute the search direction. L-BFGS uses a fixed,
low-cost formula requiring no extra derivative information. TN uses an elaborate, variable-cost
iteration with partial second-derivative information. Both of them use the cubic interpolation to
obtain the strong Wolfe conditions. The numerical experiments showed that neither algorithm is
clearly superior to the other one. In terms of the CPU computing time, neither algorithm is a clear
winner: the higher iteration cost of TN is compensated by a much lower iteration count, on average.
The performance of these algorithms appears to be correlated with the degree of nonlinearity: for
quadratic and approximately quadratic problems, TN outperforms L-BFGS. For highly nonlinear
problems, L-BFGS is the best. In terms of function evaluations, L-BFGS is preferable to TN for more
highly nonlinear problems. However, TN almost always requires fewer iterations than L-BFGS, and
therefore, if the number of the gradient evaluations in the inner iteration could be significantly
reduced, TN would be competitive or more efficient than L-BFGS. Subject to the distribution of
the eigenvalues of the Hessian, there is no clear correlation between the success of the methods and
the eigenvalue structure. For problems with ill-conditioned Hessians, TN seems to be better. The
clustering of the eigenvalues at the solution does not seem to be beneficial to one method more than to
the other one.
In the following, we shall present the performances of the L-BFGS (m ¼ 5) in the implementation
of Liu and Nocedal (1989) and of TN in the implementation of Nash (1985) for solving 80 uncon-
strained optimization test problems from our UOP collection, with n ¼ 1000, . . ., 10000. Figure 7.2
presents the Dolan and Moré performance profiles of TN versus L-BFGS (m ¼ 5) for solving this set
328 7 Inexact Newton Methods

Fig. 7.2 Performance

profiles of L-BFGS (m ¼ 5)
versus TN (truncated
Newton) based on
iterations calls, function
calls, and CPU time,
respectively

of 800 unconstrained optimization test problems subject to the iterations calls, function calls, and
CPU time metric, respectively.
By comparing L-BFGS versus TN (see Fig. 7.2) subject to the number of iterations, we can see that
L-BFGS is better in 310 problems (i.e., it achieved the minimum number of iterations in
310 problems), while TN is better in 387 problems, etc. Out of 800 problems considered in this set
of numerical experiments, only for 728 problems does the criterion (1.3) hold.
Note that, subject to the number of iterations and to the number of function calls, TN is a top
performer. However, subject to the CPU time metric, L-BFGS is more efficient and more robust. Both
these algorithms are reliable for solving a large variety of large-scale unconstrained optimization
7.6 Solving Large-Scale Applications 329

problems. In our numerical experiments, we noticed that the performances of these methods do not
depend on the structure of the Hessian matrix of the problems.
Let us now present the performances of these algorithms for solving the applications from the
MINPACK-2 collection (see Appendix D). Table 7.1 presents the performances of TN in the
implementation of Nash (1985) for solving these applications, where n is the number of variables
(n ¼ nx  ny, nx ¼ 200, ny ¼ 200), #iter is the number of iterations to get the solution, #fg is the
number of function and its gradient calls, and cpu is the CPU computing time in seconds for solving
the application.
Comparing Table 7.1 with Tables 6.2, 6.3, 6.4, 6.5, and 6.6, we can see that, subject to the number
of iterations and to the function calls, TN is better than L-BFGS, but subject to the CPU computing
time metric, both L-BFGS and TN have similar performances, L-BFGS being slightly faster.

7.6 Solving Large-Scale Applications

The inexact or truncated Newton methods are designed for solving large-scale unconstrained optimi-
zation problems. Since for large-scale problems the factorization of the Hessian is prohibitive, it
follows that it is preferable to compute an approximate solution to the Newton step by using iterative
linear algebra techniques. TN by Nash implements such a technique. The resulting algorithm has the
global convergence property and with appropriate parameters may be superlinear convergent to the
solution. Besides, TN finds an effective search direction when the Hessian ∇2f(xk) is indefinite, being
implemented in a “Hessian-free” manner without explicit computation or storage of the Hessian.
Table 7.2 shows the performances of TN for solving five applications from the MINPACK-
2 collection, each of them with 250,000 variables (n ¼ nx  ny, nx ¼ 500, ny ¼ 500).
Comparing the performances of TN subject to the CPU computing time versus the modern
conjugate gradient algorithms CG-DESCENT (Table 5.13) with 2401.01 seconds versus DESCON
(Table 5.14) with 1537.55 seconds and versus DKþ (Table 5.15) with 2643.24 seconds, we can see
that TN (Table 7.2) with 1223.89 seconds is a top performer. In the same realm of comparisons,

Table 7.1 Performances of TN for solving five applications from the MINPACK-2 collection (40,000 variables)

n #iter #fg cpu

A1 40,000 14 320 10.35
A2 40,000 40 790 13.92
A3 40,000 53 1791 53.37
A4 40,000 26 501 21.87
A5 40,000 20 312 5.06
Total - 153 3714 104.57

Table 7.2 Performances of TN for solving five applications from the MINPACK-2 collection (250,000 variables)

n #iter #fg cpu

A1 250,000 12 649 45.61
A2 250,000 56 1933 173.11
A3 250,000 139 4205 652.80
A4 250,000 29 943 285.13
A5 250,000 17 726 67.24
Total - 253 8456 1223.89
330 7 Inexact Newton Methods

subject to the CPU time metric, we can see that TN (Table 7.2) with 1223.89 seconds is slightly faster
than L-BFGS (m ¼ 5) (Table 6.9) with 1247.09 seconds.

Notes and References

A deep analysis of the inexact Newton method is given by Conn, Gould, and Toint (2000). The choice
of the forcing terms in an inexact Newton method for solving nonlinear algebraic systems of
equations was presented by Eisenstat and Walker (1996) and An, Mo, and Liu (2007). A matrix-
free line-search algorithm for the large-scale equality constrained optimization that allows for inexact
step computations was given by Byrd, Curtis, and Nocedal (2010). Numerical experience with the
truncated Newton method for the unconstrained optimization was presented by Dixon and Price
(1988). Freely available software for the unconstrained optimization includes TN (Nash, 1984a,
1984b) and TNPACK (Schlick & Fogelson, 1992a, 1992b). In TNPACK, a sparse modified Cholesky
factorization based on the Yale Sparse Matrix Package is used to factor the preconditioner, which
need not be positive definite. Two modified Cholesky factorizations have been implemented in
TNPACK (Schlick, 1993). The package TNBC (Nash, 1984a, 1984b) implements the truncated
Newton method for solving simple bound optimization problems (see Chap. 12). A survey of the
truncated Newton methods was presented by Nash (2000).
 The Trust-Region Method
8

In the unconstrained optimization, two approaches are fundamental: the line-search and the trust-
region. Both of them generate steps by using a quadratic model of the minimizing function, but in
different ways. The line-search methods, presented in the previous chapters, generate a descent search
direction d and then determine a suitable stepsize α along this direction, hoping that the function
values will be reduced. On the other hand, the trust-region methods define a region around the current
iterate within which we trust the quadratic model to be an adequate representation of the minimizing
function and to choose the step which is the approximate minimizer of the model in this region.
Therefore, the trust-region methods choose the direction and the stepsize simultaneously. Of course,
if a step is not acceptable, the size of the region will be reduced and a new minimizer will be found.
The size of the trust-region is important in the economy of each step. If the region is too small, then
the algorithm will take small steps. If it is too large, the minimizer of the model may be far from the
minimizer of the function. The size of the region is selected based on the performance of the
algorithm at the previous iteration.
The purpose of this chapter is to present the trust-region method as well as its properties for
solving unconstrained optimization problems. For this, the concept of trust-region, the convergence
of the corresponding algorithm, and different techniques for solving the subproblems associated to
this method are to be developed. One of the best descriptions of the trust-region method is given by
Nocedal and Wright, (2006). We follow their developments.

8.1 The Trust-Region

Consider the problem min{f(x) : ℝn ! ℝ}, where f is continuously differentiable. Assume that for the
function f, around the current point xk, a quadratic model mk is used, which is based on the Taylor
series development
1
f ðxk þ dÞ ¼ f ðxk Þ þ gTk d þ d T ∇2 f ðxk þ td Þd, ð8:1Þ
2
where gk ¼ ∇ f(xk) and t is a scalar in the interval (0,1). Considering an approximation Bk to the
Hessian of the minimizing function f, the model mk is defined as
1
mk ðd Þ ¼ f ðxk Þ þ gTk d þ dT Bk d, ð8:2Þ
2

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 331
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_8
332 8 The Trust-Region Method

where Bk is a symmetric matrix. Observe that the difference between f(xk þ d) and mk(d) is of order
O(kdk2), which is small when kdk is small.
When Bk is equal to the true Hessian ∇2f(xk), the above difference is of order O(kdk3), showing that
the model is accurate when kdk is small. This choice of Bk ¼ ∇2f(xk) defines the trust-region Newton
method.
The step of the trust-region method is computed as solution of the following subproblem:
1
min mk ðd Þ ¼ f ðxk Þ þ gTk d þ dT Bk d,
d2ℝn 2
subject to ð8:3Þ
kd k  Δ k ,

where Δk > 0 is the trust-region radius.

In (8.3), k.k is the Euclidian norm, so the solution dk of (8.3) is the minimizer of mk in the ball of
radius Δk. Therefore, the trust-region approach involves the solving of a sequence of subproblems
(8.3)
in which
the objective function and the constraints are both quadratic. If Bk is positive definite
and
B1 k g

k  Δk , then the solution of (8.3) is easy to compute as the unconstrained minimum
dBk ¼ B1 B
k gk of the quadratic mk(d). In this case, d k is called the full step. In other cases, the solution
of (8.3) is not so obvious, and the computational effort associated to the trust-region method is to
solve the subproblem (8.3). However, in the trust-region method, we need only an approximate
solution of (8.3), which is not so difficult to obtain.
Considering Bk ¼ ∇2f(xk), the optimality conditions for (8.3) show that dk is the solution of the
linear system
 2 
∇ f ðxk Þ þ λI dk ¼ ∇f ðxk Þ,

where the scalar λ  0 is the Lagrange multiplier associated to the trust-region constraints, the matrix
(∇2f(xk) þ λI) is positive semidefinite and
λðΔk  kdk kÞ ¼ 0:

If ∇2f(xk) is positive definite and Δk is sufficiently large, then the solution of (8.3) is the solution of
the linear system

∇2 f ðxk Þd ¼ ∇f ðxk Þ,

the Newton system. Otherwise, the trust-region method guarantees that

 1




Δk  kd k k ¼
∇2 f ðxk Þ þ λI ∇f ðxk Þ
,

and so, if Δk ! 0, then λ ! 1 and dk   1λ ∇f ðxk Þ: Therefore, dk is a function of λ and indirectly a

function of Δk. As λvaries between 0 and 1, it can be shown that dk ¼ dk(λ) continuously varies
between the Newton direction (in the positive definite case) and a multiple of  ∇ f(xk). (More
discussion on the solution of (8.3) is given in Sect. 8.5, where an iterative technique for solving the
subproblem (8.3) is presented.)
The most important ingredient in any trust-region method is the strategy for selecting the trust-
region radius Δk at each iteration. The selectin is based on the agreement between the model mk and
the minimizing function f at the previous iteration.
8.1 The Trust-Region 333

Given the step dk, let us define the ratio

f ðxk Þ  f ðxk þ dk Þ
rk ¼ , ð8:4Þ
mk ð0Þ  mk ðd k Þ

where f(xk)  f(xk þ dk) is called the actual reduction and mk(0)  mk(dk) is called the predicted
reduction (predicted by the model). The value of rk is crucial in the trust-region method.
Since the step dk is obtained by minimizing the model mk over a domain that includes d ¼ 0, it
follows that the predicted reduction will always be nonnegative. Therefore, if rk is negative, then the
new value of the minimizing function f(xk þ dk) is greater than the current value f(xk), and in this case,
this step must be rejected. On the other hand, if rk is close to 1, then there is a good agreement between
the model mk and the function f over this step, so it is safe to expand the trust-region for the next
iteration. Finally, if rk is positive but significantly smaller than 1, then the trust-region is not altered,
but if it is close to zero or negative, the trust-region is shrank by reducing Δk at the next iteration. This
strategy is described on steps as the trust-region algorithm.

Algorithm 8.1 Trust-region

1. Initialization. Select an initial point x0 and the initial trust-region radius Δ0 > 0. Choose the constants
0 < μ < η < 1 (e.g., μ ¼ 1/4 and η ¼ 3/4). Set k ¼ 0
2. For k ¼ 0, 1, . . .
(i) If xk is optimal, stop
(ii) Compute dk as an approximate solution of (8.3)
(iii) Evaluate the ratio rk from (8.4)
(iv) If rk  μ, then set xkþ1 ¼ xk (unsuccessful step), else set xkþ1 ¼ xk þ dk (successful step)
(v) Update Δk:
If rk  μ, then set Δkþ1 ¼ Δk/2,
If μ < rk < η, then set Δkþ1 ¼ Δk,
If rk  η, then set Δkþ1 ¼ 2Δk.
End for ♦

In Algorithm 8.1, observe that the radius is increased only if kdkk actually reaches the boundary of
the trust-region. Otherwise, if the step stays strictly inside the region, then the value of the radius is
not changed for the next iteration. The value of rk shows how well the model predicts the reduction in
the function value. If rk is small, that is, rk  μ, then the actual reduction in the function value is much
smaller than the one predicted by mk(dk), that is, the model cannot be trusted for a radius as large as
Δk. In this case, the step dk will be rejected and Δk will be reduced. On the other hand, if rk is large,
that is, rk  η, then the model is adequately predicting the reduction in the function value, suggesting
that the model can be trusted over an even wider region. In this case, the radius Δk will be increased.
We now state a convergence theorem of the trust-region methods (see Griva, Nash, & Sofer,
2009). Additional theoretical results and discussions of the second-order optimality conditions can be
found in the paper by Moré (1983).

Theorem 8.1 Let f : ℝn ! ℝ be a real-valued function. Consider x0 as an initial point and

{xk} the sequence of points generated by the trust-region algorithm. Assume that the level set
S ¼ {x : f(x)  f(x0)} is a compact set and f(x), ∇f(x), and ∇2f(x) are continuous for all x 2 S. Then,
lim k∇f ðxk Þk ¼ 0:
k!1
334 8 The Trust-Region Method

Proof Before presenting the proof, some comments are needed. Observe that the theorem does not
state that the sequence {xk} converges to a local minimum of f. It only states that ∇f(xk) ! 0. The
proof has two parts.

In the first part, it is proved that a subsequence of {k∇f(xk)k} converges to zero. The proof is by
contradiction. If no such subsequence converges to zero, then, for all sufficiently large values of k, it
follows that k∇f(xk)k  ε > 0, where ε is a constant. However, we are interested only in the asymptotic
behavior of the trust-region algorithm, and therefore, the early iterations may be ignored, that is, we
may as well assume that k∇f(xk)k  ε for all k. The first part of the proof has five major steps. The first
two steps establish the relationships among the quantities f(xk)  f(xkþ1), mk(dk), Δk, and k∇f(xk)k.
The remaining steps use the assumption that k∇f(xk)k  ε to obtain a contradiction. Step 3 shows that
limk ! 1Δk ¼ 0. If Δk is small, then kdkk is too, and the quadratic model must be a good prediction of
the actual reduction in the function value, that is, limk ! 1rk ¼ 1. If this it true, then the trust-region
algorithm will not reduce Δk, that is, limk ! 1Δk 6¼ 0, thus contradicting the result of step 3 and
proving the overall result. Let M be a constant satisfying k∇2f(xk)k  M for all k. Observe that this
upper bound M exists because ∇2f(x)is continuous on the level set S that is closed and bounded
(compact).
The first part of the proof is as follows:

1. A bound on the predicted reduction: Let us prove that

 
1 k∇f ðxk Þk
f ðxk Þ  mk ðd k Þ  k∇f ðxk Þk min Δk ,
2 M

by investigation how small mk could be if dk is a multiple of  ∇ f(xk). For this, define the function
 
∇f ðxk Þ
ΦðαÞ ¼ mk α  f ðxk Þ
k∇f ðxk Þk
T 2 
∇f ðxk ÞT ∇f ðxk Þ 1 2 ∇f ðxk Þ ∇ f ðxk Þ ∇f ðxk Þ
¼ α þ α
k∇f ðxk Þk 2 k∇f ðxk Þk2
1
¼ αk∇f ðxk Þk þ α2 Mk ,
2
where Mk ¼ ∇ f(xk)T(∇2f(xk)) ∇ f(xk)/k∇f(xk)k2  k∇2f(xk)k  M. Let α be the minimizer of Φ on the
interval [0, Δk]. Observe that α > 0. If 0 < α < Δk, then α can be determined by setting Φ0(α) ¼ 0,
showing that α ¼ k∇f(xk)k/Mk and
1 1
Φðα Þ ¼  k∇f ðxk Þk2 =Mk   k∇f ðxk Þk2 =M:
2 2
On the other hand, supposing that α ¼ Δk, it follows that MkΔk  k∇f(xk)k. This follows from the
fact that if Mk  0, then this is trivially satisfied; otherwise, this is a consequence of setting Φ0(α) ¼ 0,
since the solution of this equation must be Δk. Thus,
1 1
Φðα Þ ¼ ΦðΔk Þ ¼ Δk k∇f ðxk Þk þ Δ2k Mk   Δk k∇f ðxk Þk:
2 2
Therefore, the desired result is obtained by observing that mk(dk)  f(xk)  Φ(α).
8.1 The Trust-Region 335

2. A bound on f(xk)  f(xkþ1): If a successful step is taken, then

f ðxk Þ  f ðxkþ1 Þ
μ  rk ¼ ,
f ðxk Þ  mk ðd k Þ

where μ is the constant used to test rk in the trust-region algorithm. Hence, from step 1, it follows that
f ðxk Þ  f ðxkþ1 Þ  ðf ðxk Þ  mk ðdk ÞÞμ 
1 k∇f ðxk Þk
 μk∇f ðxk Þk min Δk , :
2 M

3. limk!1Δk ¼ 0: Since f is bounded below on S and the trust-region algorithm ensures that
f cannot increase at any iteration, it follows that limk!1f(xk) exists and is finite. Assume that
k∇f(xk)k  ε > 0. If k is a successful step, then step 2 shows that

n o
1 ε
f ðxk Þ  f ðxkþ1 Þ  με min Δk , :
2 M
The limit of the left-hand side is zero, so lim ki !1 Δki ¼ 0, where {ki} are the indices of the
iterations where successful steps are taken. At successful steps, the trust-region radius is either kept
constant or doubled; at unsuccessful steps, the radius is reduced. Therefore, between successful steps,
2Δki  Δki þ1  ⋯  Δkiþ1 : Thus, limk ! 1Δk ¼ 0.

4. limk ! 1rk ¼ 1: From the Taylor series of f(xk þ dk), it follows that

jf ðxk þ d k Þ  mk ðdk Þj

1
¼ f ðxk Þ þ ∇f ðxk ÞT dk þ dTk ∇2 f ðxk þ ξk dk Þdk  mk ðdk Þ
2

1 1
¼  dTk ∇2 f ðxk Þd k þ d Tk ∇2 f ðxk þ ξk d k Þdk
2 2
1 1
 Mkdk k2 þ Mkd k k2 ¼ Mkdk k2  MΔ2k :
2 2
Using the bound from step 1 and the result of step 3, for large values of k, we get
1
f ðxk Þ  mk ðd k Þ  εΔk :
2
Therefore,

f ðx k Þ  f ðx k þ d k Þ

jr k  1j ¼  1
f ðxk Þ  mk ðd k Þ
j f ðx k þ d k Þ  m k ðd k Þj
¼
jf ðxk Þ  mk ðd k Þj
MΔ2k 2M
 ¼ Δ ! 0:
εΔk =2 ε k
336 8 The Trust-Region Method

5. limk!1Δk 6¼ 0: If limk!1rk ¼ 1, then, for large values of k, the trust-region algorithm will not
decrease Δk. Therefore, Δk will be bounded away from zero.

This is a contradiction establishing that a subsequence of the sequence {k∇f(xk)k} converges to

zero, thus completing the first part of the proof.
The second part of the proof is as follows. By contradiction, it is proved that limk!1k∇f(xk)k ¼ 0.
If this result is not true, then k∇f ðxki Þk  ε > 0 for a subset {ki} of the iterations of the trust-region
algorithm. However, since a subsequence of {k∇f(xk)k} converges to zero, there must exist a set of
indices {li} such that
1
k∇f ðxk Þk  ε for ki  k < li ,
4
1
k∇f ðxli Þk < ε:
4
If ki  k < li and iteration k is successful, then step 2 shows that

  
1 1 ε=4
f ðxk Þ  f ðxkþ1 Þ  μ ε min Δk , :
2 4 M

The left-hand side of the above inequality goes to zero, so that

f ðxk Þ  f ðxkþ1 Þ  ε1 kxkþ1  xk k,

where ε1 ¼ (με)/8. Because kxkþ1  xkk ¼ 0 for an unsuccessful step, it follows that this result holds
for ki  k < li. Using this result repeatedly, we obtain

ε 1 kx k i  x l i k
 ε1 ðkxki  xki þ1 k þ kxki þ1  xki þ2 k þ ⋯ þ kxli 1  xli kÞ
 f ðxki Þ  f ðxki þ1 Þ þ f ðxki þ1 Þ  f ðxki þ2 Þ þ ⋯ þ f ðxli 1 Þ  f ðxli Þ
¼ f ðxki Þ  f ðxli Þ:

Since the right-hand side of the above result goes to zero, it follows that the left-hand side can be
made arbitrarily small. Since ∇f(x) is continuous on S and S is a compact, by choosing i large enough
it is possible to guarantee that
ε
k∇f ðxki Þ  ∇f ðxli Þk  :
4
With this, the following contradiction is obtained:
ε  k∇f ðxki Þk ¼ kð∇f ðxki Þ  ∇f ðxli ÞÞ þ ∇f ðxli Þk
ε ε ε
 k∇f ðxki Þ  ∇f ðxli Þk þ k∇f ðxli Þk  þ ¼ < ε:
4 4 2
Therefore, limk!1k∇f(xk)k ¼ 0. ♦

The main problem with Algorithm 8.1 is how to solve the trust-region subproblem (8.3). In the
following, three strategies for obtaining an approximate solution of the subproblem (8.3) are
presented. They are based on the Cauchy point. This point is the minimizer of mk along the steepest
descent direction given by the negative gradient subject to the trust-region bound. The first approxi-
mate strategy is the dogleg method, which is appropriate when the Hessian Bk is positive definite. The
8.2 Algorithms Based on the Cauchy Point 337

second strategy, known as the two-dimensional subspace minimization, can be used when Bk is
indefinite. The third strategy, known as the trust-region Newton-CG strategy, is based on the
conjugate gradient method to minimize mk and is appropriate when Bk is large and sparse.

8.2 Algorithms Based on the Cauchy Point

We emphasize that we seek an optimal solution for (8.3), but for the global convergence, it it enough
to find an approximate solution dk that lies inside the trust-region and gives a sufficient reduction in
the model.

The Cauchy Point

The sufficient reduction in the model can be measured in terms of the Cauchy point, denoted by dCk ,
which is computed by the following procedure.

Algorithm 8.2 Cauchy point computation

1. Find the vector d sk solution of the linear

 version of (8.3), that is,
dsk ¼ arg min f ðxk Þ þ gTk d subject to kdk  Δk (8.5)
 
2. Compute the scalar τk > 0 that minimizes mk τk d sk subject to the trust-region bound, that is,
  s 


τk ¼ arg min mk τd k subject to
τdsk
 Δk (8.6)
τ>0
3. Set dCk ¼ τk dsk ♦

Observe that in closed form the solution of (8.5) is simply obtained as

Δk
dsk ¼  g: ð8:7Þ
kgk k k

It is more elaborate to obtain τk from (8.6), and for that, according to the sign of gTk Bk gk , the
 
following two cases should be considered. When gTk Bk gk  0, the function mk τd sk monotonically
decreases with τ whenever gk 6¼ 0. Therefore, in this case, τk is the largest value that satisfies the trust-
 
region bound, that is, τk ¼ 1. For the case gTk Bk gk > 0, mk τdsk is a convex quadratic function in τ. In
 
this case, τk is either the unconstrained minimizer of this quadratic, that is, kgk k3 = Δk gTk Bk gk , or the
boundary value 1, whichever comes first. In conclusion, the solutions of (8.5) and (8.6) are as follows:
Δk
d Ck ¼ τk g, ð8:8Þ
k gk k k

where
8
< 1, if gTk Bk gk  0,
n  o
τk ¼ 3  ð8:9Þ
: min kgk k = Δk gTk Bk gk , 1 , otherwise:

Observe that the Cauchy point is not difficult to compute and is of a crucial importance in the
economy of the trust-region algorithm. Besides, the Cauchy point does not strongly depend on the
matrix Bk, which is used only in the computation of the step length. Obviously, a rapid convergence of
338 8 The Trust-Region Method

the algorithm is expected only if Bk is used in determining the direction and the length of the step and
if Bk contains useful curvature information about the minimizing function f. Some variants of the
trust-region algorithms compute the Cauchy point as above and then try to improve it. The improve-
ment strategy is designed in such a way so that the full step d Bk ¼ B1
k gk is chosen whenever Bk is

B




positive definite and dk  Δk : When Bk is the exact Hessian ∇ f(xk) or a quasi-Newton approxi-
2

mation of it, then this improvement strategy can be expected to yield superlinear convergence. Now,
let us describe the methods for finding the approximate solutions to the subproblem (8.3).

The Dogleg Method

This method can be used when Bk is positive definite. In this case, the unconstrained minimizer of mk
is exactly dBk ¼ B1
k gk : When this point is feasible for (8.3), it follows that it is a solution, so we

B




have dk ðΔk Þ ¼ dk when dk  Δk : When Δk is small relative to dBk , the restriction kdkk  Δk
B

ensures that the quadratic term in mk has little effect on the solution of (8.3). For such a small Δk, an
approximation to dk(Δk) can be obtained by omitting the quadratic term in (8.3) and by writing
gk
dk ðΔk Þ  Δk , ð8:10Þ
kgk k

For the intermediate values of Δk, the solution dk ðΔk Þ follows a curved trajectory. The dogleg
method finds an approximate solution by replacing the curved trajectory for dk ðΔk Þ with a path
consisting of two line segments. The first-line segment runs from the origin to the minimizer of mk
along the steepest descent direction, which is

gTk gk
k ¼
dU g, ð8:11Þ
gTk Bk gk k

e
while the second-line segment runs from dU k to d k : This trajectory is denoted by d k ðτÞ for τ 2 [0, 2],
B

where
(
e τd U
k, 0  τ  1,
d k ðτ Þ ¼  B  ð8:12Þ
dk þ ðτ  1Þ d k  dk , 1  τ  2:
U U

The dogleg method chooses dk to minimize the model mk along this path subject to the trust-region
bound. The following proposition shows that the minimum along the dogleg path can be easily found.

Proposition 8.1 Let Bk be positive definite. Then,

(i)
dek ðτÞ
is an increasing function of τ,


(ii) mk dek ðτÞ is a decreasing function of τ.

Proof We restrict the analysis to the case τ 2 [1, 2]. For τ 2 [0, 1], both (i) and (ii) are easy to show.
For (i), let us define the function h(α) by
8.2 Algorithms Based on the Cauchy Point 339

2
1


hðαÞ ¼
dek ð1 þ αÞ

2
1
 
2
¼
d U
k þ α dBk  d U
k


2
1

   



2 þ α d U T dB  d U þ 1 α2
dB  d U
2 :
¼
d U
k k k k k k
2 2
To prove (i), we must show that h0(α)  0 for α 2 (0, 1). However,
 T  U 


h0 ð α Þ ¼  d U k dk  dBk þ α
dU k  dk
B
2

 T  U 
  dU k dk  dBk
 
gTk gk T gTk gk 1
¼ T g  T g þ B k gk
g k Bk g k k gk Bk g k k
 T 2 !
gTk B1
k gk gk gk
¼ gk gk T
T
1 T    0,
gk Bk g k gk Bk gk gTk B1
k gk

where the final inequality follows from the Cauchy-Schwarz inequality.


For proving (ii), let us define hbðαÞ ¼ mk dek ð1 þ αÞ and show that hbðαÞ  0 for α 2 (0, 1).
Substituting (8.12) in (8.3) and by differentiation, we get
 T    B   B
U T

h0 ðαÞ ¼ dBk  d U
k k þ α dk  dk
g k þ Bk d U Bk d k  d U
k
 T   B 
 dBk  d U
k gk þ Bk d U
k þ Bk d k  d k
U

 T  
¼ dBk  d U
k gk þ Bk dBk ¼ 0,

proving the result. ♦

Therefore, from this proposition, if
dBk
 Δk , it follows that the path dek ðτÞ intersects the trust-
region boundary kdkk ¼ Δk at exactly one point and nowhere else. Since mk is decreasing along the



path, the chosen value of dk will be at d Bk if
dBk
 Δk , otherwise at the point of intersection of the
dogleg trajectory and the trust-region boundary. In the latter case, the appropriate value of τ is
computed by solving the following scalar quadratic equation:

U  


d þ ðτ  1Þ dB  dU
2 ¼ Δ2 :
k k k k

Let us now discuss the case in which the exact Hessian ∇2f(xk) is available to be used in the model
 1
problem (8.3). When ∇2f(xk) is positive definite, simply set Bk ¼ ∇2f(xk), that is, dBk ¼ ∇2 f ðxk Þ gk,
and apply the above procedure to find the Newton-dogleg step. Otherwise, dBk can be computed by
choosing Bk to be one of the positive definite modified Hessian, as described in Sect. 4.5, and proceed
to find the dogleg step as above. d Bk will be set to the usual Newton step, near a solution that satisfies
the second-order sufficient conditions (see Theorem 11.13), thus allowing a rapid local convergence
of the Newton method. However, to use the modified Hessian in the Newton-dogleg method is not
benefic. A modified factorization perturbs the diagonal elements of ∇2f(xk) in an arbitrary way, and so
340 8 The Trust-Region Method

the benefits of the trust-region method may not be achieved. Anyway, the Newton-dogleg method
is most appropriate when the minimizing function is convex (i.e., ∇2f(xk) is always positive
semidefinite).

The Two-Dimensional Subspace Minimization

When Bk is positive definite, the dogleg method can be made more sophisticated by widening the
search for dk to the entire two-dimensional subspace spanned by dU B
k and d k (equivalently gk and
1
Bk gk ). The subproblem (8.3) is replaced by
1
min mk ðd Þ ¼ f ðxk Þ þ gTk d þ dT Bk d,
d2ℝn 2
subject to ð8:13Þ
kd k  Δ k ,
 
d 2 span gk , B1k gk :

The Cauchy point dCk is feasible for (8.13), so the optimal solution of this subproblem yields at
least as much reduction in the model mk as the Cauchy point, thus ensuring the global convergence of
the algorithm. Observe that the two-dimensional subspace minimization strategy is obviously an
 
extension of the dogleg method, since the entire dogleg path lies in span gk , B1k gk : The case of
indefinite Bk was considered by Byrd, Schnabel, and Schultz (1988) and Schultz, Schnabel, and Byrd
(1985). When Bk has negative eigenvalues, the two-dimensional subspace in (8.13) is changed to
h i
span gk , ðBk þ αI Þ1 gk for some α 2 ðλ1 , 2λ1 , ð8:14Þ

where λ1 is the most negative eigenvalue of Bk.Of course, this choice of α ensures that Bk þ αI is
positive definite. When k(Bk þ αI)1gkk  Δk, then the subspace search (8.13) and (8.14) is
discarded; instead, the step is defined as

dk ¼ ðBk þ αI Þ1 gk þ vk , ð8:15Þ

where vk is a vector satisfying vTk ðBk þ αI Þ1 gk  0: This condition ensures that
1
kdkk  k(Bk þ αI) gkk. When Bk has zero eigenvalues but no negative eigenvalues, the step is
defined to be the Cauchy point, i.e., dk ¼ dCk : Observe that the computational effort with this method
lies in a single factorization of Bk or of Bk þ αI.

8.3 The Trust-Region Newton-CG Method

In the following, let us discuss the finding of an approximate solution to the trust-region subproblem
(8.3) that produces improvements of the Cauchy point. This strategy is based on a modified conjugate
gradient algorithm and was designed by Steihaug (1983). The following algorithm, Algorithm 8.3,
generates the step dk which is used in Algorithm 8.1 for some choice of tolerance εk at each iteration.
The Steihaug algorithm finds an approximate solution to the subproblem (8.3), where Bk ¼ ∇2f(xk).
Denote the search directions of the modified conjugate gradient iteration by dj and the sequence of
iterates that it generates by zj.
8.4 The Global Convergence 341

Algorithm 8.3 Newton conjugate gradient (Steihaug)

1. Initialization. Set: z0 ¼ 0, u0 ¼ ∇ f(xk), d0 ¼  u0 ¼  ∇ f(xk). Set the the tolerance εk > 0

2. If ku0k  εk, then return with dk ¼ z0 ¼ 0, otherwise go to step 3
3. For j ¼ 0, 1, . . .
If dTj Bk dj  0, then find τ such that dk ¼ zj þ τdj minimizes mk(dk) in (8.3) and
satisfies kdkk ¼ Δk, return dk; otherwise, set αj ¼ uTj uj =d Tj Bk d j ,
zjþ1 ¼ zj þ αjdj.
End if
If kzjþ1k  Δk, then find τ  0 such that dk ¼ zj þ τdj satisfies kdkk ¼ Δk, return dk;
Otherwise, set ujþ1 ¼ uj þ αjBkdj
End if
If kujþ1k < εk, then return dk ¼ zjþ1, set βjþ1 ¼ uTjþ1 ujþ1 =uTj uk , djþ1 ¼  ujþ1 þ βjþ1dj
End if
End for ♦

Some remarks are as follows. The first “if” statement inside the loop stops the algorithm if its
current search direction dj is a direction of nonpositive curvature along Bk, while the second “if”
statement inside the loop determines the termination if zjþ1 violates the trust-region bound. In both
cases, the algorithm returns the step dk obtained by intersecting the current search direction with the
trust-region boundary. The choice of the tolerance εk at each iteration of Algorithm 8.3 is important in
keeping the overall cost of the trust-region Newton-CG method at a low level. Near a well-behaved
solution x, the trust-region bound becomes inactive, and the method reduces to the inexact (truncated)
Newton method analyzed in Chap. 7 (see Theorems 7.4 and 7.5). Therefore, the rapid convergence can
 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
be obtained by choosing εk ¼ min 0:5, k∇f ðxk Þk k∇f ðxk Þk (see Algorithm 7.1).
The initialization of z0 to zero in Algorithm 8.3 is a crucial feature of the algorithm. Provided
 
k∇f(xk)k2  εk, Algorithm 8.3 terminates at a point dk for which mk ðdk Þ  mk dCk , that is, when the
reduction in the model equals or exceeds the reduction of the Cauchy point. To show it, consider
several cases. First, if dT0 Bk d0 ¼ ð∇f ðxk ÞÞT Bk ∇f ðxk Þ  0, then the condition in the first “if”
statement is satisfied, and the algorithm returns the Cauchy point dk ¼  Δk(∇f(xk))/k∇f(xk)k.
Otherwise, Algorithm 8.3 defines z1 as follows:

uT0 u0 ð∇f ðxk ÞÞT ∇f ðxk Þ

z1 ¼ α 0 d 0 ¼ T d0 ¼  ∇f ðxk Þ:
d 0 Bk d 0 ð∇f ðxk ÞÞT Bk ∇f ðxk Þ

If kz1k < Δk, then z1 is exactly the Cauchy point. Subsequent steps of Algorithm 8.3 ensure that
the final dk satisfies mk(dk)  mk(z1). On the other hand, when kz1k  Δk, the second “if” statement
is activated, and Algorithm 8.3 terminates at the Cauchy point. Another crucial property of Algorithm
8.3 is that the sequence of vectors {zj} generated by the algorithm satisfies





0 ¼ kz0 k2 < ⋯ <
zj
2 <
zjþ1
2 < ⋯ < kdk k2  Δk :

8.4 The Global Convergence

We start the global convergence analysis in this section by giving an estimate of the decrease in the mk
model achieved by the Cauchy point. Then, by using this estimate, we prove that the sequence of
gradients {gk} generated by Algorithm 8.1 has an accumulation point at zero, that is, this sequence
342 8 The Trust-Region Method

converges to zero when η is strictly positive. The first result is that the dogleg, the two-dimensional
subspace minimization, and the Steihaug algorithm produce approximate solutions dk to the subprob-
lem (8.3), solutions that satisfy the following estimate of the decrease in the model function
 
kgk k
mk ð0Þ  mk ðdk Þ  c1 kgk k min Δk , , ð8:16Þ
k Bk k

where c1 2 (0, 1] is a constant. In the next proposition, we show that the Cauchy point satisfies (8.16)
with c1 ¼ 1/2.

Proposition 8.2 The Cauchy point dCk satisfies (8.16) with c1 ¼ 1/2, that is,
 
 C 1 kgk k
mk ð0Þ  mk d k  kgk k min Δk , : ð8:17Þ
2 k Bk k

Proof Consider the case gTk Bk gk  0: We have

 
mk dCk  mk ð0Þ ¼ mðΔk gk =kgk kÞ  f ðxk Þ
Δk 1 Δ2k T
¼ k gk k2 þ g Bg
kgk k 2 kgk k2 k k k
 Δk kgk k
 
kgk k
 kgk k min Δk , ,
k Bk k

proving that (8.17) holds.

For the next case, consider gTk Bk gk > 0 and

kgk k 3
 1: ð8:18Þ
Δk gTk Bk gk
 
From (8.9), it follows that τk ¼ kgk k3 = Δk gTk Bk gk . Now, from (8.8), it follows that

  kg k4 1  kg k4
mk d Ck  mk ð0Þ ¼  T k þ gTk Bk gk  k 2
gk B k gk 2 g T Bk g k k
4
1 kgk k
¼
2 gTk Bk gk
4
1 kgk k

2 k Bk k kg k k 2
2
1 kgk k
¼
2 kB k k
 
1 kgk k
  kgk k min Δk , ,
2 k Bk k

showing that (8.17) holds.

8.4 The Global Convergence 343

The last case is the one in which (8.18) does not hold, that is,

kgk k3
gTk Bk gk < : ð8:19Þ
Δk

In this case, from (8.9), τk ¼ 1. Therefore, from (8.19), it follows that

  Δ 1 Δ2k T
mk dCk  mk ð0Þ ¼  k kgk k2 þ g Bg
kgk k 2 kgk k2 k k k
1 Δ2k kgk k
3
¼ Δk kgk k þ
2 kgk k2 Δ k
1
¼  Δ k kgk k
2
 
1 kgk k
  kgk k min Δk , ,
2 k Bk k

showing that (8.17) holds. ♦

To satisfy (8.16), the approximate solution dk has only to achieve a reduction that is at least some
fixed fraction c2 of the reduction achieved by the Cauchy point. This is shown in the following
theorem.

Theorem 8.2 Let dk be any vector such that kdkk  Δk and

  
mk ð0Þ  mk ðdk Þ  c2 mk ð0Þ  mk dCk :

Then, dk satisfies (8.16) with c1 ¼ c2/2. If dk is the exact solution dk of (8.3), then it satisfies (8.16)
with c1 ¼ 1/2.

Proof Since kdkk  Δk, from Proposition 8.2, it follows that

 
   1 kgk k
mk ð0Þ  mk ðd k Þ  c2 mk ð0Þ  mk dCk  c2 kgk k min Δk , ,
2 kB k k

proving the result. ♦

Observe that both the dogleg and the two-dimensional subspace minimization algorithms satisfy
 
(8.16) with c1 ¼ 1/2, because they all produce approximate solutions dk for which mk ðdk Þ  mk dCk
(Nocedal & Wright, 2006).
The convergence to the stationary points depends on whether the parameter η in Algorithm 8.1 is
set to zero or to some small positive values. When η ¼ 0, then we can prove that the sequence of
gradients {gk} has a limit point at zero. When η > 0, then a stronger result follows, that is, gk ! 0.
Assume that the approximate Hessians Bk are uniformly bounded in norm and the minimizing
function f is bounded below on the level set
S ¼ fx : f ðxÞ  f ðx0 Þg: ð8:20Þ

Define an open neighborhood of this set by

344 8 The Trust-Region Method

SðR0 Þ ¼ fx : kx  yk < R0 for some y 2 Sg, ð8:21Þ

where R0 is a positive constant.

In order to have a more general result, we allow the length of the approximate solution dk of (8.3)
to exceed the trust-region bound, provided that it stays within some fixed multiple of the bound, that
is,
kdk k  γΔk , for some constant γ  1: ð8:22Þ

The following two theorems, proved by Nocedal and Wright (2006), show the global convergence
of Algorithm 8.1, both for the case η ¼ 0 and for η > 0.

Theorem 8.3 Let η ¼ 0 in Algorithm 8.1. Suppose that kBkk  β for some constant β, f is bounded
below on the level set S defined by (8.20) and Lipschitz continuously differentiable in the neighbor-
hood S(R0) for some R0 > 0 and that all the approximate solutions of (8.3) satisfy the inequalities
(8.16) and (8.22) for some positive constants c1 and γ. Then,

lim inf kgk k ¼ 0: ♦

k!1

Theorem 8.4 Let η 2 (0, 1/4) in Algorithm 8.1. Suppose that kBkk  β for some constant β, f is
bounded below on the level set S defined by (8.20) and Lipschitz continuously differentiable in the
neighborhood S(R0) for some R0 > 0 and that all the approximate solutions dk of (8.3) satisfy the
inequalities (8.16) and (8.22) for some positive constants c1 and γ. Then,
lim kgk k ¼ 0: ♦
k!1

8.5 Iterative Solution of the Subproblem

In this section, let us describe a technique for solving the subproblem (8.3). This is based on the
characterization of the exact solutions of (8.3), which is given by the following theorem due to Moré
and Sorensen (1983). The theorem shows that the solution dk of (8.3) satisfies

ðBk þ λI Þdk ¼ gk , ð8:23Þ

for some λ  0.

Theorem 8.5 The vector d k is a global solution of the trust-region problem

1
minn mk ðd Þ ¼ f ðxk Þ þ gTk d þ dT Bk d,
d2ℝ 2
subject to ð8:24Þ
kd k  Δ k ,

if and only if dk is feasible, and there is a scalar λ  0 such that the following conditions are satisfied:

ðBk þ λI Þdk ¼ gk : ð8:25aÞ

8.5 Iterative Solution of the Subproblem 345




λ Δk 
d k
¼ 0 ð8:25bÞ

ðBk þ λI Þ is positive definite: ð8:25cÞ

Before proving the theorem, let us discuss some aspects of the conditions (8.25). The condition
(8.25b) is a complementarity condition that states that at least one of the nonnegative quantities λ and



Δk 
dk
must be zero. Therefore, when the solution lies strictly inside the trust-region, we must
have λ ¼ 0 and, therefore, Bk dk ¼ gk with Bk positive semidefinite, from (8.25a) and (8.25c),
respectively.
Theorem 8.5 suggests an algorithm for finding the solution dk of (8.24). Either λ ¼ 0 satisfies
(8.25a) and (8.25c) with kdkk  Δk, or else define

dk ðλÞ ¼ ðBk þ λI Þ1 gk

for λ sufficiently large such that (Bk þ λI) is positive definite and seeks a value λ > 0 such that
kdk ðλÞk ¼ Δk : ð8:26Þ

Of course, this problem is a one-dimensional root finding problem in variable λ.

To find a value of λ with all the above properties, consider the eigendecomposition of Bk and use it
to study the properties of kdk(λ)k. Since Bk is symmetric, there is an orthogonal matrix Q and a
diagonal matrix Λ such that Bk ¼ Qk Λk QTk , where Λk ¼ diag {λ1, . . ., λn} and λ1  λ2  . . .  λn are

the eigenvalues of Bk. (It should be written Λk ¼ diag λk1 , . . . , λkn , but we have simplified the
notation.) Obviously, Bk þ λI ¼ Qk ðΛk þ λI ÞQTk , and for λ 6¼ λj, it follows that
Xn qTj gk
d k ðλÞ ¼ Qk ðΛk þ λI Þ1 QTk gk ¼  q, ð8:27Þ
λ þλ j
j¼1 j

where qj is the j-th column of Qk.

Therefore, by the orthonormality of q1, . . ., qn, it follows that

2
Xn qTj gk
kd k ð λ Þ k2 ¼  2 : ð8:28Þ
j¼1 λj þ λ

Observe that if λ >  λ1, it follows that λj þ λ > 0 for all j ¼ 1, . . ., n. Therefore, kdk(λ)k is a
continuous, nonincreasing function of λ in the interval (λ1, 1). In fact, we have
lim kdk ðλÞk ¼ 0: ð8:29Þ
λ!1

Moreover, when qTj gk 6¼ 0, then

lim kd k ðλÞk ¼ 1: ð8:30Þ

λ!λj

Therefore, when qT1 gk 6¼ 0, then there is a unique value λ 2 (λ1, 1) such that kdk(λ)k ¼ Δ. Of
course, there might be other smaller values of λ for which kdk(λ)k ¼ Δ, but they will fail to satisfy
(8.25c).
346 8 The Trust-Region Method

Now, we are in the position to describe a procedure for computing λ 2 (λ1, 1) for which
kdk(λ)k ¼ Δ. Actually, there are two procedures: one for the case in which qT1 gk 6¼ 0 and the other
one for the case in which qT1 gk ¼ 0, as follows:



Case qT1 gk 6¼ 0: When Bk is positive definite and
B1

k gk  Δk , then the value λ ¼ 0 satisfies

(8.25), so the procedure can be terminated with λ ¼ 0. Otherwise, by using the Newton method, we
can find the value λ >  λ1 that solves
φ1 ðλÞ ¼ kd k ðλÞk  Δk ¼ 0: ð8:31Þ

Nocedal and Wright (2006) show that the disadvantage of this approach can be seen by consider-
ing the form of kdk(λ)k when λ is greater than, but close to λ1. For such a λ, the function φ1 can be
approximated by a rational function as
C1
φ1 ð λ Þ  þ C2 ,
λ þ λ1

where C1 > 0 and C2 are constants. Obviously, this approximation is highly nonlinear and the Newton
method for the root-finding will be unreliable or slow. If the problem (8.31) is reformulated to be
almost linear near the optimal λ, the results obtained are better. By defining
1 1
φ2 ðλÞ ¼  ,
Δk kd k ðλÞk

from (8.28), it can be shown that for λ slightly greater than λ1, we have
1 λ þ λ1
φ 2 ðλÞ   ,
Δk C3

for some C3 > 0. Therefore, φ2 is almost linear near λ1 and the Newton method for root-finding will
perform well, provided that it maintains λ >  λ1. For solving the equation φ2(λ) ¼ 0, the Newton
method generates a sequence of iterates λ(l ) by setting
 
ðlþ1Þ ðlÞ φ2 λðlÞ
λ ¼ λ  0  ðlÞ  , l ¼ 0, 1, . . . ð8:32Þ
φ2 λ

The following algorithm is a practical implementation of (8.32).

Algorithm 8.4 Trust-region subproblem

1. Initialization. Consider λ(0) and Δk > 0

2. For l ¼ 0, 1, . . .
Factor: Bk þ λðlÞ I ¼ RTk Rk
Solve: RTk Rk dl ¼ gk , RTk ql ¼ dl
Set:

2

kd l k kd l kΔk
λðlþ1Þ ¼ λðlÞ þ kql k Δk
End for ♦

Algorithm 8.4 can be made practical by adding some safeguarding, for instance, when λ(l ) <  λ1,
the Cholesky factorization Bk þ λðlÞ I ¼ RTk Rk will not exist. Anyway, an improved version of
8.5 Iterative Solution of the Subproblem 347

Algorithm 8.4 always converges to a solution of (8.26). The most difficult operation in Algorithm 8.4
is the Cholesky factorization of Bk þ λ(l )I. Of course, it is not necessary to get an accurate solution for
(8.26). Algorithm 8.4 is stopped after two or three iterations when an approximate solution is
computed.
Case qT1 gk ¼ 0: This case is a little more difficult because the limit (8.30) does not hold for λj ¼ λ1,
and therefore, there might not be a value λ 2 (λ1, 1) such that kdk(λ)k ¼ Δk. Of course, in this case,
the root-finding techniques described above will not work, since there is no solution for λ in the open
interval (λ1, 1). However, Theorem 8.4 assures us that the right value of λ lies in the interval
[λ1, 1), so there is only one possibility: λ ¼  λ1.
Observe that (Bk  λ1I ) is singular. Therefore, there is a vector z such that kzk ¼ 1 and
(Bk  λ1I)z ¼ 0. In fact, z is an eigenvector of Bk corresponding to the eigenvalue λ1, so, by the
orthogonality of Q, we have qTj z ¼ 0 for λj 6¼ λ1. From this property, it follows that, if we set

X qTj gk
d k ðλÞ ¼ q þ τz ð8:33Þ
λj þ λ j
j:λj 6¼λ1

for any scalar τ, we have

2
X qTj gk
kdk ðλÞk2 ¼  2 þ τ ,
2

j:λj 6¼λ1 λj þ λ

so it is always possible to choose τ to ensure that kdk(λ)k ¼ Δk. Therefore, the condition (8.25) holds
for this choice of dk and for λ ¼  λ1.
The following proposition deals with the unconstrained minimization of quadratics with the
Hessian positive semidefinite and constitutes a support for proving Theorem 8.5.

Proposition 8.3 Let m be the quadratic function

1
mðd Þ ¼ gT d þ dT Bd, ð8:34Þ
2
where B is any symmetric matrix. Then, the following statements are true:

(i) m attains a minimum if and only if B is positive semidefinite and g is in the range of B. If B is
positive semidefinite, then every d satisfying Bd ¼  g is a global minimizer of m.
(ii) m has a unique minimizer if and only if B is positive definite.

Proof
(i) The “if” part. Since g is in the range of B, there is a d such that Bd ¼  g. For all w 2 ℝn, we can
write
1
mðd þ wÞ ¼ gT ðd þ wÞ þ ðd þ wÞT Bðd þ wÞ
2


1 T 1
¼ g d þ d Bd þ gT w þ ðBd ÞT w þ wT Bw
T
2 2 ð8:35Þ
1 T
¼ mðdÞ þ w Bw
2
 mðdÞ,

since B is positive semidefinite. Hence, d is a minimizer of m.

348 8 The Trust-Region Method

For the “only if” part, let d be a minimizer of m. Since ∇m(d ) ¼ Bd þ g ¼ 0, it follows that g is in
the range of B. Also, we have ∇2m(d ) ¼ B positive semidefinite, giving the result.

(ii) The “if” part. The same argument as in (i) suffices with the additional argument that wTBw > 0
whenever w 6¼ 0. For the “only if” part, deduce as above that B is positive semidefinite. If B is not
positive definite, then there is a vector w 6¼ 0 such that Bw ¼ 0. Therefore, from (8.35), it follows
that m(d þ w) ¼ m(d), so the minimizer is not unique, giving a contradiction. ♦

Proof of Theorem 8.5

Assume that there is λ  0 such that the conditions (8.25) are satisfied. Proposition 8.3 implies that d k
is a global minimum of the quadratic
1 λ
bk ðdÞ ¼ gTk d þ dT ðB þ λI Þd ¼ mk ðdÞ þ dT d:
m ð8:36Þ
2 2
 
bk dk , it follows that
bk ðd Þ  m
Since m
  λ
  T  
mk ðdÞ  mk dk þ dk dk  dT d : ð8:37Þ
2



 T 
Since λ Δk 
d k
¼ 0 and, therefore, λ Δ2k  d k d k ¼ 0, we have

  λ 
mk ðdÞ  mk dk þ Δ2k  d T d :
2
 
Therefore, from λ  0, it follows that mk ðd Þ  mk dk for all d with kdk  Δk. Therefore, d k is a
global minimizer of (8.24).
Conversely, we assume that d k is a global solution of (8.24) and show that there is a λ  0 that



satisfies (8.25). In the case
d k
< Δk , d k is an unconstrained minimizer of mk, and so
   
∇mk d k ¼ Bk dk þ gk , ∇2 mk dk ¼ Bk is positive semidefinite, and so the properties (8.25) hold
for λ ¼ 0.



For the remainder of the proof, assume that
dk
¼ Δk : Then, (8.25b) is immediately satisfied, and
therefore, d k also solves the constrained problem
min mk ðdÞ subject to kdk ¼ Δk :

By applying the optimality conditions for this problem, we find that there is a λ such that the
Lagrangian function defined by
λ T 
Lðd, λÞ ¼ mk ðdÞ þ d d  Δ2k
2
 
has a stationary point at dk : By setting ∇d L d k , λ to zero, it follows that

Bk dk þ gk þ λdk ¼ 0 ) ðBk þ λI Þd k ¼ gk , ð8:38Þ

   T
so that (8.25a) holds. Since mk ðdÞ  mk dk for any d with dT d ¼ d k dk ¼ Δ2k , for such vectors d,
we have
8.5 Iterative Solution of the Subproblem 349

  λ
  T  
mk ðdÞ  mk dk þ dk dk  dT d :
2
Now, substituting the expression of gk from (8.38) into the above expression, we get
1 T  
d  dk ðBk þ λI Þ d  dk  0: ð8:39Þ
2
Since the set of directions
(
d  dk
w : w ¼ 



d  d
, for some d with kd k ¼ Δk g
k

is dense on the unit sphere, (8.39) suffices to prove (8.25c).

It remains to show that λ  0. As in Nocedal and Wright (2006), since (8.25a) and (8.25c) are
satisfied by d k , from Proposition 8.3(i), it follows that d k minimizes m
bk , so (8.37) holds. Suppose
that there are only negative values of λ that satisfy (8.25a) and (8.25c). Then, from (8.37), we have
 


mk ðdÞ  mk d k whenever kdk 
dk
¼ Δk : Since we already know that d k minimizes mk for
kdk  Δk, it follows that d k is in fact the global unconstrained minimizer of mk. From Proposition 8.3
(i), it follows that Bkd ¼  gk and Bk is positive semidefinite. Therefore, conditions (8.25a) and
(8.25c) are satisfied by λ ¼ 0, which contradicts our assumption that only negative values of λ can
satisfy these conditions. ♦

The key to attaining the fast rate of convergence is that the solution of the trust-region subproblem
is well inside the trust-region and becomes closer and closer to the true Newton step. The steps that
satisfy this property are said to be asymptotically similar to the Newton steps. The following result
applies to any algorithm of the form of Algorithm 8.1 that generates steps that are asymptotically
similar to the Newton steps whenever the Newton steps satisfy the trust-region bound.

Theorem 8.6 Let f be Lipschitz continuously differentiable in a neighborhood of a point x at which

the second-order sufficient conditions (Theorem 11.13) are satisfied. Suppose that the sequence {xk}
converges to x and that for all k sufficiently large the trust-region algorithm based on (8.3) with
Bk ¼ ∇2f(xk) chooses the steps dk that satisfy the Cauchy point based on the reduction criterion (8.16)



and are asymptotically similar to the Newton steps d Nk whenever
dNk
 Δk =2, that is,






d k  d N
¼ o
d N
: ð8:40Þ
k k

Then, the trust-region bound Δk becomes inactive for all k sufficiently large, and the sequence {xk}
converges superlinearly to x.

Proof Firstly, we seek a lower bound on the predicted reduction mk(0)  mk(dk) for all the



sufficiently large k. Assume that k is sufficiently large, so that o
dNk
in (8.40) is smaller than

N

N

N

N






d
: When
d
 Δk =2, it follows that kd k k 
d
þ o
d
 2
dN
, while if
dN
>
k k

k k k k
Δk =2, it follows that kdk k  Δk < 2
dNk
: Therefore, in both cases,





kd k k  2
dNk
 2
∇2 f ðxk Þ1
kgk k,



and so kgk k  12 kdk k=
∇2 f ðxk Þ1
:
350 8 The Trust-Region Method

From the reduction (8.16), it follows that

!
kgk k
mk ð0Þ  mk ðd k Þ  c1 kgk k min Δk ,
2


∇ f ðxk Þ

!
kd k k kd k k
 c1
2
min kd k k,
2



2
∇ f ðxk Þ1
2
∇ f ðxk Þ

∇2 f ðxk Þ1

kd k k2
¼ c1



2 :
4
∇2 f ðxk Þ

∇2 f ðxk Þ1

Since xk ! x, by the continuity of ∇2f(x) and by the positive definiteness of ∇2f(x), it follows that
the following bounds hold for all ksufficiently large
c1 c1

2

2
2 
2

2
2 ≜ c 3 ,
4
∇ f ðxk Þ

∇ f ðxk Þ

1
8
∇ f ðx Þ

∇ f ðx Þ1

where c3 > 0 is a constant. Therefore,

mk ð0Þ  mk ðdk Þ  c3 kd k k2 ,

for all k sufficiently large. By the Lipschitz continuity of ∇2f(x) near x and by using Taylor’s theorem,
it follows that

jðf ðxk Þ  f ðxk þ d k ÞÞ  ðmk ð0Þ  mk ðdk ÞÞj

Z1
1 T 2 1

¼ dk ∇ f ðxk Þdk  dk ∇ f ðxk þ td k Þdk dt
T 2
2 2
0
L
 kd k k3 ,
4
where L > 0 is the Lipschitz constant associated to ∇2f(.). Therefore, from (8.4), it follows that

kd k k3 ðL=4Þ L L
j r k  1j  ¼ kd k  Δ: ð8:41Þ
c 3 kd k k2 4c3 k 4c3 k

As we know, the trust-region can be reduced only if rk < 1/4, so, from (8.41), it follows that the



sequence {Δk} is bounded away from zero. Since xk ! x,
d Nk
! 0, and therefore, from (8.40),
kdkk ! 0. Therefore, for k sufficiently large, the trust-region bound is inactive, and eventually, the



bound
dNk
 Δk =2 is always satisfied.
To prove the superlinear convergence, from (8.40), we have





kxk þ dk  x k 
xk þ dNk  x
þ
d Nk  d k


 


¼ o kxk  x k2 þ o
d Nk
¼ oðkxk  x kÞ,

thus proving the superlinear convergence. ♦

Observe that if d k ¼ d Nk for all k sufficiently large, it follows that the sequence {xk} converges to

x . Under the condition of Theorem 8.6, a reasonable implementation of the dogleg and the subspace
8.6 The Scaled Trust-Region 351

Table 8.1 Performances of TRON for solving five applications from the MINPACK-2 collection

n #iter #f #g #h #Hv #cg cpu

A1 40,000 21 8 8 8 72 43 2.95
A2 40,000 21 8 8 8 72 73 3.50
A3 40,000 2074 1117 480 480 4320 1763 319.90
A4 40,000 15 6 6 6 54 35 4.69
A5 40,000 149 74 39 39 351 142 22.71
Total - 2280 1213 541 541 4869 2056 353.75

In this table, n is the number of variables, #iter is the number if iterations for solving the application, #f is the number
of function evaluations, #g is the number of gradient evaluations, #h is the number of the Hessian evaluations, #Hv is the
number of the Hessian-vector evaluations, #cg is the number of the conjugate gradient iterations, and cpu is the CPU
computing time in seconds

minimization with Bk ¼ ∇2f(xk) will eventually use the steps dk ¼ dNk , and therefore, the convergence
is quadratic. In the case of the dogleg and of two-dimensional subspace minimization methods, the
exact step d Nk is a candidate for dk, since d Nk lies inside the trust-region along the dogleg path and
inside the two-dimensional subspace (Nocedal & Wright, 2006).

Numerical Study: TRON for the Unconstrained Optimization

In the following, let us present the performances of TRON (Lin & Moré, 1999) for solving five
applications from the MINPACK-2 collection (see Appendix D). TRON implements a trust-region
method for the solution of large bound-constrained optimization problems. The Cauchy step is
generated by a gradient projection method. The search direction is computed by a preconditioned
conjugate gradient method with an incomplete Cholesky factorization. The projected searches allow
TRON to examine the faces of the feasible set by generating a small number of minor iterates, even
for problems with a large number of variables. As a result, TRON is remarkably efficient for solving
large bound-constrained optimization problems. In our numerical experiments, the large bounds on
variables are fixed, thus obtaining unconstrained optimization applications. Table 8.1 shows the
performances of TRON for solving these applications, each of them with 40,000 variables (n ¼ nx ny,
nx ¼ 200, ny ¼ 200).

8.6 The Scaled Trust-Region

The poor scaling of the problem is when the minimizing function is highly sensitive to small changes
in certain components of the variable x and relative insensitive to changes in other components of the
variable x. Poor scaling of function f means that x lies in a narrow valley, that is, the contours of the
minimizing function near x are eccentric ellipses. Since the trust-region around the current iterate
within the model mk(.) is an adequate representation of the minimizing function f, it is easy to see that
a spherical trust-region may not be appropriate when f is poorly scaled. Therefore, better is to use the
elliptical trust-regions in which the axes are short in the sensitive directions and longer in the less
sensitive directions. Elliptical trust-regions are defined by
kDd k k  Δk ,

where D is a diagonal matrix with positive diagonal elements. Elliptical trust-region yield to the
following scaled trust-region subproblem:
352 8 The Trust-Region Method

1
min mk ðd Þ ≜ f ðxk Þ þ dT gk þ d T Bk d, subject to kDd k k  Δk :
d2ℝn 2

When the minimizing function f is highly sensitive to the value of the i-th component xi of vector x,
then the corresponding diagonal element dii of D is assigned to a large value, while dii is small for
less-sensitive components of x. Obviously, the diagonal elements of the scaling matrix D may be
2
obtained from the second derivatives ∂ f =∂x2i :
For scaled trust-region method, the Cauchy point calculation is changed according to the following
algorithm.

Algorithm 8.5 Generalized Cauchy point

1. Find the vector d sk as solution of the subproblem:

min f ðxk Þ þ gTk d, subject to kDdk  Δk
d2ℝn
 
2. Compute the scalar τk > 0 that minimize mk τd sk subject to satisfying the trust-region bound:
 s


τk ¼ arg min mk τd k subject to
τDd sk
 Δk ♦
τ>0

For this scaled trust-region version, we find that

Δk
2
dsk ¼ 


D1 g
D gk
k

and
8
>
> 1, if gTk D2 Bk D2 gk  0,
< (
1
3 )
τk ¼
D g

>
> min 1,
k
, otherwise:
: Δk gTk D2 Bk D2 gk

The generalized Cauchy point is computed as

dCk ¼ τk dsk :

The scaling matrix D may be changed from iteration to iteration; most of the theoretical results on
trust-region methods will still apply with minor modifications under condition that each diagonal
element dii of D satisfies 0 < dlo  dii < dhi < 1 , where dlo and dhi define a range [dlo,dhi].

Notes and References

The idea of trust-region method was first proposed by Levenberg (1944) and Marquardt (1963) as a
technique for solving nonlinear least-squares problems. The proof of the convergence theorem of the
trust-region algorithm is originally due to Powell (1975). Methods for computing the search
directions within a trust-region method are described by Gay (1981) and Sorensen (1982). Conver-
gence theory is discussed by Moré (1983). Advances in trust-region algorithms were presented by
Yuan (2015). The content of this chapter is taken from Nocedal and Wright (2006), which is one of
the best descriptions of the trust-region methods. The numerical experiments for solving the
applications from MINPACK-2 collection were given by the package TRON (Lin & Moré, 1999),
one of the most respectable software. The developments of algorithms and the corresponding
8.6 The Scaled Trust-Region 353

software for trust-region methods before 1982 were presented by Moré (1983). Dennins and Schnabel
(1989) survey trust-region methods emphasizing important developments in literature. The general
theory of the inexact trust-region methods was presented by Byrd, Schabel, and Schultz (1988). They
introduced the two-dimensional subspace minimization method for trust-region and discussed the
case of indefinite B to ensure stronger convergence results than that given by Theorems 8.3 and 8.4.
An exhaustive presentation of the state of the art in trust-region methods both for unconstrained and
constrained optimization could be found in the monograph of Conn, Gould, and Toint (2000). For the
convergence of trust-region algorithms for unconstrained minimization without derivatives, see
Powell (2011).
In our presentation, the trust-region method was defined in Euclidian norm, but it could be defined
in other norms as kdkk1  Δk or kdkk1  Δk, or scaled as kDdkk1  Δk or kDdkk1  Δk, where D is a
diagonal positive definite matrix. The most advantageous is the 1-norm, in which case the feasible
region is the rectangular box defined by xk þ d  0, d   Δke, and d  Δke, where e ¼ [1, 1, . . ., 1]T.
In this case, the solution of the subproblem is easily computed by using the simple bound-constrained
optimization (see Chap. 12).
The trust-region algorithm is also implemented in filterSQP (Fletcher & Leyffer, 1998) which is a
sequential quadratic programming solver with a filter to promote global convergence and in
KNITRO/INTERIOR-CG (Byrd, Hribar, & Nocedal, 1999) with sequential quadratic programming.
 Direct Methods for Unconstrained
Optimization 9

Consider the problem

min f ðxÞ, ð9:1Þ

x2ℝn

where f : ℝn ! ℝ is bounded from below on ℝn and n is relatively small, let us say n  50. Many
algorithms have been proposed to solve this problem, but in this chapter, we are interested in dealing
with the case in which the derivatives of this function are unavailable, impractical to obtain, or
unreliable. The optimization methods that use only the function values are called derivative-free
optimization or direct search methods or zeroth-order methods.
Excellent reviews and perspectives with an emphasis on highlighting recent developments of the
derivative-free optimization methods both for unconstrained and constrained optimization with
deterministic, stochastic, or structured objectives were given by Kolda, Lewis, and Torczon (2003),
Rios and Shainidis (2013), and Larson, Menickelly, and Wild (2019). A review of the derivative-free
algorithms followed by a systematic comparison of 22 related implementations using a test set of
502 problems was given by Rios and Shainidis (2013). Their conclusion is not definitive. They
emphasize that the ability of all these solvers to obtain good solutions diminishes with the increasing
problem size. Besides, attaining the best solutions even for small problems is a challenge for most
current derivative-free solvers, and there is no single solver whose performance dominates the
performances of all the others. The dimensionality of the problems and the non-smoothness rapidly
increase the complexity of the search and decrease the performances of all the solvers. In conclusion,
there is a large diversity of derivative-free optimization methods with an impressive development.
More details on derivative-free methods can be found in Andrei (2021a).
Addressing the optimization problems that arise from scientific engineering, from artificial
intelligence applications, or from some other areas of activity for which the objective and the
constraints are available as a mixture of a mathematical model and the output of a black-box or a
simulation oracle that does not provide derivative information, this chapter has a special position in
the architecture of the book. From the multitude of derivative-free optimization algorithms, we have
selected only the algorithm of Nelder-Mead (NELMED) (see Nelder & Mead, 1965), Powell
(NEWUOA) (see Powell, 2003, 2004, 2006), and Andrei (DEEPS) (see Andrei, 2021a).

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 355
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_9
356 9 Direct Methods for Unconstrained Optimization

9.1 The NELMED Algorithm

The Nelder-Mead algorithm has numerous interpretations. In the following, one of the simplest but
sufficiently complete will be discussed. This method is based on simplexes. A simplex in ℝn is a n þ 1
element set of n-dimensional vertices with a nondegenerate set of edges. More exactly, a simplex
S ⊂ ℝn is a set of n þ 1 vertices {x1, . . ., xnþ1} ⊂ ℝn connected by the edges

E ¼ ðx2  x1 , x3  x1 , . . . , xnþ1  x1 Þ

which form a basis of ℝn, i.e., span(E) ¼ ℝn.

At iteration k, the Nelder-Mead simplex algorithm applied to the function f from (9.1) maintains an


ordered set of simplex vertices xk1 , . . . , xknþ1 : The ordering of the simplex vertices is




f xk1  f xk2  ⋯  f xknþ1 : ð9:2Þ

In this context, the vertex xk1 with the lowest function value is referred to as the lowest vertex. The
vertex with the highest value is the highest vertex.
At the beginning of each iteration, the centroid point is computed as

1X k
n
x¼ x:
n i¼1 i

At each step, a trial point is computed. Each iteration may have one of the following two
outcomes: (1) one of the computed trial points with a lower function value replaces the highest
vertex or (2) a shrink operation is performed.
The Nelder-Mead simplex algorithm uses four different trial points xr, xe, xoc, and xic as follows:


1: Re flection ðρ > 0Þ : xkr ¼ x þ ρ x  xknþ1 , ð9:3Þ


2: Expansion ðη > max f1, ρgÞ : xke ¼ x þ η xkr  x , ð9:4Þ


3: Outside contraction ð0 < γ < 1Þ : xkoc ¼ x þ γ xkr  x , ð9:5Þ


4: Inside contraction ð0 < γ < 1Þ : xkic ¼ x  γ x  xknþ1 : ð9:6Þ

If none of the trial points yields improvement of the minimizing function, then a shrink operation is
performed. Standard values of the parameters in (9.3, 9.4, 9.5, and 9.6) are ρ ¼ 1, η ¼ 2, γ ¼ 1/2. A
 
shrink operation transforms the simplex vertices xk2 , . . . , xknþ1 according to the following rule:


xeki ¼ xk1 þ σ xki  xk1 , ð9:7Þ

where 0 < σ < 1. (σ ¼ 1/2).

One iteration of the Nelder-Mead algorithm is presented as Algorithm 9.1.
9.2 The NEWUOA Algorithm 357

Algorithm 9.1 Nelder-Mead algorithm (NELMED) (one iteration)

1. Order. Order the simplex vertices according to (9.2)

2. Compute the centroid x
3. Reflect. Compute xkr as in (9.3)




4. If f xk1  f xkr < f xkn , then set xknþ1 ¼ xkr

k



5. Expand. If f xr < f xk1 , then compute xke as in (9.4). If f xke < f xkr , then set xknþ1 ¼ xke , else set
xknþ1 ¼ xkr



6. Contract. If f xkr  f xkn , then

Contract

outside:




If f xkn  f xkr < f xknþ1 , then compute xkoc as in (9.5). If f xkoc  f xkr , then
set xknþ1 ¼ xkoc
else
Go to step 7
End if

Contract

inside:



If f xkr  f xknþ1 , then compute xkic as in (9.6). If f xkic < f xknþ1 , then set
xknþ1 ¼ xkic
else
Go to step 7
End if
7. Shrink. Apply (9.7) to the vertices of the current simplex ♦

The hope is that the successive decreases of the values of the function f at the simplex vertices
eventually lead to the convergence to x. The convergence of the algorithm was established only for
problems with a small number of variables (Lagarias, Reeds, Wright, & Wright, 1998).
Since the algorithm is based only on comparisons among the function values, then a reasonable



stopping criterion is f xknþ1  f xk1   ε, where ε is a positive constant, sufficiently small. Another
stopping criterion is based on the volume of the simplex V(Sk)  ε, where V(Sk)is computed as

jdetðEk Þj
V ð Sk Þ ¼ :
n!
As regards the initial simplex, a practical choice is to compute the vertices as x0iþ1 ¼ x01 þ λi ei ,
i ¼ 1, . . ., n, where x01 is the given starting point x0, λi for i ¼ 1, . . ., n, are given scalars, and ei are the
unit vectors corresponding to the i-th coordinate axis. In this case, the volume of the initial simplex
can be computed as

1 Y
nþ1 
V ð S0 Þ ¼ x0  x0 :
n! i¼2 i 1

Several details on the properties and convergence of the Nelder-Mead algorithm can be found in
Lagarias, Reeds, Wright, and Wright (1998). A convergent variant of the Nelder-Mead algorithm was
given by Price, Coope, and Byatt (2002).

9.2 The NEWUOA Algorithm

NEWUOA is an iterative algorithm which seeks the least value of a function f : ℝn ! ℝ, when f can
be computed for any vector of variables x. A short description of NEWUOA is as follows. At the
358 9 Direct Methods for Unconstrained Optimization

beginning of each iteration, the NEWUOA algorithm uses a quadratic model Q which is embedded
into a trust region procedure for adjusting the variables. The model Q is revised at each iteration.
When Q is revised, the new Q interpolates f at m points, where the recommended value of m is 2n þ 1.
The remaining freedom in the new Q is taken up by minimizing the Frobenius norm of the change to
∇2Q. Only one interpolating point is altered at each iteration. Therefore, the amount of work per
iteration is only of order (m þ n)2.
At the beginning of each iteration, the model Q(x) has to satisfy only m interpolating conditions
Q(xi) ¼ f(xi), i ¼ 1, . . ., m, where the position of different points xi, i ¼ 1, . . ., m, is generated
automatically. The algorithm implements a technique that is suggested by the symmetric Broyden
method for updating ∇2Q when the first derivatives of f are available. Let the old model Qold be known
and the new model Qnew be asked to satisfy some conditions that are compatible and that leave some
freedom in the parameters of Qnew. The technique tries to minimize k∇2Qnew  ∇2QoldkF, where k.kF
is the Frobenius norm. The conditions on the new model Q ¼ Qnew are the interpolating equations
Q(xi) ¼ f(xi), i ¼ 1, . . ., m. Thus, ∇2Qnew is uniquely defined and Qnew itself is also unique.
At the beginning of the current iteration, the variables are updated as xnew ¼ xopt þ d, where xopt
corresponds to f(xopt), the least calculated value of f so far. If the error |f(xnew)  Qold(xnew)| is
relatively small, then the model predicted well the new value of f, even if the errors of the
approximation ∇2Q  ∇2f are substantial. On the other hand, if |f(xnew)  Qold(xnew)| is relatively
large, then by satisfying Qnew(xnew) ¼ f(xnew), the updating technique should improve the accuracy of
the model. The change in the variables d is an approximate solution to the trust region subproblem


min Q xopt þ d subject to kdk  Δ:

The algorithm uses a number of parameters: ρbeg, ρend, ρold, ρnew, and Δ. The initial interpolating
points xi, i ¼ 2, . . ., m, include x0, while the other points have the property kxi  x0k1 ¼ ρbeg. The
parameter ρend, which has to satisfy ρend  ρbeg, should have the magnitude of the required accuracy
in the final values of variables. The parameter ρ is a lower bound on the trust region radius Δ from the
interval [ρend, ρbeg]. The purpose of ρ is to maintain enough distance between the interpolating points.
The value of Δ is revised on most iterations.
On steps, the algorithm NEWUOA is as follows:

Algorithm 9.2 Powell algorithm (NEWUOA)

1. Select the initial interpolating points. Let xopt be an initial point where f is the least
2. Compute d by approximately minimizing Q(xopt þ d ) subject to kdk < Δ
3. If kdk  ρ/2, then go to step 4; otherwise, go to step 10
4. Calculate f(xopt þ d ) and set r ¼ (f(xopt)  f(xopt þ d ))/(Q(xopt)  Q(xopt þ d )). Thus, Δ is revised subject to
Δ  ρ. The parameter move is set to zero or to the index of the interpolating point that will be dropped next
5. If move > 0, then Q is modified so that Q interpolates f at xopt þ d instead of xmove. If f(xopt þ d ) < f(xopt),
then xopt is overwritten by xopt þ d. Go to step 6
6. If r  0.1, then go to step 2; otherwise, go to step 7
7. Let xmove be the current interpolating point that maximizes the distance dist ¼ kxmove  xoptk
8. If dist  2Δ, then xmove is replaced by xopt þ d, set r ¼ 1 and go to step 5. Otherwise, if max{kdk, Δ}  ρ
and r  0, go to step 9
9. If ρ ¼ ρend, then compute f(xopt þ d ) and stop. Otherwise, reduce ρ about a factor of 10 subject to
ρ  ρend. Reduce Δ ¼ max [ρold/2, ρnew] and go to step 2
10. If three recent values of kdk and f  Q are small, then go to step 9; otherwise, reduce Δ by a factor of
10 or to its lower bound ρ. Set r ¼  1. Set xmove to be the current interpolating point that maximizes the
distance dist ¼ kxmove  xoptk and go to step 8 ♦

Details on NEWUOA are given in Powell (2002, 2003, 2004, 2006).

9.3 The DEEPS Algorithm 359

9.3 The DEEPS Algorithm

In the following, we present a new derivative-free random search method at two levels, which is
completely different from the known randomized methods for the unconstrained optimization.
Roughly speaking, our method is close to pure random search, but extended at two levels (Andrei,
2021a)
Suppose that all the variables are in a domain D ⊂ ℝn defined by some bounds on variables. Some
of these bounds are imposed by the physical constraints which define the problem. Others are
artificially introduced, for being used in our algorithm. Anyway, we consider that all the variables
are bounded by some known bounds, the same for all of them. Suppose that the minimum value of the
function f is known as f opt. Assume that the function f is continuous and bounded below on ℝn.
The idea of the algorithm is as follows. Set the current iteration iter ¼ 1 and assign a value to the
maximum number of iterations admitted by the algorithm as maxiter. Suppose that the searching of
the minimum of the function f starts from the initial point x0 2 ℝn. Consider an initial domain
D ⊂ ℝn defined by its bounds (lower and upper) to each of the n variables. These limits may be the
same for all the variables, let us say lobnd and upbnd. In other words, if on components we have

xi ¼ x1i , . . . , xni , then assume that lobnd  xli  upbnd for all l ¼ 1, . . ., n. Of course, the bounds
lobnd and upbnd can be different for different components of the variables xi, but in the following, we
assume that they are the same for each component xli , l ¼ 1, . . . , n: We emphasize that these bounds
lobnd and upbnd may be the bounds defining the optimization problem and imposed by the
engineering constructive specifications of the minimizing problem.
Around the initial point x0, we randomly generate over D a number N of trial points, x1, . . ., xN,
where xj 2 ℝn, for j ¼ 1, . . ., N. If N ¼ n þ 1, then these points define a simplex in Rn, i.e., the convex
hull of its n þ 1 points x1, . . ., xnþ1. Otherwise, if N 6¼ n þ 1, we say that we have a geometric
structure which we call a complex in ℝn. Obviously, N may be larger or smaller than the number
of variables n, or even equal to n. Evaluate the minimizing function f in these points, thus obtaining
f(x1), . . ., f(xN).
Now, for every trial point xj, j ¼ 1, . . ., N, let us define the local domains Dj ⊂ ℝn by specifying
the bounds, let us say lobndc and upbndc. Of course, these bounds may be different for every trial
point, but in our development, we consider them equal for every trial point. Using these bounds
lobndc and upbndc around each trial point xj, we randomly generate M local trail points x1j , . . . xM j ,
where xij 2 ℝn , for i ¼ 1, . . ., M. In other words, around the trial point xj, the local trial points
x1j , . . . xM
j are bounded by the limits lobndc and upbndc. Usually, the local domains Dj are smaller
than the domain D, but this is not compulsory. Now, evaluate the function f in these local trial points,



thus obtaining f x1j , . . . , f xM
j for any j ¼ 1, . . ., N.
With this, determine the local point xkj corresponding to the minimum value from the set
n

o
 n

o
f x1j , . . . , f xM
j , i.e., f xkj ¼ min f x1j , . . . , f xMj : The following decision is now



taken: if f xkj < f xj , then replace the trial point xj with the local trial point xkj and the value f(xj)


with f xkj :
The above procedure is repeated for every trial point xj, j ¼ 1, . . ., N. At the end of this cycle, if the
case, the trial points xj, j ¼ 1, . . ., N, are replaced by the local trial points xkj , j ¼ 1, . . ., N, for which


the function value f xkj might be smaller than f(xj). Let us denote these new trial points by y1, . . ., yN,
360 9 Direct Methods for Unconstrained Optimization

i.e., yj ¼ xkj , j ¼ 1, . . ., N, and the function values in these points by f(y1), . . ., f(yN), i.e.,



f yj ¼ f xkj , j ¼ 1, . . ., N.
Now, determine the point yk corresponding to the minimum values of the minimizing function,
i.e., f(yk) ¼ min {f(y1), . . ., f(yN)}.
For j ¼ 1, . . ., N, j 6¼ k, where k is the index corresponding to the minimum value f(yk), compute the
middle point zj ¼ (yk þ yj)/2 and evaluate the minimizing function in this point, thus obtaining f(zj).
For j ¼ 1, . . ., N, j 6¼ k, if f(zj) < f(yj), then replace the point yj with zj and the value f(yj) with f(zj);
otherwise, do not replace them. Set zk ¼ yk and f(zk) ¼ f(yk). At this step of the algorithm, another set
of new trial points denoted as z1, . . ., zN is obtained, for which the function values f(z1), . . ., f(zN) might
be smaller than the previous values of the minimizing function f.
Now, determine f max ¼ max {f(z1), . . ., f(zN)}, corresponding to the trial point zmax from the set of
points {z1, . . ., zN}. Similarly, determine f min ¼ min {f(z1), . . ., f(zN)}, corresponding to the trial
point zmin from the set of points {z1, . . ., zN}. Compute the middle point zm ¼ (zmax þ zmin)/2 and the
function value in this point f(zm).
In the following, for j ¼ 1, . . ., N, where the index j is different from the indices corresponding to
the minimum and maximum points zmin and zmax, respectively, if f(zm) < f(zj), then compute the
reflected point zr ¼ αzm  zj, and if f(zr) < f(zj), then replace zj by the reflected point zr, i.e., set zj ¼ zr
and f(zj) ¼ f(zr). Notice that α is a positive coefficient known as the reflection coefficient, often
selected as α 2 [1, 2], usually close to 2.
At the end of this cycle, some of the trial points zj, j ¼ 1, . . ., N, have been replaced by their
reflected points. Therefore, we have a new set of trial points {z1, . . ., zN} for which the function
values {f(z1), . . ., f(zN)} might be smaller than the previous values.
Intensive numerical experiments have shown that the values of the minimizing function are
significantly reduced in the first few iterations, after which the algorithm is stalling, i.e., the function
values are very slowly reduced along the iterations. In order to accelerate the algorithm, the following
step is implemented. If the difference between the minimum values of the minimizing function at two
successive iterations is smaller than a positive, small enough threshold δ, then a simple line-search
with a positive step β or with a negative step β is initiated from the current minimum point along the
direction determined by the minimum point and the second minimum point. Thus, if the function
value is reduced, then the minimum point is replaced by the new trial point. Again, at the end of this
step, it is possible for the minimum trial point zmin to be replaced by another point for which the
function value is smaller. Therefore, another set of trial points {z1, . . ., zN} is obtained, for which the
function values {f(z1), . . ., f(zN)} might be smaller than the previous values.
Now, determine f max ¼ max {f(z1), . . ., f(zN)} corresponding to the trial point zmax from the set of
trial points {z1, . . ., zN}. Similarly, determine f min ¼ min {f(z1), . . ., f(zN)} corresponding to the trial
point zmin from the set of trial points {z1, . . ., zN}.
From this set of trial points {z1, . . ., zN}, at the iteration iter, determine the point zk corresponding
to the minimum values of the minimizing function, i.e., f(zk)  fiter ¼ min {f(z1), . . ., f(zN)}.
At the iteration iter, compute the difference: diter ¼ |f(zk)  f opt|  |fiter  f opt|. With these, the
following decisions are implemented:

• If |f max| > B, where B is a positive constant sufficiently large, then reduce the bounds lobnd and
upbnd, as, for example, lobnd/2 and upbnd/2, respectively. Set iter ¼ iter þ 1 and go to generate
around the initial point x0 a new set of trial points by using the reduced bounds lobnd/2 and upbnd/
2. At the end of this decision, the bounds lobnd and upbnd are likely to be reduced.
9.3 The DEEPS Algorithm 361

• If diter 6¼ diter1, then, if diter > ε, set iter ¼ iter þ 1 and diter  1 ¼ diter. If iter > max iter, then go to
the final step, else go to the next step. If diter ¼ diter  1, then reduce the bounds of the local domain,
i.e., set lobndc ¼ lobndc/2 and upbndc ¼ upbndc/2. If diter > ε, then set iter ¼ iter þ 1, and if
iter > max iter, then go to the final step, else go to the next step. At the end of this decision, the
lower and upper bounds of the local domains lobndc and upbndc might be reduced.
• Here we have the next step, where the following decision is taken: if |fiter  fiter1| < t, where t is a
prespecified small threshold, then reduce the complex, i.e., for j ¼ 1, . . ., N, compute
zj ¼ zmin þ (zj  zmin)/2.

At this iteration, set fiter1 ¼ fiter, and for j ¼ 1, . . ., N, set xj ¼ zj and go to the next iteration by
generating M local trail points x1j , . . . xM
j around each new trial points xj, j ¼ 1, . . ., N.
The final step: improve the minimum point obtained as above by randomly selecting a number
Q of points in a domain C defined by the bounds lobndcc and upbndcc, smaller than lobndc and
upbndc, respectively. Evaluate the minimizing function in these Q points, and select the minimum
value from them, which corresponds to the solution of the problem.
More exactly, on steps, the algorithm may be described as follows:

Algorithm 9.3 Andrei algorithm (DEEPS)

1. Initialization. Select: an initial point x0, the number N of a set of trial points and the number M of the set
of local trial points around each trial point. Select ε > 0 small enough. Select the bounds lobndand upbnd
which determine the set of trial points, the bounds lobndc and upbndcwhich determine the set of local
trail points, the bounds lobndcc and upbndcc which determine the final trail points around the minimum,
as well as the threshold t small enough. Select B a positive constant sufficiently large. Select a value for
the coefficient α used in the reflection of points and a value for the coefficient β used in a line-search
along the direction determined by the minimum trial point and by the second minimum trial point. Select
a value for the parameter δ used in initiating the above line-search. Set iter ¼ 1 and maxiter as the
maximum number of iterations admitted by the algorithm
2. Around x0, randomly generate N trial points x1, . . ., xN in a domain D defined by the lower bound
lobndand by the upper bound upbnd on variables
3. Evaluate the minimizing function f in the trial points f(x1), . . ., f(xN)
4. For j ¼ 1, . . ., N, around xj do:
(a) Generate M points x1j , . . . xMj in the domains Dj defined by the bounds lobndc and upbndc,
smaller than
thebounds of
the domain D. Evaluate the function in these points, thus obtaining
the values f x1j , . . . , f xM
j


(b) Determine the point xkj corresponding to the minimum values f xkj ¼
n

o
min f x1j , . . . , f xM
j




(c) If f xkj < f xj , then replace the trial point xj by xkj and the value f(xj) by f xkj
End for
[By these replacements, at the end of this cycle, another set of new trial points denoted as y1, . . ., yN is
obtained, for which the function values are f(y1), . . ., f(yN).]
5. Determine the point yk corresponding to the minimum values of the minimizing function:
f(yk) ¼ min {f(y1), . . ., f(yN)}
6. For j ¼ 1, . . ., N, j 6¼ k, do:
(a) Compute the point zj ¼ (yk þ yj)/2
(b) Evaluate the minimizing function in this point, thus obtaining f(zj)
(c) If f(zj) < f(yj), then replace the point yj by zj and the value f(yj) by f(zj)
End for
362 9 Direct Methods for Unconstrained Optimization

7. Set zk ¼ yk and f(zk) ¼ f(yk)

[At the end of steps 6 and 7, another set of new trial points denoted as z1, . . ., zN is obtained, for which
the function values are f(z1), . . ., f(zN).]
8. Determine f max ¼ max {f(z1), . . ., f(zN)} and the corresponding trial point zmax Determine
f min ¼ min {f(z1), . . ., f(zN)} and the corresponding trial point zmin
9. Compute the middle point zm ¼ (zmax þ zmin)/2 and the function value in this point f(zm)
10. For j ¼ 1, . . ., N, where the index j is different from the indices corresponding to the minimum and
maximum points zmin and zmax respectively, do:
If f(zm) < f(zj), then
Compute the reflected point zr ¼ αzm  zj and f(zr)
If f(zr) < f(zj), then
Set zj ¼ zr and f(zj) ¼ f(zr)
End if
End if
End for
[At the end of this cycle another set of trial points z1, . . ., zN is obtained, for which the function values are
f(z1), . . ., f(zN).]
11. At iteration iter determine:
max
f iter ¼ max ff ðz1 Þ, . . . , f ðzN Þg and the corresponding trial point zmax
f iter ¼ min ff ðz1 Þ, . . . , f ðzN Þg and the corresponding trial point zmin
min
 
12. If f min  f min   δ, then do:
iter iter1
(a) In the current set of trial points determine the second minimum trial point zmins
(b) Compute the point zs ¼ zmin þ β(zmin  zmins)
(c) Compute f s ¼ f(zs)
If f s < f iter
min
, then
Set z ¼ zs, f iter
min min
¼ f s and go to step 13
Else
(d) Compute the point zs ¼ zmin  β(zmin  zmins)
(e) Compute f s ¼ f(zs)
If f s < f iter
min
, then
Set z ¼ zs, f iter
min min
¼ f s and go to step 13
Else
Go to step 13
End if
End if
End if
13. If upbnd < 1 and lobnd >  1, then go to step 15
 max 
14. If f iter  > B, then:
(a) Reduce the bounds as lobnd/2 and upbnd/2
(b) Set iter ¼ iter þ 1. If iter > max iter, then go to step 19, else go to step 2
End if
 min 
15. At iteration iter compute the difference:f iter  f opt iter
 min   
16. If f iter  f opt  6¼ f min  f opt iter1 , then
 min iteropt  iter1
If f iter  f iter > ε, then
   
Set iter ¼ iter þ 1 and f min  f opt 
iter1 iter1
¼ f min  f opt 
iter iter
If iter > max iter, then go to step 19, else go to step 17
Else
Go to step 19
End if
Else
Set lobndc ¼ lobndc/2
 and upbndc ¼ upbndc/2
If f iter
min
 f opt iter > ε, then
Set iter ¼ iter þ 1
If iter > max iter, then go to step 19, else go to step 17
9.3 The DEEPS Algorithm 363

Else
Go to step 19
End if
End if
 min 
17. If f iter min 
 f iter1 < t, then reduce the complex:
For j ¼ 1, . . ., N
compute zj ¼ zmin þ (zj  zmin)/2
End for
End if
18. min
Set f iter1 ¼ f iter
min

For j ¼ 1, . . ., N, set xj ¼ zj and go to step 4

19. Improve the search. Around the minimum point zk obtained so far, randomly select a number Q of points
in the domain C defined by the bounds lobndcc and upbndcc, smaller than lobndc and upbndc,
respectively. Evaluate the minimizing function in these Q points, and from them, select the minimum as
the solution of the problem ♦

To ensure the convergence, the algorithm implements three actions in steps 14, 16, and 17.

1. At step 14, if the maximum absolute value of the minimizing function f computed in the current
trial points is greater than a prespecified value B sufficiently large, let us say B ¼ 105, then the
bounds lobnd and upbnd are halved. Therefore, the algorithm generates a sequence of points in
bounded domains.
2. Based on the differences between the minimum values of the minimizing function and the known
optimum value at two successive iterations, step 16 implements a procedure for halving the bounds
lobndcand upbndc of the local domains, thus shrinking the local search domains.
3. If the difference between the minimum values of the minimization function at two successive
iterations is smaller than a threshold t, then the algorithm reduces the complex in step 17.

The algorithm keeps a balance between the threshold δ for initiating the line-searches in step
12 and the threshold t for reducing the complex in step 17. In general, the values for δ are selected as
being smaller than those of t. Usually, δ ¼ 104 and t ¼ 102, but some other values are acceptable.
Numerical experiments have shown that the algorithm has two phases: the reduction phase and the
stalling one. In the reduction phase, for some problems, the function values are significantly reduced
for the first few iterations. After that, the algorithm gets into the stalling phase, where the function
values are reduced very slowly. Both thresholds δ and t are introduced to accelerate the algorithm, i.e.,
to diminish the stalling. Depending on the value of the threshold δ, a line-search is initiated from the
minimum point along the direction determined by the minimum point and the second minimum point,
corresponding to the current set of trial points. Similarly, depending on the value of the threshold t,
the algorithm reduces the complex.
In the following, let us discuss and present some key elements of the algorithm by illustrating the
evolution of the maximum distance among the trial points, as well as the evolution of the maximum
distance among the trial points and the local trial points along the iterations.
At iteration iter,for any two points xi and xj, from the set of trials points, let us define the distance
 
 
ij ¼ xi  xj 2 , for i, j ¼ 1, . . ., N, i 6¼ j. Of course, this distance depends
between these points as d iter
on the bounds lobnd and upbnd. Now, let us determine a pair of trial points xu and xv, from the set of
trials points, so that

uv ¼ kxu  xv k2 ¼
d iter max diter
ij , ð9:8Þ
i, j¼1, ..., N, j>i
364 9 Direct Methods for Unconstrained Optimization

iter
i.e., d iter
uv is the maximum distance among the trial points. With this, define the sphere S centered in
point cuv ¼ (xu þ xv)/2 of ray r uv ¼ d uv =2: Obviously, the volume of this sphere is
iter iter

π
3
V iter ¼ d iter
uv : ð9:9Þ
6

Proposition 9.1 At iteration iter, all the trial points belong to the sphere Siter.

Proof Suppose by contradiction that a trial point xp 2 = Siter. Then, if kxu  xpk  kxu  xvk, then
kxv  xpk  kxu  xvk, i.e., d uv is not the maximum distance among the trail points at iteration iter.
iter

Similarly, if kxv  xpk  kxu  xvk, then kxu  xpk  kxu  xvk, i.e., diter
uv is not the maximum distance
among the trail points at iteration iter. Therefore, xp has to belong to Siter. ♦
 min 
Proposition 9.2 At iteration iter, if f iter min 
 f iter1 < t, then Viter ¼ Viter  1/8.

Proof Let d iter1 be the maximum distance among the trial points at iteration iter  1 determined by
uv  min 
the trial points xu and xv. Observe that at iteration iter, if f iter min 
 f iter1 < t, then algorithm DEEPS
reduces the complex, i.e., for j ¼ 1, . . ., N, the new trial points are computed as xj ¼ xmin þ (xj  xmin)/
2, where xmin is the trial point corresponding to the minimum value of function f. Therefore, the
distance from any trial point xi to any trial point xj is
  

 
diter
ij ¼ xi  xj 2 ¼ x min þ xi  x min =2  x min  xj  x min =22
1  1 ð9:10Þ
¼ xi  xj 2 ¼ diter1 :
2 2 ij
In particular, for the points xu and xv which define the maximum distance among the trial points at
uv ¼ d uv
iteration iter, it follows that diter iter1
=2: Therefore, from (9.9), Viter ¼ Viter  1/8. ♦
 min 
Hence, at iteration iter, if f iter min 
 f iter1 < t, then the complex is reduced, and the DEEPS
algorithm generates the trial points in a sphere of smaller volume.
Now, let us consider the situation in which at a certain iteration iter  1, around the point x0, a
number N of trial points are generated as

xiter1
j ¼ x0 þ r j ðupbnd  lobnd Þ þ eðlobnd Þ, j ¼ 1, . . . , N,
h iT
where r j ¼ r 1j , . . . , r nj is a vector with the components r kj , k ¼ 1, . . ., n, random numbers in and
e ¼ [1, . . ., 1] 2 ℝn and lobndand upbnd are the bounds of the domain D,(see step 2). As we know, the
distance among these trial points is
   
 iter1 iter1 
d iter1
ij ¼ x i  x j  ¼ r i  r j 2 jupbnd  lobnd j, i, j ¼ 1, . . . , N, i 6¼ j: ð9:11Þ
2

 2 1=2
  Pn
pffiffiffi pffiffiffi
However, r i  r j 2 ¼ k
k¼1 r i  r kj  n: Therefore, diter1
ij  njupbnd  lobnd j,
for any i, j ¼ 1, . . ., N, i 6¼ j.
 max 
Proposition 9.3 If at the iteration iter f iter  > B, where B is a large enough prespecified parame-
ter, then, at this iteration, the distance among the trial points is reduced.
9.3 The DEEPS Algorithm 365

Proof In this case, at step 14 the algorithm DEEPS reduces the bounds lobnd and upbnd as lobnd/2
and upbnd/2, respectively, and go to generate a new set of trial points in step 2 using these
new bounds. Hence, at this iteration, the new trial points are computed as xiter j ¼ x0 þ
2 r j ðupbnd
 lobnd Þ þ 12 eðlobnd Þ, for j ¼ 1, . . ., N, where again rj is a vector with components
1

random numbers in [0, 1]. Therefore, at this iteration, the distance among the new trial points is
 
 iter  1


 1 iter1
ij ¼ xi
diter  xiter
j  ¼ 2 r i  r j 2 jupbnd  lobnd j ¼ 2 d ij , ð9:12Þ
2

for any i, j ¼ 1, . . ., N, i 6¼ j. ♦
 min   min 
If f iter  f opt iter ¼ f iter1  f opt iter1 , then, at step 16 of the algorithm DEEPS, the local bounds
lobndcand upbndc are halved. Now, at iteration iter, for every trail point xj, j ¼ 1, . . ., N, let us define
j , j ¼ 1, . . ., N, from xj, j ¼ 1, . . ., N, to the local points xj , k ¼ 1, . . .M, generated by
the distance diter k

the algorithm, i.e.,

 
 
d iter
j ¼ max xj  xkj  , j ¼ 1, . . . N: ð9:13Þ
k¼1, ..., M 2

With this, at iteration iter, let us determine a trial point xw for which
n o
d iter
w ¼ max d iter
j : j ¼ 1, . . . , N : ð9:14Þ

In other words, d iter

w is the maximum distance among all trial points and all local trial points. As in
Proposition 9.3, the following proposition can be proved.
 min   min 
Proposition 9.4 If at the iteration iter f iter  f opt iter ¼ f iter1  f opt iter1 holds, then, at this
iteration, the distance diter
w among the trial points and the local trial points is reduced. ♦

Again, since at every iteration the algorithm DEEPS chooses the minimum point among the
generated local points and at some specific iteration the bounds lobndc and upbndc are halved, it
follows that the maximum distance between the trial points and the local trial points tends to zero.
With these, the convergence of the algorithm DEEPS can be proved as follows.

Theorem 9.1 Let f be a continuous function bounded from below. Then, the algorithm DEEPS
initialized in a point x0 generates a sequence {xk} convergent to a point x where f(x)  f(x0).

Proof Since at every iteration k DEEPS chooses the minimum point xk from the trial points, it
follows that the sequence {f(xk)} is monotonously decreasing, i.e., f(xk)  f(xk  1), for any k ¼ 1,
2, . . . . Function f is bounded from below, and therefore, the sequence {f(xk)} is convergent. By


continuity, it follows that there is a point x such that lim f ðxk Þ ¼ f lim xk ¼ f ðx Þ: ♦
k!1 k!1

Observe that x is only a point where f(x)  f(x0). Since we do not have access to the gradient (and
the Hessian) of function f, nothing can be said about its optimality. Having in view the derivative
scarcity of information about the minimizing function f, the result given by DEEPS may be
acceptable from a practical point of view. More details on DEEPS algorithm can be found in Andrei
(2021a).
366 9 Direct Methods for Unconstrained Optimization

In this presentation of DEEPS, there are some (open) problems which need to be clarified:
selection of the parameters N and M, selection of the lower and upper bounds, the use of the line-
search procedure in step 12, etc. Our intensive numerical experiments with DEEPS show that all these
elements have a marginal effect on the performances of DEEPS (Andrei, 2021a).

9.4 Numerical Study: NELMED, NEWUOA, and DEEPS

In the first set of numerical experiments, consider the minimization of the following function:
X
n1
f ðx Þ ¼ ðjxi j þ 2jxiþ1 j  9Þ2 þ ð2jxi j þ jxiþ1 j  13Þ2 ,
i¼1

where x0 ¼ [1, . . ., 1]. For minimizing this functions with different values for n, Tables 9.1, 9.2, and
9.3 show the performances of NELMED, NEWUOA with ρend ¼ 106, and DEEPS, respectively.
For minimizing this function for different values of n, from Tables 9.2 and 9.3 observe that
NEWUOA needs a total of 859.23 seconds, while DEEPS needs only 14.08 seconds. Observe that
both NEWUOA and DEEPS give a highly accurate solution.
In the second set of numerical experiments, let us solve the applications from the SMUNO
collection presented in Appendix B. This collection includes 16 real applications from different

Table 9.1 Performances of NELMED

n #iter #nf cpu vfomin vf0

50 28658 32523 0.81 605.627290 6664.0
100 157965 170796 12.15 1047.023215 13464.0
200 551573 576315 193.49 2544.360161 27064.0

FORTRAN77 version by R. O’Neill (1971), modifications by John Burkardt

Table 9.2 Performances of NEWUOA

n #iter #nf cpu vfomin vf0

50 6 1037 1.10 389.333333 6664.0
100 6 1527 6.07 789.333333 13464.0
200 6 3665 117.55 1589.333333 27064.0
300 6 7616 734.51 2389.333333 40664.0

FORTRAN 77 version by Powell

Table 9.3 Performances of DEEPS

n #iter #nf cpu vfomin vf0

50 1164 37037 0.09 389.333333 6664.0
100 1510 47962 0.22 789.333333 13464.0
200 3521 112367 1.03 1589.333333 27064.0
300 11053 349293 12.74 2389.333333 40664.0

FORTRAN 77 version by Andrei

9.4 Numerical Study: NELMED, NEWUOA, and DEEPS 367

Table 9.4 Performances of NELMED for solving 16 small-scale applications from the SMUNO collection

n #iter #nf cpu vfomin vf0 name

2 4321 14260 1 – 0.2644531398164E+03 –0.374731373714E+02 1. WEBER-1
2 5749 17111 0 0.9560739834844E+01 0.7859432489718E+02 2. WEBER-2
2 485 1515 0 0.8749843722120E+01 0.7860286479337E+02 3. WEBER-3
4 58196 298649 8 0.3075923582382E – 03 0.5313172272109E– 02 4. ENZIMES
6 601448 2963265 50 0.3370929785002E – 03 0.1961733675060E+08 5. REACTOR
8 18824 131208 2 0.6829467973914E – 08 0.5334258881257E+01 6. ROBOT
4 44064 225895 35 0.6872367777921E+01 0.9958700480657E+01 7. SPECTR
4 220110632 999990007 26078 0.3822441651590E – 01 0.2905300235663E+01 8. ESTIMP
5 16640 96881 1 0.2887516043798E – 03 0.3312269269234E+08 9. PROPAN
2 3584 11541 1 0.1744152005589E+01 0.2563325000000E+04 10. GEAR-1
8 1232994 12451823 231 0.9315130266680E – 05 0.1905692553768E+00 11. HHD
6 539699 4132548 52 0.2463694921725E – 05 0.2391760016000E+02 12. NEURO
10 19060786 155723303 4155 0.8484442736605E – 10 0.1219988990749E+03 13. COMBUST
9 39546 390639 71 0.1022362061183E – 01 0.2964578187893E+04 14. CIRCUIT
3 54473 136162 83 0.1750977749414E+03 0.2335910048036E+10 15. THERM
4 8026086 36117397 533 0.7515500076120E – 19 0.7370818569964E– 03 16. GEAR-2

FORTRAN77 version by R. O’Neill (1971), modifications by John Burkardt

Table 9.5 Performances of NEWUOA for solving 16 small-scale applications from the SMUNO collection

n #iter #nf cpu vfomin vf0 name

2 7 129 0 – 0.2644531378352E+03 –0.3747313737140E+02 1. WEBER-1
2 6 72 0 0.9560739742798E+01 0.7859432489718E+02 2. WEBER-2
2 6 89 0 0.8749841736967E+01 0.7860286479337E+02 3. WEBER-3
4 6 306 1 0.3075056038514E– 03 0.5313172272109E– 02 4. ENZIMES
6 6 4824 15 0.5297708382383E –11 0.1961733675060E+08 5. REACTOR
8 6 262 1 0.3301153056066E –13 0.5334258881257E+01 6. ROBOT
4 6 246 1 0.8312307692327E+01 0.9958700480657E+01 7. SPECTR
4 7 1588 2 0.3185717487911E– 01 0.2905300235663E+01 8. ESTIMP
5 7 4114 9 0.4282211642674E– 04 0.3312269269234E+08 9. PROPAN
2 6 63 0 0.1744152005588E+01 0.2563325000000E+04 10. GEAR-1
8 5 9000 47 0.6277417900490E– 04 0.1905692550902E+00 11. HHD
6 5 282 1 0.4539057615182E+01 0.2391760016000E+02 12. NEURO
10 6 2812 21 0.3419994182218E– 11 0.1219988990749E+03 13. COMBUST
9 6 7370 44 0.1768702508139E– 09 0.2964578187893E+04 14. CIRCUIT
3 4 118 1 0.6575396003806E+05 0.2335910048036E+10 15. THERM
4 8 82 0 0.4058913495309E– 21 0.7370818569964E– 03 16. GEAR-2

FORTRAN 77 version by Powell

areas of activity: chemical reaction, solution of a chemical reactor, robot kinematics problem, solar
spectroscopy, propane combustion in air-reduced variant, gear train of minimum inertia, human
heart dipole, neurophysiology, combustion application, circuit design, thermistor, and optimal design
of a gear train. The largest application has 10 variables. Table 9.4 shows the performances of
NELMED for solving these applications.
Observe that NELMED needs 249817527 iterations and 31.301 seconds. Table 9.5 includes the
performances of NEWUOA. We can see that for solving all the 16 applications, NEWUOA needs
97 iterations and 14.3 seconds. Table 9.6 presents the performances of DEEPS. The total number of
368 9 Direct Methods for Unconstrained Optimization

Table 9.6 Performances of DEEPS for solving 16 small-scale applications from the SMUNO collection

n #iter #nf cpu vfomin vf0 name

2 119 1485 0 – 0.2644531378352E+03 –0.3747313737140E+02 1. WEBER-1
2 67 1318 0 0.9560744054913E+01 0.7859432489718E+02 2. WEBER-2
2 113 6198 1 0.8749847970082E+01 0.7860286479337E+02 3. WEBER-3
4 28 42903 2 0.3087657632221E– 03 0.5313172272109E– 02 4. ENZIMES
6 1375 6989983 197 0.7472925581554E– 04 0.1961733675060E+08 5. REACTOR
8 43 2459 0 0.1828017526180E– 07 0.5334258881257E+01 6. ROBOT
4 4 233 0 0.8316382216967E+01 0.9958700480657E+01 7. SPECTR
4 54 1705727 99 0.3185724691657E– 01 0.2905300235663E+01 8. ESTIMP
5 10 10013013 274 0.2467602087429E– 04 0.3312269269234E+08 9. PROPAN
2 15 342 0 0.1744152005590E+01 0.2563325000000E+04 10. GEAR-1
8 2404 36265585 1487 0.9966084682095E– 04 0.1905692550902E+00 11. HHD
6 1044 7999853 219 0.8545018926146E– 04 0.2391760016000E+02 12. NEURO
10 25 856 0 0.4061987800161E– 08 0.1219988990749E+03 13. COMBUST
9 3731 9654682 1244 0.1036184837525E– 03 0.2964578187893E+04 14. CIRCUIT
3 15 752179 251 0.1742216236340E+03 0.2335910048036E+10 15. THERM
4 6 3090 0 0.3886716443010E– 13 0.7370818569964E– 03 16. GEAR-2

FORTRAN version by Andrei

Table 9.7 Performances of NEWUOA for solving 10 large-scale applications from the SMUNO collection

n #iter #nf cpu vfomin vf0 name

100 6 13048 205.96 0.1822103437144E–10 0.3976246221006E+03 DENSCHNA
200 7 25965 868.09 0.7241454319422E–10 0.6585000000000E+06 DENSCHNB
200 7 11503 381.19 0.5939581794857E– 08 0.4160000000000E+05 DENSCHNF
300 5 3691 321.24 0.1315442201683E– 09 0.5710855664408E+01 BROWN
200 7 17026 539.42 0.2200999786590E+03 0.1174100000000E+05 ENGVAL1
100 4 100000 888.07 0.2343611622985E– 05 0.7942000000000E+04 NONDQAR
200 6 15063 523.91 0.2386448816338E– 09 0.5970000000000E+03 ARWHEAD
200 3 100000 3588.24 0.3474833628182E+00 0.3256542280009E+17 VARDIM
200 7 20557 708.15 0.1000000000068E+01 0.1626000000000E+04 DIXMAANA
300 7 30590 2533.33 0.1000000000360E+01 0.4494375000000E+04 DIXMAANB

FORTRAN 77 version by Powell

iterations required by DEEPS for solving all these 16 applications is 9053, while the CPU computing
time is of 37.74 seconds. Observe that NEWUOA is more efficient than NELMED and DEEPS.
In the last set of numerical experiments, Tables 9.7 and 9.8 present the numerical performances of
NEWUOA with ρend ¼ 106 and DEEPS with ε ¼ 106 for solving 10 unconstrained optimization
problems from the LACOP collection, with the number of variables in the range [100,300]. To get an
accurate solution in NEWUOA, the number of function’s evaluations was limited to 100000. Observe
that both these algorithms are able to solve problems with a relative large number of variables, but the
differences are significant. For solving all these 10 problems, NEWUOA needs a total of 59 iterations
and 10557.60 seconds. On the other hand, DEEPS needs a total of 3976 iterations and 239.04 seconds.
In the above tables, we have n ¼ the number of variables, #iter ¼ the number of iterations to get a
local solution, #nf ¼ the number of function evaluations, cpu ¼ the CPU computing time (seconds),
vfomin ¼ the minimum value of the minimizing function at solution, vf0 ¼ the value of the
minimizing function in the initial point, and name ¼ the name of the application.
9.4 Numerical Study: NELMED, NEWUOA, and DEEPS 369

Table 9.8 Performances of DEEPS for solving 10 large-scale applications from the SMUNO collection

n #iter #nf cpu vfomin vf0 name

100 54 2726250 44.64 0.1933041622350E– 05 0.3976246221006E+03 DENSCHNA
200 576 5066817 70.13 0.7798508331993E– 05 0.6585000000000E+06 DENSCHNB
200 1919 110060 1.72 0.6170389679663E– 03 0.4160000000000E+05 DENSCHNF
300 12 30930 3.82 0.2519366320438E– 04 0.5710855664408E+01 BROWN
200 56 285028 4.28 0.2460315490512E+03 0.1174100000000E+05 ENGVAL1
100 11 57306 0.27 0.6641255924711E– 07 0.7942000000000E+04 NONDQAR
200 277 717905 10.82 0.3723608045636E– 04 0.5970000000000E+03 ARWHEAD
200 1032 5258097 85.49 0.7836853799080E– 04 0.3256542280009E+17 VARDIM
200 11 643 0.05 0.1000005648123E+01 0.1626000000000E+04 DIXMAANA
300 28 143708 17.82 0.1000012240963E+01 0.4494375000000E+04 DIXMAANB

FORTRAN 77 version by Andrei

Subject to CPU computing time the most critical resources, it seems that DEEPS is more efficient
than NEWUOA. However, the solution given by NEWUOA is more accurate. The direct optimization
methods are recommended for minimizing functions with a small number of variables for which the
derivative information are difficult to obtain or are unreliable.

Notes and References

The derivative-free methods can be classified as direct and model-based. Both of them may be
randomized. Briefly, the direct algorithms determine the search directions by computing the values of
the function f directly, whereas the model-based algorithms construct and utilize a surrogate model of
the minimizing function to guide the search process. Furthermore, these methods are classified as
local or global. Finally, these algorithms can be stochastic or deterministic, depending upon whether
they require random search steps or not. The direct search methods include the methods of a
simplicial type, where one typically moves away from the worst point (a point at which the function
value is the highest), and the methods of a directional type, where one tries to move along a direction
defined by the best point. The simplex methods construct and manipulate a collection of n þ 1 affinely
independent points in ℝn called vertices, which define a simplex and make use of some rules
(reflection, expansion, inner contraction, and shrink) to modify these vertices of the simplex until
the best vertex has been determined. The most popular and representative simplex method is of
Nelder and Mead (1965). Each iteration of the directional direct-search methods generates a finite set
of points around the current point xk, called poll points, by taking xk and by adding terms of the form
αkd, where αk is a positive step and d is an element from a finite set of directions Dk, the poll
direction set.
Some of the very well-known direct methods for solving unconstrained optimization problems are
computing a zero of a nonlinear function in a given interval (Andrei, 1975a, 1975b); the univariate
search; the Fibonacci and golden section search; the coordinate search; the pattern move of Hooke-
Jeeves (1961); the simplex method by Spendley, Hext, and Himsworth (1962); the conjugate
directions of Powell (1964); turning the coordinates of Rosenbrock (1960); UOBYQA (Uncon-
strained Optimization BY Quadratic Approximation) of Powell (1994, 2002); the implicit filtering
by Kelley (1999) and Choi and Kelley (2000) (see Andrei, 1999a, 2009e); etc. A comparison of the
numerical performances of these direct methods is given by Andrei (2009e). It seems that the direct
methods based on the interpolating quadratics are some of the best direct methods. The derivative-
free methods for one-dimensional problems are described by Brent (1973). Recent surveys of the
direct methods for optimization can be found in the papers of Wright (1996); Conn, Scheinberg, and
370 9 Direct Methods for Unconstrained Optimization

Toint (1997b); Powell (1998); Lewis, Torczon, and Trosset (2000); Kolda, Lewis, and Torczon
(2003); Rios and Shainidis (2013); and Andrei (2021a, b, c, d).
The main software packages for the derivative-free optimization include COBYLA (Powell,
2003), DFO (Conn, Scheinberg, & Toint, 1997b), UOBYQA (Powell, 2002), WEDGE (Marazzi, &
Nocedal, 2002), NEWUOA (Powell, 2004) implementing the model-based methods using linear or
quadratic models, APPS (Hough, Kolda, & Torczon, 2001) using a pattern-search method, NELMED
(O’Neill, 1971), DIRECT (Jones, Perttunen, & Stuckman, 1993; Finkel, 2003) for the global optimi-
zation, and DEEPS (Andrei, 2021a, b, c, d). E04FCC (NAG – The Numerical Algorithms Group) is a
comprehensive algorithm for finding an unconstrained minimum of a sum of squares of m nonlinear
functions in n variables (m  n).
 Constrained Nonlinear Optimization
Methods: An Overview 10

Consider a particular form of a nonlinear optimization problem

min f ðxÞ
subject to ð10:1Þ
cðxÞ ¼ 0, x  0,

where f : ℝn ! ℝ is the objective function and c : ℝn ! ℝm are the constraints assumed to be twice
continuously differentiable. For the problem (10.1), denote the multipliers corresponding to the
equality constraint by y and the multipliers of the inequality constraints by z  0. Observe that
(10.1) may have inequality constraints, simple bounds on variables, or even range constraints of the
form li  ci(x)  ui, i ¼ 1, . . ., m, which are here omitted for the sake of simplifying the presentation
(Leyffer & Mahajan, 2010). To solve (10.1), the iterative methods are used in a reductionist approach.
The problem (10.1) is approximated by a sequence of subproblems. Their solving gives a sequence of
solutions {xk} starting from an initial point x0. The solution of a subproblem is the initial guess for the
next subproblem in this sequence, and all the process has a number of the so-called major iterations.
In this frame, every subproblem is also solved by an iterative process through the so-called minor
iterations. A simple prototype for solving (10.1) to generate the sequence {xk} is as follows:

Algorithm 10.1 Prototype for the nonlinear optimization algorithm

1. Choose an initial estimate x0 and set k ¼ 0

2. Evaluate a criterion for stopping the iterations
3. At the current point xk elaborate a local model of (10.1)
4. Approximately solve the local model (sub-problem) to determine an improved estimation xkþ1 of the
solution of (10.1)
5. Set k ¼ k þ 1 and continue with step 2 ♦

In the following, the main components of Algorithm 10.1 for solving the problem (10.1) are
discussed in a critical manner. The real sense of the word “criticism” does not mean “destruction,”
but “creation.” Consequently, some issues may be looked upon: how the approximate subproblem
that determines an improved new iterate is computed, the globalization strategy that ensures the

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 371
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_10
372 10 Constrained Nonlinear Optimization Methods: An Overview

convergence of the algorithm initialized in remote starting points by indicating whether a new
estimate of the solution is better than the current estimate, the refining mechanisms that reduces the
step computed by the local model to enforce the globalization strategy to work far away from the
solution, and the convergence tests as well as the termination conditions that check for the optimal
solution or for the failure of the algorithm. The nonlinear optimization algorithms are classified by the
choice with which they implement each of these fundamental components of Algorithm 10.1.

10.1 Convergence Tests

Usually, the convergence tests are based on the Karush-Kuhn-Tucker conditions (see Theorem
11.15). For example, for the problem (10.1), suitable approximate convergence tests may be
presented as
kc ð x k Þ k  ε 1 , ð10:2aÞ

k∇f ðxk Þ  ∇cðxk Þyk  zk k  ε2 , ð10:2bÞ

k min fxk , zk gk  ε3 , ð10:2cÞ

where εi > 0, i ¼ 1, 2, 3, are tolerances and the “min” in (10.2c) corresponding to the complementarity
condition is taken componentwise. (Here ∇f(xk) is the gradient of f and ∇c(xk) is the Jacobian of
c computed at point xk.) It is quite possible for the convergence of the algorithm not to be ensured to
an approximate KKT point (see Theorem 11.15), for example, if the constraints do not satisfy a
constraint qualification (see Remark 11.2). In these cases, the test (10.2b) is replaced by

k∇cðxk Þyk þ zk k  ε2 , ð10:3Þ

corresponding to a Fritz John point. It is also possible for the termination conditions to be modified to
include the “relative error,” as used in some algorithms. The specific norm used in the above criteria
depends upon the number of variables. For small-scale problems, the two-norm is satisfactory, while
for large-scale problems, the infinity norm is recommended.
The limits of these convergence tests are the feasibility conditions and those of optimality are
separated, so that we may obtain a feasible solution which is not optimal and vice versa. However, the
feasibility of the constraints is always the most important criterion for terminating the iteration. From
the practical point of view, the decision that the iterate xk is an adequate approximation to the
solution is based on two major tests: whether xk almost satisfies the sufficient conditions for optimality
and whether the sequence {xk} appears to have converged.

10.2 Infeasible Points

As already seen in Algorithm 10.1, one needs an initial point x0 to start the iterations. The computa-
tion of an initial point is a problem in itself. Unless the problem is convex or some restrictive
assumptions are made, Algorithm 10.1 cannot guarantee the convergence even to a feasible point.
Besides, there are many cases in which the points of the sequence {xk} determined by Algorithm 10.1
must be feasible. In these cases, a procedure for certifying the feasibility is needed. An appropriate
converge test and feasibility subproblem is based on the following feasibility problem:
10.3 Approximate Subproblem: Local Models and Their Solving 373

min x0 kcðxÞk, ð10:4Þ

which, as can be seen, may be formulated as a smooth optimization problem by introducing the slack
variables. For solving the problem (10.4), one can use Algorithm 10.1, as the feasibility problem can
be reformulated as a smooth nonlinear optimization problem by introducing additional variables. The
objective function of (10.4) can be replaced by a weighted norm. In this case, suitable tests are

k∇cðxk Þyk  zk k  ε and k min ðxk , zk Þk  ε, ð10:5Þ

where yk are the multipliers or the weights corresponding to the norm used in the objective of (10.4).

10.3 Approximate Subproblem: Local Models and Their Solving

In step 3 of Algorithm 10.1, the elaboration of a local model of the problem (10.1) around the current
point xk is needed. The key difference among the nonlinear optimization algorithms is how the local
model is constructed around the current point. The idea is to generate a local model of the problem
for which a solution can be easily obtained and which can improve the current iterate. Three broad
classes of local models are known: sequential linear models, sequential quadratic models, and
interior point models.
The sequential linear programming (SLP) methods construct a linear approximation to (10.1).
Usually, these linear programming programs are unbounded. Therefore, these methods require an
additional trust-region constraint

min d mk ðdÞ≜∇f ðxk ÞT d

subject to ð10:6Þ
cðxk Þ þ ∇cðxk ÞT d ¼ 0,
xk þ d  0,
k d k1  Δ k ,

where Δk > 0 is the radius of the trust-region. To ensure the convergence, in general, Δk ! 0 must
converge to zero. This method was used by Griffith and Stewart (1961) with some success, without
introducing a trust-region constraint. The sequential linear programming method can be considered as
a steepest descent method and typically converges only linearly. If at the solution point there are
exactly n active constraints and the normals of these constraints are linearly independent, then the
sequential linear programming method reduces to the Newton method for solving a square system of
nonlinear equations, which is superlinear convergent.
The sequential quadratic programming (SQP) methods minimize a quadratic model mk(d ) subject
to a linearization of the constraints about the current point xk
1
min d mk ðdÞ≜∇f ðxk ÞT d þ dT Bk d
2
subject to ð10:7Þ
cðxk Þ þ ∇cðxk ÞT d ¼ 0,
xk þ d  0,

where Bk ≜∇2xx Lðxk , λk Þ is the Hessian of the Lagrange function associated to the problem (10.1) and
λk are the Lagrange multipliers, to obtain the search direction d ≜ x  xk. The new iterate is given by
xkþ1 ¼ xk þ d together with the Lagrange multipliers λkþ1 associated to the linearized constraints of
374 10 Constrained Nonlinear Optimization Methods: An Overview

(10.7). If the matrix Bk is not positive definite on the null space of the active constraint normals, then
the quadratic subproblem (10.7) is nonconvex and the SQP methods determine a local minimum of
(10.7). In case of large-scale problems, the solution of the quadratic subproblems can become
computationally expensive because the null space method for solving the quadratic subproblem
requires the factorization of a dense-reduced Hessian matrix.
The sequential linear-quadratic programming (SLQP) methods combine the advantages of the
SLP method (fast solution of the linear programming subproblem) and of the SQP methods (fast local
convergence) by adding an equality constraint to the SLP method [(Fletcher & de la Maza, 1989),
(Chin & Fletcher, 2003), (Byrd, Gould, Nocedal, & Waltz, 2002, 2004a)]. In these methods, two
subproblems are solved: the first one is a linear programming subproblem which gives a step for the
n o
next iteration and also an estimate of the active set Ak ≜ i : ½xk  þ dbi ¼ 0 obtained from a solution
i

db of (10.6). (Here [xk]i is the component i of vectorxk.) This estimation of the active set is used to
construct and solve the second problem, which is an equality constrained quadratic programming
problem defined by the active constraints
1
min d qk ðdÞ≜∇f ðxk ÞT d þ dT Bk d
2
subject to ð10:8Þ
cðxk Þ þ ∇cðxk ÞT d ¼ 0,
½xk i þ d i ¼ 0, i 2 Ak :

If Bk is positive definite on the null space of the constraint normals, then the solution of (10.8) is
equivalent to the following linear algebraic system:
2 32 3 2 3
Bk ∇cðxk Þ I k x ∇f ðxk Þ þ Bk xk
6 T 76 7 6 7
4 ∇cðxk Þ 0 0 54 yk 5 ¼ 4 cðxk Þ þ ∇cðxk ÞT xk 5, ð10:9Þ
T
Ik 0 0 zk 0

obtained by applying the KKT conditions to the subproblem (10.8). (Here, I k ¼ ½ei i2Ak are the
normals of the active inequality constraints; yk and zk are the Lagrange multipliers.) It is always
possible to choose a basis from the linear programming subproblem (by simplex algorithm) such that
the augmented matrix [Ak Ik] has full rank. If Bk is not positive definite on the null space of the active
constraint normals, then the inertia of this matrix can be corrected by adding to Bk a multiple of the
identity matrix, thus ensuring the descent step of the subproblem (10.8).
The sequential quadratic-quadratic programming (SQQP) methods have been recently introduced
by Gould and Robinson (2008, 2010). From the very beginning, they construct and solve a quadratic
programming subproblem by using a positive definite Hessian of the Lagrangian. The solution of this
convex subproblem is used into an equality constrained quadratic programming model with the exact
second derivative of the Lagrangian. More precisely, the SQQP procedure is a second derivative
method that is globalized via the l1 merit function. Given an estimate xk of the solution to the problem
(10.1), a search direction at this point is generated from the combination of three steps: a predictor
step defined as a solution to a strictly convex quadratic programming subproblem, a Cauchy step
(driving convergence of the algorithm) which is computed from a special univariate global minimi-
zation problem, and a SQP step computed from a local solution of a special nonconvex quadratic
programming subproblem. The algorithm is embedded into the trust-region approach.
10.3 Approximate Subproblem: Local Models and Their Solving 375

Remark 10.1 As a general characterization of these sequential linear and quadratic programming
methods, the following theoretical aspects can be mentioned. If Bk is the exact Hessian of the
Lagrange function and the Jacobian of the active constraints has full rank, then the SQP methods
are quadratic convergent near a minimizer that satisfies both a constraint qualification and a second-
order sufficient condition (the Hessian of the Lagrangian is positive definite on the null space of the
active constraint normals) (Boggs & Tolle, 1995). Moreover, under the additional assumption of
strict complementarity, all the four methods based on sequential linear or quadratic programming
identify the optimal active set in a finite number of iterations. The above sequential methods are also
known as active-set methods because the solution of each linear or quadratic subproblem provides
not only a suitable new iterate but also an estimate of the active set at the solution point (Leyffer, &
Mahajan, 2010).
Additionally, two major concerns are associated to the SQP methods: incompatible linearized
constraints and unbounded solutions of the quadratic subproblems. For handling the unbounded
solutions of (10.7), two approaches are possible. The first one is to use a positive definite approxima-
tion to the Hessian in (10.7). Thus, a strictly convex-bounded quadratic program is obtained with a
unique minimizer. The second approach allows for a nonconvex quadratic program by explicitly
bounding the solution via a trust-region constraint. Both approaches are effective in practice. The
issue of incompatible subproblems is more challenging. Observe that the quadratic programming
subproblem (10.7) may be naturally incompatible. In this case, different techniques have been
suggested for dealing with incompatible subproblems: the “constraint shifting” (Vardi, 1985), the
use of the “elastic mode” (Gill, Murray, & Saunders, 2002, 2005) (see Chap. 15), and a “feasibility
restoration phase” (Fletcher & Leyffer, 2002) (see Chap. 18). ♦

The interior point methods are an alternative approach to the active set methods. Essentially, these
are perturbed Newton methods which postpone the determination of the active constraints set at the
end of the iterative process for solving the problem. Plenty of interior point methods are known, but
the most successful one is the primal-dual, which can be viewed as the Newton method applied to the
perturbed first-order optimality conditions of problem (10.1). The first-order optimality conditions
include the stationary condition of the Lagrange function, the feasibility condition (satisfying the
constraints of the problem), and the complementarity condition (see Definition 11.14). In the interior
point methods, only these complementarity conditions are perturbed, as in the following nonlinear
algebraic system:
2 3
∇f ðxÞ  ∇cðxÞT y  z
6 7
Fμ ðx, y, zÞ ¼ 4 c ðx Þ 5 ¼ 0, ð10:10Þ
Xz  μe

where μ > 0 is the barrier parameter, X ¼ diag (x) is a diagonal matrix with the components of the
vector x on the main diagonal, and e is a vector with all the components equal to one. The interior
point methods start the computations from an interior point x0, z0 > 0 and generate a sequence of
interior points xk, zk > 0 as approximate solutions of the system (10.10), for a decreasing sequence of
barrier parameters. By applying the Newton method to the nonlinear primal-dual system (10.10)
around xk, we get the local model (i.e., the approximate subproblem):
2 32 3
Bk ∇cðxk Þ I Δx
6 T 76 7
4 ∇cðxk Þ 0 0 54 Δy 5 ¼ Fμ ðxk , yk , zk Þ, ð10:11Þ
Zk 0 Xk Δz
376 10 Constrained Nonlinear Optimization Methods: An Overview

where Bk is an approximate to the Hessian of the Lagrange function and Zk ¼ diag (zk) is a diagonal
matrix with the components of the vector zk on the main diagonal. (Xk ¼ diag (xk).) The next step is
computed as
   
xkþ1 , ykþ1 , zkþ1 ¼ ðxk , yk , zk Þ þ αx Δx, αy Δy, αz Δz ,

where the stepsizes αx, αy, and αz are computed to ensure that xkþ1, zkþ1 > 0 remain strictly positive. A
simple variant of the interior point algorithms is as follows.

Algorithm 10.2 Prototype of the interior point algorithm

1. Choose an initial point estimate (x0, y0, z0) such that x0 > 0 and z0 > 0. Choose a value for the barrier
parameter μ0, the parameter 0 < σ < 1 and a decreasing sequence εk. Set k ¼ 0
2. Evaluate a criterion for stopping the iterations concerning the optimality of (xk, yk, zk).
3. Set (xk, 0, yk, 0, zk, 0) ¼ (xk, yk, zk) and j ¼ 0
  
4.  
If Fμk xk,j , yk,j , zk,j   εk , then go to step 6. Otherwise, continue with step 5
5. Approximately solve the Newton system (10.11) for a new iterate (xk, jþ1, yk, jþ1, zk, jþ1). Set j ¼ j þ 1 and
go to step 4
6. Reduce the barrier parameter μkþ1 ¼ σμk set k ¼ k þ 1 and go to step 2 ♦

Observe that this algorithm has two loops: one responsible for solving the problem and the other
one for approximately solving the Newton system. Notice that this is a general prototype interior
point algorithm, and many other ingredients are needed to make it efficient and robust.
At this point, it is very instructive to see the relationship of the interior point methods with the
barrier methods developed by Fiacco and McCormick (1990). The importance of the barrier methods
comes from the fact that they can provide polynomial-time algorithms for the linear programming
problems [(Wright, 1991), (Forsgren, Gill, & Wright, 2002), (Nemirovskii, & Todd, 2008)]. The
barrier methods approximately solve a sequence of barrier subproblems of the following form:
X
n
min x2ℝn f ðxÞ  μ log ðxi Þ
i¼1
subject to ð10:12Þ
cðxÞ ¼ 0,

for a decreasing sequence of the barrier parameters μ > 0. The first-order optimality conditions for the
problem (10.12) are given by

∇f ðxÞ  μX1 e  ∇cðxÞy ¼ 0, ð10:13aÞ

cðxÞ ¼ 0: ð10:13bÞ

Now, applying the Newton method to the nonlinear system (10.13), the following linear algebraic
system is obtained:
" #  " #
Bk þ μX2
k ∇cðxk Þ Δx ∇f ðxk Þ  μX1
k e  ∇cðxk Þyk
¼ :
∇cðxk ÞT 0 Δy cðxk Þ

Introducing the first-order multiplier estimates Zðxk Þ≜μX1

k , which can also be written as
Z(xk)Xk ¼ μe, the following linear system is obtained:
10.4 Globalization Strategy: Convergence from Remote Starting Points 377

" #  " #
Bk þ Zðxk ÞX1
k ∇cðxk Þ Δx ∇f ðxk Þ  μX1
k e  ∇cðxk Þyk
¼ :
∇cðxk ÞT 0 Δy c ðx k Þ

This system is equivalent to the primal-dual Newton system (10.11), where

Δz ¼ X1 ZΔx  Ze  μX1 e

has been eliminated. Therefore, the main difference between the classical barrier methods as
elaborated by Fiacco and McCormick and the primal-dual interior point methods is that the matrix
Zk is not free for the barrier methods, but it is chosen as the primal multiplier Zðxk Þ≜μX1 k : This
freedom in the primal-dual interior point methods avoids some difficulties with the ill-conditioning of
the Hessian to the barrier function. This was the main reason for rejecting the barrier methods in
solving nonlinear optimization problems.

Remark 10.2 As a characterization of the interior point methods, we can state: if there exists a
compact set of isolated local minimizers of the problem (10.1) with at least one point in the closure of
the strictly feasible set, then it follows that the barrier methods converge to a local minimum of the
problem (10.1) (Wright, 1991). On the other hand, for the convex nonlinear optimization problems,
the interior point methods, as in the case of linear programming, lead to polynomial-time algorithms
(Nesterov, & Nemirovskii, 1994). ♦

10.4 Globalization Strategy: Convergence from Remote Starting Points

The local models discussed above guarantee the convergence only in a small neighborhood of a
regular point (see Definitions 11.13 and 11.16). However, the initialization of the algorithms near the
unknown optimal solution is a difficult task. Therefore, the globalization strategies are concerned
with ensuring the convergence from the remote starting points to the stationary points (see Theorem
11.3). The globalization strategy should not be confused with the global optimization. To ensure the
convergence from the remote starting points, the progress of iterates generated by the approximate
subproblems must be monitored. For the unconstrained optimization, monitoring is very simple
because the progress of the algorithm can be measured by comparing the values of the objective
function. For the constrained optimization, it is necessary to take into account the constraint violation.
Three broad classes of strategies exist: the augmented Lagrangian methods, the penalty and merit
function methods, and the filter methods.
The augmented Lagrangian methods for the problem (10.1) use the augmented Lagrangian as
ρ
Lðx, y, ρÞ ¼ f ðxÞ  cðxÞT y þ kcðxÞk22 , ð10:14Þ
2
where ρ > 0 is a penalty parameter. In this context, there are two possibilities to develop the
algorithms based on the augmented Lagrangian.

(a) Linearly constrained Lagrangian methods. These methods minimize a modified Lagrangian
subject to a linearization of the constraints. The modified Lagrangian is defined as
ρ
Lðx, y, ρÞ ¼ f ðxÞ  pk ðxÞT y þ kpk ðxÞk22 , ð10:15Þ
2
378 10 Constrained Nonlinear Optimization Methods: An Overview

where pk(x) collects the higher-order nonlinear terms from the Taylor series at the current iterate xk,
i.e.,

pk ðxÞ ¼ cðxÞ  cðxk Þ  ∇cðxk ÞT ðx  xk Þ: ð10:16Þ

Therefore, this approach consists in forming and solving the following local model:

min x Lðx, yk , ρk Þ
subject to ð10:17Þ
cðxk Þ þ ∇cðxk ÞT ðx  xk Þ ¼ 0,
x  0,

for a fixed value of the penalty parameter ρk. The Lagrange multipliers are updated by using a first-
order multiplier update rule:
ykþ1 ¼ yk  ρk cðxkþ1 Þ, ð10:18Þ

where xkþ1 is the solution of (10.17). Therefore, the local model (10.17) is used in an iterative scheme
for a decreasing sequence of the penalty parameters. This approach is implemented in MINOS
(Chap. 14) (Murtagh & Saunders, 1980, 1982, 1987, 1995). Another modification of the augmented
Lagrangian as a combination of penalty and barrier terms is used in SPENBAR (Chap. 14) (Andrei,
1996a, 1996b, 1996c, 1998a).

(b) Bound-constrained Lagrangian methods. These methods approximately minimize the augmented
Lagrangian for the problem (10.1) subject to simple bounds on the variables,

min x Lðx, yk , ρk Þ
subject to ð10:19Þ
x  0,

for a sequence of increasing penalty parameters. The advantage of this approach is that efficient
methods for the simple bounds on the variables optimization can be applied: gradient projection
conjugate gradient approach by Moré and Toraldo (1991); limited memory BFGS with bounds by
Byrd, Lu, Nocedal, and Zhu (1995b); spectral projected gradient method by Birgin, Martı́nez, and
Raydan (2000, 2001); truncated Newton with simple bounds by Nash (1984a, b, 1985); etc. (see
Chap. 12). The idea of an iteration which uses the local model given by (10.19) is as follows. For a
fixed value of the penalty parameter ρk, solve the subproblem (10.19). Thus, an approximate solution
    
xck is obtained. If c xck   ηk , then update the Lagrange multipliers as ykþ1 ¼ yk  ρk c xck ;
otherwise, increase the value of the penalty parameter, for example, ρkþ1 ¼ 10ρk. Here, {ηk} is a
forcing sequence which controls the progress to the feasibility of the nonlinear constraints. Each
minimization of (10.19) can be started from the previous iterate. Representative for the bound-
constrained Lagrangian method is LANCELOT (Conn, Gould, & Toint, 1992a, 1992b).

Remark 10.3 As a general characterization of the augmented Lagrangian methods, we mention the
linearly constrained augmented Lagrangian method can be made globally convergent by adding
slack variables in order to handle the infeasible subproblems (Friedlander & Saunders, 2005). On the
other hand, a bound-constrained Lagrangian method converges globally if the sequence {xk} is
bounded and if the Jacobian of the constraints at all the limit points of {xk} has the column rank not
10.4 Globalization Strategy: Convergence from Remote Starting Points 379

smaller than m (the number of constraints) (Conn, Gould, & Toint, 1991a, b). The difficulty with the
augmented Lagrangian methods is the procedure for selecting the penalty parameters. Introducing
the filter methods by Fletcher and Leyffer (2002) avoids this difficulty. ♦

The penalty and merit function methods combine the objective function and a measure of the
constraint violation into a single function whose local minimizers correspond to the local minimizers
of the problem (10.1). The convergence from the remote starting points can be ensured by forcing the
descent of the penalty or the merit function by using one of the mechanisms discussed in the next
section. The exact penalty functions are alternative to the augmented Lagrangian and are defined as
Pρ ðxÞ ¼ f ðxÞ þ ρkcðxÞk, ð10:20Þ

where ρ > 0 is the penalty parameter. In (10.20), the most used norm is l1. If ρ  kykD, where y is the
Lagrange multiplier corresponding to the nonlinear constraints and k.kD is the dual norm of k.k (i.e.,
the l1norm in the case of the l1 exact penalty function), then the local minimum x of the exact
penalty function Pρ(x) is a local minimum of the problem (10.1) (Fletcher, 1987). Therefore, the
classical approach minimizes a sequence of penalty subproblems Pρ(x) for an increasing sequence of
the penalty parameters. Some other merit functions are known, for example, the quadratic penalty
function f ðxÞ þ ρkcðxÞk22 , the oldest one. The corresponding algorithm based on this penalty function
is convergent to a local solution only if the sequence of the penalty parameters is divergent to infinity.
The global convergence is also ensured by using the augmented Lagrange and the penalty functions
of the type f(x) þ c(x)Ty þ ρkc(x)k. A major difficulty with these penalty functions is the possibility
that near a strictly isolated minimum, the step given by the Newton method should not be accepted.
This phenomenon is known as the Maratos effect (Maratos, 1978). The remedy for the Maratos effect
is the introduction of second-order corrections or the use of the nonmonotone line-search. However,
the greatest difficulty with the penalty methods is the selection and the updating of the penalty
parameters.
The filter methods try to harmonize two objectives, always present in any optimization problem:
minimization of the objective function and minimization of the violation of the constraints (see
Chap. 18). The filter methods keep a record of the objective function value fl ¼ f(xl) and of the
constraint violation hl ¼ kc(xl)k for a number of previous iterates xl, l 2 Fk, where Fk is the filter at
iteration k (Fletcher & Leyffer, 2002). A new iterate is acceptable in the filter if it improves either the
objective function or the constraint violation, compared to all the previous iterates recorded in the
filter. In other words, the point xb is acceptable in the filter if f ðxbÞ  f ðxl Þ  γhðxl Þ or hðxbÞ  βhðxl Þ,
for any l 2 Fk, where γ > 0 and 0 < β < 1 are constants which ensure that the iterates cannot accumulate
at infeasible limit points. To ensure the convergence to a local minimizer, the filter algorithm uses a
standard sufficient reduction condition taken from the unconstrained optimization:
f ðxk Þ  f ðxk þ dÞ  σmk ðdÞ, ð10:21Þ

where σ > 0 is the fraction of the predicted decrease and mk(d ) is the model reduction from the
approximate subproblem. This condition is used only if the model predicts a decrease in the objective
function. Therefore, the filter method uses a switching condition mk(d)  γh(xk)2 to decide when
(10.21) should be enforced. A new iterate that satisfies both these conditions is called an iteration of
type f, while an iterate for which the switching condition fails is called an iteration of type h.

Remark 10.4 The general characteristic of the filter method is that it avoids the complications with
selecting and updating the penalty parameter from the augmented Lagrangian or the penalty function
methods. However, the difficulty with this method is the selection of the parameters which determine
380 10 Constrained Nonlinear Optimization Methods: An Overview

the convergence as well as the specification of the heuristics for ensuring the feasibility of the
subproblems and the convergence to a local solution. ♦

10.5 The Refining the Local Model

The idea of refining the local model is to reduce the stepsize that is computed by the approximate
subproblem. Two mechanisms for refining the local model are known: line-search methods and trust-
region methods. Both these mechanisms can be used in the frame of any of the approximate
subproblems and of any globalization strategies for the convergence from the remote starting points.
Therefore, a great variety of nonlinear optimization algorithms result, with different capabilities for
solving complex and large-scale optimization problems.
The line-search methods enforce the convergence by using a backtracking line-search along the
direction s. For the sequential quadratic programming methods, the searching direction is given by the
quadratic programming problem (10.7), s ¼ d. For the interior point methods, the searching direction
s ¼ (Δx, Δy, Δz) is the solution of the primal-dual linear algebraic system (10.11). To ensure that the
model produces a descent direction, one must have ∇P(xk)Ts < 0, where P(x) is a merit or a penalty
function. A very popular line-search is the Armijo rule (Armijo, 1966) (see (Andrei, 2009e)). A
prototype algorithm with line-search is as follows.

Algorithm 10.3 Prototype of the line-search method for the nonlinear optimization

1. Choose an initial point x0 2 ℝn, as well as 0 < σ < 1. Set k ¼ 0

2. Evaluate a criterion for stopping the iterations. If xk is the optimal solution of the problem, stop;
otherwise, go to step 3
3. Approximately solve an approximate sub-problem of (10.1) around xk to find a search direction s
4. Verify whether s is a descent direction if ∇P(xk)Ts < 0,for a given merit function P(x)
5. Set α0 ¼ 1 and m ¼ 0
6. If P(xk þ αms)  f(xk) þ αmσsT ∇ P(xk), then continue with step 7; otherwise, go to step 8
7. Set αmþ1 ¼ αm/2 and evaluate P(xk þ αmþ1s). Set m ¼ m þ 1. Continue with step 6
8. Set k ¼ k þ 1 and go to step 2 ♦

Observe that the algorithm has two loops. The inner loop determines the stepsize by backtracking.
The outer loop is responsible for the search direction determination, i.e., the solution of the problem.
The line-search methods with filters can be defined in a similar way, but, instead of checking the
descent in the merit function, a filter method is used to check the acceptance to a filter (Leyffer, &
Mahajan, 2010).
The trust-region methods restrict the step in an explicit way by adding a trust-region constraint of
the form kdk  Δk, where the used norm is l1, l2, or l1. Most methods use the l1 norm in the trust-
region constraints, which can be represented by the simple bounds on the variables. The trust-region
radius Δk > 0 is adjusted at every iteration, depending on how well the approximate subproblem
agrees with the nonlinear optimization problem (10.1). A prototype of the optimization algorithm
with trust-region is as follows.
10.5 The Refining the Local Model 381

Algorithm 10.4 Prototype of the trust-region method for the nonlinear optimization

1. Choose an initial point x0 and a value for the trust-region radius Δ0 > 0. Set k ¼ 0
2. Evaluate a criterion for stopping the iterations. If xk is the optimal solution of the problem, stop;
otherwise, go to step 3
3. Approximately solve the local model of the problem (10.1), i.e. the sub-problem (10.7) in which a trust-
region constraint kdk  Δk is introduced
4. If xk þ d is sufficiently better than xk, then increase Δk and continue with step 5; otherwise, reduce Δk, for
example, Δk ¼ Δk/2 and go to step 3
5. Set k ¼ k þ 1 and go to step 2 ♦

The trust-region methods are related to the regularization techniques, which refer to the positive
definiteness of the Hessian matrix Bk. These techniques are implemented by adding a multiple of the
identity matrix to the Hessian. Locally, the solution of the regularized problem is equivalent to the
solution of a trust-region problem with an l2 norm trust-region. Clearly, the disadvantage of the trust-
region methods is that the subproblems may become inconsistent as Δk ! 0. This situation can be
dealt with in three different ways: a penalty function approach (Nocedal & Wright, 2006) (see
Chap. 14), a restoration phase in which the algorithm minimizes the constraint violation (Fletcher
& Leyffer, 2003) (see Chap. 18), or a composite step approach (Omojokun, 1989) (see Chap. 15). All
these refining mechanisms of the local model are used in nonlinear optimization algorithms. The
difficulties are related to the computation of the stepsize, to the selection of the merit function, and to
the techniques for regularization.
In conclusion, a large variety of constrained nonlinear optimization algorithms are known. All of
them use four major ingredients: a local model of the problem around the current point, a globaliza-
tion strategy which ensures the convergence of the algorithm from remote initial points, a refining
mechanism which reduces the stepsize computed by the local model to enforce the convergence, and
some convergence tests which certify the type of the limit point obtained by the algorithm. The
diversity of the optimization algorithms is given by the way in which all these ingredients are
implemented. Besides, the optimization algorithms implement a multitude of other ingredients
which are crucial in the efficiency and robustness for solving complex large-scale nonlinear optimi-
zation problems. Example of such ingredients are sparse or dense linear algebra techniques, factori-
zation of indefinite matrices, inertia correction of matrices, complement Schur computation,
identification of the active constraints, initialization of the algorithms with respect to the primal
and dual variables, updating procedures for the barrier parameter or a trust-region radius, updating the
parameter of the central trajectory, solving the infeasibilities (elastic programming), etc.
The most efficient and robust nonlinear optimization algorithms use combinations of the active-set
methods (local models based on sequential linear or quadratic programming), or interior point
methods with globalization strategies (augmented Lagrangian, penalty and merit function or filters
methods) and refining mechanisms (line-search or trust-region methods), all of them in a frame where
advanced computational linear algebra techniques are used. The most advanced nonlinear optimi-
zation algorithms implementing the above concepts are illustrated in the chapters of this book.

Notes and References

This overview of the constrained nonlinear optimization methods is based on a deep analysis of the
nonlinear optimization problems and algorithms for solving them. All the nonlinear optimization
problems are solved in a reductionist approach. A current point is considered as an approximation to
the solution of the original problem. In this current point, the problem is approximated through a
382 10 Constrained Nonlinear Optimization Methods: An Overview

linear or quadratic model. This model is solved in a specific way. Thus, the obtained solution is
considered as the next approximation to the solution of the original problem and as the starting point
for solving the next approximation of the problem. For the termination of this iterative process, some
criteria for stopping the iterations are used. A large variety of nonlinear optimization algorithms is
known. The differences among them are the strategy for initialization, the construction of the local
models, the globalization, the refining mechanism, and the stopping criteria. Extremely important is
the computational linear algebra for implementing the different steps of the nonlinear optimization
algorithms. These ingredients of an advanced nonlinear optimization algorithm are found in different
publications, like Luenberger (1973, 1984); Gill, Murray, and Wright (1981); Dennis and Schnabel
(1983); Peressini, Sullivan, and Uhl (1988); Bazaraa, Sherali, and Shetty (1993); Bertsekas (1999);
Conn, Gould, and Toint (2000); Nocedal, and Wright (2006); Sun and Yuan (2006); Bartholomew-
Biggs (2008); Kelley (1999); Andrei (1999a, 1999b, 2009e, 2009f, 2015a, 2017c, 2020a); Leyffer and
Mahajan (2010); and Luenberger and Ye (2016).
It is noteworthy that the optimization methods and techniques have two main traditions: the linear
programming and the calculus of variations. These two topics have evolved along two disjoint paths
with no connection between them. The situation dramatically changed in 1984, when Karmarkar
(1984) announced a new polynomial-time interior-point algorithm for linear programming, for which
he reported performance times that were 50 times faster than those of the simplex algorithm.
Immediately after, in 1985, it was shown that there was a formal equivalence between Karmarkar’s
method and the classical logarithmic barrier method applied to linear programming problems. Since
then, interior methods have continued to transform both the theory and the practice of constrained
optimization. This was the first unification between linear programming and nonlinear optimization.
At present, both the simplex algorithm and different variants of the interior-point algorithms coexist
as important methods for solving linear and nonlinear optimization problems.
On the other hand, a wide variety of problems arising in the system and control theory can be
reduced to a few standard convex or quasiconvex optimization problems involving linear matrix
inequalities. In other words, many constraints in the system and control theory, including convex
quadratic inequalities, matrix norm inequalities, Lyapunov matrix inequalities, the positive-real
lemma, the bounded-real lemma, etc., can be expressed as linear matrix inequalities. Interior-point
methods are important since these optimization problems can be solved numerically very efficiently
using the polynomial-time algorithms (see Boyd, El Ghaoui, Feron, & Balakrishnan, 1994), thus
achieving the second unification of optimization and of the system and control theory.
We can see that the interior-point methods were the instrument that unified linear and nonlinear
optimization with the system and control theory by emphasizing the problem of complexity of the
optimization algorithms. Modern constrained optimization methods are based on a combination of
the interior-point with the sequential linear-quadratic programming with line-search or filters in an
advanced computational linear algebra environment, as we can to see in the chapters of this book.
 Optimality Conditions for Nonlinear
Optimization 11

The optimization problems considered in this book involve the minimization or maximization of a
function of several real variables subject to one or more constraints. The constraints may be the
non-negativity of variables, the simple bounds on variables, and the equalities or inequalities as
functions of these variables. These problems are known as continuous nonlinear constrained optimi-
zation or nonlinear programming.
The purpose of this chapter is to introduce the main concepts and the fundamental results in
nonlinear optimization, known as optimality conditions. Plenty of very good books dedicated to these
problems are known in literature: Luenberger (1973); Gill, Murray, and Wright (1981); Peressini,
Sullivan, and Uhl (1988); Bazaraa, Sheraly, and Shetty (1993); Bertsekas (1999); Boyd and
Vandenberghe (2004); Nocedal and Wright (2006); Sun and Yuan (2006); Chachuat (2007); Andrei
(2009e, 2015a, 2017c); etc.
The general continuous nonlinear optimization problem is expressed as
min f ðxÞ
subject to ð11:1Þ
ci ðxÞ  0, i ¼ 1, . . . , m,
hi ðxÞ ¼ 0, i ¼ 1, . . . , p,

where x 2 ℝn, f : ℝn ! ℝ, ci : ℝn ! ℝ, i ¼ 1, . . ., m, and hi : ℝn ! ℝ, i ¼ 1, . . ., p, are continuously

differentiable functions. Usually, the function f is called the objective function. Each of the constraints
ci(x)  0, i ¼ 1, . . ., m, is called an inequality constraint, and each hi(x) ¼ 0, i ¼ 1, . . ., p, is called an
equality constraint. Often, (11.1) is called a nonlinear program. A vector x satisfying all the equality
and inequality constraints is called a feasible solution (point) to the problem (11.1). Define
X ¼ fx : ci ðxÞ  0, i ¼ 1, . . . , m, hi ðxÞ ¼ 0, i ¼ 1, . . . , pg

as the feasible region (or feasible domain).

In this chapter, we are interested to specify what is meant by optimality for the general nonlinear
optimization problem and to give conditions under which a solution for the problem (11.1) exists.
Both necessary and sufficient conditions for optimality are presented, starting with unconstrained
problems and continuing with problems with inequality constraints and equality constraints and
finally for general nonlinear optimization problems with equality and inequality constraints. We
emphasize that conditions which are satisfied at a local minimizer x are the necessary conditions;

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 383
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_11
384 11 Optimality Conditions for Nonlinear Optimization

conditions which guarantee that x is a local minimizer are the sufficient conditions. The key to
understand the nonlinear optimization is the Karush-Kuhn-Tucker (KKT) optimality conditions. This
is a major result which identifies an algebraic system of equalities and inequalities, which
corresponds to the solution to any nonlinear optimization problem. This system can often be used
to develop algorithms for computing a solution for the problem or to get some additional information
about the sensitivity of the minimum value of the problem subject to changes in the constraints. In
general, many optimization algorithms can be interpreted as methods for numerically solving the
KKT nonlinear system of equations.
In the mathematical optimization, the KKT conditions are first-order necessary conditions for a
solution in nonlinear optimization to be optimal, provided that some regularity conditions are
satisfied. For problems with inequality constraints, the KKT approach generalizes the method of
Lagrange multipliers, which allows only equality constraints. For the development of the KKT
optimality conditions, three possible approaches can be used. One is based on the separation and
support theorems from the convex set theory. Another one uses the penalty functions, and the third
one comes from the theory of Lagrange multipliers. Each of these approaches has its own virtues and
provides its own insights on the KKT Theorem. In this text, we consider the optimality conditions for
the continuous nonlinear optimization (the mathematical programming), using the formalism of
Lagrange.

11.1 General Concepts in Nonlinear Optimization

In the following, we shall present some definitions and results used in the context of nonlinear
programming. At the same time, we shall define a particular class of nonlinear programs, the convex
programming. In this section, X ⊂ ℝ denotes a nonempty set of real numbers.

Definition 11.1 (Upper Bound, Lower Bound). A real number α is called an upper bound for X if
x  α for all x 2 X. The set X is said to be bounded above if it has an upper bound. Similarly, a real
number α is called a lower bound for Xif x  α for all x 2 X. The set X is said to be bounded below if it
has a lower bound.

Definition 11.2 (Least Upper Bound, Greatest Lower Bound). A real number α is called the least
upper bound (or supremum, or sup) of X, if (i) α is an upper bound for X; and (ii) there does not exist
an upper bound for X that is strictly smaller than α. The supremum, if it exists, is unique and is
denoted by supX. A real number α is called the greatest lower bound (or infimum, or inf) of X, if (i) α
is a lower bound forX; and (ii) there does not exist a lower bound forX that is strictly greater than α.
The infimum, if it exists, is unique and is denoted by infX.

It is worth saying that for sups and infs, the following equivalent definition is useful.

Definition 11.3 (Supremum, Infimum). The supremum of X, provided it exists, is the least upper
bound for X, i.e., a real number α, satisfying (i) z  α for any z 2 Xand (ii) for any α < α, there exists
z 2 X such that z > α: Similarly, the infimum of X,provided it exists, is the greatest lower bound for X,
i.e., a real number α satisfying (i) z  α for any z 2 Xand (ii) for any α > α, there exists z 2 X such
that z < α:

Definition 11.4 (Maximum, Minimum). The maximum of a set X is its largest element if such an
element exists. The minimum of a set X is its smallest element if such an element exists.
11.1 General Concepts in Nonlinear Optimization 385

The key differences between the supremum and the maximum concepts are as follows. If a set has a
maximum, then the maximum is also a supremum for this set, but the converse is not true. A finite set
has always a maximum which is also its supremum, but an infinite set does not need to have a
maximum. The supremum of a set X does not need to be an element of the set X itself, but the
maximum of X must always be an element of X.
Concerning the existence of infima and suprema in ℝ, fundamental is the axiom of completeness.
“If a nonempty subset of real numbers has an upper bound, then it has a least upper bound. If a
nonempty set of real numbers has a lower bound, it has a greatest lower bound.” In other words, the
completeness axiom guarantees that, for any nonempty set of real numbers that is bounded above, a
supremum exists (in contrast to the maximum, which may or may not exist).
Let us consider the minimization problem
min ff ðxÞ : x 2 Xg, ð11:2Þ

where X ⊂ ℝn represents the feasible set. Any point x 2 X is a feasible point or an admissible point.
Any point x 2 ℝn\X is called to be infeasible.

Definition 11.5 (Global Minimum, Strict Global Minimum). A point x 2 Xis said to be a global
minimum of f on X if f(x)  f(x) for any x 2 X. A point x 2 Xis said to be a strict global minimum of f
on X if f(x) > f(x) for any x 2 Xwith x 6¼ x.

Definition 11.6 (Global Maximum, Strict Global Maximum). A point x 2 Xis said to be a global
maximum of f on X if f(x)  f(x) for any x 2 X. It is a strict global maximum of f on X if f(x) < f(x) for
any x 2 Xwith x 6¼ x.

The point x is called an optimal solution of the optimization problem. The real number f(x) is
known as the optimal value of the objective function subject to the constraints x 2 X.
Observe the distinction between the minimum/maximum and the infimum/supremum. The value
min{f(x) : x 2 X} must be attained at one or more points x 2 X. On the other hand, the value inf
{f(x) : x 2 X} does not necessarily have to be attained at any points x 2 X. However, if a minimum
(maximum) exists, then its optimal value equals the imfimum (supremum).
If a minimum exists, it is not necessarily unique. That is, there may be a finite number or even an
infinite number of feasible points x that satisfy the inequality f(x)  f(x) for any x 2 X. The notation
arg min ff ðxÞ : x 2 Xg≜fx 2 X : f ðxÞ ¼ inf ff ðxÞ : x 2 Xgg

is reserved for the set of the minima of function fon X,that is, a set in ℝn.

Definition 11.7 (Local Minimum, Strict Local Minimum). A point x 2 X is said to be a local
minimum of f on X if there exists ε > 0 such that f(x)  f(x) for any x 2 B(x, ε) \ X, where B(x, ε) is
the open ball centered at x of radius ε. Similarly, a point x 2 X is said to be a strict local minimum of
f on X if there exists ε > 0 such that f(x) > f(x) for any x 2 B(x, ε)\{x} \ X.

Definition 11.8 (Local Maximum, Strict Local Maximum). A point x 2 X is said to be a local
maximum of f on X if there exists ε > 0 such that f(x)  f(x) for any x 2 B(x, ε) \ X. Similarly, it is
said to be a strict local maximum of f on X if there exists ε > 0 such that f(x) < f(x) for any x 2 B(x, ε)
\{x} \ X.
386 11 Optimality Conditions for Nonlinear Optimization

A fundamental problem in optimizing a function on a given set is whether a minimum or a

maximum point exists in the given set. This result is known as the theorem of Weierstrass. It shows
that if X is nonempty, closed, and bounded, and f is continuous on X, then a minimum of f on X exists.

Theorem 11.1 (Weierstrass). Let X be a nonempty and compact set. Assume that f : X ! ℝ is
continuous on X. Then, the problem min{f(x) : x 2 X} attains its minimum.

Proof If f is continuous on X and X is both closed and bounded, it follows that f is bounded below on
X. Now, since X is nonempty, from the axiom of completeness, there exists a greatest lower bound
α ¼ inf {f(x) : x 2 X}. Let 0 < ε < 1 and consider the set Xk ¼ {x 2 X : α  f(x)  α þ εk}, k ¼ 1, 2, . . .
By the definition of the infimum, for each k it follows that Xk 6¼ ∅ . Therefore, a sequence of points
{xk} ⊂ X can be constructed by selecting a point xk for each k ¼ 1, 2, . . . Since X is bounded, there
exists a convergent subsequence {xk}K ⊂ X indexed by the set K ⊂ ℕ with x as its limit. Since
X is closed, it follows that x 2 X. By continuity of f on X, since α  f(xk)  α þ εk, we have
α ¼ limk!1 , k2Kf(xk) ¼ f(x). Therefore, there exists a solution x 2 X so that f(x) ¼ α ¼ inf {f(x) : x 2 X},
i.e., x is a minimizing solution. ♦

All the hypotheses of this theorem are important. The feasible set must be nonempty; otherwise,
there are no feasible points at which the minimum is attained. The feasible set must be closed, i.e., it
must contain its boundary points. The objective function must be continuous on the feasible set;
otherwise, the limit at a point may not exist or it may be different from the value of the function at that
point. Finally, the feasible set must be bounded; otherwise, even continuous functions can be
unbounded on the feasible set.

Definition 11.9 (Convex Program). Let C be a convex set in ℝn and let f : C ! ℝn be a convex
function on C. Then, min{f(x) : x 2 C} is called a convex optimization problem, or a convex program.

The fundamental result in convex programming is the following theorem.

Theorem 11.2 Let x be a local minimum of a convex program. Then, x is also a global minimum.

Proof If x is a local minimum, then there exists ε > 0 such that f(x)  f(x) for any x 2 B(x, ε). Now,
suppose that x is not a global minimum. Then, there exists y 2 C such that f( y) < f(x). Let λ 2 (0, 1)
be chosen such that the point z ¼ λy þ (1  λ)x 2 B(x, ε). By convexity of C, z 2 C. Therefore,

f ðzÞ  λf ðyÞ þ ð1  λÞf ðx Þ < λf ðx Þ þ ð1  λÞf ðx Þ ¼ f ðx Þ,

which is a contradiction, since x is a local minimum. ♦

11.2 Optimality Conditions for Unconstrained Optimization

Let us consider the problem of minimizing a function f(x) without constraints on the variables x 2 ℝn:

min ff ðxÞ : x 2 ℝn g:

For a given point x 2 ℝn, the optimality conditions determine whether or not a point is a local or a
global minimum of f. To formulate the optimality conditions, it is necessary to introduce some
concepts which characterize an improving direction along which the values of the function fdecrease.
11.2 Optimality Conditions for Unconstrained Optimization 387

Definition 11.10 (Descent Direction). Suppose that f : ℝn ! ℝ is continuous at x. A vector d 2 ℝn is

a descent direction for f at x if there exists δ > 0 such that f(x þ λd ) < f(x) for any λ 2 (0, δ). The
cone of descent directions at x, denoted by Cdd(x), is given by
Cdd ðx Þ ¼ fd : there exists δ > 0 such that f ðx þ λdÞ < f ðx Þ, for any λ 2 ð0, δÞg:

Assume that f is a differentiable function. To get an algebraic characterization for a descent

direction for f at x let us define the set
n o
C0 ðx Þ ¼ d : ∇f ðx ÞT d < 0 :

The following result shows that every d 2 C0(x) is a descent direction at x.

Proposition 11.1 (Algebraic Characterization of a Descent Direction). Suppose that f : ℝn ! ℝ is

continuously differentiable at x. If there exists a vector d such that ∇f(x)Td < 0, then d is a descent
direction for f at x, i.e., C0(x) ⊆ Cdd(x).

Proof Since f is continuously differentiable at x, it follows that

f ðx þ λdÞ ¼ f ðx Þ þ λ∇f ðx ÞT d þ λkdkoðλdÞ,

where limλ ! 0o(λd) ¼ 0.Therefore,

f ðx þ λd Þ  f ðx Þ
¼ ∇f ðx ÞT d þ kdkoðλdÞ:
λ
Since ∇f(x)Td < 0 and limλ ! 0o(λd) ¼ 0, it follows that there exists a δ > 0 such that
∇f(x)Td þ kdko(λd) < 0 for all λ 2 (0, δ). ♦

Theorem 11.3 (First-Order Necessary Conditions for a Local Minimum). Suppose that f : ℝn ! ℝ is
continuously differentiable at x. If x is a local minimum, then ∇f(x) ¼ 0.

Proof Suppose that ∇f(x) 6¼ 0. If we consider d ¼  ∇ f(x), then ∇f(x)Td ¼  k∇f(x)k2 < 0. By
Proposition 11.1, there exists a δ > 0 such that for any λ 2 (0, δ), f(x þ λd) < f(x). However, this is in
contradiction with the assumption that x is a local minimum for f. ♦

Observe that the above necessary condition represents a system of n algebraic nonlinear equations.
All the points x which solve the system ∇f(x) ¼ 0 are called stationary points. Clearly, the stationary
points need not all be local minima. They could very well be local maxima or even saddle points. In
order to characterize a local minimum, we need more restrictive necessary conditions involving the
Hessian matrix of the function f.

Theorem 11.4 (Second-Order Necessary Conditions for a Local Minimum). Suppose that f : ℝn ! ℝ
is twice continuously differentiable at point x. If x is a local minimum, then ∇f(x) ¼ 0 and ∇2f(x) is
positive semidefinite.

Proof Consider an arbitrary direction d. Then, using the differentiability of f at x, we get
388 11 Optimality Conditions for Nonlinear Optimization

1
f ðx þ λdÞ ¼ f ðx Þ þ λ∇f ðx ÞT d þ λ2 dT ∇2 f ðx Þd þ λ2 kdk2 oðλd Þ,
2
where limλ!0o(λd) ¼ 0. Since x is a local minimum, ∇f(x) ¼ 0. Therefore,
f ðx þ λd Þ  f ðx Þ 1 T 2 
¼ d ∇ f ðx Þd þ kdk2 oðλdÞ:
λ2 2

Since x is a local minimum, for λ sufficiently small, f(x þ λd)  f(x). For λ ! 0, it follows from
the above equality that dT∇2f(x)d  0. Since d is an arbitrary direction, it follows that ∇2f(x) is
positive semidefinite. ♦

In the above theorems, we have presented the necessary conditions for a point x to be a local
minimum, i.e., these conditions must be satisfied at every local minimum solution. However, a point
satisfying these necessary conditions need not be a local minimum. In the following theorems, the
sufficient conditions for a global minimum are given, provided that the objective function is convex
on ℝn (see Appendix A).

Theorem 11.5 (First-Order Sufficient Conditions for a Strict Local Minimum). Suppose that
f : ℝn ! ℝ is continuously differentiable at x and convex on ℝn. If ∇f(x) ¼ 0, then x is a global
minimum of f on ℝn.

Proof Since f is convex on ℝn and continuously differentiable at x, then from the property of convex
functions given by the Proposition A4.5, it follows that for any x 2 ℝn f(x)  f(x) þ ∇ f(x)T(x  x).
However, x is a stationary point, i.e., f(x)  f(x) for any x 2 ℝn. ♦

The following theorem gives the second-order sufficient conditions characterizing a local mini-
mum point for those functions which are strictly convex in a neighborhood of the minimum point.

Theorem 11.6 (Second-Order Sufficient Conditions for a Strict Local Minimum). Suppose that
f : ℝn ! ℝ is twice continuously differentiable at point x. If ∇f(x) ¼ 0 and ∇2f(x) is positive
definite, then x is a local minimum of f.

Proof Since f is twice continuously differentiable, for any d 2 ℝn, we can write
1
f ðx þ d Þ ¼ f ðx Þ þ ∇f ðx ÞT d þ dT ∇2 f ðx Þd þ kdk2 oðdÞ,
2
where limd!0o(d ) ¼ 0. Let λ be the smallest eigenvalue of ∇2f(x). Since ∇2f(x) is positive definite, it
follows that λ > 0 and dT∇2f(x)d  λkdk2. Therefore, since ∇f(x) ¼ 0, we can write
 
λ
f ðx þ d Þ  f ðx Þ  þ oð d Þ kd k2 :
2

Since limd!0o(d ) ¼ 0, then there exists a η > 0 such that |o(d )| < λ/4 for any d 2 B(0, η). Hence,
λ
f ðx þ dÞ  f ðx Þ  kd k2 > 0
4
for any d 2 B(0, η)\{0}, i.e., x is a strict local minimum of function f. ♦
11.3 Optimality Conditions for Problems with Inequality Constraints 389

If we assume f to be twice continuously differentiable, we observe that, since ∇2f(x) is positive

definite, ∇2f(x) is positive definite in a small neighborhood of x and so f is strictly convex in a small
neighborhood of x. Therefore, xis a strict local minimum, that is, it is the unique global minimum
over a small neighborhood of x.

11.3 Optimality Conditions for Problems with Inequality Constraints

In the following, we shall discuss the nonlinear optimization problems with inequality constraints:

min f ðxÞ
subject to ð11:3Þ
x 2 X,

where X is a general set. Further on, we will be more specific and will define the problem as to
minimize f(x) subject to c(x)  0, where c(x) is the vector of constraint functions.

Definition 11.11 (Feasible direction). Let X be a nonempty set in ℝn.A nonzero vector d 2 ℝn is a
feasible direction at x 2 cl(X) if there exists a δ > 0 such that x þ ηd 2 X for any η 2 (0, δ).
Moreover, the cone of feasible directions at x, denoted by Cfd(x), is given by
Cfd ðx Þ≜fd 6¼ 0, there is δ > 0 such that x þ ηd 2 X, for any η 2 ð0, δÞg:

Clearly, a small movement from x along the direction d 2 Cfd(x) leads to feasible points. On the
other hand, a similar movement along a direction d 2 C0(x) (see Definition 11.10) leads to solutions
which improve the value of the objective function. The following theorem, which gives a geometrical
interpretation of the local minima, shows that a necessary condition for local optimality is that every
improving direction is not a feasible direction.

Theorem 11.7 (Geometric Necessary Condition for a Local Minimum). Let X be a nonempty set in
ℝn, and let f : ℝn ! ℝ be a continuously differentiable function. Suppose that xis a local minimum of
the problem (11.3). Then, C0(x) \ Cfd(x) ¼ ∅ .

Proof Suppose that there exists a nonzero vector d 2 C0(x) \ Cfd(x). By the Proposition 11.1 of the
algebraic characterization of a descent direction, there exists δ1 > 0 such that f(x þ ηd) < f(x) for any
η 2 (0, δ1). On the other hand, by Definition 11.11 of feasible direction, there exists δ2 > 0 such that
x þ ηd 2 Xfor any η 2 (0, δ2). Therefore, there exists x 2 B(x, η) \ X such that f(x þ ηd) < f(x), for
every η 2 (0, min{δ1, δ2}), which contradicts the assumption that x is a local minimum of f on X (see
Definition 11.7). ♦

So far, we have obtained a geometric characterization of the optimality condition for the problem
(11.3) given by Theorem 11.7 where Cfd(x) is the cone of feasible directions. To get a practical
optimality condition, implementable in computer programs, we need to convert this geometric
condition into an algebraic one. For this, we introduce the concept of active constraints at x and
define a cone Cac(x) ⊆ Cfd(x) in terms of the gradients of these active constraints. Now, we specify
the feasible set X as
390 11 Optimality Conditions for Nonlinear Optimization

X≜fx : ci ðxÞ  0, i ¼ 1, . . . , mg, ð11:4Þ

where ci : ℝn ! ℝ, i ¼ 1, . . ., m, are continuous functions. Define the vector c(x) ¼ [c1(x), . . ., cm(x)].

Definition 11.12 (Active Constraint, Active Set). Let ci : ℝn ! ℝ, i ¼ 1, . . ., m, which define the
feasible set X ¼ {x : ci(x)  0, i ¼ 1, . . ., m}, and consider x 2 X a feasible point. For each i ¼ 1, . . .,
m, the constraint ci is said to be active or binding at x if ci(x) ¼ 0. It is said to be inactive at x if
ci(x) < 0. The set
Aðx Þ≜fi : ci ðx Þ ¼ 0g

denotes the set of active constraints at x.

The following proposition gives an algebraic characterization of a feasible direction showing the
relation between a cone Cac(x) expressed in terms of the gradients of the active constraints and the
cone of the feasible directions.

Proposition 11.2 (Algebraic Characterization of a Feasible Direction). Let ci : ℝn ! ℝ,

i ¼ 1, . . ., m, be continuously differentiable functions, and consider the feasible set X ¼ {x : ci(x)  0,
i ¼ 1, . . ., m}. For any feasible point x 2 X, we have
n o
Cac ðx Þ≜ d : ∇ci ðx ÞT d < 0, i 2 Aðx Þ ⊆ Cfd ðx Þ:

Proof Suppose that Cac(x) is a nonempty set. Let d 2 Cac(x). Observe that ∇ci(x)Td < 0 for each
i 2 A(x). Therefore, by Proposition 11.1, the algebraic characterization of a descent direction, it
follows that d is a descent direction for ci at x, i.e., there exists δ2 > 0 such that ci(x þ ηd ) < ci(x) ¼ 0
for any η 2 (0, δ2) and for any i 2 A(x). On the other hand, since ci is differentiable at x, it follows
that it is continuous at x. Therefore, since ci(x) < 0 and ci is continuous at x for each i 2 = A(x), there
 
exists δ1 > 0 such that ci(x þ ηd) < 0 for any η 2 (0, δ1) and for any i 2 = A(x ). Besides, for all
η 2 (0, min{δ1, δ2}), the points x þ ηd 2 X. Therefore, by Definition 11.11 of feasible direction,
d 2 Cfd(x). ♦

Remark 11.1 From the Theorem 11.7, we know that C0(x) \ Cfd(x) ¼ ∅. However, from
Proposition 11.2, we have that Cac(x) ⊆ Cfd(x). Therefore, C0(x) \ Cac(x) ¼ ∅, for any local
optimal solution x. ♦

The above geometric characterization of the local optimal solution (see Theorem 11.7) holds either
at the interior points intX ≜ {x 2 ℝn : ci(x) < 0, i ¼ 1, . . ., m}, or at the boundary points. For the
interior points, any direction is feasible, and the necessary condition C0(x) \ Cac(x) ¼ ∅ reduces to
the very well-known condition ∇f(x) ¼ 0, which is identical to the necessary optimal condition for
the unconstrained optimization (see Theorem 11.3).
It is important to notice that the condition C0(x) \ Cac(x) ¼ ∅ can be satisfied by non-optimal
points, i.e., this condition is necessary but not sufficient for a point x to be a local minimum of the
function f on X. For example, at any point x for which ∇ci(x) ¼ 0, for an arbitrary index i 2 A(x), the
condition C0(x) \ Cac(x) ¼ ∅ is trivially satisfied.
In the following, in order to get an algebraic necessary optimality condition to be used in
numerical computation, we want to transform the geometric necessary optimality condition
C0(x) \ Cac(x) ¼ ∅ to a statement in terms of the gradient of the objective function and the
11.3 Optimality Conditions for Problems with Inequality Constraints 391

gradient of the constraints. Thus, the first-order optimality conditions, known as the Karush-Kuhn-
Tucker (KKT) necessary conditions, are obtained. In order to formulate the KKT conditions, we need
to introduce the concept of regular point and of KKT point.

Definition 11.13 (Regular Point–Inequality Constraints). Let ci : ℝn ! ℝ, i ¼ 1, . . ., m, be

continuously differentiable functions, and consider the feasible set X ¼ {x 2 ℝn : ci(x)  0, i ¼ 1, . . ., m}.
A point x 2 X is a regular point if the gradient vectors ∇ci(x), i 2 A(x), are linear independent, i.e.,
rank½∇ci ðx Þ, i 2 Aðx Þ ¼ card ðAðx ÞÞ:

Definition 11.14 (KKT Point). Let f : ℝn ! ℝ and ci : ℝn ! ℝ, i ¼ 1, . . ., m, be continuously

differentiable functions. Consider the problem min{f(x) : c(x)  0}. If a point (x, μ) 2 ℝn  ℝmsatisfies
the algebraic conditions:

∇f ðx Þ þ ðμ ÞT ∇cðx Þ ¼ 0, ð11:5Þ

μ  0, ð11:6Þ

cðx Þ  0, ð11:7Þ

ðμ ÞT cðx Þ ¼ 0: ð11:8Þ

then (x, μ) is called a KKT point.

In Definition 11.14, the scalars μi, i ¼ 1, . . ., m, are called the Lagrange multipliers. The first
condition (11.5) is known as the primal feasibility condition. The conditions (11.6) and (11.7) are
known as dual feasibility conditions. The last condition (11.8), expressed as μi ci ðx Þ ¼ 0, i ¼ 1, . . .,
m, is the complementarity slackness (or transversality) condition.
We are now in the position to present the KKT necessary conditions for the optimality of the
nonlinear optimization problem with inequality constraints. For this, a very useful result is given by
the Theorem of Gordan (see Theorem A4.3). This is extensively used in the derivation of the
optimality conditions of linear and nonlinear programming problems.

Theorem 11.8 (KKT Necessary Conditions). Let f : ℝn ! ℝ and ci : ℝn ! ℝ, i ¼ 1, . . ., m, be

continuously differentiable functions. Consider the problem min{f(x) : c(x)  0}. If x is a local
minimum and a regular point of the constraints, then there exists a unique vector μ such that (x, μ)
is a KKT point.

Proof We know that x is an optimal solution for the problem min{f(x) : c(x)  0}. Therefore, using
the Remark 11.1, no direction d 2 ℝn exists such that ∇f(x)Td < 0 and ∇ci(x)Td < 0, for any i 2 A(x),

are simultaneously satisfied. Now, let A 2 ℝðcardðAðx ÞÞþ1Þn be the matrix whose rows are ∇f(x)T
 T 
and ∇ci(x ) , i 2 A(x ). By the Gordan Theorem (see Theorem A4.3), there exists a nonzero vector
  
p ¼ u0 , u1 , . . . , ucardðAðx ÞÞ  0 in ℝcardðAðx ÞÞþ1 such that ATp ¼ 0. Therefore,
X
u0 ∇f ðx Þ þ ui ∇ci ðx Þ ¼ 0,
i2Aðx Þ

 
where u0  0 and ui  0 for i 2 A(x) and u0 , u1 , . . . , ucardðAðx ÞÞ is not the vector zero. Considering
= A(x), the following conditions are obtained:
ui ¼ 0 for all i 2
392 11 Optimality Conditions for Nonlinear Optimization

u0 ∇f ðx Þ þ uT ∇cðx Þ ¼ 0,
uT cðx Þ ¼ 0,
u0  0, u  0,
ðu0 , uÞ 6¼ ð0, 0Þ,

where u is the vector with components ui for i ¼ 1, . . ., m, some of them being u0 , u1 , . . . , ucardðAðx ÞÞ
and the others being zero. Observe that u0 6¼ 0, because otherwise the assumption that the gradient of
the active constraints are linear independent at x is not satisfied. Now, considering the vector μ as
the vector u whose components are divided by u0, we get that (x, μ) is a KKT point. ♦

The above theorem shows the importance of the active constraints. A major difficulty in applying
this result to get optimization algorithms for solving (11.3) with (11.4) is that we do not know in
advance which constraints are active and which are inactive at the solution of the problem. In other
words, we do not know the active set. The majority of algorithms for solving this optimization
problem with inequalities face this difficulty of identifying the active set. Of course, the idea of
investigating all possible active sets of a problem in order to get the points satisfying the KKT
conditions is usually impractical. More exactly, this difficulty may be detailed as follows:

Combinatorial Difficulty for Solving the Inequality Constrained Optimization

In direct parlance, the difficulty of inequality constrained optimization problems lies in deciding
which of constraints are active at the solution and which are not. One approach used in the active-set
methods starts by making a guess of the optimal active set A(x), that is, the set of constraints that are
satisfied as equalities at the solution x. This set is called the working set and is denoted by W. With
this, solve the problem in which the constraints from the working set are imposed as equalities and the
constraints which are not in W are ignored. Then, check to see if there is a choice of the Lagrange
multipliers such that the solution x obtained for this W satisfies the KKT conditions. If so, then x is
accepted as a local solution of the problem. Otherwise, a different choice of the working set W is
considered, and then, the above procedure is repeated. This approach is based on the observation that,
in general, it is easier to solve equality constrained problems than nonlinear programs with
inequalities. Observe that the number of choices for the working set W may be very large, up to
2m, where m is the number of inequality constraints. Since the number of the possible working sets
grows exponentially with the number of inequalities, this phenomenon is called the combinatorial
difficulty of the inequality constraints optimization. Obviously, to design a practical algorithm by
considering all the possible choices for W is impossible.

A different approach for solving inequality constrained problems which avoid the combinatorial
difficulty of nonlinear optimization with inequalities is given by the interior point (or barrier) methods
(see Chap. 17). These methods generate iterates that stay away from the boundary of the feasible set
defined by the inequality constraints. While the solution of the nonlinear optimization problem is
being approached, the barrier effects are weakened, thus allowing an increasingly accurate estimation
of the solution to be obtained. Thus, interior point methods avoid the combinatorial difficulty of the
inequality constrained optimization.

Remark 11.2 (Constrained Qualification). Observe that not every local minimum is a KKT point.
For a local minimum x to be a KKT point, an additional condition must be introduced on the
behavior of the constraints. Such a condition is known as the constraint qualification. Observe that the
first-order Taylor series expansion of the functions defining the problem about x is used to form an
11.4 Optimality Conditions for Problems with Equality Constraints 393

approximate problem in which both the objective and the constraints are linear. If, near the current
point x, the linearization is fundamentally different from the feasible set, then the linear approxima-
tion of the problem does not yield useful information about the original problem. Therefore, certain
assumptions about the nature of the constraints ci that are active at x must be introduced to ensure that
near x, the linearized approximation is similar to the feasible set. Given the point x and the active set
A(x),the linear independence constraint qualification (LICQ) holds if the set of active constraint
gradients {∇ci(x), i 2 A(x)} is linear independent. In Theorem 11.8, such a constraint qualification is
that x is a regular point, which is also known as the linear independence constraint qualification. The
Lagrange multipliers are guaranteed to be unique in Theorem 11.8 if LICQ holds. In general, if LICQ
holds, none of the active constraint gradients can be zero. Another (weaker) constraint qualification is
the Mangasarian-Fromovitz constraint qualification (MFCQ). It requires that there exists (at least)
one direction d 2 Cac(x), i.e., such that ∇ci(x)Td < 0, for each i 2 A(x). The MFCQ is weaker than
LICQ, i.e., the Lagrange multipliers are guaranteed to be unique if LICQ holds, while this uniqueness
property may be lost under MFCQ. Finally, observe that the constraint qualifications are sufficient
conditions for the linear approximation of the problem to be adequate, they are not necessary
conditions. ♦

In the following theorem, we present a sufficient condition which guarantees that any KKT point
of an inequality constrained nonlinear optimization problem is a global minimum of the problem. Of
course, this result is obtained under the convexity hypothesis.

Theorem 11.9 (KKT Sufficient Conditions). Let f : ℝn ! ℝ and ci : ℝn ! ℝ, i ¼ 1, . . ., m, be

convex and continuously differentiable functions. Consider the problem min{f(x) : c(x)  0}. If
(x, μ) is a KKT point, then x is a global minimum of the problem.
P 
Proof Let us define the function LðxÞ≜f ðxÞ þ m i¼1 μi ci ðxÞ: Since f and ci, i ¼ 1, . . ., m, are convex

functions and μi  0, i ¼ 1, . . ., m, it follows that L is also convex. Now, the dual feasibility
conditions determine that ∇L(x) ¼ 0. Therefore, by Theorem 11.5, x is a global minimum for L on
ℝn, i.e., L(x)  L(x) for any x 2 ℝn. Therefore, for any x such that ci(x)  ci(x) ¼ 0, i 2 A(x), it
follows that
X
f ðx Þ  f ðx  Þ   μi ½ci ðxÞ  ci ðx Þ  0:
i2Aðx Þ

On the other hand, the set {x 2 ℝn : ci(x)  0, i 2 A(x)} contains the feasible set {x 2 ℝn : ci(x)  0,
i ¼ 1, . . ., m}. Therefore, x is a global minimum for the problem with inequality constraints. ♦

11.4 Optimality Conditions for Problems with Equality Constraints

In this section, the nonlinear optimization problem with equality constraints is considered:
min f ðxÞ
subject to ð11:9Þ
hi ðxÞ ¼ 0, i ¼ 1, . . . , p,

where f : ℝn ! ℝ and hi : ℝn ! ℝ, i ¼ 1, . . ., p, are continuously differentiable functions. The

functions hi(x) ¼ 0, i ¼ 1, . . ., p, of the above problem define the vector h(x) ¼ [h1(x), . . .hp(x)]. If x
394 11 Optimality Conditions for Nonlinear Optimization

satisfies the constraints from (11.9), i.e., hi(x) ¼ 0, i ¼ 1, . . ., p, it is said to be feasible. Otherwise, it
is called infeasible.
The optimality of x can be seen as a balance between the function minimization and the constraint
satisfaction. A move away from xcannot be made without either violating a constraint or increasing
the value of the objective function. Formally, this can be stated as the following proposition.

Proposition 11.3 (Balance Between Function and Constraints). If x is a solution of (11.9) and
x þ δx is a nearby point, then:

1. If f(x þ δx) < f(x), then hi(x þ δx) 6¼ 0 for some i.

2. If h1(x þ δx) ¼ . . . ¼ hp(x þ δx) ¼ 0, then f(x þ δx)  f(x). ♦

In order to establish the optimality conditions for the nonlinear optimization problems with
equality constraints, we need to introduce some relevant concepts. An equality constraint h(x) ¼ 0
defines in ℝn a set which can be viewed as a hypersurface. When there are p equality constraints
hi(x) ¼ 0, i ¼ 1, . . ., p, then their intersection defines a (possible empty) set:
X≜f x 2 ℝn : hi ðxÞ ¼ 0, i ¼ 1, . . . , pg:

If the functions defining the equality constraints are continuously differentiable, then the set X is
said to be a differentiable manifold, or a smooth manifold.
Now, in any point on a differentiable manifold, the tangent set can be defined as follows. A curve η
on a manifold X is a continuous application η : I ⊂ ℝ ! X, i.e., a family of points η(t) 2 X
continuously parameterized by t in the interval I ⊂ ℝ. Clearly, a curve passes through the point x if
x ¼ η(t) for some t 2 I. The derivative of a curve at t, if it exists, is defined in a classical manner as
ηðt þ ξÞ  ηðt Þ
η_ ðt Þ≜ lim ξ!0 :
ξ

A curve is differentiable or smooth if a derivative exists for each t 2 I.

Definition 11.15 (Tangent Set). Let X be a differentiable manifold in ℝn and a point x 2 X. Consider
the collection of all the continuously differentiable curves on X passing through x.Then, the
collection of all the vectors tangent to these curves at x is the tangent set to X at x, denoted by
TX(x).

Definition 11.16 (Regular Point–Equality Constraints). Let hi : ℝn ! ℝ, i ¼ 1, . . ., p, be continu-

ously differentiable functions on ℝn, and consider the set X ≜ {x 2 ℝn : hi(x) ¼ 0, i ¼ 1, . . ., p}.
A point x 2 X is a regular point if the gradient vectors ∇hi(x), i ¼ 1, . . ., p, are linearly independent,
i.e.,
 
rank ∇h1 ðx Þ, . . . , ∇hp ðx Þ ¼ p: ð11:10Þ

If the constraints are regular in the sense of the above definition, then X is a subspace of dimension
n  p. In this case, TX(x) is a subspace of dimension n  p, called tangent space. At regular points,
the tangent space can be characterized in terms of the gradients of the constraints (Luenberger, 1973).
11.4 Optimality Conditions for Problems with Equality Constraints 395

Proposition 11.4 (Algebraic Characterization of a Tangent Space). Let hi : ℝn ! ℝ, i ¼ 1, . . ., p, be

continuously differentiable functions on ℝn, and consider the set X ≜ {x 2 ℝn : hi(x) ¼ 0, i ¼ 1, . . ., p}.
At a regular point x 2 X, the tangent space is such that
n o
T X ðx Þ ¼ d : ∇hðx ÞT d ¼ 0 : ð11:11Þ

Proof Let TX(x) be the tangent space at x and M(x) ¼ {d : ∇ h(x)Td ¼ 0}. Consider any curve η(t)
passing through x at t ¼ t, having derivative η_ ðt Þ such that ∇hðx ÞT η_ ðt Þ 6¼ 0: Since such a curve
would not lie on X, it follows that TX(x) ⊂ M(x). Now to prove that TX(x) ⊃ M(x), we must show
that if d 2 M(x), then there is a curve on X passing through x with derivative d. In order to construct
such a curve, we consider the equations
 
h x þ td þ ∇hðx ÞT uðtÞ ¼ 0,

where for fixed t the vector u(t) 2 ℝp is unknown. Observe that we have a nonlinear system of
pequations with p unknowns, continuously parameterized by t. At t ¼ 0, there is a solution u(0) ¼ 0.
The Jacobian matrix of the above system with respect to u at t ¼ 0 is the matrix ∇h(x) ∇ h(x)T which
is nonsingular, since ∇h(x) is of full rank if x is a regular point. Thus, by the implicit function
theorem (see Theorem A2.6), there is a continuous solution u(t) for a  t  a. The curve
η(t) ¼ x þ td þ ∇ h(x)Tu(t) by construction is a curve on X. By differentiating the above nonlinear
system with respect to t at t ¼ 0, we get

d 
0 ¼ hðηðtÞÞ ¼ ∇hðx ÞT d þ ∇hðx Þ∇hðx ÞT u_ ð0Þ:
dt t¼0

By definition of d, we have ∇h(x)Td ¼ 0. Therefore, since ∇h(x) ∇ h(x)T is nonsingular, it

follows that u_ ð0Þ ¼ 0: Therefore, η_ ð0Þ ¼ d þ ∇hðx ÞT u_ ð0Þ ¼ d and the constructed curve has
derivative d at x. ♦

The Method of Lagrange Multipliers

Let us now present the optimality conditions for the nonlinear optimization problems with equality
constraints by using the method of Lagrange multipliers. The idea is to restrict the search of a
minimum of (11.9) to the manifold X ≜ {x 2 ℝn : hi(x) ¼ 0, i ¼ 1, . . ., p}. The following theorem
gives the geometric necessary condition for a local minimum of a nonlinear optimization problem
with equality constraints. It is shown that the tangent space TX(x) at a regular local minimum point x
is orthogonal to the gradient of the objective function at x.

Theorem 11.10 (Geometric Necessary Condition for a Local Minimum). Let f : ℝn ! ℝ and
hi : ℝn ! ℝ, i ¼ 1, . . ., p, be continuously differentiable functions. Suppose the x is a local minimum
point of the problem min{f(x) : h(x) ¼ 0}. Then, ∇f(x) is orthogonal to the tangent space TX(x), i.e.,

C0 ðx Þ \ T X ðx Þ ¼ ∅:

Proof Assume that there exists a d 2 TX(x) such that ∇f(x)Td 6¼ 0. Let η : I ¼ [a, a] ! X, a > 0, be
any smooth curve passing through x, with η(0) ¼ x and η_ ð0Þ ¼ d: Also let φ be the function defined
as φ(t) ≜ f(η(t)) for any t 2 I. Since x is a local minimum of f on X ≜ {x 2 ℝn : h(x) ¼ 0}, by
396 11 Optimality Conditions for Nonlinear Optimization

Definition 11.7, it follows that there exists δ > 0 such that φ(t) ¼ f(η(t))  f(x) ¼ φ(0) for any
t 2 B(0, δ) \ I. Therefore, t ¼ 0 is an unconstrained local minimum point for φ, and

0 ¼ ∇φð0Þ ¼ ∇f ðx ÞT η_ ð0Þ ¼ ∇f ðx ÞT d:

However, this is in contradiction with the assumption that ∇f(x)Td 6¼ 0. ♦

The conclusion of this theorem is that if x is a regular point of the constraints h(x) ¼ 0 and a local
minimum point of f subject to these constraints, then all d 2 ℝn satisfying ∇h(x)Td ¼ 0 must also
satisfy ∇f(x)Td ¼ 0.
The following theorem shows that this property that ∇f(x) is orthogonal to the tangent space
implies that ∇f(x) is a linear combination of the gradients of hi(x), i ¼ 1, . . ., p, at x. This relation
leads to the introduction of Lagrange multipliers and the Lagrange function.

Theorem 11.11 (First-Order Necessary Optimality Conditions). Let f : ℝn ! ℝ and hi : ℝn ! ℝ,

i ¼ 1, . . ., p, be continuously differentiable functions. Consider the problem min{f(x) : h(x) ¼ 0}. If
x is a local minimum and it is a regular point of the constraints, then there exists a unique vector
λ 2 ℝp such that

∇f ðx Þ þ ∇hðx ÞT λ ¼ 0: ð11:12Þ

Proof Since x is a local minimum of f on X ¼ {x 2 ℝn : h(x) ¼ 0},by Theorem 11.10, it follows

that C0(x) \ TX(x) ¼ ∅ ,i.e., the system

∇f ðx ÞT d < 0, ∇hðx ÞT d ¼ 0,

is inconsistent. Now, consider the following two sets:

n o
C1 ≜ ðz1 , z2 Þ 2 ℝpþ1 : z1 ¼ ∇f ðx ÞT d, z2 ¼ ∇hðx ÞT d ,

C2 ≜ ðz1 , z2 Þ 2 ℝpþ1 : z1 < 0, z2 ¼ 0 :

Observe that C1 and C2 are convex sets and C1 \ C2 ¼ ∅ . Therefore, by the separation of two
convex sets, given by Proposition A4.2, there exists a nonzero vector (μ, λ) 2 ℝp þ 1 (μ 2 ℝ, λ 2 ℝp)
such that for any d 2 ℝn and for any (z1, z2) 2 C2
h i
μ∇f ðx ÞT d þ λT ∇hðx ÞT d  μz1 þ λT z2 :

Now, considering z2 ¼ 0 and having in view that z1 can be made an arbitrary large negative
number, it follows that μ  0. Additionally, considering (z1, z2) ¼ (0, 0), we must have
[μ ∇ f(x) þ λT ∇ h(x)]Td  0, for any d 2 ℝn. In particular, letting d ¼  [μ ∇ f(x) þ λT ∇ h(x)],
it follows that kμ ∇ f(x) þ λT ∇ h(x)k2  0, and thus,

μ∇f ðx Þ þ λT ∇hðx Þ ¼ 0, with ðμ, λÞ 6¼ ð0, 0Þ:

11.4 Optimality Conditions for Problems with Equality Constraints 397

Observe that μ > 0, for otherwise the above relation would contradict the assumption that ∇hi(x),
i ¼ 1, . . ., p, are linear independent. The conclusion of the theorem follows letting λ ¼ λ/μ and noting
that the linear independence assumption implies the uniqueness of the λ. ♦

Remark 11.3 The first-order necessary optimality conditions given by the Theorem 11.11 together
with the constraints of the problem (11.9):

∇f ðx Þ þ ∇hðx ÞT λ ¼ 0, ð11:13aÞ

hðx Þ ¼ 0, ð11:13bÞ

represent a total of n þ p nonlinear equations in the variables (x, λ). These conditions determine, at
least locally, a unique solution (x, λ). However, as in the unconstrained case, a solution to the first-
order necessary optimality conditions does not have to be a local minimum of the problem (11.9). ♦

Definition 11.17 (Lagrange multipliers). The scalars λ1 , . . . , λp in (11.12) are called the Lagrange
multipliers.

Definition 11.18 (Constraint normals). The vectors ∇h1(x), . . ., ∇ hp(x) are called the constraint
normals.

The condition (11.12) shows that ∇f(x) is linearly dependent on the constraint normals. Therefore,
a constrained minimum occurs when the gradients of the objective function and the gradients of the
constraints interact in such a way that any reduction in f can only be obtained by violating the
constraints (see Proposition 11.3).

Definition 11.19 (Lagrange function–Lagrangian). The function L : ℝn  ℝp ! ℝ associated to the

nonlinear optimization problem (11.9) is defined as

Lðx, λÞ≜f ðxÞ þ λT hðxÞ: ð11:14Þ

Remark 11.4 (Regularity Assumption). For a local minimum to satisfy the above first-order neces-
sary conditions (11.13) and, in particular, for unique Lagrange multipliers to exist, it is necessary that
the equality constraints hi(x) ¼ 0, i ¼ 1, . . ., p, satisfy a regularity condition. As already seen, for a
local minimum of an inequality constrained nonlinear optimization problem to be a KKT point, a
constrained qualification is needed. For the equality constrained nonlinear optimization problems, the
condition that the minimum point is a regular point corresponds to linear independence constrained
qualification. ♦

If xis a regular local minimum of (11.9), then the first-order necessary optimality conditions
(11.13) can be rewritten as
∇x Lðx , λ Þ ¼ 0, ð11:15aÞ

∇λ Lðx , λ Þ ¼ 0: ð11:15bÞ

Observe that the second condition (11.15b) is a restatement of the constraints. The solution of the
optimization problem (11.9) corresponds to a saddle point of the Lagrangian.
398 11 Optimality Conditions for Nonlinear Optimization

Theorem 11.12 (Second-Order Necessary Optimality Conditions). Let f : ℝn ! ℝ and hi : ℝn ! ℝ,

i ¼ 1, . . ., p, be continuously differentiable functions. Consider the problem min{f(x) : h(x) ¼ 0}.
If x is a local minimum and it is a regular point of the constraints, then there exists a unique
vector λ 2 ℝp such that:

∇f ðx Þ þ ∇hðx ÞT λ ¼ 0, ð11:16Þ

and
 
d T ∇ 2 f ð x  Þ þ ∇2 h ð x  Þ T λ  d  0 ð11:17Þ

for any d 2 ℝn such that ∇h(x)Td ¼ 0.

Proof The first condition ∇f(x) þ ∇ h(x)Tλ ¼ 0 follows from the Theorem 11.11. Now we
concentrate to the second condition. Let x be a regular point and consider d an arbitrary direction
from TX(x), i.e., ∇h(x)Td ¼ 0. Let η : I ¼ [a, a] ! X, a > 0, be an arbitrary twice-differentiable
curve passing through x with η(0) ¼ x and η_ ð0Þ ¼ d: Consider φ a function defined as φ(t) ≜ f(η(t)),
for any t 2 I. Since x is a local minimum of f on X ≜ {x 2 ℝn : h(x) ¼ 0}, it follows that t ¼ 0 is an
unconstrained local minimum point for φ. Therefore, by Theorem 11.4, it follows that

∇2 φð0Þ ¼ η_ ð0ÞT ∇2 f ðx Þη_ ð0Þ þ ∇f ðx ÞT €ηð0Þ  0:

On the other hand, differentiating the relation h(η(t))Tλ ¼ 0 twice, we get

   T
η_ ð0ÞT ∇2 hðx ÞT λ η_ ð0Þ þ ∇hðx ÞT λ €ηð0Þ ¼ 0:

Now, adding the last two relations we obtain

 
dT ∇2 f ðx Þ þ ∇2 hðx ÞT λ d  0,

which must hold for every d such that ∇h(x)Td ¼ 0. ♦

The above theorem says that if TX(x) is the tangent space to X at x, then the matrix
∇2xx Lðx , λ Þ ¼ ∇2 f ðx Þ þ ∇2 hðx ÞT λ is positive semidefinite on TX(x).

Remark 11.5 (Feasible Directions and Second-Order Conditions) An n-vector d is said to be a

feasible direction at x if ∇h(x)d ¼ 0, where ∇h(x) is the Jacobian of the constraints at x. Let us
assume that d is a feasible direction normalized so that kdk ¼ 1. Considering the Taylor’s expansion:
 
hðx þ εd Þ ¼ hðx Þ þ ε∇hðx Þd þ O kεdk2 ,

then h(x þ εd) ¼ O(ε2). Therefore, a move away from x along d keeps the constraints satisfied to
first-order accuracy. In particular, if all the constraints in (11.9) are linear, then x þ εd is a feasible
point for all ε > 0. On the other hand, if any of the hi(x) in (11.9) are nonlinear, then d is a direction
tangential to the constraints at x. It is easy to see that the condition (11.12) implies that, for any
feasible direction d,
11.4 Optimality Conditions for Problems with Equality Constraints 399

dT ∇f ðx Þ ¼ 0:

To distinguish a minimum from a maximum or a saddle-point, the second-order optimality

condition must be used. These conditions can be stated as follows:

1. If the constraint functions hi are all linear, the second-order condition that guarantees x is a
minimum of problem (11.9) is

dT ∇2 f ðx Þd > 0

for any feasible direction d.

2. If the constraint functions hi are nonlinear, the second-order condition that guarantees x is a
minimum of problem (11.9) is

dT ∇2 Lðx , λ Þd > 0

for any feasible direction d. ♦

Remark 11.6 (Eigenvalues in Tangent Space). Geometrically, the restriction of the matrix
 
∇2xx Lðx , λ Þ to TX(x) corresponds to the projection PT X ðx Þ ∇2xx Lðx , λ Þ : A vector y 2 TX(x) is
 2 
an eigenvector of the projection PT X ðx Þ ∇xx Lðx , λ Þ if there is a real number ν such that
 
PT X ðx Þ ∇2xx Lðx , λ Þ y ¼ νy:

 
The real number ν is called the eigenvalue of PT X ðx Þ ∇2xx Lðx , λ Þ : To obtain a matrix represen-
 
tation for PT X ðx Þ ∇2xx Lðx , λ Þ , it is necessary to introduce a basis of the tangent subspace TX(x). It is
best to introduce an orthonormal basis, say E ¼ [e1, . . ., en  p]. Any vector y 2 TX(x) can be written
as y ¼ Ez, where z 2 ℝnp. Now, ∇2xx Lðx , λ ÞEz represents the action of ∇2xx Lðx , λ Þ on such a
vector. To project this result back into TX(x) and to express the result in terms of the basis E ¼ [e1,
. . ., enp], it is necessary to multiply by ET. Therefore, ET ∇2xx Lðx , λ ÞEz is the vector whose
components give the representation in terms of the basis E. The (n  p)  (n  p) matrix
ET ∇2xx Lðx , λ ÞE is the matrix representation of ∇2xx Lðx , λ Þ restricted to TX(x). The eigenvalues
of ∇2xx Lðx , λ Þ restricted to TX(x) can be determined by computing the eigenvalues of
ET ∇2xx Lðx , λ ÞE: These eigenvalues are independent of the particular choice of the basis E. ♦

Recall that the conditions given in Theorems 11.11 and 11.12 are necessary conditions. These
must hold at each local minimum point. However, a point satisfying these conditions may not be a
local minimum. As in the unconstrained case, it is possible to derive second-order conditions for
constrained optimization problems. The following theorem provides sufficient conditions for a
stationary point of the Lagrange function to be a local minimum.

Theorem 11.13 (Second-Order Sufficient Conditions). Let f : ℝn ! ℝ and hi : ℝn ! ℝ, i ¼ 1, . . ., p,

be twice continuously differentiable functions. Consider the problem min{f(x) : h(x) ¼ 0}. If x
and λ satisfy
400 11 Optimality Conditions for Nonlinear Optimization

∇x Lðx , λ Þ ¼ 0, ð11:18aÞ

∇λ Lðx , λ Þ ¼ 0, ð11:18bÞ

and

yT ∇2xx Lðx , λ Þy > 0 ð11:19Þ

for any y 6¼ 0 such that ∇h(x)Ty ¼ 0, then x is a strict local minimum.

Proof Consider the augmented Lagrange function:

c
Lðx, λÞ ¼ f ðxÞ þ λT hðxÞ þ khðxÞk2 ,
2
where c is a scalar. Clearly,
 
∇x Lðx, λÞ ¼ ∇x L x, λ ,
 
∇2xx Lðx, λÞ ¼ ∇2xx L x, λ þ c∇hðxÞT ∇hðxÞ,

where λ ¼ λ þ chðxÞ: Since (x, λ) satisfy the sufficient conditions, by the Theorem A4.4, we obtain
that ∇x Lðx , λ Þ ¼ 0 and ∇2xx Lðx , λ Þ > 0, for sufficiently large c. L being positive definite at
(x, λ), it follows that there exist ρ > 0 and δ > 0 such that
ρ
Lðx, λ Þ  Lðx , λ Þ þ kx  x k2
2
for kx  xk < δ. Besides, since Lðx, λ Þ ¼ f ðxÞ when h(x) ¼ 0, we get
ρ
f ðxÞ  f ðx Þ þ kx  x k2
2
if h(x) ¼ 0, kx  xk < δ, i.e., x is a strict local minimum. ♦

Sensitivity: Interpretation of the Lagrange Multipliers

The i-th Lagrange multiplier can be viewed as the measuring of the sensitivity of the objective
function with respect to the i-th constraint, i.e., how much the optimal value of the objective function
would change if that constraint was perturbed.
At the very beginning, let us consider p ¼ 1, i.e., the problem (11.9) has one constraint h1(x) ¼ 0.
Now, suppose that x is a local solution of the problem
min ff ðxÞ : h1 ðxÞ ¼ 0g,

and consider the perturbed problem

min ff ðxÞ : h1 ðxÞ ¼ δg,

where δ is a known scalar. If the solution of the perturbed problem is x þ u, then by using the
Taylor’s expansion, a first-order estimate of the optimum function value is f(x þ u)  f(x) þ uT ∇ f(x).
However, the optimality condition for the original problem given by (11.12) states that ∇f ðx Þ ¼
λ1 ∇h1 ðx Þ, where λ1 is the Lagrange multiplier. Hence,
11.4 Optimality Conditions for Problems with Equality Constraints 401

f ðx þ uÞ  f ðx Þ  λ1 uT ∇h1 ðx Þ:

Since x þ u solves the perturbed problem, it follows that h1(x þ u) ¼ δ, hence

h1(x) þ uT ∇ h1(x)  δ. However, h1(x) ¼ 0. Therefore, uT ∇ h1(x)  δ, that is,
f ðx þ uÞ  f ðx Þ  δλ1 :

In other words, the Lagrange multiplier is an approximate measure of the change in the objective
function that will occur if a unit amount is added to the right-hand side of the constraint. In general,
we have the following theorem.

Theorem 11.14 (Interpretation of the Lagrange Multipliers). Consider the family of problems min
{f(x) : h(x) ¼ w}, where f : ℝn ! ℝ and h : ℝn ! ℝp are twice continuously differentiable. Suppose
for w ¼ 0 there is a local solution x that is a regular point and that, together with its associated
Lagrange multiplier vector λ, satisfies the second-order sufficient conditions for a strict local
minimum. Then, for every w 2 ℝp in a region containing 0, there is a x(w), depending continuously
on w, such that x(0) ¼ x and such that x(w) is a local minimum of the problem. Furthermore,
∇w f ðxðwÞÞjw¼0 ¼ λ:

Proof Consider the system of equations:

∇f ðxÞ þ ∇hðxÞT λ ¼ 0,

hðxÞ ¼ w:

By hypothesis, when w ¼ 0, there is a solution x, λ to this system. The Jacobian matrix of this
system, at this solution, is
" #
Lðx Þ ∇hðx ÞT
,
∇hðx Þ 0

where L(x) ¼ ∇2f(x) þ ∇2h(x)Tλ.

Since xis a regular point and L(x) is positive definite on {y : ∇ h(x)Ty ¼ 0}, it follows that this
matrix is nonsingular. Thus, by the implicit function theorem (see Theorem A2.6), there is a solution
x(w), λ(w) to the system which is twice continuously differentiable. Therefore,

∇w f ðxðwÞÞjw¼0 ¼ ∇f ðx ÞT ∇w xð0Þ,

∇w hðxðwÞÞjw¼0 ¼ ∇hðx Þ∇w xð0Þ:

However, since h(x) ¼ w, it follows that ∇h(x)∇wx(0) ¼ I. On the other hand, from
∇f(x) þ ∇ h(x)Tλ ¼ 0, it follows that ∇wf(x(c))|w ¼ 0 ¼  λ. ♦
402 11 Optimality Conditions for Nonlinear Optimization

11.5 Optimality Conditions for General Nonlinear Optimization Problems

We now present a generalization of Theorems 11.8, 11.11, 11.12, and 11.13 to nonlinear optimization
problems with equality and inequality constraints:

min f ðxÞ
subject to ð11:20Þ
ci ðxÞ  0, i ¼ 1, . . . , m,
hj ðxÞ ¼ 0, j ¼ 1, . . . , p,

where x 2 ℝn, f : ℝn ! ℝ, ci : ℝn ! ℝ, i ¼ 1, . . ., m, and hj : ℝn ! ℝ, j ¼ 1, . . ., p, are continuously

differentiable functions. Define the vectors c(x) ¼ [c1(x), . . ., cm(x)] and h(x) ¼ [h1(x), . . ., hp(x)].

Remark 11.7 (Discarding the Inactive Constraints). Let us consider the nonlinear optimization
problem min{f(x) : ci(x)  0, i ¼ 1, . . ., m}. Suppose that x is a local minimum point for this
problem. Clearly, x is also a local minimum of the above problem, where the inactive constraints
= A(x) have been discarded. Therefore, the inactive constraints at x can be ignored in the
ci(x)  0, i 2
statement of the optimality conditions. On the other hand, the active constraints can be treated as
equality constraints at a local minimum point. Hence, x is also a local minimum point to the equality
constrained problem:
min ff ðxÞ : ci ðxÞ ¼ 0, i 2 Aðx Þg

The difficulty is that we do not know the set of the active constraints at x.
From Theorem 11.11, it follows that if x is a regular point, there exists a unique Lagrange
multiplier vector μ 2 ℝm such that
X
∇f ðx Þ þ μi ∇ci ðx Þ ¼ 0:
i2Aðx Þ

Now, assigning zero Lagrange multipliers to the inactive constraints, we get

∇f ðx Þ þ ∇cðx ÞT μ ¼ 0,

2Aðx Þ:
μi ¼ 0, i=

Clearly, the last condition can be rewritten as μi ci ðx Þ ¼ 0, i ¼ 1, . . ., m.

It remains to show that μ > 0. For this, assume that μq < 0 for some q 2 A(x). Now, let A 2 ℝ(mþ1)n
be the matrix whose rows are ∇f(x) and ∇ci(x), i ¼ 1, . . ., m. Since x is a regular point, it follows that
the Lagrange multiplier vector μ is unique. Therefore, the condition ATy ¼ 0 can only be satisfied by
y ¼ γ(1 μ)T with γ 2 ℝ. However, μq < 0. Therefore, by Gordan’s Theorem A4.3, there exists a
direction d 2 ℝn such that Ad < 0: In other words, d 2 C0 ðx Þ \ Cac ðx Þ 6¼ ∅, which contradicts
the hypothesis that x is a local minimum of the problem. All these results represent the KKT
optimality conditions as stated by Theorem 11.8. Although this development is straightforward, it
is somewhat limited by the regularity-type assumption at the optimal solution. ♦

Definition 11.20 (Regular Point–General Case). Let ci : ℝn ! ℝ, i ¼ 1, . . ., m, and hj : ℝn ! ℝ,

j ¼ 1, . . ., p, be continuously differentiable functions. Consider the set X ¼ {x 2 ℝn : ci(x)  0, i ¼ 1, . . ., m,
11.5 Optimality Conditions for General Nonlinear Optimization Problems 403

hj(x) ¼ 0, j ¼ 1, . . ., p}. A point x 2 X is a regular point of the constraints from (11.20) if the
gradients ∇ci(x), i 2 A(x), and ∇hj(x), j ¼ 1, . . ., p, are linearly independent. ♦

Definition 11.20 introduces the linear independence constraint qualification (LICQ) for general
nonlinear optimization problems, i.e., the gradients of the active inequality constraints and the
gradients of the equality constraints are all linearly independent at x. Another constraint qualification
is the linear constraint qualification (LCQ), i.e., ci(x), i ¼ 1, . . ., m, and hj : ℝn ! ℝ, j ¼ 1, . . ., p, are
affine functions. Another one is the Slater condition for a convex problem, i.e., there exists a point x
such that ci ðxÞ < 0, i ¼ 1, . . ., m, and hðxÞ ¼ 0:
We emphasize that the constraint qualification ensures that the linearized approximation to the
feasible set X captures the essential shape of X in a neighborhood of x.

Theorem 11.15 (First- and Second-Order Necessary Conditions). Let f : ℝn ! ℝ, ci : ℝn ! ℝ, i ¼ 1,

. . ., m, and hi : ℝn ! ℝ, i ¼ 1, . . .p, be twice continuously differentiable functions. Consider the
problem min{f(x) : c(x)  0, h(x) ¼ 0}. If x is a local minimum for this problem and it is a regular
point of the constraints, then there exist unique vectors μ 2 ℝm and λ 2 ℝp such that

∇f ðx Þ þ ∇cðx ÞT μ þ ∇hðx ÞT λ ¼ 0, ð11:21aÞ

μ  0, ð11:21bÞ

cðx Þ  0, ð11:21cÞ

hðx Þ ¼ 0, ð11:21dÞ

ðμ ÞT cðx Þ ¼ 0, ð11:21eÞ

and
 
yT ∇2 f ðx Þ þ ∇2 cðx ÞT μ þ ∇2 hðx ÞT λ y  0, ð11:22Þ

for all y 2 ℝn such that ∇ci(x)Ty ¼ 0, i 2 A(x) and ∇h(x)Ty ¼ 0. ♦

Proof Observe that since μ  0 and c(x)  0, (11.21e) is equivalent to the statement that a
component of μ is nonzero only if the corresponding constraint is active. Since x is a minimum
point over the constraint set, it is also a minimum over the subset of that set defined by setting the
active constraints to zero. Therefore, for the resulting equality constrained problem defined in a
neighborhood of x,there are Lagrange multipliers. Hence, (11.21a) holds with μi ¼ 0 if ci(x) 6¼ 0.

It remains to show that μ  0. This is a little more elaborate. Suppose that for some k 2 A(x),
μk< 0: Let X and T be the surface and the tangent space, respectively, defined by all the other active
constraints at x. By the regularity assumptions, there is a d such that d 2 T and ∇ck(x)Td < 0. Let η(t)
be a curve on X passing through x at t ¼ 0 with η_ ð0Þ ¼ d: Then, for small t  0, it follows that η(t) is
feasible and

df 
ðηðtÞÞ ¼ ∇f ðx ÞT d < 0
dt t¼0

by (11.21a), which contradicts the fact that xis a minimum point. ♦

404 11 Optimality Conditions for Nonlinear Optimization

The conditions (11.21) are known as the Karush-Kuhn-Tucker conditions, or KKT conditions. The
conditions (11.21e) written as μi ci ðx Þ ¼ 0, i ¼ 1, . . ., m, are the complementary conditions. They
show that either constraint i is active or the corresponding Lagrange multiplier μi ¼ 0, or possibly
both. For a given nonlinear optimization problem (11.20) and a solution point x, there may be many
Lagrange multipliers (μ, λ) for which the conditions (11.21) and ((11.22) are satisfied. However,
when x is a regular point (the LICQ is satisfied), the optimal (μ, λ) is unique.
The KKT conditions motivate the following definition which classifies constraints according to
whether or not their corresponding Lagrange multiplier is zero.

Definition 11.21 (Strongly Active (Binding)–Weakly Active Constraints). Let x be a local solution
to the problem (11.20) and the Lagrange multipliers (μ, λ) which satisfy the KKT conditions
(11.21). We say that an inequality constraint ci(x) is strongly active or binding if i 2 A(x) and the
corresponding Lagrange multiplier μi > 0: We say that ci(x) is weakly active if i 2 A(x) and the
corresponding Lagrange multiplier μi ¼ 0: ♦

Definition 11.22 (Strongly Active Simple Bound Constraints–Binding Simple Bound Constraints).
Let xbe a local solution to the problem min{f(x) : x 2 ℝn, li  xi  ui, i ¼ 1, . . ., n}. Let g ¼ ∇ f(x)
be the gradient of the minimizing function in x. Then, the inequality constraint li  xi  ui is a
strongly active simple bound constraint in x if xi ¼ li and gi  0, or xi ¼ ui and gi  0, where gi is
the i-th component of g. ♦

The non-negativity condition (11.21b) on the Lagrange multiplier for the inequality constraints
ensures that the function f(x) will not be reduced by a move off any of the binding constraints at x to
the interior of the feasible region.
A special case of complementarity is important because it introduces the concept of degeneracy in
optimization.

Definition 11.23 (Strict Complementarity). Let x be a local solution to the problem (11.20) and the
Lagrange multipliers (μ, λ) which satisfy the KKT conditions (11.21). We say that the strict
complementarity holds if exactly one of μi and ci(x) is zero for each index i ¼ 1, . . ., m. In other
words, we have μi > 0 for each i 2 A(x). ♦

Usually, satisfaction of strict complementarity is beneficial for algorithms and makes it easier to
determine the active set A(x) so that convergence is more rapid.

Remark 11.8 (Degeneracy). A property that causes difficulties for some optimization algorithms is
degeneracy. This concept refers to the following two situations:

• The gradients of the active constraints ∇ci(x), i 2 A(x), are linearly dependent at the solution
point x. Linear dependence of the gradients of the active constraints can cause difficulties during
the computation of the step direction because certain matrices that must be factorized become rank
deficient.
• Strict complementarity fails to hold, that is, there is some index i 2 A(x) such that all the Lagrange
multipliers satisfying the KKT conditions (11.21) have μi ¼ 0: In the case when the problem
contains weakly active constraints, it is difficult for an algorithm to determine whether these
constraints are active at the solution. For some optimization algorithms (active-set algorithms and
gradient projection algorithms), the presence of weakly active constraints can cause the algorithm
to zigzag as the iterates move on and off the weakly constraints along the successive iterations. ♦
11.5 Optimality Conditions for General Nonlinear Optimization Problems 405

Theorem 11.16 (Second-Order Sufficient Conditions). Let f : ℝn ! ℝ, ci : ℝn ! ℝ, i ¼ 1, . . ., m, and

hj : ℝn ! ℝ, j ¼ 1, . . .p, be twice continuously differentiable functions. Consider the problem min
{f(x) : c(x)  0, h(x) ¼ 0}. If there exist x, μ, and λ satisfying the KKT conditions (11.21a),
(11.2b), (11.2c), (11.2d), and (11.21e) and

yT ∇2xx Lðx , μ , λ Þy > 0,

for all y 6¼ 0 such that

∇ci ðx ÞT y ¼ 0, i 2 Aðx Þ with μi > 0, ð11:23aÞ

∇ci ðx ÞT y  0, i 2 Aðx Þ with μi ¼ 0, ð11:23bÞ

∇hðx ÞT y ¼ 0, ð11:23cÞ

where L(x, μ, λ) ¼ f(x) þ μTc(x) þ λTh(x), then x is a strict local minimum of the problem.

Proof The theorem says that the Hessian of the Lagrangian is positive definite on the critical cone
C(x, μ, λ) defined by (11.23) for x, μ and λ satisfying the KKT conditions (11.21a), (11.2b),
(11.2c), (11.2d), and (11.21e).
Assume that x is not a strict local minimum, and let {yk} be a sequence of feasible points
converging to x such that f(yk)  f(x). Consider yk of the form yk ¼ x þ δksk with δk > 0 and
kskk ¼ 1. Assume that δk ! 0 and sk ! s. Clearly, ∇f(x)Ts  0 and ∇hj(x)s ¼ 0 for j ¼ 1, . . ., p.
On the other hand, for each active constraint ci, we have ci(yk)  ci(x)  0. Therefore,
∇ci(x)Ts  0.

If ∇ci(x)Ts ¼ 0, for all i 2 l : cl ðx Þ ¼ 0, μl > 0 , then the proof is similar to that in Theorem

11.13. If ∇ci(x)Ts < 0 for at least one i 2 l : cl ðx Þ ¼ 0, μl > 0 , then

0  ∇f ðx ÞT s ¼ λT ∇hðx Þs  μT ∇cðx Þs > 0,

which represents a contradiction. ♦

The KKT sufficient conditions for convex programming with inequality constraints given in
Theorem 11.9 can immediately be generalized to nonlinear optimization problems with convex
inequalities and affine equalities.

Theorem 11.17 (KKT Sufficient Conditions for General Problems). Let f : ℝn ! ℝ and ci : ℝn ! ℝ,
i ¼ 1, . . ., m, be convex and continuously differentiable functions. Also, let hi : ℝn ! ℝ, i ¼ 1, . . ., p,
be affine functions. Consider the problem minf(x) subject to x 2 X ≜ {x 2 ℝn : c(x)  0, h(x) ¼ 0}. If
(x, μ, λ) satisfies the KKT conditions (11.21a)–(11.21e), then x is a global minimum for fon X. ♦

Sensitivity: Interpretation of the Lagrange Multipliers for General Problems

As we have already seen, Theorem 11.14 presents an interpretation of the Lagrange multipliers for
nonlinear optimization problems with equality constraints. Each Lagrange multiplier tells us some-
thing about the sensitivity of the optimal objective function value f(x) with respect to the
corresponding constraint. Clearly, for an inactive constraint i 2 = A(x), the solution x and the function
value f(x) are independent of whether this constraint is present or not. If we slightly perturb ci by a
tiny amount, it will still be inactive, and therefore, x will still be a local solution of the optimization
problem. Since μi ¼ 0 from (11.21e), the Lagrange multiplier shows that the constraint i has no
406 11 Optimality Conditions for Nonlinear Optimization

importance in the system of the constraints. Otherwise, as in Theorem 11.14, the following theorem
can be presented.

Theorem 11.18 (Interpretation of the Lagrange Multipliers for General Problems). Consider the
family of problems min{f(x) : c(x)  v, h(x) ¼ w}, where f : ℝn ! ℝ, c : ℝn ! ℝm and h : ℝn ! ℝp
are twice continuously differentiable. Suppose for v ¼ 0, w ¼ 0 there is a local solution x that is a
regular point and that, together with its associated Lagrange multiplier μ  0, λ, satisfies the
second-order sufficient conditions for a strict local minimum. Then, for every (v, w) 2 ℝm þ p, in a
region containing (0, 0) 2 ℝm þ p, there is a solution x(v, w) continuously depending on (v, w), such
that x(0, 0) ¼ xand such that x(v, w) is a local minimum of the problem. Furthermore,
∇v f ðxðv, wÞÞj0,0 ¼ μ ,

∇w f ðxðv, wÞÞj0,0 ¼ λ : ♦

11.6 Duality

In optimization, the duality theory shows how to construct an alternative problem from functions and
data that define the original optimization problem. In this context, the original problem is called the
primal problem. In some cases, the dual problem is easier to solve than the original problem. Besides,
the dual problem can be used to obtain a lower bound on the optimal value of the objective for the
primal problem.
Let us consider the nonlinear optimization problem (primal problem)
min f ðxÞ subject to cðxÞ  0, ð11:24Þ
x2ℝn

where f : ℝn ! ℝ is a continuously differentiable function and c(x) ≜ [c1(x), . . ., cm(x)]T, with ci(x),
i ¼ 1, . . ., m, are all convex functions. The Lagrangian function for (11.24) is

Lðx, λÞ ¼ f ðxÞ  λT cðxÞ, ð11:25Þ

where λ 2 ℝm is the Lagrange multiplier vector. With this, the dual objective function q : ℝn ! ℝ is
defined as

qðλÞ≜ inf Lðx, λÞ: ð11:26Þ

x

Obviously, for some problems, the infimum in (11.26) is 1 for some values of λ. The domain of
q is defined as the set of λ for which q is finite, that is, D ≜ {λ : q(λ) >  1}. Observe that the
infimum in (11.26) involves finding the global minimizer of the function L(., λ) for a given λ, which is
a very difficult problem. However, when f and ci, i ¼ 1, . . ., m, are all convex functions and λ  0, it
follows that the function L(., λ) is also convex. In this case, all the local minimizers are also global
minimizers. The dual problem of (11.24) is defined as
max qðλÞ subject to λ  0: ð11:27Þ
λ2ℝm

In the following, we show how these problems are related.

11.6 Duality 407

Theorem 11.19 The function q defined by (11.26) is concave and its domain D is convex.

Proof For any λ0 2 ℝm and λ1 2 ℝm, any x 2 ℝn, and any α 2 [0, 1], it follows that
Lðx, ð1  αÞλ0 þ αλ1 Þ ¼ ð1  αÞLðx, λ0 Þ þ αLðx, λ1 Þ:

As we know, the infimum of a sum is greater than or equal to the sum of the infimums. Therefore,
taking the infimum of both sides in the above expression and using (11.26), we obtain
qðð1  αÞλ0 þ αλ1 Þ  ð1  αÞqðλ0 Þ þ αqðλ1 Þ,

showing the concavity of q. Now, if both λ0 and λ1 belong to D, it follows that q((1  α)λ0 þ αλ1)   1.
That is, (1  α)λ0 þ αλ1 2 D, i.e., D is convex. ♦
 
Theorem 11.20 For any xb feasible for (11.24) and any b
λ  0, it follows that q b
λ  f ðxbÞ:

Proof We have b
λ  0 and cðxbÞ  0: Therefore,
 
q bλ ¼ inf f ðx Þ  b
λ cðxÞ  f ðxbÞ  b
T T
x
λ cðxbÞ  f ðxbÞ: ♦

Theorem 11.20, called the weak duality theorem, tells us that value of the dual problem (11.27)
gives a lower bound on the optimal objective value for the primal problem (11.24).
The KKT optimality conditions for the primal problem (11.24) are as follows:

∇f ðxbÞ  ∇cðxbÞb
λ ¼ 0, ð11:28aÞ

cðxbÞ  0, ð11:28bÞ

b
λ  0, ð11:28cÞ

b
λi ci ðxbÞ ¼ 0, i ¼ 1, . . . , m, ð11:28dÞ

where ∇c(x) 2 ℝn  m is the Jacobian matrix defined by ∇c(x) ¼ [∇c1(x), . . ., ∇cm(x)]. The next
theorem, due to Wolfe (1961), shows that the optimal Lagrange multipliers for (11.24) are solutions
of the dual problem (11.27).

Theorem 11.21 Suppose that xb is a solution of (11.24) and that f and ci, i ¼ 1, . . ., m, are convex
 
functions on ℝn that are continuously differentiable. Then, any b
λ for which xb, bλ satisfies the KKT
conditions (11.28) is a solution of the dual problem (11.27).
   
Proof Suppose that xb, bλ satisfies (11.28). Since b
λ  0, it follows that L :, b
λ is a convex and
differentiable function. Therefore, having in view (11.28a), for any x, it follows that
     T  
L x, b
λ  L xb, bλ þ ∇x L xb, b
λ ðx  xbÞ ¼ L xb, b λ :

Now, from (11.28d), we have

408 11 Optimality Conditions for Nonlinear Optimization

     
q b
λ ¼ inf L x, b
λ ¼ L xb, bλ ¼ f ðxbÞ  b
T
λ cðxbÞ ¼ f ðxbÞ:
x

 
From Theorem 11.20, it follows that qðλÞ  f ðxbÞ for all λ  0. Therefore, from q bλ ¼ f ðxbÞ, it
follows that bλ is a solution of the dual problem (11.27). ♦

We have dealt so far with the duality in the Lagrange sense. In the following, we present a slightly
different form of the duality, the so-called the Wolfe duality (1961), which is very convenient for
computations. The Wolfe dual of (11.24) is
max Lðx, λÞ ð11:29aÞ
x, λ
subject to
ð11:29bÞ
∇x Lðx, λÞ ¼ 0, λ  0:

The following theorem shows the relationship between these dual problems.

Theorem 11.22 Suppose that f and ci, i ¼ 1, . . .m, are convex and continuously differentiable on
 
ℝn. Suppose that xb, b λ is a solution of (11.24) at which the linear independence constrained
 
qualification holds. Then, xb, b
λ solves the dual problem (11.29).

 
Proof From the KKT optimality conditions (11.28), it follows that xb, b λ satisfies (11.29b) and
 
L xb, b
λ ¼ f ðxbÞ: Therefore, having in view the convexity of L(., λ), for any pair (x, λ) that satisfies
(11.29b), we have
 
L xb, bλ ¼ f ðxbÞ  f ðxbÞ  λT cðxbÞ ¼ Lðxb, λÞ
 Lðx, λÞ þ ∇x Lðx, λÞT ðxb  xÞ ¼ Lðx, λÞ,
 
showing that xb, bλ maximizes L over the constraints (11.29b), i.e., solvers (11.29). ♦

Example 11.1 For the linear programming problem

min cT x subject to Ax  b, ð11:30Þ

the Lagrange dual is

max bT λ subject to AT λ ¼ c, λ  0: ð11:31Þ

λ

The Wolfe dual is

max cT x  λT ðAx  bÞ subject to AT λ ¼ c, λ  0: ð11:32Þ

λ

Obviously, substituting ATλ  c ¼ 0 into the objective of (11.32), we obtain (11.31). ♦

11.6 Duality 409

Example 11.2 For the quadratic problem

1 T
min x Gx þ cT s subject to Ax  b  0, ð11:33Þ
2
where G is symmetric and positive definite, the dual objective is
1 T
qðλÞ ¼ inf Lðx, λÞ ¼ inf x Gx þ cT s  λT ðAx  bÞ:
x x 2

Since G is positive definite and L(., λ) is a strictly convex quadratic function, it follows that the
infimum is achieved when ∇xL(x, λ) ¼ 0, that is, when Gx þ c  ATλ ¼ 0. Therefore, x ¼ G1(ATλ  c).
Introducing it in the infimum, we get the dual objective as
1 T T  
qð λ Þ ¼  A λ  c G1 AT λ  c þ bT λ:
2

The dual Wolfe of (11.33) is

1 T
max x Gx þ cT x  λT ðAx  bÞ subject to Gx þ c  AT λ ¼ 0, λ  0: ð11:34Þ
λ, x 2

However, (c  ATλ)Tx ¼  xTGx. Therefore, (11.34) can be rewritten as

1
max  xT Gx þ bT λ subject to Gx þ c  AT λ ¼ 0, λ  0, ð11:35Þ
λ, x 2

which is the Wolfe dual of (11.33). ♦

Notes and References

Plenty of books and papers are dedicated to the theoretical developments of the optimality conditions
for continuous nonlinear optimization. Many details and properties of the theoretical aspects of the
optimality conditions and duality can be found in Bertsekas (1999), Nocedal and Wright (2006), Sun
and Yuan (2006), etc. The content of this chapter is based on the books by Chachuat (2007); Bazaraa,
Sherali, and Shetty (1993); Luenberger and Ye (2016); and Nocedal and Wright (2006). A thorough
description of the constraint qualification and duality is given in Mangasarian (1995). As for the
optimality conditions for the problems with inequality constraints, the material is inspired by
Bazaraa, Sherali, and Shetty (1993). The derivation of the necessary and sufficient optimality
conditions for problems with equality constraints follows the developments presented by Luenberger
(1973). The sensitivity analysis and the interpretation of the Lagrange multipliers for nonlinear
optimization are derived from Luenberger (1973). The duality for nonlinear programming is
described by Bertsekas (1999). Our presentation of the duality follows the ideas of Nocedal and
Wright (2006) and Griva, Nash, and Sofer (2009).

The KKT conditions were originally named after Harold W. Kuhn (1925–2014) and Albert
W. Tucker (1905–1995), who first published them in 1951 (Kuhn & Tucker, 1951). Later on, the
scholars discovered that the necessary conditions for this problem had been stated by William Karush
(1917–1997) in his master’s thesis in 1939 (Karush, 1939). Another approach of the optimality
conditions for the nonlinear optimization problem was given in 1948 by Fritz John (1910–1994) (John
1948). Also see Cottle (2012).
 Simple Bound Constrained
Optimization 12

The simple bound constrained optimization is a class of nonlinear optimization problems with a
special structure, found in many real practical applications. The mathematical model of these
problems is as follows:
min f ðxÞ
subject to ð12:1Þ
x 2 X ¼ fx 2 ℝn , li  xi  ui , i ¼ 1, . . . , ng:

The function f : ℝn ! ℝ is supposed to be at least twice continuously differentiable. The set

X defined by the real numbers  1 < li  ui < þ 1 , i ¼ 1, . . ., n, which represents the bounds on the
variables, is the feasibility domain of the problem (12.1). Suppose that for any i ¼ 1, . . ., n, li  ui, that
is, the feasibility domain X is nonempty. Since X is a compact set, it follows that (12.1) always has a
local optimum solution (see Theorem 11.1). The inequalities  1 < li  xi  ui < þ 1 , i ¼ 1, . . ., n,
are called simple bounds. The i-th simple bound constraint is active at x 2 X if xi ¼ li, or xi ¼ ui. In the
first case, if xi ¼ li, we say that the variable xi is assigned to its lower bound. In the second case, if
xi ¼ ui, the variable xi is assigned to its upper bound. The set of active constraints at the point x 2 X is
denoted by A(x). The set of inactive constraints at x 2 X is denoted by I(x). If li ¼ ui, then the i-th
component of x is fixed to the value li ¼ ui.
This problem is often a sub-problem of the augmented Lagrangian or of the penalty computational
schemes for solving the general constrained optimization (Facchinei & Lucidi, 1992a, b; Conn,
Gould, & Toint, 1997a; Nocedal & Wright, 2006; Sun & Yuan, 2006). Therefore, the development of
numerical algorithms to efficiently solve (12.1), especially for large-scale problems, is important in
both theory and practice.
The purpose of this chapter is to present this class of problems, the main aspects of the optimality
conditions, as well as the main computational methods for solving these problems. For this, we follow
the developments given by Kelley (1999), together with some details from Bertsekas (1976, 1982a, b,
1999); Lin and Moré (1999); Birgin and Martı́nez (2001); Birgin, Martı́nez, and Raydan (2000,
2001); Byrd, Lu, and Nocedal (1995a); Byrd, Lu, Nocedal, and Zhu (1994a, 1995b); and Hager and
Zhang (2006a, b). At the same time, we present some computational results, comparisons among the
algorithms, as well as some nonlinear optimization applications with simple bounds. From the
multitude of algorithms dedicated to this problem, we insist on the spectral projected gradient

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 411
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_12
412 12 Simple Bound Constrained Optimization

method, on the limited memory BFGS with simple bounds, and on the truncated Newton with simple
bounds.

12.1 Necessary Conditions for Optimality

As we know, for a continuously differentiable function of one variable, the necessary conditions for
the unconstrained optimality at x are simply f 0(x) ¼ 0, and if f is twice continuously differentiable,
00
f (x)  0. For a simple bound constrained problem in which the variables restrict the domain of f to
an interval [a, b], the necessary condition must be changed in order to admit the possibility for the
minimizer to be one of the endpoints of the interval [a, b]. If x ¼ a is a local minimizer, then
00
f(x)  f(a) for all a  x sufficiently near a. Therefore, f 0(a)  0. Nothing can be said about f .
 0  
Similarly, if x ¼ b is a local minimizer, then f (b)  0. Hence, all three possibilities x ¼ a, x ¼ b,
and a < x < b can be expressed by the following theorem:

Theorem 12.1 Let f be a continuously differentiable function of one variable on the interval
[a, b]. Let x be a local minimum of f on [a, b]. Then,

f 0 ðx Þðx  x Þ  0 for all x 2 ½a, b, ð12:2Þ

and if f is twice continuously differentiable on [a, b],

f 00 ðx Þðx  aÞðb  x Þ  0: ð12:3Þ

A point x 2 X is stationary for the problem (12.1) if

∇f ðx ÞT ðx  x Þ  0 for all x 2 X: ð12:4Þ

As in the unconstrained case, the stationary points are said to satisfy the first-order necessary
conditions.

In order to present the second-order necessary conditions for the problem (12.1), the reduced
Hessian is introduced. Let f be twice continuously differentiable at x 2 X. The reduced Hessian
∇2R f ðxÞ is the matrix
(

2  δij , i 2 AðxÞ, j 2 AðxÞ,
∇R f ðxÞ ij ¼
2  ð12:5Þ
∇ f ðxÞ ij , otherwise:

Theorem 12.2 Let f be twice Lipschitz continuously differentiable and let x be the solution of the
problem (12.1). Then, the reduced Hessian ∇2R f ðx Þ is positive semidefinite.

Proof Assume that at point x there are t inactive indices and n  t active indices. With this, the vector
x 2 X can be partitioned by reordering the variables as x ¼ [z, y], where z corresponds to the inactive
indices and y to the active ones. Then, the map ψ(z) ¼ f(z, y) has an unconstrained local minimizer at
z 2 ℝt, and hence, ∇2ψ is positive semidefinite. However, the reduced Hessian can be written as
12.1 Necessary Conditions for Optimality 413

" #
∇2 ψ ðx Þ
∇2R f ðx Þ ¼
I

if the variables are partitioned as above. ♦

Let P be the projection onto X, i.e., the map that takes x to the nearest point (in the l2 norm) in X to
x. Then,
8
>
< li , if ðxÞi  li ,
PðxÞi ¼ ðxÞi , if li < ðxÞi < ui , ð12:6Þ
>
:
ui , if ðxÞi  ui :

The following theorem proved in Kelley (1999) states the necessary condition for optimality.

Theorem 12.3 Let f be continuously differentiable. A point x 2 X is stationary for the problem
(12.1) if and only if
x ¼ Pðx  α∇f ðx ÞÞ ð12:7Þ

for all α  0. ♦

In a Lagrangian formalism, supposing that no li is 1 and no ui is + 1 , i ¼ 1, . . ., n, then the KKT

conditions for x to solve the problem (12.1) are

∇f ðx Þ  λ þ μ ¼ 0,
λ  0, ðl  x ÞT λ ¼ 0,
μ  0, ðx  uÞT μ ¼ 0,
l  x  u,

where λ, μ 2 ℝn are the KKT multipliers. In this context, the strict complementarity is said to hold at
the KKT point (x, λ, μ) as follows: if xi ¼ li implies λi > 0 and xi ¼ ui implies μi > 0: Another
equivalent way to present the KKT conditions is as follows:

l  x  u,
li < xi < ui ) ∇f ðx Þi ¼ 0,
xi ¼ li ) ∇f ðx Þi  0,
xi ¼ ui ) ∇f ðx Þi  0:

If the projected gradient ∇Pf(x) is defined by

8  

< min 0, ∇f ðx Þi ,
> if xi ¼ li ,
∇P f ðx Þi ≜ ∇f ðx Þi , if li < xi < ui ,
>
:  
max 0, ∇f ðx Þi , if xi ¼ ui ,

then the first-order necessary conditions for optimality can be written as

l  x  u, ∇P f ðx Þ ¼ 0:
414 12 Simple Bound Constrained Optimization

12.2 Sufficient Conditions for Optimality

The sufficient conditions are formulated by using the definition of the reduced Hessian. Observe that
if x is stationary, i 2 I(x), and ei is a unit vector in the i-th coordinate direction, then x  tei 2 X for
all t sufficiently small. Since

df ðx  tei Þ
¼ ∇f ðx ÞT ei  0,
dt
it follows that
ð∇f ðx ÞÞi ¼ 0 for all i 2 I ðx Þ: ð12:8Þ

In order to formulate the sufficient conditions for optimality, the concept of nondegenerate
stationary point is introduced. A point x 2 X is a nondegenerate stationary point for the problem
(12.1) if x is a stationary point and
ð∇f ðx ÞÞi 6¼ 0 for all i 2 Aðx Þ: ð12:9Þ

If x is a solution of the problem (12.1), then x is a nondegenerate local minimizer.

Nondegeneracy is important in the formulation of sufficient conditions and in the design of the
termination criteria. Let S be an arbitrary set of indices. Then, define

xi , i 2 S,
P S ðxÞi ¼ ð12:10Þ
0, 2S:
i=

The following theorem proved in Kelley (1999) gives the sufficiency conditions associated to
problem (12.1).

Theorem 12.4 Let x 2 X be a nondegenerate stationary point for the problem (12.1). Let f be twice
continuously differentiable in a neighborhood of x, and assume that the reduced Hessian at x is
positive definite. Then, x is a solution of the problem (12.1) and, hence, a nondegenerate local
minimizer.

Proof Let x 2 X and define φ(t) ¼ f(x þ t(x  x)). We prove that either (i) φ0(0) > 0 or (ii) φ0(0) ¼ 0,
00
φ (0) > 0. Consider e ¼ x  x. Observe that

φ0 ð0Þ ¼ ∇f ðx ÞT e ¼ ∇f ðx ÞT ðPA ðeÞ þ PI ðeÞÞ:

However, from the stationarity, we have ∇f(x)TPI(e) ¼ 0. Now, if PA(e) 6¼ 0, then, from
nondegeneracy, it follows that ∇f(x)TPA(e) > 0, i.e., (i) holds. On the other hand, if PA(e) ¼ 0, then


φ00 ð0Þ ¼ ðx  x ÞT PI ∇2 f ðx Þ PI ðx  x Þ ¼ ðx  x ÞT ∇2R f ðx Þðx  x Þ > 0,

which proves (ii). ♦

12.3 Methods for Solving Simple Bound Optimization Problems

One of the methods for solving simple bound optimization problems is the active-set method. Polyak
(1969) proposed an extension of the conjugate gradient method to this class of problems with
12.3 Methods for Solving Simple Bound Optimization Problems 415

quadratic objective. In this method, the conjugate gradient method is used to explore a face of the
feasible set, and the negative gradient is used to leave a face. At every iteration, the algorithm of
Polyak only added or dropped one constraint. Dembo and Tulowitzki (1983) proposed an algorithm
which could add and drop many constraints at an iteration. Later, Yang, and Tolle (1991) further
developed Polyak’s algorithm to obtain a finite termination even when the problem was degenerate at
the local minimizer. Another variant of this algorithm, which includes a rigorous convergence
analysis, was given by Wright (1990). Moré and Toraldo (1991) improved this algorithm by
proposing a gradient projection method to identify a working face of the feasible set, followed by a
conjugate gradient method to explore the face. For a simple bound optimization problem, at point
x 2 X, the following two sets L(x) ¼ {i : xi ¼ li} and U(x) ¼ {i : xi ¼ ui} can be defined. Clearly,
A(x) ¼ L(x) [ U(x). If x is a local minimizer of f in X, then x is a local minimizer of f(x) subject to
xi ¼ li, i 2 L(x) and xi ¼ ui, i 2 U(x). The active-set methods aim at predicting L(x) and U(x) by
using in an iterative way the disjoint sets L, U ⊆ {1, . . ., n}. The idea of the active-set method is as
follows: given the sets L and U, the following problem

minf ðxÞ
subject to
xi ¼ li , i 2 L and xi ¼ ui , i 2 U

is approximately solved as an unconstrained optimization problem over the variables xi, i 2 = A ¼ L [ U.

For the quadratic programming case, for which f is quadratic, the active-set methods were developed
by Coleman and Hulbert (1989). The main difficulty with the active-set method is the selection of the
sets L and U in an iterative process either by adding variables which violate one of their bounds or by
removing those for which further progress is predicted.
An active-set algorithm was developed by Hager and Zhang (2006a), which consists of a
nonmonotone gradient projection step, an unconstrained optimization step, and a set of rules for
branching between steps. The implementation of this algorithm uses the cyclic Barzilai-Borwein
algorithm for the gradient projection step and the conjugate gradient algorithm CG-DESCENT by
Hager and Zhang (2005) for the unconstrained optimization step. An attractive feature of this
algorithm is that the search directions are always sufficient descent directions, and when the objective
function is strongly convex quadratic, the convergence of the algorithm is achieved in a finite number
of iterations, even when the strict complementarity slackness does not hold (see Definition 11.14)
(Hager & Zhang, 2006a).
Other methods for solving these problems is the gradient projected (Rosen, 1960; Levitin &
Polyak, 1966; Bertsekas, 1976). These methods are extensions of the steepest-descent method to deal
with convex constraints. The iteration is computed as xkþ1 ¼ PX(xk  αk ∇ f(xk)), where PX(v) projects
v onto X and αk is the stepsize. In the case of simple bounds, PX(v) is trivial to compute. Bertsekas
(1976) showed that the method has the following important feature. For the nondegenerate problems
(i.e., those problems for which the removing of one or more active constraints necessarily changes the
solution), the optimal “face” of the active constraints will be determined in a finite number of
iterations. If the active faces visited by consecutive iterates of the gradient projected algorithm are
identical, then the Newton-like method should be used to investigate this face. Therefore, the
projected Newton method is generated (Bertsekas, 1982a). For the quadratic case, this idea was
considered by Moré and Toraldo (1991). In this case, if the initial point is sufficiently near a
nondegenerate local minimizer x, then the search direction is computed by using the reduced
Hessian. The resulting projected Newton method will take full steps, i.e., αk ¼ 1 along the iterations.
On the other hand, if the initial point is far from x and the reduced Hessian is not symmetric and
positive definite, then the line-search may fail. Clearly, this possibility of indefiniteness is the main
416 12 Simple Bound Constrained Optimization

weakness in any line-search method which uses ∇2f when far from the minimizer. To accelerate the
convergence of these methods, more research has been developed on the Newton and trust-region
methods. For the nondegenerate bound optimization problems, the superlinear and quadratic conver-
gence was established by Conn, Gould, and Toint (1988a, b, 1991a) and Facchinei, Júdice, and Soares
(1998). For the degenerate problems, these convergence results were established by Facchinei,
Lucidi, and Palagi (2002); Friedlander, Martı́nez, and Santos (1994); Lescrenier (1991); and Lin
and Moré (1999). Computing the Newton step can be expensive. Therefore, approximation
techniques, such as the sparse or incomplete Cholesky factorization, have been suggested in order
to reduce the computational expense.
A gradient projection method with the limited memory BFGS matrix to approximate the Hessian of
the objective function was developed by Byrd, Lu, Nocedal, and Zhu (1995b). In this algorithm, the
gradient projection method is used to determine a set of active constraints at each iteration. This is a
line-search algorithm (as opposed to trust-region) that uses the limited memory BFGS matrices.
Another algorithm that uses the limited-memory quasi-Newton methods to update the inactive
variables and a projected gradient method to update the active variables was given by Ni and
Yuan (1997).
A trust-region Newton method is the TRON method (Lin & Moré, 1999). It uses the gradient
projection to generate a Cauchy step, a preconditioned conjugate gradient method with an incomplete
Cholesky factorization to generate a direction, and a projected search to compute the step. Using the
projected searches allows TRON to examine the faces of the feasible domain by generating a small
number of minor iterations.
The affine-scaling interior point method by Coleman and Li (1994, 1996, 1997) represents a
different approach, related to the trust-region method. Some developments of this strategy are
presented in Dennis, Heinkenschlos, and Vicente (1998); Heinkenschlos, Ulbrich, and Ulbrich
(1999); and Ulbrich, Ulbrich, and Heinkenschlos (1999). These methods are characterized by a
reformulation of the necessary optimality conditions obtained by multiplication with a scaling matrix.
The resulting system is solved by using the Newton method. In another way of research, Zhang (2004)
proposes an interior point approach for solving this linear system.
As we have already said, a projected Newton-like method for the solution of (12.1) is the truncated
Newton method for the large-scale box constrained optimization given by Facchinei, Lucidi, and
Palagi (2002). This method is based on the works of Facchinei and Lucidi (1992a, b). At each
iteration k, the estimates L(xk) and U(xk) of the variables are defined that will supposedly be at their
lower and upper bounds at the solution, respectively. At the same time, the estimate F(xk) of the
variables which we believe to be free is computed. This partition of variables suggests performing an
unconstrained minimization in the space of the free variables, a typical approach in the active-set
methods. For a locally fast convergent method, an obvious choice for the unconstrained minimization
in the subspace of the free variables is the Newton method. In order to compensate the loss of the
curvature information that we have in the subspace of those variables which are active but with zero
multiplier, Facchinei, Lucidi, and Palagi (2002) introduced a correction term in the right-hand term of
the Newton system. Other approaches also based on the truncated Newton method are those given by
Schlick and Fogelson (1992a, b) and Nash (1984a, b, 1985).
A nonmonotone spectral projected gradient (SPG) for solving simple bound optimization
problems is the one given by Birgin, Martı́nez, and Raydan (2000, 2001). This algorithm combines
the projected gradient method of Bertsekas (1976) with two new features in optimization. The first
one is based on the nonmonotone line-search developed by Grippo, Lampariello, and Lucidi (1986).
The second one uses the spectral step length introduced by Barzilai and Borwein (1988) and further
analyzed by Raydan (1993, 1997).
12.4 The Spectral Projected Gradient Method (SPG) 417

Another approach that we mention and that was given by Sainvitu and Toint (2006) is the filter-
trust-region method for the simple bound constrained optimization. The algorithm combines the
filter-trust-region algorithm of Gould, Sainvitu, and Toint (2005b) with a gradient projection method.
Finally, a direct method for solving (12.1) is BOBYQA, by Powell (2009), where a quadratic
approximation of the objective function is minimized at each iteration. Recent advances in the simple
bound constrained optimization have been presented by Hager and Zhang (2006b).
In the following, we present the spectral projected gradient method (SPG) by Birgin, Martı́nez, and
Raydan (2000, 2001), the limited-memory BFGS algorithm with gradient projection (L-BFGS-B) by
Byrd, Lu, Nocedal, and Zhu (1995b), and the truncated Newton with simple bounds (TNBC) by Nash
(1984a, b, 1985) for solving simple bound constrained optimization problems.

12.4 The Spectral Projected Gradient Method (SPG)

The gradient projected algorithm and its spectral variant are natural extensions of the steepest descent
algorithm to simple bound constrained problems (Bertsekas, 1976). Let us consider a current iteration
x. The new iteration is computed as

xþ ¼ Pðx  α∇f ðxÞÞ,

where α is the stepsize computed by the Armijo rule or by some other line-search procedures. The
gradient projected algorithm determines a sequence of iterations which satisfies the simple bounds of
the problem (12.1) and achieves a sufficient decrease of the function f. For α > 0, define

xðαÞ ¼ Pðx  α∇f ðxÞÞ: ð12:11Þ

The sufficient decrease condition for the line-search in simple bound constrained problems is
expressed as
ρ
f ðxðαÞÞ  f ðxÞ  kx  xðαÞk2 , ð12:12Þ
α
where ρ is a positive parameter (ρ ¼ 104 (Dennis & Schnabel, 1983)). Therefore, the general
gradient projected algorithm can be presented as the following algorithm.

Algorithm 12.1 Gradient projected for simple bounds

1. Select an initial point x0 and a value for the parameter β 2 (0, 1). Set k ¼ 1
2. Test a criterion for stopping the iterations
3. Compute f(xk) and ∇f(xk)
4. (Armijo rule) Find the least integer m such that (12.12) holds for α ¼ βm
5. Set xkþ1 ¼ x(α), k ¼ k þ 1 and go to step 2 ♦

The spectral projected gradient method is a method of projected gradient which includes two
ingredients. The first one is an extension to the simple bound optimization problem and an extension
of the globalization techniques used in the unconstrained optimization and based on the
non-monotone line-search by Grippo, Lampariello and Lucidi (1986). The second one consists in
using the spectral stepsize introduced by Barzilai and Borwein (1988) and analyzed by Raydan
(1993). Mainly, the scaled gradient projected algorithm is a version of the gradient projected
algorithm by Bertsekas, which uses the Armijo rule along a nonlinear trajectory of projections.
418 12 Simple Bound Constrained Optimization

Consider the problem (12.1) where the function f is defined and has continuous partial derivatives
on an open set that contains X. Assume that f is bounded from below on X. Consider a point x 2 ℝn.
Then, the orthogonal projection of x onto X is PX(x) with
ðPX ðxÞÞi ¼ max fli , min fxi , ui gg, i ¼ 1, . . . , n: ð12:13Þ

For any x 2 X, the algorithm uses the spectral projected gradient defined as
gt ðxÞ ¼ ½PX ðx  t∇f ðxÞÞ  x, ð12:14Þ

where t > 0 is a spectral selection of the stepsize. Observe that zeroing the spectral projected gradient
is equivalent to the optimality conditions of the first order. Therefore, the norm of the spectral
projected gradient is a criterion for stopping the iterations.
The algorithm starts with an initial point x0 2 ℝn and uses an integer m  1 used in the
nonmonotone line-search, a small parameter αmin > 0, a large parameter αmax > αmin, a small
parameter ε1 > 0, a sufficient decrease parameter γ 2 (0, 1), and the safeguarding parameters
0 < σ 1 < σ 2 < 1 used as protection in the quadratic interpolation. Initially, α0 2 [αmin, αmax] is
arbitrary. The step length is computed by the quadratic or cubic interpolation.

Algorithm 12.2 Spectral projected gradient—SPG

1. Initialization. Consider some numerical values for the above described parameters, as well as the initial
point x0 2 ℝn. If x0 2
= X, then set x0 ¼ PX(x0). Set k ¼ 0
2. Compute f(xk) and ∇f(xk)
3. Compute the gradient projected PX(xk  ∇ f(xk))
4. Compute the initial spectral step length
n as n oo
α ¼ min α max , max α min , 1
kPX ðxk ∇f ðxk ÞÞxk k1

5. Test for stopping the iterations. If

kPX(xk  ∇ f(xk))  xkk1  ε1,
stop; otherwise, continue with step 6
6. Compute the spectral direction of the projected gradient as
d ¼ PX(xk  α ∇ f(xk))  xk
7. Initialization of the nonmonotone line-search. Set α ¼ 1 and compute xkþ1 ¼ xk þ αd. Compute f(xkþ1)
8. If
f(xkþ1)  max0  j  min {k, m}{f(xk  j)} þ γα(dT ∇ f(xk)),
then go to step 11; otherwise, continue with step 9, where the quadratic or cubic interpolation is
implemented
9. Quadratic interpolation. If α  0.1, then set α ¼ α/2; otherwise, compute
α2 ðdT ∇f ðxk ÞÞ
at ¼  2 f ðx Þf ðx Þα dT ∇f ðx Þ
ð kþ1 k ð k ÞÞ

If at < σ 1 or at > σ 2α, then set at ¼ at/2. Set α ¼ at. Compute xkþ1 ¼ xk þ αd and go to step 8.
Cubic interpolation. If α ¼ 1, then compute
α2 ðdT ∇f ðxk ÞÞ
αt ¼  2 f ðx Þf ðx Þα dT ∇f ðx Þ
ð kþ1 k ð k ÞÞ

and go to step 10.

Otherwise, compute
p ¼ f(xkþ1)  f(xk)  α(d ∇ f(xk)), q ¼ fp  f(xk)  αp(d ∇ f(xk)),
T T

pαp
a ¼ α2  α2 ααp , b ¼ α2 þ α2 ααp ,
p q 1 qα 1
p p

r ¼ b2  3a(dT ∇ f(xk)).
12.4 The Spectral Projected Gradient Method (SPG) 419

If a ¼ 0, then set
ðd T ∇f ðxk ÞÞ
αt ¼ 2b ,
otherwise,
pffi
αt ¼ bþ
3a :
r

If αt > α/2, then set αt ¼ α/2

10. Set αp ¼ α, fp ¼ f(xkþ1). If αt  α/10, then set α ¼ α/10; otherwise, α ¼ αt. Compute xkþ1 ¼ xk þ αd,
f(xkþ1) and continue with step 8
11. Compute sk ¼ xkþ1  xk, yk ¼ ∇ f(xkþ1)  ∇ f(xk) and determine the spectral step length. If yTk sk  0,
then set α ¼ αmax; otherwise,
n n oo
sT s
α ¼ min α max , max α min , ykT skk ,
k

k ¼ k þ 1 and continue with step 5 ♦

The algorithm is based on the spectral projected gradient direction PX(xk  αk ∇ f(xk))  xk, where





αk is the safeguarded inverse Rayleigh quotient sTk sk = yTk sk : Observe that sTk sk = yTk sk is a
R1
Rayleigh quotient corresponding to the average Hessian matrix ∇2 f ðxk þ tsk Þdt:
0
The line-search uses the protected quadratic interpolation or the cubic interpolation implemented
in steps 9 and 10, according to a value of a parameter, not explained here. The protection of the
quadratic interpolation is given by the parameters σ 1 and σ 2, which are initialized in step 1 of the
algorithm. The quadratic interpolation acts when the minimum of the one-dimensional quadratic
function q(.) defined in such a way that q(0) ¼ f(xk), q(α) ¼ f(xk þ αdk), and ∇qð0Þ ¼ dTk ∇f ðxk Þ lies
outside the interval [σ 1, σ 2α] and not when this minimum lies outside the interval [σ 1α, σ 2α], as
usually implemented. This means that when the interpolation tends to reject 90% (for σ 1 ¼ 0.1) of the
original search interval (let us say [0,1]), then we say that its prediction is not reliable and a more
conservative bisection is preferred. On the other hand, the cubic interpolation considers an approxi-
mation of the minimizing function f by a cubic polynomial and uses four interpolation conditions
based on the function values and its derivatives, each of them computed in two different points.
The convergence of this algorithm is proved by Birgin, Martı́nez, and Raydan (1999).

Theorem 12.5 The algorithm SPG is well defined, and any accumulation point of the sequence it
generates is a constrained stationary point for the problem (12.1). ♦

The proof is based on the results of Bertsekas (1999), which refer to the Armijo rule in the context
of projections. Mainly, from the convexity of the domain X, it follows that for any x 2 X and
t 2 (0, αmax], the following results are true:

(i) ∇f ðxÞT gt ðxÞ   1t kgt ðxÞk22   α max

1
kgt ðxÞk22 ,
(ii) gt ðxÞ is zero if and only if x is a stationary point.

These results are used in Theorem 12.5 to prove that any accumulation point x of the sequence
{xk} generated by the algorithm SPG is a stationary point, i.e., for any x 2 X, it follows that
∇f ðxÞT ðx  xÞ  0:
420 12 Simple Bound Constrained Optimization

Numerical Study—SPG: Quadratic Interpolation versus Cubic Interpolation

The SPG algorithm with both quadratic and cubic interpolation was implemented in double precision
Fortran, compiled with f77 (default compiler settings), and run on a Workstation Intel Pentium 4 with
1.8 GHz. We selected a number of 80 large-scale simple bound optimization test functions in
generalized or extended form presented in the UOP collection (Andrei, 2020a). For each test function,
we considered 10 numerical experiments with the number of variables increasing as n ¼ 1000,
2000, . . ., 10000. Therefore, a set of 800 simple bound optimization problems is obtained. Each
problem is solved with SPG in two variants: SPG with quadratic interpolation (SPGp) and SPG
with cubic interpolation (SPGc). The parameters used in SPG were initialized as m ¼ 10, αmin ¼ 103,
αmax ¼ 103, γ ¼ 104, σ 1 ¼ 0.1, σ 2 ¼ 0.9, ε1 ¼ 0.
Figure 12.1 shows the Dolan and Moré (2002) performance profiles of SPGp versus SPGc, subject
to the CPU time metric. From Fig. 12.1, we can see that, subject to the CPU time metric, SPGp is
more efficient than SPGc, but they have the same robustness for solving this set of simple bound
constrained optimization problems, SPGp being slightly more robust. Comparing SPGp versus SPGc
subject to the number of iterations, we can see that SPGp was better in 255 problems (i.e., it achieved
the minimum number of iterations in solving 255 problems). SPGc was better in 162 problems, and
they achieved the same number of iterations in solving 295 problems. Therefore, subject to the
number of iterations, SPGp is more efficient than SPGc. Similarly, SPGp is faster than SPGc.
Both these algorithms find local optimal solutions. Out of 800 problems considered in this
numerical study, only for 712 problems does the criterion (1.3) hold. The conclusion of this numerical
study is that both these scaled gradient projected algorithms with quadratic or cubic interpolation in
the line-search represent efficient and robust algorithms for solving a large variety of simple bound
optimization problems.

Fig. 12.1 SPG: Quadratic

interpolation versus cubic
interpolation
12.5 L-BFGS with Simple Bounds (L-BFGS-B) 421

12.5 L-BFGS with Simple Bounds (L-BFGS-B)

This section presents a very efficient and reliable algorithm for solving the simple bound optimization
problem (12.1) based on the limited memory BFGS update which approximates the Hessian of the
function f. The algorithm is described by Byrd, Lu, Nocedal, and Zhu (1995b) and is based on the
developments given by Conn, Gould, and Toint (1988a) and Moré and Toraldo (1989). It uses the
gradient projection method to determine a set of active constraints at each iteration. The main
ingredient is the compact representation of the limited memory BFGS matrices described by Byrd,
Nocedal, and Schnabel (1994b).

Description of the algorithm Suppose that at the current iterate xk, the following elements are
known: the value of the objective function fk, the gradient gk ¼ ∇ f(xk), and a positive definite
limited memory approximation Bk of the Hessian ∇2f(xk). Therefore, the following quadratic model of
f can be formed:
1
mk ðxÞ ¼ f ðxk Þ þ gTk ðx  xk Þ þ ðx  xk ÞT Bk ðx  xk Þ: ð12:15Þ
2
The algorithm approximately minimizes mk(x) subject to the simple bounds li  xi  ui, i ¼ 1, . . .,
n. It is done by using the gradient projection method to find a set of active constraints, followed by a
minimization of mk(x), where the bounds are equality constraints.
For this, let us consider the piecewise linear path

xðtÞ ¼ PX ðxk  tg k , l, uÞ

obtained by projecting the steepest descent direction onto the feasible set X, where
8
>
< li , xi < li ,
PX ðx, l, uÞi ¼ xi , li  xi  ui , ð12:16Þ
>
:
ui , x i > ui :

In the following, the generalized Cauchy point xc is computed, which is defined as the first local
minimizer of the univariate piecewise quadratic

qk ðtÞ ¼ mk ðxðtÞÞ:

All the variables whose value at point xc are at the lower or at the upper bound and which form the
active set A(xc) are held fixed. With this, the following quadratic programming problem over the
subspace of free variables
 
min mk ðxÞ : xi ¼ xci , 8i 2 Aðxc Þ
subject to ð12:17Þ
li  xi  ui , 8i=
2Aðxc Þ,

is considered. Firstly, an approximate solution of (12.17) is computed by ignoring the bounds on the
free variables and by using xc as the starting point. After an approximate solution xkþ1 of the problem
(12.17) has been obtained, the new iteration xkþ1 is computed by line-search along the direction
dk ¼ xkþ1  xk using the strong Wolfe line-search conditions
422 12 Simple Bound Constrained Optimization

f ðxkþ1 Þ  f ðxk Þ þ ραk gTk dk , ð12:18Þ

T

g d k
 σ
gT d k
, ð12:19Þ
kþ1 k

where αk is the step length and ρ and σ are positive parameters used in the Wolfe line-search. With all
these elements, the gradient at xkþ1 is evaluated, a new limited-memory Hessian approximation Bkþ1
is computed, and the process is repeated.
Observe that the generalized Cauchy point xc, which is a minimizer of mk(x) on the projected
steepest descent direction, satisfies mk(xk) > mk(xc) if the projected gradient is nonzero. Since xkþ1 is
on the path from xc to the minimizer of (12.17), along which mk decreases, it follows that the value of
mk at xkþ1 is not larger than its value at xc, i.e.,
1
f ðxk Þ ¼ mk ðxk Þ > mk ðxc Þ  mk ðxkþ1 Þ ¼ f ðxk Þ þ gTk dk þ d Tk Bk dk : ð12:20Þ
2
Therefore, if Bk is positive definite and dk is not zero, the inequality (12.20) implies that gTk d k < 0:
In conclusion, since in our algorithm every Hessian approximation Bk is positive definite, it follows
that the approximate solution xkþ1 of the quadratic problem (12.17) defines a descent direction
dk ¼ xkþ1  xk for the objective function f.

Limited-memory BFGS updates The limited memory BFGS matrices used in the algorithm are
represented in compact form. At every iteration xk, the algorithm stores a small number, let us say m,
of correction pairs {si, yi}, i ¼ k  1, . . ., k  m, where sk ¼ xkþ1  xk and yk ¼ gkþ1  gk. These
correction pairs contain information about the curvature of the function f, and in the frame of the
BFGS formula, they define the limited-memory iteration matrix Bk. In the following, as described in
Byrd, Nocedal, and Schnabel (1994b), we shall represent these matrices without explicitly
forming them.

Firstly, the following n  m correction matrices are formed:

Y k ¼ ½ykm , . . . , yk1 , Sk ¼ ½skm , . . . , sk1 : ð12:21Þ

Now, if θ is a positive scaling parameter and if the m correction pairs {si, yi}, i ¼ k  1, . . ., k  m,
satisfy the condition yTi si > 0, i ¼ k  1, . . ., k  m, then the matrix obtained by updating θI m-times
using the BFGS updating formula and the pairs {si, yi}, i ¼ k  1, . . ., k  m, can be written as

Bk ¼ θI  W k Mk W Tk , ð12:22Þ

where
W k ¼ ½Y k θSk , ð12:23Þ
" #1
Dk LTk
Mk ¼ , ð12:24Þ
Lk θSTk Sk

and where Lk and Dk are the m  m matrices

 T
ykm1þj skm1þi , i > j,
ðLk Þij ¼ i, j ¼ 1, . . . , m, ð12:25Þ
0, i  j,
12.5 L-BFGS with Simple Bounds (L-BFGS-B) 423

 
Dk ¼ diag yTkm skm , . . . , yTk1 sk1 : ð12:26Þ

Since Mk is a 2m  2m matrix and since m is chosen to be a small integer, the cost of computing the
inverse in (12.24) is negligible. The compact representation (12.22) is very efficient in numerical
computations and involves the product of Bk with a vector, which often occurs in the algorithm.
In this context, as it is explained in Byrd, Nocedal, and Schnabel (1994b), there is a similar
representation of the inverse limited-memory BFGS matrix Hk that approximates the inverse Hessian
matrix
1 T
H k ¼ I þ W k Mk W k , ð12:27Þ
θ
where
h i
1
Wk ¼ Y Sk , ð12:28Þ
θ k
2 3
0 R1
k
Mk ¼ 4   5, ð12:29Þ
1
RT RT Dk þ Y Tk Y k R1
k k θ k


yTkm1þj skm1þi , i  j,
ðRk Þij ¼ ð12:30Þ
0, i > j:

Since the bounds on variables possibly prevent the line-search from satisfying the second Wolfe
condition (12.19), there is no guarantee that the curvature condition yTk sk > 0 always holds. Therefore,
in order to maintain the positive definiteness of the limited-memory BFGS matrix, if the curvature
condition yTk sk > εkyk k2 is not satisfied for a small positive constant ε, then the correction pairs
(sk, yk) are discarded.

The generalized Cauchy point A very important element in the frame of the L-BFGS-B algorithm is
the computation of the generalized Cauchy point as the first local minimizer of the quadratic model
along the piecewise linear path obtained by projecting the points along the steepest descent direction
xk  tgk onto the feasible domain. For this, define x0 ¼ xk and drop the index k, so that x, g, and B stand
for xk, gk, and Bk, respectively. However, subscripts are used to denote the component of a vector; for
example, gi is the i-th component of g. Superscripts will be used to represent the iterates during the
piecewise search for the Cauchy point.
To define the breakpoints in each coordinate direction, the following elements are computed

8

<
xi  ui =gi ,
> gi < 0,
0

ti ¼ x0i  li =gi , gi > 0, i ¼ 1, . . . , n, ð12:31Þ

>
:
1, gi ¼ 0,

and sorted in an increasing order to get the ordered set {t j : t j  t jþ1, j ¼ 1, . . ., n}. In the following,
a search along PX(x0  tg, l, u) is done; thus a piecewise linear path expressed as
424 12 Simple Bound Constrained Optimization

(
x0i  tg i , t  ti ,
xi ðt Þ ¼ i ¼ 1, . . . , n, ð12:32Þ
x0i  ti gi , t > ti ,

is obtained. Now, suppose that the interval [t j1, t j] is examined. Let us define the ( j  1)-th
breakpoint as x j1 ¼ x(t j1), such that on [t j1, t j] we have x(t) ¼ x j1þΔtd j1, where Δt ¼ t  t j1
and

gi , tj1 < ti ,
dj1
i ¼ i ¼ 1, . . . , n: ð12:33Þ
0, tj1  ti ,

With this notation, on the line segment [x(t j  1), x(t j)], the quadratic (12.15) can be written as
1
T

mðxÞ ¼ f þ gT ðx  x0 Þ þ x  x0 B x  x0
2 ð12:34Þ

 1
T

¼ f þ gT zj1 þ Δtd j1 þ zj1 þ Δtd j1 B zj1 þ Δtd j1 ,
2
where

zj1 ¼ xj1  x0 : ð12:35Þ

Therefore, on the line segment, [x(t j  1), x(t j)] m(x) can be written as a quadratic in Δt
   
1
T
T
mbðΔtÞ ¼ f þ gT zj1 þ zj1 Bzj1 þ gT d j1 þ d j1 Bzj1 Δt
2
 
1
j1 T j1
þ d Bd Δt2 ð12:36Þ
2
1
¼ f j1 þ f 0j1 Δt þ f 00j1 Δt2 ,
2
where
1
j1 T j1
f j1 ¼ f þ gT zj1 þ z Bz , ð12:37Þ
2

T
f 0j1 ¼ gT dj1 þ dj1 Bzj1 , ð12:38Þ

T
f 00j1 ¼ dj1 Bd j1 : ð12:39Þ

From the equation dm bðΔtÞ=dΔt ¼ 0 , we get Δt ¼ f 0j1 =f 00j1 : Since B is positive definite, it
follows that this represents a minimizer, provided that t j1 þ Δt lies on [t j1, t j). Otherwise, the
generalized Cauchy point lies at x(t j1) if f 0j1  0, or beyond it, or even at x(t j) if f 0j1 < 0:
If, after exploring the interval [t j1, t j], the generalized Cauchy point has not been found, we set

xj ¼ xj1 þ Δtj1 d j1 , Δtj1 ¼ tj  tj1 ð12:40Þ

and update the directional derivatives f 0j and f 00j as the search moves to the next interval. As in Byrd,
Lu, Nocedal, and Zhu (1995b), let us assume that only one variable becomes active at t j, and let us
denote its index by b. Then, tb ¼ t j and we zero out the corresponding component of the search
direction as
12.5 L-BFGS with Simple Bounds (L-BFGS-B) 425

dj ¼ d j1 þ gb eb , ð12:41Þ

where eb is the b-th unit vector. From (12.34) and (12.40), it follows that

zj ¼ zj1 þ Δtj1 dj1 : ð12:42Þ

Therefore, using (12.38), (12.39), (12.41), and (12.42), we get

T
f 0j ¼ gT dj þ d j Bzj

T
T
¼ gT dj1 þ g2b þ d j1 Bzj1 þ Δtj1 dj1 Bdj1 þ gb eTb Bzj ð12:43Þ
¼ f 0j1 þ Δtj1 f 00j1 þ g2b þ gb eTb Bzj

and

T
f 00j ¼ dj Bdj

T
¼ dj1 Bd j1 þ 2gb eTb Bdj1 þ g2b eTb Beb ð12:44Þ
¼ f 00j1 þ 2gb eTb Bd j1
þ g2b eTb Beb :

The only expensive computations in (12.43) and (12.44) are eTb Bzj , eTb Bd j1 , and eTb Beb , which
may require O(n) operations since B is a dense limited-memory matrix. Therefore, it seems that the
computation of the generalized Cauchy point could require O(n2) operations, since, in the worst case,
n segments of the piecewise linear path can be examined. For large-scale problems, this cost would be
prohibitive. However, by using the limited-memory BFGS formulae (12.22) and (12.33), the updating
formulae (12.43) and (12.44) become

f 0j ¼ f 0j1 þ Δtj1 f 00j1 þ g2b þ θgb zjb  gb wTb MW T zj , ð12:45Þ

f 00j ¼ f 00j1  θg2b  2gb wTb MW T dj1  g2b wTb Mwb , ð12:46Þ

where wTb stands for the b-th row of the matrix W. The only O(n) operations remaining in (12.45) and
(12.46) are WTz j and WTd j1. However, from (12.41) and (12.42), it follows that z j and d j are updated
at every iteration by a simple computation. Therefore, if we store the two 2m-dimensional vectors


pj ≜W T dj ¼ W T dj1 þ gb eb ¼ pj1 þ gb wb , ð12:47Þ


cj ≜W T zj ¼ W T zj1 þ Δtj1 dj1 ¼ cj1 þ Δtj1 pj1 , ð12:48Þ

then, by means of the expressions

f 0j ¼ f 0j1 þ Δtj1 f 00j1 þ g2b þ θgb zjb  gb wTb Mcj , ð12:49Þ

f 00j ¼ f 00j1  θg2b  2gb wTb Mpj1  g2b wTb Mwb , ð12:50Þ

the updating of f 0j and f 00j will require only O(m2) operations. If more than one variable becomes active
at t j, then the above updating process is repeated before examining the new interval [t j, t jþ1].
Observe that, during the computation of the generalized Cauchy point, the examination of the first
segment of the projected steepest descent path requires O(n) operations. All the subsequent segments
require only O(m2) operations, where m is the number of correction pairs stored in the limited
426 12 Simple Bound Constrained Optimization

memory matrix. Since m is usually small, let us say less than 10, the cost of examining all the
segments after the first segment is negligible. Besides, observe that it is not necessary to keep track of
z j 2 ℝn, since only the component zjb corresponding to the bound that has become active is needed to
update f 0j and f 00j :

Algorithm 12.3 Computation of the generalized Cauchy point

1. Initialization. Consider: x, l, u, g and B ¼ θI  WMWT

2. For i ¼ 1, . . ., n compute: 8
> ðxi  ui Þ=gi , gi < 0,
< 
gi , ti 6¼ 0,
ti ¼ ðxi  li Þ=gi , gi > 0, di ¼
>
: 0, ti ¼ 0:
1, gi ¼ 0,
3. Compute:
F ¼ {i : ti > 0}, p 00¼ WTd, c ¼ 0,
f ¼ g d ¼  d00 d, f ¼ θdTd  dTWMWTd ¼  θf0  pTMp,
0 T T

Δtmin ¼  f 0/f , told ¼ 0, t ¼ min {ti, i 2 F},

b ¼ i such that ti ¼ t (remove b from F), Δt ¼ t
4. Examination of the subsequent segments:
While Δtmin  Δt do:

ub , db > 0,
xcp ¼ zb ¼ xcp
b  xb , c ¼ c þ Δtp,
b
lb , db < 0,
f 0 ¼ f 0 þ Δtf 00 þ g2b þ θgb zb  gb wTb Mc,
f 00 ¼ f 00  θg2b  2gb wTb Mp  g2b wTb Mwb ,
00
p ¼ p þ gbwb, db ¼ 0, Δtmin ¼  f 0/f ,
told ¼ t, t ¼ min {ti : i 2 F},
b ¼ i such that ti ¼ t (remove b from F),
Δt ¼ t  told
end while
5. Compute:
Δtmin ¼ max {Δtmin, 0}, told ¼ told þ Δtmin,
xcp
i ¼ xi þ told d i , for all i such that ti  t,
for all i 2 F with ti ¼ t, remove i from F,
c ¼ c þ Δtminp ♦

In the last step of Algorithm 12.3, the 2m-vector c is updated so that in the end, c ¼ WT(xc  xk).
This vector will be used to initialize the subspace minimization when the primal direct method or the
conjugate gradient method is used.
Once the generalized Cauchy point xc has been computed, an approximate minimum of the
quadratic model mk over the space of the free variable is determined. In Byrd, Lu, Nocedal, and
Zhu (1994a, 1995b), three approaches to minimize the model are developed: a direct primal method
based on the Sherman-Morrison-Woodbury formula, a primal iterative method using the conjugate
gradient method, and a direct dual method using the Lagrange multipliers. In all these approaches,
the minimization of mk is done by ignoring the bounds. After that, an appropriate point truncates the
move so as to satisfy the bound constraints. In the direct primal approach, n  t variables are fixed at
their bounds at the generalized Cauchy point xc, and the quadratic model (12.17) is solved over the
subspace of the remaining t free variables, starting from xc and imposing the free variables at the
corresponding bounds from (12.17).
In the algorithm L-BFGS-B, the iterations are terminated when the projected gradient is small
enough, i.e.,
12.5 L-BFGS with Simple Bounds (L-BFGS-B) 427

kPX ðxk  gk , l, uÞ  xk k1 < εg , ð12:51Þ

where εg > 0 is a parameter. Observe that PX(xk  gk, l, u)  xk is the projected gradient. With this, a
formal description of the L-BFGS-B algorithm can be presented as follows:

Algorithm 12.4 L-BFGS-B

1. Consider an initial point x0, as well as an integer m that determines the number of limited memory
corrections stored (si, yi), i ¼ 1, . . ., m. Select the values of the parameters ε > 0 and εg > 0
2. If the convergence test (12.51) is satisfied, stop
3. Using Algorithm 12.3, compute the generalized Cauchy point
4. Compute the search direction dk by the direct primal method, the conjugate gradient method or by the
dual method
5. Using the strong Wolfe line-search (12.18) and (12.19), perform a line-search along dk subject to the
bounds on the problem in order to compute a stepsize αk
6. Set xkþ1 ¼ xk þ αkdk. Compute f(xkþ1) and ∇f(xkþ1)
7. If yk satisfies the curvature condition yTk sk > εkyk k2 , then the correction pair (sk, yk) is added to the
matrices Sk and Yk. If more than m updates are stored, delete the oldest columns from Sk and Yk and place
the new ones instead
8. Update: STk Sk , Y Tk Y k , Lk, Rk and set θ ¼ yTk yk =yTk sk
9. Set k ¼ k þ 1 and continue with step 2 ♦

More details on the L-BFGS-B algorithm are given in Byrd, Lu, Nocedal, and Zhu (1994a, 1995b).
In the following, we present a numerical study and comparisons versus the SPG algorithm.

Numerical Study: L-BFGS-B Versus SPG

Consider the same set of 800 simple bound optimization problems from the UOP collection used in
the previous numerical experiment on the SPG algorithm with quadratic or cubic interpolation. L-
BFGS-B implements two criteria for stopping the iterations
f k  f kþ1
   τεm or kPX ðxk  gk , l, uÞ  xk k1 < εg ,
max f k , f kþ1 , 1

where τ is a factor controlling the accuracy of the solution, εm is the precision of the machine, and εg is
the tolerance on the projected gradient. If one of these two criteria is satisfied, then the iterations are
stopped. The numerical values of the parameter τ are as follows: τ ¼ 1. d þ 12 for small accuracy,
τ ¼ 1. d þ 7 for medium accuracy, and τ ¼ 1. d þ 1 for high accuracy. In this numerical study, we
have considered τ ¼ 1. d þ 7 and εg ¼ 1. d  5. The number of correction pairs stored in computing
BFGS with limited memory is m ¼ 5.
Figures 12.2 and 12.3 show the performance profiles of L-BFGS-B (m ¼ 5) versus SPG with
quadratic interpolation (SPGp) and versus SPG with cubic interpolation (SPGc), respectively. In
Figs. 12.2 and 12.3, we have the computational evidence that both SPGp and SPGc are more efficient
than L-BFGS-B, but subject to robustness, we can see that L-BFGS-B is more robust than both SPGp
and SPGc. The algorithm SPG is more simple than L-BFGS-B, at each iteration requiring a
projection, the computation of the spectral gradient, as well as a nonmonotone linear search. On
the other hand, L-BFGS-B is more complicated, involving the computation of the generalized Cauchy
428 12 Simple Bound Constrained Optimization

Fig. 12.2 L-BFGS-B

versus SPG with quadratic
interpolation (SPGp)

Fig. 12.3 L-BFGS-B

versus SPG with cubic
interpolation (SPGc)

point, the determination of the search direction, a linear search by strong Wolfe conditions (12.18)
and (12.19), as well as the limited-memory BFGS update.

12.6 Truncated Newton with Simple Bounds (TNBC)

This method, called TNBC, elaborated by Nash (1984a, b, 2000) is especially developed to solve
simple bound constrained optimization problems. The method is based on a line-search with a
classical active-set strategy for treating the bounds. TNBC is a truncated Newton method (Dembo
12.6 Truncated Newton with Simple Bounds (TNBC) 429

& Steihaug, 1983) that uses the conjugate gradient projection method to obtain a new search
direction. However, it computes only an approximation to the Newton direction because it is
truncated before the solution to the subspace minimization problem is obtained. In other words, in
the truncated Newton method, the current estimate of the solution is updated (i.e., a step is computed)
by the approximate solving of the Newton system by making use of an iterative algorithm. Therefore,
the algorithm implements a double iterative method: an outer iteration for the nonlinear optimization
problem and an inner iteration for the Newton system. The inner iteration is stopped or “truncated”
before the solution to the Newton system is obtained. The motivation of this approach is that far away
from the solution; the Newton system does not need to be solved very accurately. Therefore, the
Newton iterations can be truncated. The search direction dk is determined such that
2
∇ f ðxk Þdk þ ∇f ðxk Þ  ηk k∇f ðxk Þk,

where {ηk} is known as the “forcing sequence.” The Hessian matrix with the second-order informa-
tion is not given. Therefore, the Hessian vector product Bkv for a given vector v required by the inner
conjugate gradient algorithm is obtained by finite differencing as
∇f ðxk þ hvÞ  ∇f ðxk Þ
∇2 f ðxk Þv ffi ,
h

pffiffiffi
where h ¼ 1 þ kxk k2 ε and ε is the relative machine precision. Each matrix vector product
requires one gradient evaluation. The truncated Newton algorithm differs from the standard Newton
approach, mainly in its use of the parameter ηk which determines the accuracy with which the Newton
system ∇2f(xk)dk ¼  ∇ f(xk) is solved in order to obtain a search direction.
The conjugate gradient inner algorithm is preconditioned by a scaled two-step limited-memory
BFGS method with Powell’s restarting strategy used to reset the preconditioner periodically (Nash,
1985).
The line-search is based on the cubic interpolation and is terminated when the strong Wolfe
conditions are satisfied, as described in Gill and Murray (1979).
Consider the same set of 800 simple bound optimization problems used in the previous numerical
experiments, where, this time, n ¼ 100, 200, . . ., 1000. Figures 12.4 and 12.5 present the

Fig. 12.4 TNBC versus L-

BFGS-B (m ¼ 5) and
versus SPGp
430 12 Simple Bound Constrained Optimization

Fig. 12.5 TNBC versus L-

BFGS-B (m ¼ 5) and
versus SPGc

performances of the truncated Newton method with simple bounds in the implementation of Nash
(1984b) versus L-BFGS-B (m ¼ 5) and SPGp and SPGc, respectively.
From Figs. 12.4 and 12.5, we can see that L-BFGS-B is more robust than TNBC, SPGp, and SPGc.
Both SPGp and SPGc are more efficient than L-BFGS-B and TNBC. From Fig. 12.4, we can see that
SPGp is faster in solving 531 problems. In contrast, L-BFGS-B is faster in solving 425 problems,
while TNBC is faster in solving 379, etc.

12.7 Applications

This section presents the numerical performances of the simple bounds optimization algorithms SPG
and L-BFGS-B for solving five real industrial applications taken from the MINPACK-2 test problem
collection (Averick, Carter, & Moré, 1991; Averick, Carter, Moré, & Xue, 1992). The mathematical
models of these applications are expressed as unconstrained optimization problems and are described
in Appendix D. All these applications without simple bounds have been solved with the SCALCG
algorithm, which implements a double BFGS preconditioned conjugate gradient (Andrei 2008a,
2010b). In these numerical experiments, for all the applications, some simple bounds on the variables
were introduced and solved with SPG and L-BFGS-B, respectively.

Application A1 (Elastic-Plastic Torsion)

Considering D ¼ (0, 1)  (0, 1), c ¼ 5, and nx ¼ 200, ny ¼ 200, a minimization problem with 40,000
variables is obtained. The solution of this application without simple bounds is illustrated in Fig. 12.6.
By introducing the simple bounds 0  v  0.01, the algorithm SPG gives the solution presented in
Fig. 12.7.
12.7 Applications 431

Fig. 12.6 Solution of the

application A1 without
simple bounds. nx ¼ 200, 0.4
ny ¼ 200
0.3

0.2

0.1

0
200
150 200
100 150
100
50 50
0 0

Fig. 12.7 Solution of the

application A1 with simple
bounds 0  v  0.01. 0.01
nx ¼ 200, ny ¼ 200
0.008

0.006

0.004

0.002

0
200
150 200
100 150
50 100
50
0 0

Table 12.1 Elastic-plastic torsion. SPG. 40,000 variables. M ¼ 10

Quadratic interpolation Cubic interpolation

#iter #f #g cpu #iter #f #g cpu
0 v 1 3311 5433 3312 113.25 2663 3600 2664 83.21
0 v 0.1 981 1504 982 59.49 719 940 720 40.59
0 v 0.01 152 190 153 7.63 118 132 119 5.66

Table 12.1 contains the performances of the SPG algorithm with quadratic or cubic interpolation in
the line-search for different values of the simple bounds: 0  v  1, 0  v  0.1, and 0  v  0.01. In
this table, #iter represents the number of iterations to get the solution; #f and #g are the number of
evaluations of the function and the number of evaluations of its gradient, respectively. cpu is the CPU
running time in seconds. The number of values of the objective functions retained in the
nonmonotone linear search is M ¼ 10. Observe that the cubic interpolation is slightly more benefic
versus the quadratic interpolation.
Table 12.2 presents the performances of the algorithm L-BFGS-B for solving this application. In
this table, #sg represents the total number of segments explored during the search of the generalized
Cauchy point, while #act is the number of the active constraints in the final generalized Cauchy point.
Here, m ¼ 5 is the number of the pairs {si, yi}, i ¼ 1, . . ., m, stored for the memoryless BFGS
updating.
432 12 Simple Bound Constrained Optimization

Table 12.2 Elastic-plastic torsion. L-BFGS-B. 40,000 variables. m ¼ 5

#iter #f #sg #act cpu

0 v 1 173 184 177 0 1.89
0 v 0.1 117 122 1774 13036 1.082
0 v 0.01 41 45 62 30920 2.74

Table 12.3 Elastic-plastic torsion. TNBC. 40,000 variables

#iter #fg #fgt cpu

0 v 1 34 77 733 21.68
0 v 0.4 2194 2207 13176 201.24
0 v 0.3 6459 6477 30858 550.85

Fig. 12.8 Solution of

application A2 without
simple bounds. nx ¼ 200, 0.2
ny ¼ 200
0.1

−0.1

−0.2
200
150 200
100 150
100
50 50
0 0

Table 12.3 shows the performances of TNBC for solving this application, where #fg is the number
of callings of the subroutine for the evaluation of the minimizing function and its gradient and #fgt is
the number of function and its gradient evaluations, including the evaluations in the conjugate
gradient for solving the Newton system for the line-search.
Observe that TNBC is very sensitive to the value of the bounds on variables. The closer the bounds
are, the more degraded the performances are.

Application A2 (Pressure Distribution in a Journal Bearing)

Considering b ¼ 10 and ε ¼ 0.1, as well as a discretization nx  ny of the domain D ¼ (0, 2π)  (0, 2b),
where nx ¼ 200 and ny ¼ 200, then an optimization problem with 40,000 variables is obtained. The
solution of this application without simple bounds is represented in Fig. 12.8. When introducing the
simple bounds as 0  v  1, the SPG algorithm gives the solution from Fig. 12.9.
Tables 12.4 and 12.5 present the performances of SPG and L-BFGS-B for different simple bounds,
respectively.
12.7 Applications 433

Fig. 12.9 Solution of

application A2 with simple
bound 0.2
0  v  1. nx ¼ 200,
ny ¼ 200 0.15

0.1

0.05

0
200
150 200
100 150
100
50 50
0 0

Table 12.4 Pressure distribution in a journal bearing. SPG. 40,000 variables. M ¼ 10

Quadratic interpolation Cubic interpolation

#iter #f #g cpu #iter #f #g cpu
1 v 1 4179 7071 4180 140.73 2663 3553 2664 83.96
0 v 1 4964 7796 4965 152.02 4808 6301 4809 133.78
1 v 0 3881 6179 3882 116.35 3154 4170 3155 86.99
0.5 v 0.5 4054 6820 4055 143.09 2969 4105 2970 94.84

Table 12.5 Pressure distribution in a journal bearing. L-BFGS-B. 40,000 variables. m ¼ 5

#iter #f #sg #act cpu

1 v 1 623 648 623 0 7.05
0 v 1 2419 2429 25995 12862 2.62
1 v 0 377 391 796 12904 3.84

Table 12.6 Optimal design with composite materials. SPG. 40,000 variables. M ¼ 10

Quadratic interpolation Cubic interpolation

#iter #f #g cpu #iter #f #g cpu
1 v 0 22515 34686 22516 1259.82 20969 27964 20970 1083.20
0.02 v 0 1855 2769 1856 86.21 2228 2890 2229 95.74

Application A3 (Optimal Design with Composite Materials)

Considering μ1 ¼ 1 and μ2 ¼ 2, then Tables 12.6 and 12.7 show the performances of SPG and
L-BFGS-B, respectively. Figure 12.10 presents the solution of this application obtained with SPG for
nx ¼ 200 and ny ¼ 200. Figure 12.11 shows the solution of this application given by L-BFGS-B with
the simple bounds 0.02  v  0.
434 12 Simple Bound Constrained Optimization

Table 12.7 Optimal design with composite materials. L-BFGS-B. 40,000 variables. m ¼ 5

#iter #f #sg #act cpu

1 v 0 646 649 744 0 7.94
0.02 v 0 124 127 734 7420 1.38

Fig. 12.10 Solution of

application A3 without
simple bounds. nx ¼ 200, 0
ny ¼ 200 −0.01
−0.02
−0.03
−0.04
−0.05
−0.06
200
150 200
100 150
100
50 50
0 0

Fig. 12.11 Solution of

application A3 with simple
bounds 0.02  v  0. 0
nx ¼ 200, ny ¼ 200
−0.005

−0.01

−0.015

−0.02
200
150 200
100 150
100
50 50
0 0

Application A4 (Steady-State Combustion)

Considering λ ¼ 5, Fig. 12.12 shows the solution of this application where nx ¼ 200 and ny ¼ 200.
Figure 12.13 presents the solution of the application with the simple bounds 0  v  0.2, where λ ¼ 5,
nx ¼ 200, and ny ¼ 200.
Tables 12.8 and 12.9 give the performances of SPG and of L-BFGS-B for solving this application
with different simple bounds on variables.
From these tables, we note that, subject to the CPU computing time, L-BFGS-B is the best
algorithm, followed by SPG with cubic interpolation and then by SPG with quadratic interpolation.
12.7 Applications 435

Fig. 12.12 Solution of

application A4 without
simple bounds. nx ¼ 200, 0.8
ny ¼ 200
0.6

0.4

0.2

0
200
150 200
100 150
100
50 50
0 0

Fig. 12.13 Solution of

application A4 with simple
bounds 0  v  0.2. 0.05
nx ¼ 200, ny ¼ 200
0.04

0.03

0.02

0.01

0
200
150 200
100 150
100
50 50
0 0

Table 12.8 Steady-state combustion. SPG. 40,000 variables. M ¼ 10

Quadratic interpolation Cubic interpolation

#iter #f #g cpu #iter #f #g cpu
0 v 1 5134 8418 5135 421.92 5352 7236 5353 395.13
0 v 0.2 1843 2795 1844 137.32 1764 2286 1765 121.01
0 v 0.1 1140 1724 1141 81.52 665 854 666 43.45

Table 12.9 Steady-state combustion. L-BFGS-B. 40,000 variables. m ¼ 5

#iter #f #sg #act cpu

0 v 1 360 372 424 0 5.48
0 v 0.2 185 196 1012 7960 2.76
0 v 0.1 128 130 909 14796 1.83
436 12 Simple Bound Constrained Optimization

Application A6 (Inhomogeneous Superconductors: 1-D Ginzburg-Landau)

Considering d ¼ 3.2 Å and the temperature T ¼ 5, for n ¼ 1000, the solution of this application
without simple bounds is presented in Fig. 12.14. When introducing the simple bounds on the
variables 0  v  0.001, the corresponding solution given by SPG is given in Fig. 12.15.
Tables 12.10 and 12.11 present the performances of SPG and L-BFGS-B for solving this applica-
tion with different simple bounds.

Fig. 12.14 Solution of

application A6 without
simple bounds. n ¼ 1000

Fig. 12.15 Solution of × 10−4

application A6 with simple 10.001
bounds 0  v  0.001.
n ¼ 1000
10

9.999

9.998

9.997

9.996

9.995
0 100 200 300 400 500 600 700 800 900 1000
12.7 Applications 437

Table 12.10 1-D Ginzburg-Landau problem. SPG. 1000 variables. M ¼ 10

Quadratic interpolation Cubic interpolation

#iter #f #g cpu #iter #f #g cpu
0 v 108 3465 6001 3466 0.81 4373 6001 4373 0.87
0 v 10 3466 6001 3468 1.92 4374 6001 4375 2.20

Table 12.11 1-D Ginzburg-Landau problem. L-BFGS-B. 1000 variables. m ¼ 5

#iter #f #sg #act cpu

0 v 108 34895 36076 34895 0 6.30
0 v 10 10278 10643 11438 391 1.74

Real applications of nonlinear optimization always have simple bounds on variables. In other
words, for a problem with n variable, every variable xj has a finite lower bound lj and a finite upper
bound uj, where lj  uj, j ¼ 1, . . ., n. These bounds are imposed by the engineering constructive
relationships of the process modeled as an optimization problem. Therefore, the simple bound
optimization is an important problem with numerous practical applications.
Both SPG and L-BFGS-B are able to solve simple bound optimization problems, L-BFGS-B being
slightly faster. TNBC is very sensitive to the interval in which the bounds on variables are defined.

Notes and References

Recent advances in the simple bound constrained optimization are presented in Hager and Zhang
(2006b) and Tr€oltzsch (2007). To guarantee the global convergence to a local solution, the algorithms
for solving this class of problems use either a line-search or a trust-region-oriented framework. The
algorithmic components of the line-search methods are the subspace minimization, the inexact line-
search, the active-set method, the gradient projection method, the projected Newton method, the
gradient projection–reduced Newton method, and the gradient projection–conjugate gradient method.
On the other hand, the algorithmic components of the trust-region methods are the generalized
Cauchy point and the trust-region method. For the simple bound constrained optimization, there
are some specialized solvers (software). LANCELOT/SBMIN (Conn, Gould, & Toint, 1992b) and
TRON (Lin & Moré, 1999) use the trust-region method. Both of them use a conjugate gradient
iteration to perform the subspace minimization and apply an incomplete Cholesky preconditioner.
Other solvers like DOT (Vanderplaats, 1995), L-BFGS-B, TNBC, and IPOPT (Wächter & Biegler,
2006) use line-search methods. IPOPT uses a primal-dual interior point filter line-search method (see
Chap. 19). Another approach completely different from the above categories of solvers is based on the
penalty functions method of Facchinei, Lucidi, and Palagi (2002).
Our presentation follows the papers of Kelley (1999); Byrd, Lu, Nocedal, and Zhu (1995b)
(L-BFGS-B); Birgin, Martı́nez, and Raydan (2000, 2001) (SPG); and Nash (1984a, b, 1985) (TNBC).
 Quadratic Programming
13

One of the most important nonlinear optimization problems is the quadratic programming, in which a
quadratic objective function is minimized with respect to linear equality and inequality constraints.
These problems are present in many methods as subproblems and in real applications from different
areas of activity as mathematical models of these applications. In the beginning, we consider the
equality constrained quadratic programming, after which the inequality constrained quadratic pro-
gramming is presented. Finally, methods based on the elimination of variables are discussed.

13.1 Equality Constrained Quadratic Programming

The expression of an equality constrained quadratic program is as follows:

1
minqðxÞ  xT Gx þ cT x
2
subject to ð13:1Þ

aTi x þ bi ¼ 0, i 2 E≜f1, . . . , mg,

where G is an n  n symmetric matrix and ai, i 2 E, are vectors in ℝn. All the vectors ai, i 2 E, can be
assembled into the matrix A such that the constraints from (13.1) can be compactly written as
Ax þ b ¼ 0, where A is an m  n matrix. Suppose that m  n. If the Hessian matrix G is positive
semidefinite, we say that (13.1) is a convex quadratic program. In this case, the problem is similar in
difficulty to a linear program. Strictly convex quadratic programs are those in which G is positive
definite. Nonconvex quadratic programs have G an indefinite matrix.
The first-order optimality conditions for (13.1) are

Gx þ c  AT λ ¼ 0, ð13:2aÞ

Ax þ b ¼ 0, ð13:2bÞ

where x is the solution and λ is the vector of the Lagrange multipliers. Rearranging (13.2), these
equations become a system of n þ m linear equations known as the KKT system

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 439
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_13
440 13 Quadratic Programming

" #   
G AT x c
¼ : ð13:3Þ
A 0 λ b

In the following, we consider Z to denote the n  (n  m) matrix whose columns is a basis for the
null space of A. That is, Z has full rank and satisfies AZ ¼ 0.

Factorization of the Full KKT System

For solving (13.1), the direct way, known as the factorization of the full KKT system, is to form the
linear system (13.3) and to find x and λ by solving it, let us say by the Gaussian elimination. Observe
that although the matrix from (13.3) is symmetric, the zeros on the main diagonal of this matrix imply
that it is not positive definite, that is, the Cholesky factorization is not suitable. If G is positive
definite, then the feasible stationary point obtained from (13.3) is the minimum of (13.1). Otherwise,
to confirm that x is the minimum point, the second-order condition

zT Gz > 0 for all the vectors z, suchthat Az ¼ 0

must be checked.
The system (13.3) can be rearranged in a very suitable form for computation by expressing x as

x ¼ x þ d, where x is some estimation of the solution and d is the desired step. Using this notation,
from the system (13.3), we obtain
" #   
G AT d g
 ¼ , ð13:4Þ
A 0 λ h

where g ¼ c þ Gx and h ¼ Ax  b. The matrix in (13.4) is called the KKT matrix.

Theorem 13.1 Let A be of full rank and assume that ZTGZ is positive definite. Then, the KKT matrix
" #
G AT
K¼ ð13:5Þ
A 0

is nonsingular, and therefore, there is a unique vector pair (x, λ) satisfying (13.3).

Proof Suppose that there are the vectors w and v, such that
" # 
G AT w
¼ 0: ð13:6Þ
A 0 v

Since Aw ¼ 0, from (13.6), it follows that

" # 
 T T G AT w
0¼ w v ¼ wT Gw:
A 0 v

However, w lies in the null space of A. Therefore, there is a vector u 2 ℝnm, such that w ¼ Zu.
With this, we have
13.1 Equality Constrained Quadratic Programming 441

0 ¼ wT Gw ¼ uT ZT GZu,

which by the positive definiteness of ZTGZ implies u ¼ 0. Therefore, w ¼ 0, and by (13.6), ATv ¼ 0.
Now, suppose that A is of full rank on rows. Therefore, v ¼ 0. The conclusion is that the equation
(13.6) is satisfied only if w ¼ 0 and v ¼ 0, i.e., the matrix K is nonsingular. ♦

Theorem 13.2 Let A be of full rank and assume that ZTGZ is positive definite. Then, x satisfying
(13.3) is the unique global solution of (13.1).

Proof Let x be any other feasible point satisfying Ax ¼  b. Let d ¼ x  x. Since Ax ¼  b, we
have Ad ¼ 0. Substituting x ¼ x  d in q(x) from (13.1), we get
1
qðxÞ ¼ ðx  dÞT Gðx  d Þ þ cT ðx  dÞ
2
1
¼ dT Gd  dT Gx  cT d þ qðx Þ:
2
However, from (13.3), we have Gx ¼  c þ ATλ. Therefore, from Ad ¼ 0, we have
 
dT Gx ¼ dT c þ AT λ ¼ dT c:

With this,
1
qðxÞ ¼ dT Gd þ qðx Þ:
2
Since d lies in the null space of A, there is a vector u 2 ℝnm, such that d ¼ Zu. Hence,
1
qðxÞ ¼ uT Z T GZu þ qðx Þ:
2
However, ZTGZ is positive definite, hence q(x) > q(x), except for the situation in which u ¼ 0, that
is, when x ¼ x. Therefore, x is the unique global solution of (13.1). ♦

The Schur-Complement Method

The use of (13.3) to get a solution for (13.1) may be efficient only when the matrices G and A are
sparse. Otherwise, another method, known as the Schur-complement, may be used, in which x and λ
are separately determined. For example, if we multiply (13.2a) by AG1 and then use (13.2b) to
eliminate x, we can obtain λ as solution of the following linear system:
 1 T  
AG A λ ¼ AG1 c  b: ð13:7Þ

Clearly, x is computed from

Gx ¼ AT λ  c: ð13:8Þ

A rough analysis of the computational effort shows that solving (13.7) and (13.8) requires
O(m3) þ O(n3) multiplications. This is less than O((n þ m)3) multiplications needed to solve
(13.3). However, this approach requires to perform operations with G1 and to compute a factoriza-
tion of the m  m matrix AG1AT.
442 13 Quadratic Programming

The name Schur complement derives from the fact that, by applying the block Gaussian elimina-
tion to (13.3) using G as pivot, the following block upper triangular system
" #   
G AT x c
¼ ð13:9Þ
0 AG1 AT λ b  AG1 c

is obtained. In linear algebra terminology, the matrix AG1AT is called the Schur complement of G.

The Null-Space Method

This method does not require that the matrix G should be nonsingular. It assumes only that A has full
row rank and that the matrix ZTGZ is positive definite, where Z is an n  (n  m) matrix whose
columns span the null space of A. The matrix ZTGZ is called the reduced Hessian. Clearly, this
method requires knowledge of the null space basis matrix Z.
The method considers a solution in terms of its components in two subspaces which are normal
and tangential to the constraints. As we know (see Chap. 11), the optimality conditions involve
feasible directions which lie in the tangent space of the constraints. If an n-variable optimization
problem involves m linear constraints, Ax ¼ b, then a feasible direction can be any member of the
(n  m)-dimensional subspace of the vectors z which satisfy Az ¼ 0. As we said before, let Z be an
n  (n  m) matrix whose columns span the null space of A. This means that Zw is a feasible direction
for any (n  m) vector w. Also, let Y be an n  m-dimensional arbitrary matrix, such that the matrix
½ Y Z  is nonsingular. Suppose that d in (13.4) is partitioned, so that
d ¼ YdY þ Zd Z , ð13:10Þ

where dY 2 ℝm and dZ 2 ℝnm. In this context, the components YdY and ZdZ are called the vertical step
and the horizontal step, respectively.
Substituting d given by (13.10) into the second equation from (13.4), since AZ ¼ 0, we get
ðAY Þd Y ¼ h: ð13:11Þ

However, A is a matrix of full rank m and ½ Y Z  is an n  n nonsingular matrix. Therefore, the

matrix

A½ Y Z  ¼ ½ AY 0 ð13:12Þ

is of rank m. Hence, AY is an m  m nonsingular matrix. dY is well defined as solution of the system

(13.11).
Now, substituting d given by (13.10) into the first equation from (13.4), we get

GYd Y  GZd Z þ AT λ ¼ g: ð13:13Þ

Premultiplying (13.13) by ZT, we obtain the reduced system

 T 
Z GZ dZ ¼ Z T GYd Y  Z T g: ð13:14Þ

Since ZTGZ is a symmetric positive definite matrix, it follows that dZ is well defined as solution of
the system (13.14). Since the reduced Hessian ZTGZ is positive definite, then the Cholesky factoriza-
tion can be used to solve (13.14). In conclusion, the total step d can be computed as d ¼ YdY þ ZdZ,
where dY is the solution of (13.11) and dZ is the solution of (13.14), computed in this order.
13.1 Equality Constrained Quadratic Programming 443

To obtain the Lagrange multipliers, we multiply the first line of (13.4) by YT to get the linear
system

ðAY ÞT λ ¼ Y T g þ Y T Gd: ð13:15Þ

In this method, solving the linear algebraic systems (13.11) and (13.14) requires O(m2) þ
O((n  m)3) multiplications, which can be significantly less than O((n þ m)3) multiplications needed
to solve the KKT system (13.3), especially when m  n.

Example 13.1 Let us consider the null-space method for solving the following quadratic program-
ming problem:
1 2 
min QðxÞ ¼ x1 þ x22 þ 2x23 þ x24 þ x25 þ ðx1 þ x3 þ x5 Þ
2 |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl} |fflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflffl}
xT Gx cT x

subject to the linear constraints Ax ¼ b, where

2 3
x1
2 36 x 7 2 3
6 27
1 0 2 3 1
6 7 7
4 2 1 2 1 1 56 x3 7 ¼ 4 7 5:
6 7
4 3 1 2 2 6 7
|fflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflffl}4 x4 5 |fflffl{zfflffl}
12

A x5 b

In this problem, n ¼ 5 and m ¼ 3. A basis Z for the null space of A is obtained by using the reduced
echelon technique. The matrix Y is selected, such that ½ Y Z  2 ℝnn is nonsingular. Then,
2 3 2 3
13 3 1 0 0
6 7 6 7
6 15 3 7 60 1 07
6 7 6 7
Z¼6 6 5 1 772ℝ
nðnmÞ
, Y¼6 7
60 0 17 2 ℝ :
nm
6 7 6 7
4 1 0 5 40 0 05
0 1 0 0 0

Consider x ¼ ½ 0 1 0 1 1 T an initial estimation of the solution, with which we compute

h ¼ Ax  b ¼ ½ 3 4 5 T , g ¼ Gx þ c ¼ ½ 1 1 1 1 2 T :

From (13.11), we get

2 32 3 2 3
5 6 2 3 1
1 6 76 7 6 7
dY ¼ ðAY Þ h ¼ 4 6 7 2 54 4 5 ¼ 4 0 5:
2 3 1 5 1

Similarly, from (13.14), we obtain

444 13 Quadratic Programming

 1 T

dZ ¼  Z T GZ Z GYd Y þ Z T g
" #(" # " #) " #
1 21 94 3 8 0:163064
¼ þ ¼ :
509 94 445 1 3 0:825147

Therefore, from (13.10), we get

2 3
1:355609
6 7
6 0:029481 7
6 7
d ¼ YdY þ Zd Z ¼ 6 7
6 0:990173 7:
6 7
4 0:163064 5
0:825147

Using the known initial estimation x, the local optimal solution of the problem is
2 3
1:355609
6 7
6 0:970519 7
6 7
x ¼ x þ d ¼ 6 7
6 0:990173 7:
6 7
4 1:163064 5
0:174853

The most complicated steps of the algorithm are the determination of a basis of the null space of
A and the solving of the linear algebraic systems (13.11) and (13.14). ♦

Remark 13.1 (Bartholomew-Biggs, 2008) The null-space method for quadratic programming with
equality constraints can be very easily extended to the nonlinear optimization with linear equality
constraints:

min ff ðxÞ : Ax þ b ¼ 0g, ð13:16Þ

where f : ℝn ! ℝ is continuously differentiable and A 2 ℝmn is a constant matrix. As in the

unconstrained optimization, the idea is to use an iterative scheme based on the local quadratic
approximation of f. That is, in a neighborhood of a solution estimate x, we suppose
1
f ðx þ dÞ  f ðxÞ þ ∇f ðxÞT d þ dT Bd, ð13:17Þ
2
where either B ¼ ∇2f(x) or B  ∇2f(x) is a quasi-Newton approximation of the Hessian. The following
algorithm solves the problem (13.16) by using the quadratic approximation (13.17) at each iteration.
The algorithm makes use of an update approximation to the Hessian matrix more than the exact
Hessian ∇2f(x) and of a line-search to improve the values of the nonquadratic function f.
Therefore, the problem is now to minimize (13.17) subject to Ad þ (Ax þ b) ¼ 0. The optimality
conditions for this problem are

Bd þ ∇f ðxÞ  AT λ ¼ 0,
Ad þ ðAx þ bÞ ¼ 0:

Next, we apply the technique of the null-space method. Let Z be an n  (n  m) matrix whose
columns span the null space of A. The choice of Z is not unique, but one way of getting it is by the
13.1 Equality Constrained Quadratic Programming 445

orthogonal factorization of the constraint Jacobian A. This factorization yields an orthogonal n  n

matrix Q and an m  m lower triangular matrix L, such that AQ ¼ R ¼ ½ L 0  and QTQ ¼ I. Now, let
us define Y as the matrix composed of the first m columns of Q and Z the matrix consisting of the
remaining (n  m) columns. It can be shown that AZ ¼ 0, AY ¼ L, and YTZ ¼ 0.
If d solves the quadratic approximation of (13.16), then its components in the Y-subspace and
Z-subspace can be determined as follows. Suppose that y 2 ℝm and z 2 ℝnm are vectors, such that

d ¼ Yy þ Zz:

Since AZ ¼ 0, the optimality condition Ad þ (Ax þ b) ¼ 0 implies

AYy þ ðAx þ bÞ ¼ 0:

However, AY ¼ L. Therefore, y can be found by solving a lower triangular system of equations

AYy ¼ ðAx þ bÞ:

Now, premultiplying the optimality condition Bd þ ∇ f(x)  ATλ ¼ 0 by ZT and having in view
that AZ ¼ 0, we get Z T BZz þ ZT ∇f ðxÞ þ ZT BYy ¼ 0: Therefore, z can be found by solving the
system

Z T BZz ¼ ZT ∇f ðxÞ  ZT BYy,

where ZTBZ is a symmetric positive definite matrix. Knowing y and z, we can compute d ¼ Yy þ Zz
as a correction of the estimate x.

As in the quadratic case, in this context ZT ∇ f(x) is called the reduced gradient of f and ZTBZ is the
reduced Hessian of f. Observe that both Y and Z, basis matrices for the range- and null-spaces of the
constraints matrix A, are kept constant along the iterations of the algorithm. The corresponding
algorithm is as follows:

Algorithm 13.1 Reduced gradient for linear equality constraints

1. Choose an initial feasible point x0. Choose B0 as a positive definite approximation of ∇2f(x0). Compute
Y and Z as basis matrices for the range- and the null-spaces of the constraints matrix A. Choose ε > 0
sufficiently small. Set k ¼ 0 and compute ∇f(x0)
2. Test of convergence. If kZT ∇ f(xk)k  ε, then stop
3. Determine y as solution of the system AYy ¼ ðAxk þ bÞ
 
4. Determine z as solution of the system Z T Bk Z z ¼ Z T ∇f ðxk Þ  Z T Bk Yy
5. Set dk ¼ Yy þ Zz
6. Compute the estimate of the Lagrange multipliers λkþ1 as solution of the linear system
(AY)Tλ ¼ YT ∇ f(xk) þ YTBkdk
7. Perform a line-search to determine s such that f(xk þ sdk) < f(xk). Set xkþ1 ¼ xk þ sdk as a new estimate of
the solution
8. Using a quasi-Newton update, from Bk compute a new estimate Bkþ1. Set k ¼ k þ 1 and continue with
step 2 ♦

Since the objective is not quadratic, it follows that xk þ dk is not guaranteed to be a better point
than xk. Therefore, Algorithm 13.1 is equipped with a line-search (in step 7) which may be exact or
based on the Wolfe line-search conditions.
446 13 Quadratic Programming

The algorithm uses a quasi-Newton update of Bk to get Bkþ1 as a new estimate of the Hessian. The
BFGS update has the advantage of generating positive definite matrices Bkþ1. This will ensure that the
approximation of the reduced Hessian ZTBkZ is also positive definite, and hence, dk is a descent
direction. ♦

It is noteworthy that in the nonlinear constrained optimization, the true Hessian ∇2f(x) is often
indefinite. This suggests that a positive definite updating scheme, based, for example, on BFGS
formula, may be inconsistent with making Bk a good approximation. However, this is not a problem,
because the second-order optimality conditions only relate to the null space of the binding constraint
normals. In this subspace, the optimality conditions require that the Hessian should be positive
definite.
Under reasonable assumptions about function and constraints and about the Hessian, it can be
shown that the search directions and the step lengths in the feasible subspace will satisfy the Wolfe
conditions. Therefore, the iterations will converge to a point where the reduced gradient is close
to zero.

Large-Scale Problems

Consider the quadratic programming problem with equality constraints

1
minqðxÞ≜ xT Gx þ cT x
2
subject to ð13:18Þ
Ax ¼ b,

where A 2 ℝmn of full rank, with m < n, so that the constraints Ax ¼ b are consistent. For solving
large-scale quadratic programming problems, the direct factorization methods of the KKT system
(13.3) are not very effective. The iterative methods are more suitable. As we know, the conjugate
gradient methods are dedicated to solving very large systems of linear equations with positive definite
matrices. However, the conjugate gradient methods directly applied to the KKT system (13.3) are not
recommended because they can be unstable on systems with matrices that are not positive definite. It
is better to apply the conjugate gradient methods to the reduced system (13.14).

The Conjugate Gradient Applied to the Reduced System

For this, let us express the solution of (13.18) as

x ¼ YxY þ ZxZ , ð13:19Þ

where xY 2 ℝm and xZ 2 ℝnm. Z 2 ℝn(nm) is a basis for the null space of A, i.e., AZ ¼ 0. Y 2 ℝnm
is a matrix, such that ½ Y Z  2 ℝnn is nonsingular. Observe that, since A is of full row rank,
it follows that A½ Y Z  ¼ ½ AY 0  is also of full row rank, so that the matrix AY is nonsingular.
We would like to choose Y so that the matrix AY is conditioned as much as possible. This can be
achieved by the QR factorization of AT, which has the form
13.1 Equality Constrained Quadratic Programming 447

 
R
A P ¼ ½ Q1
T
Q2  , ð13:20Þ
0

where ½ Q1 Q2  is orthogonal. The submatrices Q1 2 ℝnm and Q2 2 ℝn(nm) have orthonormal

columns, while R 2 ℝmm is an upper triangular and nonsingular matrix. P 2 ℝmm is a permutation
matrix. Now, define Y ¼ Q1 and Z ¼ Q2, so that the columns of Y and Z form an orthonormal basis of
ℝn. Now, from (13.20), it is obtained that AY ¼ PRT and AZ ¼ 0. Therefore, Y and Z have the desired
properties, and the condition number of AY is the same as that of R, which in its turn is the same as that
of A.
With this, the constraints Ax ¼ b can be written as

AYxY ¼ b, ð13:21Þ

which determines the component xY of x. Now, introducing (13.19) in (13.18), it follows that xZ is a
solution of the unconstrained reduced problem
1 T T
min x Z GZxZ þ cTZ xZ , ð13:22Þ
xZ 2 Z
where

cZ ¼ Z T GYxY þ ZT c: ð13:23Þ

Obviously, from (13.22), we can see that xZ satisfies the following linear system:

ZT GZxZ ¼ cZ : ð13:24Þ

Since ZTGZ is positive definite, it follows that the system (13.24) can be solved by applying the
conjugate gradient algorithm. In conclusion, with these algebraic constructions, we can see that since
AY is nonsingular, the component xY can be obtained from (13.21); since ZTGZ is positive definite, the
component xZ can be obtained from (13.24), and thus, the solution x of (13.18) can be computed from
(13.19).
In the following, let us present the preconditioned conjugate gradient algorithm for solving the
reduced system (13.24). Suppose that a preconditioner WZZ is given. Then, the preconditioned
conjugate gradient algorithm applied to the (n  m)-dimensional reduced system (13.24) is as
follows:

Algorithm 13.2 Preconditioned conjugate gradient for reduced systems

1. Initialization. Choose an initial point xZ

2. Compute: rZ ¼ ZTGZxZ þ cZ, gZ ¼ W 1 ZZ r Z and dZ ¼  gZ
3. Test a criterion for stopping the iterations. For example, if r TZ W 1
ZZ r Z is sufficiently small, then stop the
iterations.
4. Compute:
α ¼ rTZ gZ =dTZ Z T GZd Z , xZ ¼ xZ þ αdZ , rþ
Z ¼ r Z þ αZ GZd Z ,
T
gþ 1 þ
Z ¼ W ZZ r Z ,
 þ T þ T
β ¼ rZ gZ =r Z gZ , d Z ¼ gþ
Z þ βd Z , gZ ¼ gþZ, rZ ¼ rþZ:
5. Go to step 3 ♦
448 13 Quadratic Programming

In Algorithm 13.2, the steps produced by the conjugate gradient are denoted by dZ. It is worth
saying that it is not necessary to explicitly form the reduced Hessian ZTGZ. This is because the
conjugate gradient algorithm requires only the computation of a matrix-vector product involving this
matrix. The preconditioner WZZ 2 ℝ(nm)  (nm) is a symmetric, positive definite matrix, which may
1=2   1=2
be chosen to cluster the eigenvalues of W ZZ ZT GZ W ZZ , that is to reduce the span between the
smallest and the largest eigenvalues. One of the best selections of the preconditioner is one for which
1=2   1=2
W ZZ Z T GZ W ZZ ¼ I, that is, WZZ ¼ ZTGZ. This is the main motivation for which we take the
preconditioners of the following form:

W ZZ ¼ Z T HZ, ð13:25Þ

where H is a symmetric matrix, such that ZTHZ is positive definite.

The Projected Conjugate Gradient Method

Another algorithm for solving (13.1) can be obtained by rewriting Algorithm 13.2 to work in the
n-dimensional space. For this, the following n-vectors are introduced: x, r, g, and d, which satisfy
x ¼ ZxZ þ YxY, ZTr ¼ rZ, g ¼ ZgZ, and d ¼ ZdZ, respectively. The scaled projection matrix P 2 ℝnn is
also defined as P ¼ Z(ZTHZ)1ZT, where H is the preconditioning matrix from (13.24). Hence, the
projected conjugate gradient algorithm is as follows:

Algorithm 13.3 Projected conjugate gradient

1. Initialization. Choose an initial point x satisfying Ax ¼ b

2. Compute: r ¼ Gx þ c, g ¼ Pr and d ¼  g
3. Test a criterion for stopping the iterations. For example, if rTg is smaller that a prespecified tolerance,
then stop the iterations.
4. Compute:
α ¼ rT g=dT Gd, x ¼ x þ αd, r þ ¼ r þ αGd, gþ ¼ Pr þ ,
β ¼ ðr þ Þ gþ =r T g, d ¼ gþ þ βd, g ¼ gþ , r ¼ rþ
T

5. Go to step 3 ♦

Observe that g+, which is called preconditioned residual, is a vector defined in the null space of A.
Therefore, in exact arithmetic, all the search directions d generated by Algorithm 13.3 also lie in the
null space of A, that is, all the iterates satisfy Ax ¼ b. The iterations of Algorithm 13.3 are well defined
if ZTGZ and ZTHZ are positive definite. There are two simple choices of the preconditioning matrix H:
H ¼ diag (|Gii|) and H ¼ I. Nocedal and Wright (2006) give some details how to compute Pr without
knowing a representation of the null space basis Z. Consider the case in which H ¼ I. In this case, P is
the orthogonal projection operator onto the null space of A. This special case of P is denoted as P,
that is,
 1
P ¼ Z ZT Z ZT : ð13:26Þ
13.2 Inequality Constrained Quadratic Programming 449

The computation of the preconditioned residual gþ ¼ Pr þ can be performed in two different ways.
 1
The first one, called the normal equations approach, expresses P by P ¼ I  AT AAT A and thus
computes gþ ¼ Pr þ : Therefore, we can write g+ ¼ r+  ATv+, where v+ is the solution of the system

AAT vþ ¼ Ar þ : ð13:27Þ

The system (13.27) can be solved by using the Cholesky factorization of AAT.
The second one, called the augmented system approach, is to express the projection Pr+ as the
solution of the following augmented system:
" #   
I AT g þ rþ
¼ ð13:28Þ
A 0 vþ 0

Of course, we have
 1   1 
gþ ¼ Pr þ ¼ r þ  AT vþ ¼ r þ  AT AAT Ar þ ¼ I  AT AAT A r þ ¼ Pr þ :

All these developments allow us to compute an initial point that satisfies Ax ¼ b. The computation
of g+ ¼ Pr+ relies on the factorization of AAT from which we can compute x ¼ AT(AAT)1b. Note that
these approaches for computing g+ can give rise to significant round-off errors, so the use of the
iterative refinement is recommended in order to improve the accuracy (Nocedal & Wright, 2006).

13.2 Inequality Constrained Quadratic Programming

An inequality constrained quadratic program has the following form:

1
min qðxÞ  xT Gx þ cT x ð13:29aÞ
2
subject to
aTi x ¼ bi , i 2 E, ð13:29bÞ

aTi x bi , i 2 I, ð13:29cÞ

where G is an n  n symmetric matrix, c, x, and {ai}, i 2 E [ I, are vectors in ℝn, and E and I are finite
sets of indices. The Lagrangian for this problem is
1 X  
Lðx, λÞ ¼ xT Gx þ cT x  λi aTi x  bi :
2 i2E[I

As in Definition 11.12, the active set A(x) consists of the indices of the constraints from (13.29),
for which the equality holds at x:

Aðx Þ ¼ i 2 E [ I : aTi x ¼ bi : ð13:30Þ

Supposing that the constraint qualification LICQ is satisfied and specializing the general KKT
conditions (11.21) and (11.22) to this problem, we find that any solution x of (13.29) satisfies the
following first-order optimality conditions for some Lagrange multipliers λi , i 2 A(x):
450 13 Quadratic Programming

X
Gx þ c  λi ai ¼ 0, ð13:31aÞ
i2Aðx Þ

aTi x ¼ bi , for all i 2 Aðx Þ, ð13:31bÞ

aTi x bi , for all i 2 I∖Aðx Þ, ð13:31cÞ

λi 0, for all i 2 I \ Aðx Þ: ð13:31dÞ

If G is positive definite, then (13.29) is called convex quadratic programming. For the convex
quadratic programming, the conditions (13.31) are sufficient for x to be a global solution, as proved
in the next theorem.

Theorem 13.3 If x satisfies the conditions (13.31) for some λi , i 2 A(x), and G is positive
semidefinite, then x is a global solution of (13.29).

Proof If x is any other feasible solution for (13.29), then we have aTi x ¼ bi , for all i 2 E, and
aTi x bi , for all i 2 A(x) \ I. Hence, aTi ðx  x Þ ¼ 0 for all i 2 E and aTi ðx  x Þ 0 for all
i 2 A(x) \ I. Now, using (13.31a) and (13.31d), it follows that
X X
ðx  x ÞT ðGx þ cÞ ¼ λi aTi ðx  x Þ þ λi aTi ðx  x Þ 0: ð13:32Þ
i2E i2Aðx Þ\I

Having in view (13.32), after some algebra, we get

1
qðxÞ ¼ qðx Þ þ ðx  x ÞT ðGx þ cÞ þ ðx  x ÞT Gðx  x Þ
2
1
qðx Þ þ ðx  x ÞT Gðx  x Þ qðx Þ,
2
since G is positive semidefinite. Therefore, we have shown that q(x) q(x) for any feasible x, i.e., x
is a global solution. ♦

If the reduced Hessian ZTGZ is positive definite, where Z is a null-space basis matrix for the active
constraint Jacobian matrix, which is the matrix whose rows are aTi for all i 2 A(x), then the second-
order sufficient conditions for x to be a local minimum hold. If G is not positive definite, then the
general problem (13.29) may have more than one strict local solution. Such problems are called
nonconvex quadratic programming or indefinite quadratic programming.
For solving the inequality constrained quadratic programming problems (13.29), we can use the
active-set methods or the interior-point methods. In case of the convex quadratic programming, there
are three varieties of active-set methods: primal (Beale, 1959; Fletcher, 1971; Bunch & Kaufman,
1977), dual (Lemke, 1962), and primal-dual (Goncalves, 1972; Goldfarb & Idnani, 1983). The
active-set methods find a step from one iteration to the next one by solving a quadratic subproblem
in which some of the inequality constraints (13.29c) and all the equality constraints (13.29b) are
imposed as equalities. This subset of constraints is referred to as the working set, which at iteration
k is denoted as Wk. The case in which G is an indefinite matrix raises some complications in the
algorithms which are beyond the purpose of this book. For the nonconvex quadratic programming,
see Gould, Orban, and Toint (2005a). In the following, we shall discuss the primal active-set method
and the primal-dual active-set method by Goldfarb and Idnani (1983).
13.2 Inequality Constrained Quadratic Programming 451

The Primal Active-Set Method

The idea of the primal active-set method is as follows. Given the iterate xk and the working set Wk, test
whether xk minimizes the quadratic q in the subspace defined by the working set. If not, a step d is
computed by solving an equality constrained quadratic programming subproblem in which the
constraints corresponding to the working set Wk are regarded as equalities and the rest of constraints
are temporarily ignored. Let us define
d ¼ x  xk , gk ¼ Gxk þ c:

With this, the objective function (13.29a) becomes

1
qðxÞ ¼ qðxk þ dÞ ¼ dT Gd þ gTk d þ δk ,
2
where δk ¼ 12 xTk Gxk þ cT xk is independent of d.
Since δk can be dropped from the expression of q(x), the quadratic subproblem which must be
solved at the k-th iteration is
1 T
min d Gd þ gTk d ð13:33aÞ
2
subject to
aTi d ¼ 0, i 2 W k : ð13:33bÞ

Let dk be the solution of (13.33). Since aTi ðxk þ αd k Þ ¼ aTi xk ¼ bi for all α, observe that for each
i 2 Wk, it follows that the value of aTi x does not change as we move along dk. Therefore, since the
constraints in Wk are satisfied at xk, they are also satisfied at xk þ αdk for any value of α. Since G is
positive definite, it follows that the solution of (13.33) can be computed by any method for solving
equality constrained quadratic programming.
Supposing that the optimal dk is nonzero, the question is how far we can move along this direction
to conserve the feasibility. If xk þ dk is feasible with respect to all the constraints, we set xkþ1 ¼ xk þ dk.
Otherwise, set

xkþ1 ¼ xk þ αk dk , ð13:34Þ

where the stepsize αk is computed to be the largest value in the range [0, 1] for which all the
constraints are satisfied. We must consider the definition of αk for the constraints i 2 = Wk, since, as
we said before, the constraints in Wk are satisfied at xk þ αdk for any value of α. If ai dk 0 for some
T

i2= Wk, then for all αk 0 we have aTi ðxk þ αk dk Þ aTi xk bi : Therefore, the constraint i will be
satisfied for all nonnegative αk. On the other hand, if aTi dk < 0 for some i 2 = Wk, then
ai ðxk þ αk d k Þ bi only if
T

bi  aTi xk
αk  : ð13:35Þ
aTi d k

To maximize the decrease in q, we must take αk to be as large as possible in [0, 1] subject to the
satisfying feasibility. Hence, αk is determined as
452 13 Quadratic Programming


bi  aTi xk
αk ≜ min 1, min 2W k , aTi d k < 0
, i= : ð13:36Þ
aTi d k

The constraint i for which the minimum in (13.36) is achieved is called blocking constraint, like in
linear programming (Andrei, 2011d).
If αk < 1, that is, the step along dk was blocked by some constraint not in Wk, then a new working
set Wkþ1 is generated by adding one of the blocking constraints to Wk.
The iterations are continued in this way, by adding constraints to the working set until a point xbthat
minimizes the quadratic objective function over its current working set W b has been attained. Such a
point is easy to be recognized because the subproblem (13.33) has as solution d ¼ 0. Since d ¼ 0
satisfies the optimality conditions (13.3) corresponding to (13.33), we have
X
aib
λi ¼ g ¼ Gb
xþc ð13:37Þ
b
i2W

for some Lagrange multipliers b b It follows that xb and b

λi , i 2 W: λ satisfy the first KKT optimality
condition (13.31a) if we define the multipliers corresponding to the inequality constraints that are not
in the working set to be zero. On the other hand, because xb is also feasible with respect to all the
constraints, it follows that (13.31b) and (13.31c) are satisfied at this point.

Algorithm 13.4 Active-set method for convex quadratic programming

1. Compute a feasible initial point x0. Set W0 to be a subset of the active constraints at x0. Set k ¼ 0
2. Test a criterion for stopping the iterations
3. Solve the equality quadratic programming sub-problem (13.33) to find dk
4. If dk ¼ 0, then compute the Lagrange multipliers bλi that satisfy (13.37) with Wb ¼ W k : If bλi 0, for all
n o
i 2 Wk \ I, then stop with the solution x ¼ xk; otherwise, determine j ¼ arg min bλj : j 2 W k \ I and


set xkþ1 ¼ xk and Wkþ1 ¼ Wk\{j}.

If dk 6¼ 0, then from (13.36) compute αk and set xkþ1 ¼ xk þ αkdk. If there are blocking constraints,
then obtain Wkþ1 by adding one of the blocking constraints to Wk; otherwise, set Wkþ1 ¼ Wk and go
to step 2 ♦

Some properties of the algorithm are given by Nocedal and Wright (2006), where plenty and
important details on the primal active-set method are presented. These properties refer to the signs of
the multipliers corresponding to the inequality constraints in the working set (the indices i 2 W b \ IÞ,
various techniques that are used to determine an initial feasible point, the strategy of removing the
constraint corresponding to the most negative Lagrange multiplier, the strategy of adding or deleting
one constraint at each iteration at most, the finite termination of the active-set algorithm on strictly
convex quadratic programming, updating the factorizations, etc.
Algorithm 13.4 can be adapted to the situations in which the Hessian is indefinite by modifying the
computation of the search direction and of the stepsize. To explain the need for modification, let us
consider the computation of the step by the null-space method, that is, d ¼ Zdz, where dz is given by
(ZTGZ)dz ¼  ZTg. If the reduced Hessian ZTGZ is positive definite, then the step d is directed to the
minimizer of the subproblem (13.33), and the iteration need not be changed. On the other hand, if
ZTGZ has negative eigenvalues, then d is directed to a saddle point of (13.33), and therefore, it is not a
suitable step any more. In this case, an alternative direction sz is sought, which is a direction of
negative curvature for the reduced Hessian ZTGZ. In this case, q(xk þ αZsz) !  1 , as α ! 1 .
13.2 Inequality Constrained Quadratic Programming 453

Additionally, if necessary, the sign of sz is changed to ensure that Zsz is a non-ascent direction for q at
the current point. By moving along the direction Zsz, we encounter a constraint that can be added to
the working set for the next iteration. Now, if the reduced Hessian for this new working set is not
positive definite, the above procedure is repeated until enough constraints have been added to make
the reduced Hessian positive definite.
Another practical class of algorithms for indefinite quadratic programming is that of inertia
controlling methods. These methods never allow the reduced Hessian to have more than one negative
eigenvalue. At each iteration, the algorithm will either add or remove a constraint from the working
set. If a constraint is added, then the reduced Hessian is of smaller dimension and must remain
positive definite or be the null matrix. Therefore, an indefinite reduced Hessian can arise only when
one of the constraints is removed from the working set, which happens only when the current point is
a minimizer with respect to the current working set. In this case, we will choose the new search
direction to be a direction of negative curvature for the reduced Hessian. The algorithms for indefinite
quadratic programming differ in the way they detect the indefiniteness and they compute the negative
curvature direction and in the handling of the working set (Fletcher, 1971; Gill & Murray, 1978).

An Algorithm for Positive Definite Hessian

This active-set algorithm is based on a repeated use of the optimality conditions for an equality
quadratic programming problem. In the following, we consider the quadratic programming which has
only inequality constraints. The extension to mixed equality and inequality quadratic problems is
straightforward. Therefore, let us consider the problem
1 T
min x Gx þ cT x ð13:38aÞ
2
subject to
b þ bb 0,
Ax ð13:38bÞ

where we assume that G is positive definite. The method begins by identifying an active set of
constraints. This is an estimate of those which are binding at the solution. Now, let A be the matrix and
b the vector formed from the rows of Ab and bb corresponding to the active constraints. With this, let
 
xe, e
λ be a trial solution by minimizing (13.38a) subject to

Ax þ b ¼ 0,

obtained by using an algorithm for the equality quadratic programming problem. If

bx þ bb 0,
Ae ð13:39aÞ
 
eT
λ Ae bx þ bb ¼ 0 and eλ 0, ð13:39bÞ

then the optimality conditions (13.31) are all satisfied and the problem (13.38) is solved by x ¼ xe and
 
λ ¼ eλ: On the other hand, if xe, e
λ is not optimal, we must change the active set and solve another
equality constrained quadratic programming subproblem. This process is repeated until the active set
has become the binding set for the problem (13.38). The choice of a new active set is based on
(13.39). Two ideas are followed here. The first one is that any new constraint which is violated at xe
454 13 Quadratic Programming

can be regarded as a candidate for being added to the current active set. Secondly, any active
constraint which corresponds to a negative component of bλ is a candidate for deletion from the
current active set (Bartholomew-Biggs, 2008). The following algorithm formalizes the ideas just
b
outlined, where abi denotes the i-th row of A:

Algorithm 13.5 Active-set method with positive definite Hessian

1. Choose an initial point x and set λi ¼ 0, i ¼ 1, . . ., m

2. Test the optimality conditions. If Ax b þ bb 0, Gx þ c  AbT λ ¼ 0, and λ 0, then stop
3. Identify the active constraints as those for which
abTi x þ bbi < 0 or (b aTi x þ bbi ¼ 0 and λi 0).
Renumber these constraints so that the active set is i ¼ 1, . . ., t
4. Set g ¼ Gx þ c and bi ¼ abT x þ bbi , for i ¼ 1, . . ., t
i
5. Find d and μ as solution for the equality quadratic programming problem
min 12 d T Gd þ gT d subject to abTi d þ bi ¼ 0, i ¼ 1, . . ., t
6. Set (s ¼ 1, λi ¼ μi, i ¼ 1, . . ., t) and (λi ¼ 0, i ¼ t þ 1, . . ., m)

if abiT p <0 set

T
7. For i ¼ t þ 1, . . ., m, i.e., for all inactive constraints
ba xþb b
s ¼ min s,  i T i
bai d
8. Set x ¼ x þ sd and go to step 2 ♦

The algorithm may have different variants according to the methods for solving the problems in
step 5 and to the rules with regard to how many constraints may be added to or dropped from the
active set. The stepsize computed in step 7 checks all the inactive constraints that might be violated by
a step along the search direction p and ensures that no more than one constraint can be added to the
active set on the current iteration. Details can be found in Fletcher (1971).

Reduced Gradient for Inequality Constraints

The active-set strategy combined with the null-space approach can be very easily extended to
solving problems with the nonquadratic objective function f(x) and with linear inequality constraints
abTi x þ bbi 0, i ¼ 1, . . ., m. In this case, the Y and Z matrices used in the null-space method must be
computed whenever the active set is changed (Bartholomew-Biggs, 2008).

Algorithm 13.6 Reduced gradient for linear inequality constraints

1. Choose an initial point x0 and set λ0 ¼ 0. Choose B0 as a positive definite estimate of ∇2f(x0). Select a
tolerance ε > 0. Set k ¼ 0
2. Compute gk ¼ ∇ f(xk)
3. Select active constraints as those with abT xk þ bbi ¼ 0 and λk 0
i i
4. Consider Ak as the matrix of active constraint normals at xk
5. Compute Yk and Zk as basis matrices for the range and the null spaces of matrix Ak
6. Determine z as solution of the linear system Z Tk Bk Z k z ¼ Z Tk gk : Set dk ¼ Zkz
7. Find λkþ1 as solution of the linear system Y Tk ATk λ ¼ Y Tk gk þ Y Tk Bk dk
13.2 Inequality Constrained Quadratic Programming 455

8. Step length computation. Perform a line-search to get xkþ1 ¼ xk þ sdk so that f(xkþ1) < f(xk)
9. Compute a quasi-Newton update of Bk to get Bkþ1
 
10. Test of convergence. If Z Tk gk   ε, then stop; otherwise, set k ¼ k þ 1 go to step 2 ♦

Observe that in step 8 the computations of the step length must ensure that no new constraints are
violated. Therefore, the stepsize s is subject to an upper limit which allows at most one new constraint
to become binding at xkþ1. This can be computed as in step 7 of Algorithm 13.5. The difficulty with
this algorithm is that the matrices Y and Z used in the null-space method must be computed whenever
the active set is changed in step 4 of the algorithm.

The Reduced Gradient for Simple Bounds

An important situation in which Z can be very easily computed and, therefore, makes the reduced-
gradient approach very attractive is when the problem has only simple bounds on variables: li  xi  ui,
i ¼ 1, . . ., n. Suppose that li < ui, i ¼ 1, . . ., n. In this case, at the start of each iteration the variables are
classified in two classes: those which are fixed (i.e., on their bounds) and those which are free. If xk is
the current estimate at the start of iteration k, then the bound on the i-th variable is active if xk is fixed,
which means that
   
xki ¼ li and gki > 0 or xki ¼ ui and gki < 0 :

In this case, the matrix Z whose columns span the space of the free variables can be simply taken as
a partition of the identity matrix. When the iterate xkþ1 is computed from xk along a search direction
dk, the stepsize must ensure that no new bounds are violated. A maximum stepsize to force each
variable xki to stay within its bounds can be computed as

ðui  xki Þ=d ki if d ki > 0,
simax ¼
ðli  xki Þ=dki if dki < 0:

Algorithm 13.7 Reduced gradient for simple bounds constraints

1. Choose an initial point x0 and B0 as a positive definite estimate of ∇2f(x0). Select a tolerance ε > 0. Set
k¼0
2. Compute gk ¼ ∇ f(xk)
3. Set Zk as the n  n identity matrix
4. For i ¼ 1, . . ., n repeat: if xki satisfies (xki ¼ li and gki > 0) or (xki ¼ ui and gki < 0), then delete the i-th
column of Zk
5. Determine z as solution of the linear system Z Tk Bk Z k z ¼ Z Tk gk : Set dk ¼ Zkz
6. Use a line-search along dk to find s so that f(xk þ sdk) < f(xk)

7. For each free variable xki compute simax and set s ¼ min s, simax
8. Set xkþ1 ¼ xk þ sdk
9. Compute a quasi-Newton update of Bk to get Bkþ1
 
10. Test of convergence. If Z Tk gk   ε, then stop; otherwise, set k ¼ k þ 1 go to step 2 ♦

Details on this algorithm can be found in Bartholomew-Biggs (2008).

456 13 Quadratic Programming

The Primal-Dual Active-Set Method

In the following, we present the algorithm by Goldfarb and Idnani (1983) for solving the inequality
constrained quadratic programming problems
1
min QðxÞ≜ xT Gx þ cT x ð13:40aÞ
2
subject to
si ðxÞ ¼ aTi x þ bi ¼ 0, i ¼ 1, . . . , me , ð13:40bÞ

si ðxÞ ¼ aTi x þ bi 0, i ¼ me þ 1, . . . , m, ð13:40cÞ

l  x  u, ð13:40dÞ

where G is an n  n symmetric and positive definite matrix and ai 2 ℝn for all i ¼ 1, . . ., m. In general,
the algorithms for quadratic programming have two phases. In the first one, a feasible point is
obtained, and then, in the second phase, the optimality is achieved while maintaining the feasibility.
The algorithm of Goldfarb and Idnani is of the active-set type. By an active set, we mean a subset
of the m constraints in (13.40) that are satisfied as equalities by the current estimate x of the solution to
the quadratic programming problem (13.40). If we denote the set of indices of the constraints (without
simple bounds) by I ¼ {1, . . ., m}, then the active set is denoted by A ⊆ I. Let P(J ) be the
subproblem with the objective function (13.40a) subject only to the subset of the constraints indexed
by J ⊂ I. For example, P(∅) is the problem of finding the unconstrained minimum of (13.40). If the
solution x of a subproblem P(J ) lies on some linearly independent active set of constraints indexed by
A ⊆ J, then the pair (x, A) is called a solution. Clearly, if (x, A) is a solution for the subproblem P(J),
then it is also a solution for the subproblem P(A). By linear independence of a set of constraints, we
mean that the normals corresponding to these constraints are linearly independent. In the following,
we outline the algorithm given by Goldfarb and Idnani (1983):

1. Assume that we know an arbitrary solution (x, A).

2. Choose a violated constraint p 2 I\A.
3. If P(A [ {p}) is infeasible, then stop; problem (13.40) is infeasible.
 
4. Else, obtain a new pair x, A [ fpg , where A ⊆ A and QðxÞ > QðxÞ , and set ðx, AÞ
 
x, A [ fpg :
5. Continue with step 2 until all the constraints have been satisfied.

Since G is positive definite, it follows that the unconstrained minimum of the function Q(x) defined
in (13.40a) is x0 ¼  G1c. Therefore, the above procedure can start with the pair (x0, ∅).
For describing the algorithm, we have to introduce some notations like in Goldfarb and Idnani
(1983). The matrix of the normal vectors of the constraints in the active set A is denoted by N, and the
cardinality of A is denoted by q. Denote A+ ¼ A [ {p}, where p 2 I\A, and A is a proper subset of A,
which contains one fewer element than A. N+ and N represent the matrices of normals corresponding
to A+ and A, respectively. ei represents the i-th column of the unity matrix, and Ik is the k  k identity
matrix. n+ represents the normal vector np added to N to obtain N+; n represents the column deleted
from N to get N.
13.2 Inequality Constrained Quadratic Programming 457

If the columns from N are linearly independent, then we may introduce the following operators:
 1
N  ¼ N T G1 N N T G1 ð13:41Þ

and
 1
H ¼ G1 ðI  NN  Þ ¼ G1  G1 N N T G1 N N T G1 : ð13:42Þ

We can see that N is the pseudo-inverse, or the generalized Moore-Penrose inverse, of N in the
space of variables obtained through the transformation y ¼ G1/2x. On the other hand, H is the inverse
of the reduced Hessian of the function Q(x) with respect to the active set of constraints. If xb is a point

in the (n  q) -dimensional manifold M ¼ x 2 ℝn : nTi x ¼ bi , i 2 A and gðxbÞ  ∇QðxbÞ ¼

Gbx þ c is the gradient of the objective function Q(x) at xb, then the minimum of Q(x) over M is
attained in point x ¼ xb  HgðxbÞ: For x to be the optimal solution of the subproblem P(A), we must
have
gðxÞ ¼ NuðxÞ, ð13:43Þ

where the vector of Lagrange multipliers uðxÞ 0: From the definitions of N and H, it follows that at
such a point we have
uðxÞ ¼ N  gðxÞ 0 ð13:44Þ

and

HgðxÞ ¼ 0: ð13:45Þ

Observe that these conditions are necessary as well as sufficient for x to be the optimal solution of
P(A). Let us define r ¼ Nn+ as the Lagrange multipliers associated to the infeasible constraints.
Define H+ as in (13.42) with N replaced by N+. A similar notation is used for (N+) and u+.
The algorithm presented below follows the dual approach described above and makes use of the
operators N and H defined in (13.41) and (13.42), respectively (Goldfarb & Idnani, 1983). An
efficient implementation does not explicitly compute and does not store these operators. Instead, this
implementation stores and updates the matrices J ¼ QTL1 and R obtained from the Cholesky and QR
 
R
factorization of G ¼ LLT and L1 N ¼ Q .
0

Algorithm 13.8 Dual algorithm for quadratic programming

1. Find the unconstrained minimum of function Q(x) from (13.40a). Compute x ¼  G1c, H ¼ G1,
A ¼ ∅, and q ¼ 0
2. Choose a violated constraint, if any, and compute sj(x) for all j 2 I \ A
If V ¼ {j 2 I \ A : sj(x) < 0} ¼ ∅ , then stop; the current solution is optimal. Otherwise, choose p 2 V and
set n+ ¼ np and uþ ¼ ½ u 0 T
If q ¼ 0, then set u ¼ 0. (A+ ¼ A [ {p})
3. Determine the search direction.
Compute z ¼ Hn+ as the direction in the primal space. If q > 0, then set r ¼ Nn+ the negative of the
direction in the dual space
4. Compute the step length.
(i) Compute t1 the maximum step in the dual space without violation of the dual feasibility. If r  0 or
q ¼ 0, then set t1 ¼ 1 ; otherwise, set
458 13 Quadratic Programming

nuþ ðxÞ o
uþ ðxÞ
t1 ¼ min j
rj : r j > 0, j ¼ 1, . . . , q ¼ l rl :
In step 5 below, the element k 2 I corresponds to the l-th element form A.
(ii) Compute t2 the minimum step in the primal space such that the p-th constraint becomes feasible. If
|z| ¼ 0, then set t2 ¼ 1 ; otherwise, set
s ðxÞ
t2 ¼  zpT nþ :
(iii) Step length computation: t ¼ min (t1, t2)
5. Determine the pair and take the step.
(i) If t ¼ 1 , no step in the primal or dual space is executed and stop. The subproblem P(A+) is infeasible
and hence problem (13.40) is infeasible.
 
r
(ii) If t2 ¼ 1 , then set uþ ¼ uþ þ t and delete the constraint k, i.e., set A ¼ A\{k}, q ¼ q  1,
1

update H and N , and go to step 3.
 
r
(iii) Step in the primal and dual space. Set x ¼ x þ tz, uþ ¼ uþ þ t :
1
If t ¼ t2, then set u ¼ u and add the constraint p to A, that is, A ¼ A [ {p}, set q ¼ q þ 1, update H and N,
+

and go to step 2.
If t ¼ t1, then drop the constraint k, i.e., set A ¼ A\{k}, q ¼ q  1, update H and N, and go to
step 3 ♦

Goldfarb and Idnani (1983) prove that the algorithm will solve the quadratic programming
problem (13.40) or indicate that it has no feasible solution in a finite number of steps.
The implementation of the above algorithm in a stable manner is based on the Cholesky
factorization

G ¼ LLT ð13:46Þ

of the symmetric and positive definite matrix G and the QR factorization

   
R R
B¼Q ¼ ½ Q1 Q2  ð13:47Þ
0 0

of the n  q matrix

B ¼ L1 N: ð13:48Þ

In these relations, L is an n  n lower triangular matrix, R is a q  q upper triangular matrix, and

½ Q1 Q2  is an n  n orthogonal matrix partitioned, so that Q1 has q columns. By substituting
(13.46)–(13.48) in (13.41) and in (13.42), the operators H and N can be expressed as

H ¼ J 2 J T2 ð13:49Þ

and

N  ¼ R1 J T1 , ð13:50Þ

where
 
½ J1 J 2  ¼ LT Q1 LT Q2 ¼ LT Q ¼ J: ð13:51Þ

Although the algorithm is based on the QR factorization, the orthogonal matrix Q from (13.47) is
not stored; instead, the matrix J ¼ LTQ is stored and updated, as whenever Q is called by the
algorithm, it appears in conjunction with LT.
13.2 Inequality Constrained Quadratic Programming 459

In Algorithm 13.8, it is necessary to compute the vectors z ¼ Hn+ and r ¼ Nn+. If we compute
" #  
T þ J T1 þ d1
d¼J n ¼ T
n ¼ , ð13:52Þ
J2 d2

then, from (13.49) and (13.50), we get

z ¼ J 2 d2 , r ¼ R1 d 1 : ð13:53Þ

In the following, like in Goldfarb and Idnani (1983), let us present some details concerning the
updating of the factors J and R when a constraint is added to or deleted from the active set.

Updating the factors J and R Whenever a constraint is added to or deleted from the set of the active
constraints, the factors J and R must be updated. Numerically stable methods for updating the QR
factorization of a matrix when it is modified by the addition or deletion of a single column are well
known (Daniel, Graggs, Kaufman, & Stewart, 1976; Gill, Golub, Murray, & Saunders, 1974;
Goldfarb, 1975). In the updating scheme described by Goldfarb and Idnani (1983), the Givens
matrices Qij are used. They are equal to the identity matrix with the elements (i, i), (i, j), ( j, i), and
( j, j) replaced by t, s, s, and t, respectively, where t ¼ cos θ and s ¼ sin θ for some value of θ.
Having in view their special structure, the computations involving the Givens matrices can be
illustrated by considering the 2  2 matrix

 
b¼ t s
Q :
s t

b is orthogonal. In all the computations, Q

Since t2 þ s2 ¼ 1, it follows that Q b is chosen in such a way
T T  1=2
that the vector w ¼ ½ w1 w2  is transformed into the vector ½ ω 0  , where ω ¼
w21 þ w22 :
To accomplish this, the following elements are computed:
" 2  2 #1=2
w1 w2
μ ¼ max fjw1 j, jw2 jg, ω ¼ signðw1 Þμ þ ,
μ μ

w1 w2
t¼ and s ¼ :
ω ω

b ¼Q
To compute yb ¼ Qy bT y, where y ¼ ½ y y T , the algorithm follows the scheme given in
1 2
Gill, Golub, Murray, and Saunders (1974) and determines
w2 s
v¼ ¼ ,
ω þ w1 1 þ t
and

yb1 ¼ ty1 þ sy2 , yb2 ¼ vðy1 þ yb1 Þ  y2 : ð13:54Þ

Observe that if Qb (or Q

b ) is applied to a 2  n matrix, then the above computing scheme saves
T

n  1 multiplications, but introduces n þ 1 additions over the usual procedure for matrix
multiplications. Also, observe that the sign of ω is chosen so that there is no cancelation in the
computation of ν.
460 13 Quadratic Programming

Adding a constraint When the constraint p with the normal n+ is added to the active set A, then the
factorization (13.47) is replaced by

 
þ
 1 þ
 þ Rþ
B ¼ B L n ¼Q : ð13:55Þ
0

Therefore, from (13.38), (13.51), and (13.52), it follows that

 
T þ R d1
Q B ¼ :
0 d2

Thus, the factorization (13.55) can be obtained as

" #  
þ
I q 0 þ R d1
Q ¼Q T and R ¼ ,
0 Q 0 δ

where δ ¼ kd2k and Q ¼ Q1,2 Q2,3 ⋯Qnq1,nq is the product of the Givens matrices chosen, so that

Qd2 ¼
δe1 : ð13:56Þ

Moreover,
h i  
J þ ¼ LT Qþ ¼ J 1 J2Q
T
¼ Jþ
1 Jþ
2 , ð13:57Þ

T
where J1 has q columns, J 2 Q has n  q columns, J þ þ
1 has q þ 1 columns, and J 2 has n  q  1
columns.

Dropping a constraint When the l-th constraint is deleted from the active set, that is, when the l-th
column of N is deleted to form N+, then
 
T  T 1  R1 S
Q1 B ¼ Q1 L N ¼ ,
0 T

where the partitioned matrix on the right is equal to R but with its l-th column eliminated. If l 6¼ q,
then T is a (q  l þ 1)  (q  l ) upper-Hessenberg matrix. Again, a sequence of q  l Givens matrices
can be chosen, so that the product of these matrices Q ¼ Qql,qlþ1 ⋯Q2,3 Q1,2 reduces T to an upper
triangular matrix R2, i.e.,

QT ¼ R2 : ð13:58Þ

Thus, the new factors are obtained as

2 3
  I l1 0 0
R1 S 6 7
R ¼ and J  ¼ J 4 0
T
Q 0 5: ð13:59Þ
0 R2
0 0 I nq

The matrices Q from (13.56) and (13.58) are not computed. Instead, the Givens matrices from
which they are formed, which successively introduce zeros into d2 and T, are applied directly to the
rows of JT using the computational scheme (13.55).
13.2 Inequality Constrained Quadratic Programming 461

When the step directions z for the primal variables and r for the dual variables given by (13.53)
are computed, we need the vector d ¼ JTnþ. When a constraint is deleted from the active set, then the
same orthogonal transformations that are applied to update JT can be applied to d. If these operations
are executed, then the updated d can be used to compute the new vectors z and r after the basis change
and to determine the appropriate orthogonal transformation Q in (13.56) needed when the p-th
constraint is finally added to the active set.
Algorithm 13.8 (dual for quadratic programming) is one of the most advanced algorithms for
solving quadratic programming problems with equality and inequality constraints. Its efficiency
comes from two sources: the dual method and the implementation of an advanced computational
scheme based on the QR factorization which is updated by the Givens matrices. One of the first
implementations of Algorithm 13.8 was given by Powell (1983) (Fortran package ZQPCVX). A
professional extension was given by Schittkowski (2005) (Fortran package QLD).

Example 13.2 Let us consider the following quadratic programming problem (Andrei, 1999a,
p. 792, 2003, p. 329):

P
15 P
14
min QðxÞ  x2i þ ðxi þ xiþ1 Þ
i¼1 i¼1
subject to
0:1x1 þ 0:1x7 þ 0:3x8 þ 0:2x9 þ 0:2x11  1 ¼ 0,
0:1x2 þ 0:2x8 þ 0:3x9 þ 0:4x10 þ x11  2 ¼ 0,
0:1x3 þ 0:2x8 þ 0:3x9 þ 0:4x10 þ 2x11  3 ¼ 0,
x4 þ x8 þ 0:5x9 þ 0:5x10 þ x11  x12  3 ¼ 0,
2x5 þ x6 þ 0:5x7 þ 0:5x8 þ 0:25x9 þ 0:25x10 þ 0:5x11  x13  4 ¼ 0,
x4 þ x6 þ x8 þ x9 þ x10 þ x11  x14  5 ¼ 0,
0:1x1 þ 1:2x7 þ 1:2x8 þ 1:4x9 þ 1:1x10 þ 2x11  x15  6 ¼ 0,
0  xi  2, i ¼ 1, . . . , 15:

Table 13.1 contains information on the optimization process given by Algorithm 13.8 in the QLD
implementation by Schittkowski (2005). To get a solution, the algorithm needs eight iterations. In
Table 13.1, we can see the evolution of the values of the objective function along the iterations. Also

Table 13.1 Optimization process by QLD

#k Q( xk ) svr #nra t kkt

1 97.000000 27.800000 7 0 68.760324
2 46.471283 0.266453e−14 7 1 17.341127
3 31.302619 0.155431e−14 7 1 10.560131
4 22.807444 0 7 1 0.740935
5 22.179315 0.199840e−14 7 1 0.168103
6 22.037532 0.666133e−15 7 1 0.232076
7 21.921296 0.111022e−15 7 1 0.670174e−06
8 21.921296 0.888178e−15 7 1 0.303832e−14

In this table, we have #k ¼ the number of iterations, Q(xk) ¼ the value of the objective function, svr ¼ the sum of the
constraints violation, #nra ¼ the number of the active constraints, t ¼ the stepsize, and kkt ¼ the value of the KKT
conditions at xk
462 13 Quadratic Programming

Table 13.2 Initial point, solution, and bounds on variables

l x0 x* u
1 0 2.1 0.400273 2
2 0 2.1 0.130566 2
3 0 2.1 0 2
4 0 2.1 0 2
5 0 2.1 0.903379 2
6 0 2.1 0.416858 2
7 0 2.1 0 2
8 0 2.1 1.509334 2
9 0 2.1 1.522805 2
10 0 2.1 0.537945 2
11 0 2.1 1.013057 2
12 0 1 0.552766 2
13 0 1 0 2
14 0 1 0 2
15 0 1 0.601009 2

notice the evolution of the sum of the constraints violation, which tends to zero, as well as the
evolution of the norm of the KKT conditions. The number of the active constraints is 7; the problem
has only equality constraints. The stepsize is equal to 1 along the iterations, like in the Newton
method from the unconstrained optimization.

In Table 13.2, we can note the initial point, the bounds on variables, and the solution given by
QLD. Observe that the initial point is not feasible.

Application Q1. DYNAMIC This application is a representation of the problem concerning the
scheduling of three generators to meet the demand for power over a given period of time. The
variables x3kþi denote the output from the generator i at time tk. The constraints of the problem are
upper and lower limits on the power available from each generator, bounds on the amount by which
the output from a generator can change from time tk to tkþ1, as well as the condition that at each time
tk, the power generated must at least satisfy the demand. The mathematical model is described in Pant,
Thangaraj, and Singh (2009) and is as follows:
4 
P 
min 2:3x3kþ1 þ 0:0001x23kþ1 þ 1:7x3kþ2 þ 0:0001x23kþ2 þ 2:2x3kþ3 þ 0:00015x23kþ3
k¼0
subject to
x1 þ x2 þ x3 60,
x4 þ x5 þ x6 50,
x7 þ x8 þ x9 70,
x10 þ x11 þ x12 85,
x13 þ x14 þ x15 100,
7  x3kþ1  x3k2  6, k ¼ 1, . . . , 4,
7  x3kþ2  x3k1  7, k ¼ 1, . . . , 4,
7  x3kþ3  x3k  6, k ¼ 1, . . . , 4,
13.3 Interior Point Methods 463

Table 13.3 Dynamic. Optimization process by QLD

#k Q( xk ) svr #nra t kkt

1 1228.3246 0.42e+02 29 0 0.12e+04
2 1115.8453 0 6 1 0.11e+03
3 1006.9820 0 4 1 0.32e+03
4 762.61922 0.11e−13 4 1 0.76e+02
5 708.25997 0.89e−14 8 1 0.26e+02
6 692.67975 0.71e−14 9 1 0.15e+01
7 691.18494 0.36e−14 8 1 0.70e+01
8 684.96921 0 8 1 0.11e+02
9 674.81722 0.18e−13 8 1 0.48e+01
10 670.99302 0.67e−15 9 1 0.28e+01
11 668.92085 0.75e−14 8 1 0.21e+01
12 667.49688 0.53e−14 8 1 0.64e+00
13 666.86138 0.12e−13 7 1 0.32e+01
14 663.69246 0.18e−14 7 1 0.12e+02
15 655.26305 0.18e−14 10 1 0.48e+00
16 654.78703 0.53e−14 10 1 0.78e+00
17 654.12603 0 10 1 0.12e+01
18 653.44270 0 11 1 0.20e−30

The variables are bounded as follows:

8  x1  21, 0  x3kþ1  90, k ¼ 1, . . . , 4,
43  x2  57, 0  x3kþ2  120, k ¼ 1, . . . , 4,
3  x3  16, 0  x3kþ3  60, k ¼ 1, . . . , 4:

Considering the initial point x0i ¼ ðli þ ui Þ=2, i ¼ 1, . . ., 15, Algorithm 13.6 gives the following
solution:

x1 ¼ 8, x2 ¼ 49, x3 ¼ 3, x4 ¼ 1, x5 ¼ 52,

x6 ¼ 0, x7 ¼ 5, x8 ¼ 59, x9 ¼ 6, x10 ¼ 7,
x11 ¼ 66, x12 ¼ 12, x13 ¼ 0, x14 ¼ 95, x15 ¼ 5:

The value of the objective function at this point is Q(x) ¼ 653.4427.

Table 13.3 gives some information about the optimization process with QLD. To get the above
solution, the algorithm needs 18 iterations.

13.3 Interior Point Methods

In the following, let us present the interior point methods for solving quadratic programming
problems. We consider the convex quadratic programs with inequality constrains (Nocedal & Wright,
2006):
464 13 Quadratic Programming

1
min qðxÞ ¼ xT Gx þ cT x,
x 2
subject to ð13:60Þ
Ax b,

where G is symmetric and positive semidefinite and A ¼ [a1, . . ., am]T 2 ℝmn and b ¼ [b1, . . .,
bm]T 2 ℝm. The KKT optimality conditions for (13.60) are as follows:

Gx  AT λ þ c ¼ 0,
Ax  b 0,
ðAx  bÞi λi ¼ 0, i ¼ 1, . . . , m,
λ 0:

These conditions may be written as equalities by introducing the slack vector y 0:

Gx  AT λ þ c ¼ 0, ð13:61aÞ

Ax  y  b ¼ 0, ð13:61bÞ

ðAx  bÞi λi ¼ 0, i ¼ 1, . . . , m, ð13:61cÞ

λ 0: ð13:61dÞ

Observe that the matrix G is positive semidefinite. Therefore, these KKT conditions are both
necessary and sufficient, and consequently, the convex quadratic problem (13.60) can be solved by
finding the solutions of the system (13.61).
For this, let us consider the current iterate (x, y, λ) that satisfies y > 0 and λ > 0. Define the
complementarity measure μ as

yT λ
μ¼ : ð13:62Þ
m
With this, we define the path-following primal-dual methods by considering the perturbed KKT
conditions
2 3
Gx  AT λ þ c
6 7
Fðx, y, λ; σμÞ ¼ 4 Ax  y  b 5 ¼ 0, ð13:63Þ
YΛe  σμe

where

Y ¼ diagðy1 , . . . , ym Þ, Λ ¼ diagðλ1 , . . . , λm Þ, e ¼ ð1, 1, . . . , 1ÞT 2 ℝm

and σ 2 [0, 1]. The solutions of (13.63) for all positive values of σ and μ define the central path, which
is a trajectory that leads to the solution of the quadratic problem (13.60) as σμ tends to zero.
Now, fixing μ and applying the Newton method to (13.63), the following linear system is obtained:
2 32 3 2 3
G 0 AT Δx r d
6 76 7 6 7
4 A I 0 54 Δy 5 ¼ 4 r p 5, ð13:64Þ
0 Λ Y Δλ ΛYe þ σμe
13.3 Interior Point Methods 465

where

r d ¼ Gx  AT λ þ c, r p ¼ Ax  y  b: ð13:65Þ

The next iteration (x+, y+, λ+) is obtained as

ðxþ , yþ , λþ Þ ¼ ðx, y, λÞ þ αðΔx, Δy, ΔλÞ, ð13:66Þ

where α is chosen in such a way that y+ > 0 and λ+ > 0.

Observe that the major computational operation in the interior point method is the solution of the
system (13.64). However, by eliminating Δy ¼ AΔx þ rp from (13.64), this system may be restated in
a more compact form as
" #   
G AT Δx r d
¼   : ð13:67Þ
A Λ1 Y Δλ r p þ y þ σμΛ1 e

From (13.67), by eliminating Δλ ¼ Y1Λ(rp  y þ σμΛ1e  AΔx), a more compact form is

obtained as
   
G þ AT Y 1 ΛA Δx ¼ r d þ AT Y 1 Λ r p  y þ σμΛ1 e , ð13:68Þ

which can be solved by a modified Cholesky algorithm (see Appendix A) to get Δx.

Stepsize Selection

Like in Nocedal and Wright (2006), suppose that the new iterate is defined as

ðxþ , yþ Þ ¼ ðx, yÞ þ αprim ðΔx, ΔyÞ, ð13:69aÞ

λþ ¼ λ þ αdual Δλ, ð13:69bÞ

where α prim and αdual are the stepsizes that ensure the positivity of x+ and λ+. From (13.64) and
(13.65), it follows that the new residuals satisfy the following relations:
 
rþ
p ¼ 1α
prim
rp , ð13:70aÞ
   
rþ
d ¼ 1α
dual
r d þ αprim  αdual GΔx: ð13:70bÞ
þ
If α prim ¼ αdual ¼ α, then r þp ¼ ð1  αÞr p and r d ¼ ð1  αÞr d , that is, both residuals decrease
linearly for all α 2 (0, 1). For different stepsizes, the residual r þ
d may increase for certain values of
α prim and αdual. Therefore, the interior point algorithms use equal stepsizes, as in (13.66), and set

α ¼ min αprim τ , αdual

τ , where

αprim
τ ¼ max fα 2 ð0, 1 : y þ αΔy ð1  τÞyg, ð13:71Þ

αdual
τ ¼ max fα 2 ð0, 1 : λ þ αΔλ ð1  τÞλg: ð13:72Þ

The interior point algorithm for convex quadratic programming is based on the Mehrotra
predictor-corrector method, originally developed for linear programming (Mehrotra, 1992).
Firstly, an affine scaling step (Δxaff, Δyaff, Δλaff) is computed by setting σ ¼ 0 in (13.64). Next,
this step is improved by computing a corrector step and the centering parameter σ ¼ (μaff/μ)3, where
466 13 Quadratic Programming

 T  
μaff ¼ y þ b
αaff Δyaff λþb
αaff Δλaff =m: The total step is obtained by solving the following linear
system:
2 32 3 2 3
G 0 AT Δx r d
6 76 7 6 7
4A I 0 54 Δy 5 ¼ 4 r p 5: ð13:73Þ
0 Λ Y Δλ ΛYe  ΔΛ ΔY þ σμe
aff aff

Algorithm 13.9 Predictor-corrector algorithm for quadratic programming

1. Compute (x0, y0, λ0) such that y0 > 0 and λ0 > 0. Set k ¼ 0
2. Test a criterion for stopping the iterations
3. Set σ ¼ 0. Set (x, y, λ) ¼ (xk, yk, λk) and solve (13.64) subject to (Δxaff, Δyaff, Δλaff)
4. Compute μ ¼ yTλ/m
 

5. Compute b αaff ¼ max α 2 ð0, 1 : ðy, λÞ þ α Δyaff , Δλaff 0

 T  
6.
Compute μaff ¼ y þ b αaff Δyaff λþbαaff Δλaff =m
7. Set the centering parameter to σ ¼ (μaff/μ)3
8. Solve (13.73) subject to (Δx, Δy, Δλ)
n o
9. Choose τk 2 (0, 1) and set b
α ¼ min αprim τ k , ατ k
dual
 
10. Set xkþ1 , ykþ1 , λkþ1 ¼ ðxk , yk , λk Þ þ b
αðΔx, Δy, ΔλÞ
11. Set k ¼ k þ 1 and go to step 2 ♦

To accelerate the convergence, τk can be selected to approach 1 as the iterates approach the
solution. The performances of the algorithm are dependent on the initial point. A simple procedure for
 
selecting the initial point is as follows. Firstly, an initial point x, y, λ is defined and moved far
enough from the boundary of the region (y, λ) 0 to permit the algorithm to take long steps on the
early iterations. Next, the affine scaling step (Δxaff, Δyaff, Δλaff) from the supplied initial point
   

x, y, λ is computed. Then, x0 ¼ x, y0 ¼ max {1, jy þ Δyaffj}, λ0 ¼ max 1, λ þ Δλaff  are set,

where max and the absolute value are applied component wise.
Some comparisons between the active set and the interior point methods are as follows. The
active-set methods require a large number of iterations, in which each search direction is relatively
inexpensive to compute. On the other hand, the interior point methods take a smaller number of
iterations, each of them being more expensive. Active-set methods are complicated to implement in
computing programs, especially if the procedures for updating the matrix factorizations try to take
advantage of the sparsity or of the structure of G and A. In case of interior point methods, the nonzero
structure of the matrix to be factored at each iteration remains the same at all iterations. For very large
problems, the interior point methods are more efficient. However, if an estimate of the solution is
available, then the active-set methods may rapidly converge in just a few iterations, particularly if the
initial value of x is feasible.
13.4 Methods for Convex QP Problems with Equality Constraints 467

13.4 Methods for Convex QP Problems with Equality Constraints

Consider the convex equality constrained quadratic programming problem

1
min qðxÞ  xT Gx þ cT x
2
subject to ð13:74Þ
Ax ¼ b,

where G 2 ℝnn is a symmetric and positive definite matrix and A 2 ℝmn is of full row rank m,
where m < n. For solving this problem, the following methods can be presented:

1. Method Based on KKT Conditions

The first-order necessary optimality conditions for (13.74) are given by

Gx þ c  AT λ ¼ 0, ð13:75aÞ

Ax þ b ¼ 0: ð13:75bÞ

If G is positive definite and A has full row rank, then the matrix
" #
G AT
A 0

is nonsingular, and the solution x of (13.75) is the unique global minimizer of the convex equality
quadratic programming problem (13.74). Therefore, the solution x and the Lagrange multipliers λ
can be expressed as
 1  1 
λ ¼ AG1 AT AG c þ b , ð13:76aÞ
 
x ¼ G1 AT λ  c : ð13:76bÞ

However, the solution of the symmetric system (13.75) can be obtained by using numerical
methods that are often more efficient and more reliable than the formulae (13.76) (Golub, & Van
Loan, 1996).

2. Methods Based on Singular Value Decomposition of A

Indeed, the solutions of the system Ax ¼ b from (13.74) have the form x ¼ A+b þ Vrz, where Vr is
composed of the last n  m columns of the matrix V, and V is obtained from the singular value
decomposition (SVD) of A, namely, UΣVT (see Appendix A). With this, using this form of x in
(13.74), the constraints can be eliminated, thus obtaining an unconstrained minimization problem
1 b
min qbðzÞ ¼ zT Gz þ cbT z, ð13:77Þ
2
where
b ¼ V T GV r
G and cb ¼ V Tr ðGAþ b þ cÞ: ð13:78Þ
r

b is also positive definite, and the unique global minimizer of (13.74)

If G is positive definite, then G
is given by
468 13 Quadratic Programming

x ¼ V r z þ Aþ b, ð13:79Þ

where z is a solution of the linear system equations Gz b ¼ b c:

b
If G is positive semidefinite, then G may be either positive definite or positive semidefinite. If G b is

positive definite, then x given by (13.79) is the unique global minimizer of problem (13.74). If G b is
positive semidefinite, then there are two possibilities: (a) If cb can be expressed as a linear combination
of the columns of G,b then qbðzÞ has infinitely many global minimizers and so does q(x). (b) If cb is not a
linear combination of the columns of G, b then qbðzÞ and also q(x) have no minimizers.

3. Methods Based on QR Decomposition

This method is more economical and is based on the QR decomposition of AT, i.e.,
 
R
AT ¼ Q ,
0

where Q 2 ℝnn is an orthogonal matrix and R 2 ℝmm is an upper triangular matrix (see Appendix
A). Therefore, the constraints Ax ¼ b can be expressed as RT xb1 ¼ b, where xb1 is the vector composed
of the first m components of xb with xb ¼ QT x: Let us denote
 
xb1
xb ¼ and Q ¼ ½Q1 Q2 ,
z

where z 2 ℝnm, Q1 2 ℝnm, and Q2 2 ℝn(nm). Then,

x ¼ Q1 xb1 þ Q2 z ¼ Q2 z þ Q1 RT b,
x ¼ Qb ð13:80Þ

which is equivalent to (13.79).

Therefore, the parameterized solutions (13.80) can be used to convert problem (13.74) to the
reduced-size unconstrained minimization problem (13.77), where, this time
 
Gb ¼ QT GQ2 and cb ¼ QT GQ1 RT b þ c : ð13:81Þ
2 2

If G is positive definite, the unique global minimizer of (13.74) can be obtained as

x ¼ Q2 z þ Q1 RT b,

b ¼ b
where z is a solution of the linear system equations Gz c, where G b and cb are given by (13.81).
Of course, in both approaches presented above if G b is positive definite, then the linear system of
b ¼ b
equations Gz c can be efficiently solved by the Cholesky factorization.

13.5 Quadratic Programming with Simple Bounds: The Gradient

Projection Method

In the following, let us discuss the minimization of the quadratics in which the variables are simple
bounded by using the gradient projection approach. This is a very nice application of the optimization
with simple bounds on variables, presented in Chap. 12. Unlike the active-set method, the gradient
projection method allows the active set to rapidly change from iteration to iteration, thus reducing the
number of iterations for getting a solution. Consider the following problem:
13.5 Quadratic Programming with Simple Bounds: The Gradient Projection Method 469

1
min qðxÞ ¼ xT Gx þ cT x
2
subject to ð13:82Þ
l  x  u,

where G is a symmetric matrix and l and u are vectors of lower and upper bounds on variables. The
feasible region defined by l  x  u is called a box, because of its rectangular shape. Assume that
li < ui for all i. It is not necessary to make any positive definiteness assumptions on G, because the
gradient projection method can be applied for both convex and nonconvex problems.
Every iteration of the gradient projection method has two stages. In the first one, from the current
point x, searching is executed along the steepest descent direction, that is, along the direction Gx  c.
Whenever a bound is encountered, the search direction is bent so that it stays feasible. Therefore, the
search continues along a piecewise-linear path until the first local minimizer of q denoted by xc and
referred to as the Cauchy point is located. The working set is defined as the set of the bounds
constraints that are active at the Cauchy point, denoted by A(xc). In the second stage, a face of the
feasible box on which the Cauchy point lies is inspected by solving a subproblem in which the active
components xi, i 2 A(xc) are fixed at the values xci :

The Cauchy Point

The projection of an arbitrary point x on the feasible region l  x  u is defined as a vector whose i-th
component is
8
>
< li , if xi < li ,
Pðx, l, uÞi ¼ xi if li  xi  ui , ð13:83Þ
>
:
ui if xi > ui ,

Starting from the current point x, the piecewise-linear path x(t) obtained by projecting the steepest
descent direction at x onto the feasible region l  x  u is given by
xðtÞ ¼ Pðx  tg, l, uÞ, ð13:84Þ

where g ¼ Gx þ c. The Cauchy point xc is defined as the first local minimizer of the piecewise-
quadratic function q(x(t)), for t 0. Every line segments that define x(t) is examined to determine
this minimizer point. For this, the values of t for which each component reaches its bound along
g are determined. These values ti are computed as
8
< ðxi  ui Þ=gi , if gi < 0 and ui < þ1,
>
ti ¼ ðxi  li Þ=gi , if gi > 0 and li > þ1, ð13:85Þ
>
:
1 otherwise:

Therefore, for any t, the components of x(t) are as follows:

xi  tg i , if t  ti ,
xi ðtÞ ¼ ð13:86Þ
xi  ti gi , otherwise:

To determine the first local minimizer along P(x  tg, l, u), first eliminate the duplicate values and
the zero of ti from the set ft1 , . . . , tn g: Secondly, all these values are sorted, thus obtaining the
470 13 Quadratic Programming

reduced set of breakpoints {t1, . . ., tl} with 0 < t1 < ⋯ < tl. After that, the intervals [0, t1], [t1, t2], . . .
are examined in this order. Suppose that we have examined these intervals up to tj  1 and no local
minimizer has been found. For the interval [tj  1, tj], it follows that
 
xðtÞ ¼ x tj1 þ ðΔtÞdj1 ,

where
 
Δt ¼ t  tj1 2 0, tj  tj1 , ð13:87Þ

gi , if tj1 < ti ,
dj1
i ¼ ð13:88Þ
0, otherwise:

Therefore, on the line segment [x(tj  1), x(tj)], (13.82) can be written as
    1   T    
qðxðtÞÞ ¼ cT x tj1 þ ðΔtÞdj1 þ x tj1 þ ðΔtÞdj1 G x tj1 þ ðΔtÞdj1 ,
2
which can be rewritten as
1
qðxðtÞÞ ¼ f j1 þ f 0j1 Δt þ f 00j1 ðΔtÞ2 , ð13:89Þ
2
where Δt 2 [0, tj  tj  1], and the coefficients f j1 , f 0j1 and f 00j1 are given by
  1  T  
f j1 ≜ cT x tj1 þ x tj1 Gx tj1 ,
2
 T
f 0j1 ≜ cT d j1 þ x tj1 Gd j1 ,
 T
f 00j1 ≜ dj1 Gd j1 :

To determine Δt, the minimum of (13.89), we differentiate q(x(t)) with respect to Δt and set
0
q (x(t)) ¼ 0 to obtain Δt ¼ f 0j1 =f 00j1 : The following three cases can occur. (i) If f 0j1 > 0, then
there is a local minimizer of q(x(t)) at t ¼ tj  1. (ii) If Δt 2 [0, tj  tj  1), then there is a minimizer
at t ¼ tj  1 þ Δt. (iii) In all the other cases, continue the search with the next interval
[tj, tjþ1]. Obviously, for the next interval, the new direction d j from (13.88) needs to be computed,
with which the new values f j , f 0j , and f 00j are computed, etc.

Subproblem Minimization

After the Cauchy point xc has been computed, the active set Aðxc Þ ¼ i : xci ¼ li , or xci ¼ ui g is
determined. As we said, in the second stage of the gradient projection method, the quadratic
programming problem obtained by fixing the components xi for i 2 A(xc) at the values xci is solved.
That is, the remaining components xi for i 2
= A(xc) are determined as solution of the subproblem
13.6 Elimination of Variables 471

1
minqðxÞ ¼ xT Gx þ cT x
2
subject to ð13:90Þ
li  xi  ui , i=
2 Að x Þc

xi ¼ xci , i 2 Að x c Þ

Obviously, (13.90) does not need to be solved exactly. To obtain the global convergence of the
gradient projection algorithm, it suffices to determine an approximate solution xa of (13.90), which
satisfies the bounds and has an objective value q(xa)  q(xc). For this, Algorithm 13.2 (preconditioned
conjugate gradient for reduced systems) or Algorithm 13.3 (projected conjugate gradient) can be
used. Having in view the equalities constraints from (13.90), it follows that the Jacobian and the null-
space basis matrix Z used in Algorithm 13.2 have simple forms. The idea is to apply the conjugate
gradient to minimize q(x) from (13.90) subject to the equality constraints xi ¼ xci , i 2 A(xc) and to
terminate as soon as a bound l  x  u is encountered. Another strategy is to continue the iterations by
ignoring the bounds and finally to project the obtained solution onto the box constraints.
The gradient projection algorithm for quadratic programming with simple bounds can be presented
as follows.

Algorithm 13.10 Gradient projection for quadratic programming with box constraints

1. Compute a feasible initial point x0

2. For k ¼ 0, 1, . . .
(a) If xk satisfies the KKT optimality conditions for (13.82), stop with solution x ¼ xk; otherwise, set
x ¼ xk and determine the Cauchy point xc
(b) Find an approximate solution xa of (13.90) such that q(xa)  q(xc) and xa is feasible
(c) Set xkþ1 ¼ xa
End for ♦

If the strict complementarity holds (Definition 11.23), that is, the Lagrange multipliers associated
with all the active bounds are nonzero, then the active sets A(xc) generated by Algorithm 13.10 are
equal to the optimal active set for all k sufficiently large.
Of course, Algorithm 13.10 can be applied for minimizing the quadratic forms subject to linear
inequality constraints (13.60) aTi x bi , i ¼ 1, . . ., m. However, in this case, the difficulty is to
perform the projection on the feasible set. For this, to compute the projection of a given point xk onto
the set defined by aTi x bi , i ¼ 1, . . ., m, the following convex quadratic programming problem

max kx  xk k2 subject to aTi x bi , i ¼ 1, . . . , m,

x

has to be solved. However, this is a difficult problem; the cost of its solving may approach the cost of
solving the original problem. Therefore, for minimizing quadratic forms subject to linear inequality
constraints, it is not advisable to apply the gradient projection algorithm.

13.6 Elimination of Variables

Often, quadratic programming with equality constraints can be solved by eliminating the variables. In
this method, the equality constraints are used to eliminate the variables, thus obtaining another
simplified quadratic programming problem with a smaller number of variables. Consider the problem
472 13 Quadratic Programming

1
minqðxÞ  xT Gx þ cT x
2
subject to ð13:91Þ
Ax þ b ¼ 0,

where G is an n  n symmetric matrix, A 2 ℝmn with rank(A) ¼ m and m < n. The method of
eliminating the variables can be implemented in different manners, as follows:

1. The Elimination of Variables Method

Suppose that for the problem (13.91), rank(A) ¼ m, that is, A has full column rank, where m < n. Then,
the matrix A can be partitioned as A ¼ [B N], where B 2 ℝmm is a nonsingular submatrix.
Therefore, the system of constraints Ax þ b ¼ 0 can be written as BxB þ NxN þ b ¼ 0, where
x ¼ [xB xN]T is the corresponding partition of x, with xB 2 ℝm and xN 2 ℝnm, respectively. With
this, xB ¼  B1(NxN þ b).
Therefore, the objective function (13.91) can be written as
    
1  T T  GB GBN xB  T  xB
qð x Þ ¼ x x þ cB cN T , ð13:92Þ
2 B N GNB GN xN xN

where GB 2 ℝmm, GN 2 ℝ(nm)  (nm), GBN 2 ℝm(nm), GNB ¼ GTBN , cB 2 ℝm, and cN 2 ℝnm.
Now, substituting xB into the objective function (13.92), we obtain a reduced quadratic programming
problem in xN, i.e.,
1 Tb
min x G x þ cbTN xN ð13:93Þ
xN 2ℝnm 2 N N N

bN and cbN are obtained from the algebraic manipulation of the above substitution, that is,
where G

GbN ¼ GN þ N T BT GB B1 N  GNB B1 N  N T BT GBN ,

 
cbN ¼ cN  N T BT cB þ N T BT GB  GNB B1 b:
 
b1 cbN , which is unique. Obviously, x ¼ B1 Nx þ b :
bN is positive definite, then x ¼ G
If G N N B N
Therefore, the solution of problem (13.91) is
  " # " #

xB B1 b B1 N b1
x ¼  ¼ þ GN cbN : ð13:94Þ
xN 0 I

The Lagrange multipliers λ at x are obtained from the equation c þ Gx ¼ Aλ, using the above
partition of vectors and matrices.
If GbN is positive semidefinite, then, when

bþ Þb
bN G
ðI þ G N cN ¼ 0, ð13:95Þ
 
i.e., when cbN 2 R GbN , the minimization problem (13.93) has a solution, and its solution is
 
xN ¼ Gbþ cbN þ I  Gbþ G bþ is the generalized inverse of
bN xe, where xe 2 ℝnm is any vector and G
N N N

GbN : If (13.95) does not hold, then the problem (13.93) has no lower bound, and thus the problem
(13.91) also has no lower bound, i.e., it has no finite solution.
13.6 Elimination of Variables 473

bN has negative eigenvalues, it is clear that the minimization problem (13.93) has no lower
If G
bound.
This method of elimination of variables is simple, but when B is close to a singular matrix, then
computing the solution (13.94) is numerically unstable.

2. Generalized Elimination of Variables: The Null-Space Method

A generalized elimination of variables method is exactly the null-space method which uses the
reduced Hessian and the reduced gradient. Let {y1, . . ., ym} be a set of m linearly independent vectors
in R(A), the range of A, and {z1, . . ., zn  m} a set of n  m linearly independent vectors in N(AT), the
null space of AT. Consider the matrices

Y ¼ ½y1 , . . . , ym  2 ℝnm and Z ¼ ½z1 , . . . , znm  2 ℝnðnmÞ :

Obviously, R(Y ) ¼ R(A), R(Z ) ¼ N(AT), and ½ Y Z  is nonsingular. Besides, AY is a nonsingular

matrix and AZ ¼ 0.
With this, we can write
x ¼ Yx þ Zb
x,

where x 2 ℝm and xb 2 ℝnm :

Therefore, the constraints Ax þ b ¼ 0 of the problem (13.91) can be written as AYx ¼ b: Since
AY is nonsingular, it follows that x ¼ ðAY Þ1 b: Hence,

x ¼ Y ðAY Þ1 b þ Zb
x: ð13:96Þ

Substituting (13.96) in (13.91), we get

 T
1
x þ Z T c  ZT GY ðAY ÞT b xb:
min xbT ZT GZb ð13:97Þ
bx2ℝnm 2

Supposing that ZTGZ is positive definite, then the solution of (13.97) can be written as
 1  T 
xb ¼  ZT GZ Z c  Z T GY ðAY ÞT b : ð13:98Þ

Now, from (13.96) the solution of the problem (13.91) is

 1  T 
x ¼ Y ðAY Þ1 b  Z ZT GZ Z c  Z T GY ðAY ÞT b : ð13:99Þ

Moreover, from the KKT condition it follows that c þ Gx ¼ ATλ. Therefore, since AY is
nonsingular, we have λ ¼ (AY)TYT(c þ Gx), where x is given by (13.99).
Observe that the generalized elimination of variables uses the linear independent column vectors
{z1, . . ., znm} which form the matrix Z, which is a base of the null space of AT. This transforms the
quadratic programming problem (13.91) into an unconstrained minimization problem (13.97). That is
why this method is called the null-space method, in which the matrix ZTGZ is the reduced Hessian and
the vector ZT(c  GY(AY)Tb) is the reduced gradient. The null-space method is very useful for
small- and medium-scale problems and when the computation of the null space and the factorization
of the reduced Hessian are not very expensive.
Another approach for eliminating the variables is based on the elimination of the balance
constraints (Andrei & Bărbulescu, 1993; Andrei, 2011d). In the system of constraints from (13.91),
474 13 Quadratic Programming

a particular constraint i is called a balance constraint if bi ¼ 0. By eliminating the balance constraints,

a new quadratic programming problem is obtained, with a reduced number of constraints.

Notes and References

The quadratic programming problem has its importance in the nonlinear optimization area, many
optimization problems being modeled as quadratic programs. The most famous is the portfolio
optimization formulated by Markowitz (1952). This chapter is based on Nocedal and Wright
(2006) and Sun and Yuan (2006). Methods for general quadratic programming are presented in
Gill, Murray, Saunders, and Wright (1984) and Gould (1991). A quadratic programming bibliography
is given by Gould and Toint (2012). Quadratic programming is intensively used in sequential
quadratic programming, one of the most efficient methods in nonlinear optimization. We have
considered the active-set methods and the interior point for quadratic programming. Interior point
methods for convex quadratic programming can be found in Wright (1997) and Vanderbei (1994,
1996). The numerical comparison of the active-set and the interior point methods for convex
quadratic programming shows that the interior point methods are generally much faster on large
problems (Gould & Toint, 2002a). However, if a warm start is available, the active-set methods may
be generally preferable. Active-set methods for convex quadratic programming are implemented in
many codes: QPOPT (Gill & Murray, 1978), BQPD (Fletcher, 2000), QPA (Gould & Toint, 2002b),
DONLP (Spellucci, 1998). The interior point solvers for quadratic programming are CPLEX (ILOG
CPLEX, 2002), XPRESS-MP (Guéret, Prins, & Sevaux, 2002), MOSEK (Andersen & Andersen,
2000), etc. The problem of determining whether a feasible point for a nonconvex quadratic program is
an NP-hard problem was proven in Murty and Kabadi (1987). Also, the problem of determining
whether a given point is a local minimizer is NP-hard (Vavasis, 1990).
 Penalty and Augmented Lagrangian
Methods 14

This chapter introduces two very important concepts in the constrained nonlinear optimization. These
are penalty and augmented Lagrangian. Both concepts replace the original problem by a sequence of
subproblems in which the constraints are expressed by terms added to the objective function. The
penalty concept is implemented in two different methods. The quadratic penalty method adds a
multiple of the square of the violation of each constraint to the objective function and solves a
sequence of unconstrained optimization subproblems. Although simple and intuitive, this approach
has some major deficiencies. The nonsmooth exact penalty method, on the other hand, solves a single
unconstrained optimization problem. In this approach, a popular function is the l1 penalty function.
The problem is that the nonsmoothness may create complications in numerical implementations.
Finally, the second concept is the multipliers method or the augmented Lagrangian method, which
explicitly uses Lagrange multiplier estimates in order to avoid the ill-conditioning of the quadratic
penalty method.
The modern and best known methods for solving nonlinear optimization problems combine the
penalty concept with the augmented Lagrangian in a penalty barrier with quadratic approximation of
the inequality constraints (SPENBAR), or the minimization of a modified augmented Lagrangian
subject to the linearized constraints (MINOS), or the minimization of the augmented Lagrangian
subject to the simple bounds on variables (LANCELOT). In this chapter, we present the theory behind
the algorithms SPENBAR (Andrei, 1996a, b, c, 1998a, 2001, 2015a, 2017c) and MINOS (Murtagh &
Saunders, 1978, 1980, 1982, 1987, 1995) together with some numerical results for solving some
applications from the LACOP collection.

14.1 The Quadratic Penalty Method

Let us consider the equality constrained optimization problem

min ff ðxÞ : hi ðxÞ ¼ 0, i 2 Eg ð14:1Þ

where f : ℝn ! ℝ and hi : ℝn ! ℝ, i 2 E ≜ {1, . . ., p}, are continuously differentiable functions

defined on ℝn. The quadratic penalty function Q(x, σ) for the problem (14.1) is

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 475
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_14
476 14 Penalty and Augmented Lagrangian Methods

σX 2
Qðx, σ Þ≜f ðxÞ þ h ðxÞ, ð14:2Þ
2 i2E i

where σ > 0 is the penalty parameter. Observe that if σ is increased to 1, the constraint violations are
penalized. Therefore, the idea is to consider a sequence of {σ k} with σ k ! 1 as k ! 1 and to seek
the approximate minimum xk of Q(x, σ k) for each k. Since the penalty term in (14.2) is smooth, any
unconstrained optimization method can be used for xk computation. Therefore, a sequence of
unconstrained minimization subproblems min x2ℝn Qðx, σ k Þ is solved for each value of the penalty
parameter σ k. Clearly, in searching for xk, the approximate minimum xk1 of the previous subproblem
may be used.
Now, let us consider the general nonlinear optimization problem

min ff ðxÞ : ci ðxÞ  0, i 2 I, hi ðxÞ ¼ 0, i 2 Eg, ð14:3Þ

where f : ℝn ! ℝ, ci : ℝn ! ℝ, i 2 I ≜ {1, . . ., m} and hi : ℝn ! ℝ, i 2 E ≜ {1, . . ., p}, which

contains both inequality and equality constraints. For this problem, the following quadratic penalty
function may be defined:
σX 2 σX
Qðx, σ Þ≜ f ðxÞ þ h i ðxÞ þ ð½ c ðxÞ  Þ2 , ð14:4Þ
2 i2E 2 i2I i

where [y] ¼ max {y, 0}. In this case, the quadratic penalty function may be less smooth than the
original objective and constraints functions of problem (14.1).
The quadratic penalty method is based on the minimization of the quadratic penalty function in the
following framework.

Algorithm 14.1 Quadratic penalty method

1. Consider an initial starting point xs0 : Select an initial value σ 0 > 0 for the penalty parameter and a
nonnegative sequence {τk} with τk ! 0. Set k ¼ 0
2. Find an approximate minimum xk of the sub-problem min x2ℝn Qðx, σ k Þ starting at xsk , and terminate the
iterations when k∇xQ(x, σ k)k  τk
3. If a test of convergence of the method is satisfied, then stop with xk as an approximate solution of the
problem
4. Choose a new penalty parameter σ kþ1 > σ k
5. Choose a new starting point xskþ1 , set k ¼ k þ 1 and go to step 2 ♦

Some remarks are as follows:

1. The sequence of the penalty parameters {σ k} may be adaptively selected with respect to the
difficulty in minimizing the penalty function at each iteration k. When the minimization of
Q(x, σ k) is too expensive, then σ kþ1 can be chosen near σ k, for example, σ kþ1 ¼ 1.5σ k. On the
other hand, if the minimization of Q(x, σ k) is relatively easy to be done, then σ kþ1 can be selected
larger than σ k, for example, σ kþ1 ¼ 10σ k. Of course, some other schemes for enlarging σ k may be
adopted.
2. In case the problem has only equality constraints, then Q(x, σ k) is smooth. Therefore, the
algorithms for the unconstrained optimization (in step 2) can be used to find an approximate
solution xk. However, the minimization of Q(x, σ k) becomes more difficult as σ k becomes larger.
14.1 The Quadratic Penalty Method 477

Near the minimum, the Hessian ∇2Q(x, σ k) becomes ill-conditioned, and in this case, the quasi-
Newton or the conjugate gradient methods perform very poorly.

Theorem 14.1 Suppose that xk for any k  0 is the exact global minimum of Q(x, σ k) defined in
(14.2), determined by Algorithm 14.1. Suppose that σ k ! 1 . Then, every limit point x of the
sequence {xk} is a global solution of the problem (14.1).

Proof Let x be a global solution of (14.1), that is, f ðxÞ  f ðxÞ for all x satisfying hi(x) ¼ 0, i 2 E.
Since xk is a minimum of Q(x, σ k) for each k, we have Qðxk , σ k Þ  Qðx, σ k Þ, which leads to the
inequality
σ X 2 σ X 2
f ðxk Þ þ k hi ð x k Þ  f ð x Þ þ k h ðxÞ ¼ f ðxÞ: ð14:5Þ
2 i2E 2 i2E i

By rearranging the terms of (14.5), we get

X 2
h2i ðxk Þ  ½f ðxÞ  f ðxk Þ: ð14:6Þ
i2E
σ k

Suppose that x is a limit point of {xk} so that there is an infinite sequence K, such that
limk 2 Kxk ¼ x. By taking the limit as k ! 1 , k 2 K, on both sides of (14.6), we obtain
X X 2
h2i ðx Þ ¼ lim h2i ðxk Þ  lim ½f ðxÞ  f ðxk Þ ¼ 0,
k!1, k2K k!1, k2K σ k
i2E i2E

where the last equality follows from the fact that σ k ! 1 . Therefore, we have hi(x) ¼ 0 for all i 2 E,
so that x is feasible. Additionally, by taking the limit as k ! 1 for k 2 K in (14.5) and by the
nonnegativity of σ k and of each h2i ðxk Þ, we get
σk X 2
f ðx Þ  f ðx Þ þ lim hi ðxk Þ  f ðxÞ:
k!1, k2K 2
i2E

Since x is a feasible point whose objective function is not larger than that of the global solution x,
it follows that x is also a global solution, as claimed. ♦

Observe that this result requires finding the global minimum of each subproblem. In general, this
property of convergence to the global solution of (14.1) cannot be attained. Nocedal and Wright
(2006) prove the convergence of the sequence {xk} when the inexact minimization of Q(x, σ k) is
allowed, as in the following theorem.

Theorem 14.2 Suppose that the tolerances and the penalty parameters satisfy τk ! 0 and σ k ! 1 .
Then, if a limit point x of the sequence {xk} is infeasible, then there is a stationary point of the
function kh(x)k2. On the other hand, if a limit point x is feasible and the constraint gradients ∇hi(x),
i 2 E, are linearly independent, then x is a KKT point for the problem (14.1). For such points, for any
infinite sequence K such that limk 2 Kxk ¼ x, we have
478 14 Penalty and Augmented Lagrangian Methods

lim  σ k hi ðxk Þ ¼ λi , ð14:7Þ

k!1, k2K
h i
for all i 2 E, where λ ¼ λ1 , . . . , λp is the multiplier vector that satisfies the KKT optimality
conditions (11.21) and (11.22) for the equality constrained problem (14.1).

Proof From (14.2), we have

X
∇Qðxk , σ k Þ ¼ ∇f ðxk Þ þ σ k hi ðxk Þ∇hi ðxk Þ: ð14:8Þ
i2E

Therefore, from the termination criterion in step 2 of Algorithm 14.1, it follows that
 
 X 
 
∇f ðxk Þ þ σ k hi ðxk Þ∇hi ðxk Þ  τk : ð14:9Þ
 i2E


Having in view that kak  kbk  ka þ bk, from (14.9), we get

 
X 
  1
 hi ðxk Þ∇hi ðxk Þ  ½τk þ k∇f ðxk Þk: ð14:10Þ
 i2E  σk

Let x be a limit point of the sequence {xk}. Then, there is a subsequence K such that limk 2 Kxk ¼ x.
When k ! 1 for k 2 K, τk þ k∇f(xk)k ! k∇f(xk)k. Since σ k ! 1 , the right-hand side of (14.10)
approaches zero. Therefore, from the corresponding limit of the left-hand side of (14.10), we have
X
hi ðx Þ∇hi ðx Þ ¼ 0: ð14:11Þ
i2E

Now, if the constraints gradients ∇hi(x) are linearly dependent, then from (14.11), hi(x) 6¼ 0. In
this case, x is a stationary point of the function kh(x)k2. On the other hand, if the constraint gradients
∇hi(x) are linearly independent at the limit point x, from (14.11), it follows that hi(x) ¼ 0 for all
i 2 E, i.e., x is feasible. Hence, the KKT condition (11.13b) is satisfied. We need to check (11.13a)
and to show that (11.15) holds.
Let A(x)T ¼ [∇hi(x)]i2E be the matrix of the constraint gradients and λk ¼  σ kh(xk). From (14.9), it
follows that

Aðxk ÞT λk ¼ ∇f ðxk Þ  ∇Qðxk , σ k Þ, k∇Qðxk , σ k Þk  τk : ð14:12Þ

For all k 2 K sufficiently large, the matrix A(xk) is of full rank; therefore, A(xk)A(xk)T is
nonsingular. From the first relation of (14.12), we get
h i1
λk ¼ Aðxk ÞAðxk ÞT Aðxk Þ½∇f ðxk Þ  ∇Qðxk , σ k Þ: ð14:13Þ

Taking the limit when k 2 K goes to 1, we obtain

h i1
lim k!1,k2K λk ¼ λ ¼ Aðx ÞAðx ÞT Aðx Þ∇f ðx Þ: ð14:14Þ

Therefore, taking the limit in (14.9), we get

14.2 The Nonsmooth Penalty Method 479

∇f ðx Þ  Aðx ÞT λ ¼ 0, ð14:15Þ

i.e., λ satisfies the first KKT condition (11.13a) for the problem (14.1). In conclusion, x is a KKT
point for (14.1) with the unique Lagrange multiplier vector λ. ♦

Observe that the quantities σ khi(xk) may be used as estimates of the Lagrange multipliers λi , i 2 E.
This is important for the analysis of the augmented Lagrangian method.
Let us study the ill-conditioning of the Hessian ∇2Q(x, σ k). This analysis is extremely important in
choosing effective algorithms for the minimization in step 2 of Algorithm 14.1, as well as the linear
algebra calculations at each iteration k. From (14.2), we get
X
∇Qðx, σ k Þ ¼ ∇f ðxÞ þ σ k hi ðxÞ∇hi ðxÞ: ð14:16Þ
i2E
X
∇2 Qðx, σ k Þ ¼ ∇2 f ðxÞ þ σ k hi ðxÞ∇2 hi ðxÞ þ σ k AðxÞT AðxÞ, ð14:17Þ
i2E

where A(x)T ¼ [∇hi(x)]i2E. In the conditions of Theorem 14.2, from (14.7), it follows that the sum of
the first two terms on the right-hand side of (14.17) is approximately equal to the Hessian of the
Lagrangian function. Therefore, when x is close to the minimum point of Q(x, σ k), it follows that

∇2 Qðx, σ k Þ  ∇2 Lðx, λ Þ þ σ k AðxÞT AðxÞ: ð14:18Þ

Note that ∇2Q(x, σ k) is the sum of two matrices: a matrix whose elements are independent of σ k
and a matrix of rank card(E) whose nonzero eigenvalues are of order σ k. Usually, the number of
constraints card(E) is smaller than the number of variables n. In this case, the last term in (14.18) is
singular. Therefore, the whole matrix ∇2Q(x, σ k) has some eigenvalues approaching a constant, while
the others are of order σ k. Since σ k ! 1 , it follows that the ill-conditioning of the matrix ∇2Q(x, σ k)
is increasing as k is increasing. This is the major difficulty associated to the quadratic penalty method.
Note that this ill-conditioning is responsible for a possible inaccuracy in the computations of the
Newton step p for the subproblem min x2ℝn Qðx, σ k Þ, which is obtained by solving the linear algebraic
system

∇2 Qðx, σ k Þp ¼ ∇Qðx, σ k Þ: ð14:19Þ

Clearly, the poor conditioning of this system will lead to significant errors in the value of p,
regardless of the computational technique used to solve (14.19).

14.2 The Nonsmooth Penalty Method

As already seen, the quadratic penalty function is not exact. Its minimum point is generally not the
same as the solution of the nonlinear optimization problem for any positive value of the penalty
parameter σ. This penalty function is called inexact. In this context, the exact penalty function is
introduced, in which for certain choices of its penalty parameter, a single minimization with respect to
x can yield the exact solution of the nonlinear optimization problem (Han & Mangasarian, 1979). In
this section, we consider the nonsmooth exact penalty functions (Nocedal & Wright, 2006).
480 14 Penalty and Augmented Lagrangian Methods

For the general nonlinear optimization problem (14.3), the l1 penalty function is defined by
X X
φ1 ðx, σ Þ ¼ f ðxÞ þ σ jhi ðxÞj þ σ ½ci ðxÞ , ð14:20Þ
i2E i2I

where [y] ¼ max {0, y}. The following result is proved by Han and Mangasarian (1979).

Theorem 14.3 Suppose that x is a strict local solution of the problem (14.3) at which the first-order
necessary optimality conditions (11.21) hold with the Lagrange multipliers μi , i 2 I, and λi , i 2 E.
Then, x is a local minimum of φ1(x, σ) for all σ > σ , where
    
σ  ¼ max max i2I μi  , max i2E λi  : ð14:21Þ

If, in addition, the second-order sufficient conditions (11.22) hold and σ > σ , then x is a strict
local minimum of φ1(x, σ). ♦

The following definition characterizes the stationary points of φ1, even if φ1 is not differentiable.

Definition 14.1 A point x 2 ℝn is a stationary point of the penalty function φ1(x, σ) if

Dðφ1 ðx, σ Þ; pÞ  0, ð14:22Þ

for all p 2 ℝn. Similarly, x is a stationary point of the measure of infeasibility

X X
gð x Þ ¼ j hi ð x Þ j þ ½ci ðxÞ ð14:23Þ
i2E i2I

if DðgðxÞ, pÞ  0 for all p 2 ℝn.

If a point is infeasible for (14.3) but stationary with respect to the infeasibility measure g, then it is
an infeasible stationary point. ♦

The following theorem shows that under some mild assumptions, the stationary points of φ1(x, σ)
correspond to the KKT points of the constrained nonlinear optimization problem (14.3).

Theorem 14.4 Suppose that x is a stationary point of the penalty function φ1(x, σ) for all σ > σ > 0,
where σ is a certain threshold. Then, if x is feasible for the nonlinear optimization problem (14.3), it
satisfies the KKT optimality conditions (11.22) for (14.3). If x is not feasible for (14.3), then it is an
infeasible stationary point.

Proof Suppose that x is feasible. Therefore,

X  X h i
Dðφ1 ðx, σ Þ; pÞ ¼ ∇f ðxÞT p þ σ ∇hi ðxÞT p þ σ ∇ci ðxÞT p ,
i2E i2I\AðxÞ

where AðxÞ is the set of active constraints at x: Consider any direction p that satisfies ∇hi ðxÞT p ¼ 0
for all i 2 E and ∇ci ðxÞT p  0 for all i 2 I \ AðxÞ: Therefore,
X  X h i
∇hi ðxÞT p þ ∇ci ðxÞT p ¼ 0: ð14:24Þ
i2E i2I\AðxÞ
14.2 The Nonsmooth Penalty Method 481

By the stationarity assumption on φ1 ðxÞ, it follows that

Dðφ1 ðx, σ Þ; pÞ ¼ ∇f ðxÞT p  0,

for all p satisfying (14.24). By Theorem A4.2 (Farkas), it follows that

X
∇f ðxÞ ¼ λi ∇ci ðxÞ,
i2Aðx Þ

for some coefficients λi with λi  0, i 2 I \ AðxÞ: However, this expression implies that the KKT
optimality conditions (11.21) hold. The second part of the proof is very simple and it is omitted
here. ♦

With these results, the following algorithm based on the classical l1 penalty function may be
presented.

Algorithm 14.2 l1 penalty method

1. Consider an initial starting point xs0 : Select an initial value σ 0 > 0 for the penalty parameter and a tolerance
τ > 0. Set k ¼ 0
2. Starting with xsk , find an approximate minimum xk of φ1(x, σ k)
3. If g(xk)  τ, then stop, with xk as an approximate solution of the problem
4. Choose a new penalty parameter σ kþ1 > σ k
5. Choose a new starting point xskþ1 , set k ¼ k þ 1, and go to step 2 ♦

Because of nonsmoothness, the minimization of φ1(x, σ k) in step 2 above is the most difficult
operation in Algorithm 14.2. A practical l1 penalty method will be presented below.
The scheme for updating the penalty parameter σ k in step 4 is to increase it by an arbitrary constant
factor, for example, σ kþ1 ¼ 5σ k. If σ k is too small, then many cycles of Algorithm 14.2 may be
required. On the other hand, if σ k is too large, then the penalty function will be difficult to minimize.
Obviously, φ1(x, σ k) is nonsmooth. The gradient of φ1(x, σ k) is not defined at any x for which
ci(x) ¼ 0 for some i 2 I and hi(x) ¼ 0 for some i 2 E. In order to avoid the difficulties for minimizing
φ1(x, σ k), the strategy is to consider a simplified model of this function and try to find the minimum of
this model. The most direct idea is to define the model by linearizing the constraints and by replacing
the objective function with a quadratic function, as follows:
1
qðp; σ Þ ¼ f ðxÞ þ ∇f ðxÞT p þ pT Wp
P 2  Ph i
þσ hi ðxÞ þ ∇hi ðxÞT p þ σ ci ðxÞ þ ∇ci ðxÞT p , ð14:25Þ
i2E i2I

where W is a symmetric matrix which contains second derivatives information about the functions of
the problem (14.3). The model (14.25) is not smooth, but it can be reformulated as a smooth quadratic
programming problem by introducing the artificial variables ri, si, i 2 E, and ti, i 2 I, as follows:
482 14 Penalty and Augmented Lagrangian Methods

1 X X
min p,r,s,t f ðxÞ þ ∇f ðxÞT p þ pT Wp þ σ ðr i þ si Þ þ σ ti
2 i2E i2I
subject to: ð14:26Þ
hi ðxÞ þ ∇hi ðxÞT p ¼ r i  si , i 2 E,
ci ðxÞ þ ∇ci ðxÞT p  ti , i 2 I,
r, s, t  0:

This problem can be solved by any standard quadratic programming solver.

In this algorithm, by using the quadratic programming (14.26), the updating of the penalty
parameter σ k remains a crucial problem. Discussions about this aspect of the algorithm are presented
by Nocedal and Wright (2006). An idea is to choose an initial value of the penalty parameter and to
increase it until the feasibility has been attained. Another variant is to update the penalty parameter so
that σ k is greater than an estimate of the Lagrange multipliers computed at xk.
The development of the filter method (Fletcher & Leyffer, 2002), which does not require any
penalty parameter, placed the nonsmooth penalty methods in a cone of penumbra. However, these
methods have not been completely abandoned, partly because of their ability to handle degenerate
problems (see Remark 11.8).

14.3 The Augmented Lagrangian Method

This section discusses the augmented Lagrangian method, known as the method of multipliers.
Hestenes (1969) and Powell (1969) independently proposed these methods as a possibility of
reducing the ill-conditioning by introducing explicit Lagrange multiplier estimates into the function
to be minimized, known as the augmented Lagrange function. In the following, we consider the
problems with equalities constraints: min{f(x) : hi(x) ¼ 0, i 2 E}. As seen in Theorem 14.2, the
approximate minimum xk of Q(x, σ k) does not satisfy the feasibility constraints hi(x) ¼ 0, i 2 E.
Instead, the constraints are perturbed (see Eq. (14.7)) so that

hi ðxk Þ  λi =σ k , ð14:27Þ

for all i 2 E. Indeed, as σ k ! 1 , hi(xk) ! 0. However, is it possible to avoid this systematic

perturbation of Q(x, σ k), that is, to make the approximate minimizers better satisfy the constraints
hi(x) ¼ 0, even for moderate values of σ k?
The augmented Lagrangian function
X σX 2
LA ðx, λ, σ Þ≜f ðxÞ  λi h i ðxÞ þ h ðxÞ, ð14:28Þ
i2E
2 i2E i

achieves this goal by including the explicit estimate of the Lagrange multipliers λi. Observe that the
augmented Lagrangian (14.28) is a combination of the Lagrangian function and of the quadratic
penalty function. An algorithm based on the augmented Lagrangian function at iteration k fixes the
penalty parameter σ to a value σ k > 0 and fixes λ at the current estimate λk. Thus, a minimization of the
augmented Lagrangian with respect to x is performed. Let xk be the approximate minimum of
LA(x, λk, σ k). Therefore, by the optimality conditions for unconstrained optimization (see Theorem
11.3), we have
14.3 The Augmented Lagrangian Method 483

  X
∇LA xk , λk , σ k ¼ ∇f ðxk Þ  λki  σ k hi ðxk Þ ∇hi ðxk Þ ¼ 0: ð14:29Þ
i2E

Now, comparing this result with the optimality condition (14.15) for (14.1), it follows that

λi  λki  σ k hi ðxk Þ, ð14:30Þ

for all i 2 E. From (14.30), we get

1  
hi ð x k Þ   λi  λki ,
σk

for all i 2 E. Therefore, if λk is close to the optimal multiplier vector λ, the infeasibility at xk will be
much smaller than (1/σ k), more than being proportional to (1/σ k) as in (14.27). The relation (14.30)
suggests a formula for updating the current estimate λk of the Lagrange multiplier vector by using the
approximate minimum xk already calculated as

λkþ1
i ¼ λki  σ k hi ðxk Þ, i 2 E: ð14:31Þ

With these theoretical developments, the augmented Lagrangian algorithm for nonlinear optimi-
zation problems with equality constraints can be presented as follows.

Algorithm 14.3 Augmented Lagrangian method: Equality constraints

1. Consider the initial starting points xs0 and λ0. Select an initial value σ 0 > 0 for the penalty parameter and a
tolerance τ0 > 0. Set k ¼ 0
2. Find an approximate minimum xk of the LA(x, λk, σ k) starting at xsk : If k∇LA(xk, λk, σ k)k  τk, stop the
iterations
3. If a convergence test for (14.1) is satisfied, then stop with xk as an approximate solution of the problem
4. Update the Lagrange multipliers as in (14.31) to get λkþ1
5. Choose a new penalty parameter σ kþ1  σ k
6. Set the starting point for the new iteration as xskþ1 ¼ xk
7. Choose the tolerance τkþ1, set k ¼ k þ 1, and go to step 2 ♦

It was proved that the convergence of this method is assured without increasing σ indefinitely. In
other words, the ill-conditioning of this method is reduced; thus, the choice of the starting point xskþ1
is less critical than in the previous algorithms based on the quadratic or the nonsmooth penalty
methods. The tolerance τk could be chosen by using the infeasibility measure ∑i2Ejhi(xk)j, and the
penalty parameter σ may be updated (increased) if the reduction in this infeasibility measure is
insufficient at the current iteration (Nocedal & Wright, 2006).

Theorem 14.5 Let x be a local solution of (14.1) at which the gradients ∇hi(x), i 2 E, are linearly
independent and the second-order sufficient conditions given in Theorem 11.13 are satisfied for
λ ¼ λ. Then, there is a threshold σ such that for all σ  σ, x is a strict local minimum of the
augmented Lagrangian LA(x, λ, σ).

Proof We show that x satisfies the second-order sufficient conditions for a strict local minimum of
LA(x, λ, σ) given in Theorem 11.6, for all σ sufficiently large, i.e.,
484 14 Penalty and Augmented Lagrangian Methods

∇LA ðx , λ , σ Þ ¼ 0, ð14:32Þ

∇2 LA ðx , λ , σ Þ is positive definite: ð14:33Þ

Now, as x is a local solution of (14.1) at which LICQ is satisfied, from Theorem 11.11 (see
Remark 11.3), it follows that ∇L(x, λ) ¼ 0 and hi(x) ¼ 0, i 2 E. Therefore,
P 
∇LA ðx , λ , σ Þ ¼ ∇f ðx Þ  λi  σhi ðx Þ ∇hi ðx Þ
P
i2E
¼ ∇f ðx Þ  λi ∇hi ðx Þ ¼ ∇Lðx , λ Þ ¼ 0
i2E

verifies (14.32), independently of σ. Further on, to verify (14.33), consider

∇2 LA ðx , λ , σ Þ ¼ ∇2 Lðx , λ Þ þ σAðx ÞT Aðx Þ,

where A(x)T ¼ [∇hi(x)]i 2 E.

If ∇2LA(x, λ, σ) is not positive definite, then, for each integer k  1, we could choose a vector wk
with kwkk ¼ 1, such that

0  wTk ∇2 LA ðx , λ , kÞwk ¼ wTk ∇2 Lðx , λ Þwk þ kkAðx Þwk k22 , ð14:34Þ

and therefore,
1
kAðx Þwk k22   wTk ∇2 Lðx , λ Þwk ! 0, ð14:35Þ
k
as k ! 1 . Since the vectors {wk} lie in a compact set (the surface of the unit sphere), they have an
accumulation point w. From (14.35), it follows that A(x)w ¼ 0. Now, from (14.34), we have

wTk ∇2 Lðx , λ Þwk  kkAðx Þwk k22  0,

So, by taking the limits, we have wT∇2L(x, λ)w  0. However, this inequality contradicts the
second-order condition in Theorem 11.13, which states that we must have wT∇2L(x, λ)w > 0 for all
nonzero vectors w with A(x)w ¼ 0. Therefore, (14.33) holds for all σ sufficiently large. ♦

The following theorem, proved by Bertsekas (1999), gives the conditions under which there is a
minimum of the augmented Lagrangian LA(x, λ, σ) that lies close to x, conditions that specify some
error bounds on both xk and λkþ1, solutions of the subproblem at iteration k.

Theorem 14.6 (Bertsekas) Suppose that the assumptions of Theorem 14.5 are satisfied at x and λ.
Let σ be chosen as in the theorem. Then, there exist positive scalars δ, ε, and M, such that the
following claims hold:

(a) For all λk and σ k satisfying

 k 
λ  λ   σ k δ, σk  σ ð14:36Þ
 
the problem min x2ℝn LA x, λk , σ k subject to kx  xk  ε has a unique solution xk. Moreover,
 
kxk  x k  Mλk  λ =σ k : ð14:37Þ
14.3 The Augmented Lagrangian Method 485

(b) For all λk and σ k satisfying (14.36), we have

 kþ1   
λ  λ   Mλk  λ =σ k , ð14:38Þ

where λkþ1 is given by (14.31).

(c) For all λk and σ k satisfying (14.36), the matrix ∇2LA(xk, λk, σ k) is positive definite, and the
constraint gradients ∇hi(xk), i 2 E, are linearly independent. ♦

The theorem of Bertsekas (1999) is important. It illustrates some salient properties of the
augmented Lagrangian approach. The bound (14.37) shows that xk will be close to x if λk is accurate
or if the penalty parameter σ k is large. Therefore, in this approach of the augmented Lagrangian
method, there are two ways of improving the accuracy of xk, whereas in the quadratic penalty
approach, there is only one: increase σ k. The bound (14.37) shows that the accuracy of the multipliers
can be increased by choosing a sufficiently large value of σ k. Finally, the last observation of the
theorem shows that under the given conditions, the second-order sufficient optimality conditions for
the unconstrained minimization are satisfied for the k-th subproblem. Therefore, very good numerical
performances can be expected by applying any standard unconstrained minimization technique.

Example 14.1 Let us solve the following optimization problem by using the augmented Lagrangian
method

min f ðxÞ  x21 þ 8x22

subject to :
h1 ðxÞ  x1 þ x2  4 ¼ 0:

Observe that the augmented Lagrange function is

LA ðx, λ, σ Þ ¼ x21 þ 8x22  λðx1 þ x2  4Þ þ σ ðx1 þ x2  4Þ2 :

Consider the initial point x0 ¼ [0 4]T, the initial values for the penalty parameter σ 0 ¼ 1, and the
Lagrange multiplier λ0 ¼ 0. By using the Newton method for minimizing the augmented Lagrange
function in step 2 of Algorithm 14.3, we get the results from Table 14.1.
As a comparison, in Table 14.2, we present the optimization process for solving the problem by
using the quadratic penalty method.
This example shows that the inclusion of the Lagrange multiplier term in the augmented Lagrang-
ian LA(x, λ, σ) represents a significant improvement of the quadratic penalty method.

Table 14.1 Augmented Lagrangian method

k σ λ x1 x2 f ( xk ) h1 ( xk )
0 1 0 0 4 128 0
1 10 3.767400 1.882353 0.235294 3.986159 −1.88235
2 100 6.837930 3.418960 0.427370 13.15051 −0.153661
3 1000 7.108704 3.554350 0.444290 14.21260 −0.00135
4 10,000 7.1111 3.555555 0.444444 14.22222 −0.120e-5
486 14 Penalty and Augmented Lagrangian Methods

Table 14.2 Quadratic penalty method

k σ x1 x2 f ( xk ) h1 ( xk )
0 1 0 4 128 0
1 10 1.882353 0.235294 3.986151 −1.882350
2 100 3.265306 0.408163 11.99500 −0.326530
3 1000 3.524229 0.440528 13.97271 −0.035242
4 10,000 3.552390 0.444049 14.19697 −0.003552
5 100,000 3.555240 0.444404 14.21969 −0.000355
6 1,000,000 3.555524 0.444440 14.22197 −0.355e−4
7 10,000,000 3.555555 0.444444 14.22222 −0.355e−5

14.4 Criticism of the Penalty and Augmented Lagrangian Methods

Let us see the fundamentals of the penalty and of the augmented Lagrangian methods. For this, we
consider the nonlinear optimization problem with equality constraints:
min ff ðxÞ : hi ðxÞ ¼ 0, i 2 Eg, ð14:39Þ

where f : ℝn ! ℝ and hi : ℝn ! ℝ, i 2 E ≜ {1, . . ., p}, are continuously differentiable on ℝn. Define

h(x) ¼ [h1(x), . . ., hp(x)]. A local solution (x, λ) satisfies the first-order KKT necessary optimality
conditions

∇hðxÞT λ ¼ ∇f ðxÞ, ð14:40aÞ

hðxÞ ¼ 0, ð14:40bÞ

where λ 2 ℝp is the vector of the Lagrange multipliers.

It is common knowledge that in everyday life we always try to find a balance (an equilibrium)
between what is desirable (to minimize the objective function) and what is legally achievable
(to satisfy the constraints). This multiple objective point of view represents the essence of the penalty
methods. For example, the quadratic penalty method combines these two requirements and solves a
sequence of unconstrained optimization subproblems
1
min x2ℝn Qðx, σ Þ ¼ f ðxÞ þ σ khðxÞk2 , ð14:41Þ
2
parameterized by the penalty parameter σ > 0. Thus, a trajectory of the points x(σ) is obtained as
solutions of (14.41), as well as an increasing sequence of penalty parameters. We must let σ become
large in order to nearly achieve feasibility, but at least the penalty function should be smooth.
Therefore, we may apply the Newton method or the quasi-Newton methods for solving (14.41).
Let us introduce the Lagrange function

Lðx, λÞ ¼ f ðxÞ  λT hðxÞ: ð14:42Þ

The derivatives of the quadratic penalty function are

∇Qðx, σ Þ ¼ ∇f ðxÞ þ ∇hðxÞT λ, ð14:43Þ

14.4 Criticism of the Penalty and Augmented Lagrangian Methods 487

X
∇2 Qðx, σ Þ ¼ ∇2 f ðxÞ þ λi ∇2 hi ðxÞ þ σ∇hðxÞT ∇hðxÞ, ð14:44Þ
i2E

where λi ¼ σhi(x), i 2 E, and λ ¼ [λ1, . . ., λp].

Observe that for x(σ), ∇Q(x(σ), σ) ¼ 0. Defining λ(σ) ¼ σh(x(σ)), then (x(σ), λ(σ)) is the
exact solution of a perturbed form of the problem (14.39)
min ff ðxÞ : hðxÞ ¼ hðx ðσ ÞÞg: ð14:45Þ

If the Jacobian matrix ∇hðxÞ has full row rank and x is a unique local minimizer for (14.39), we
can show that the full Hessian ∇2Q(x, σ) is positive definite at (x(σ), λ(σ)) for sufficiently large σ.
Thus, the penalty function is convex for large σ and the minimum x(σ) exists. Therefore, the problem
(14.41) can be very well solved by the Newton method.
We note that the Newton method for minimizing (14.41) would generate a search direction d by
solving the linear system ∇2Q(x, σ)d ¼  ∇ Q(x, σ), i.e.,



∇2 Lðx, λÞ þ σ∇hðxÞT ∇hðxÞ d ¼  ∇f ðxÞ þ σ∇hðxÞT hðxÞ , ð14:46Þ

where ∇2L(x, λ) is defined with λ ¼ σh(x). The difficulty with this approach is that the system (14.46)
is ill-conditioned for large σ (assuming p < n). This is one reason why the quadratic penalty approach
(convenient enough as smoothness) proved unsuccessful for solving nonlinear optimization
problems. A solution for eliminating this deficiency was given by Gould (1986), who introduced
the vector q ¼ σ(h(x) þ ∇ h(x)d) at the current point x. With this, the Newton system (14.46) is
equivalent to
2 2 3
∇ Lðx, λÞ ∇hðxÞT  d 
∇f ðxÞ

4 5 ¼  , ð14:47Þ
1 hð x Þ
∇hðxÞ  I q
σ
which contains no large numbers and may be preferable for computing the search direction d. If (x, λ)
is close to a local optimum (x, λ) and σ is large, any ill-conditioning in (14.47) reflects the
sensitivity of (x, λ) to perturbations in the data of the problem.
Unfortunately, although d can be reliably computed from (14.47) when σ is large, this does not
save the quadratic penalty method. When h(x) is not very small, then d leads away from the
linearization of the constraints h(x) ¼ 0 at the current point x, and therefore, the Newton method is
likely to be too slow. It is obvious that the algebraic trick used does not eliminate the deficiency of the
quadratic Newton approach.
Let us now continue to study the problem (14.39) with equality constraints, bearing in mind the
difficulties encountered with the quadratic penalty method when σ becomes very large. Let (x, λ) be
a local minimizer, and assume that the Jacobian matrix ∇h(x) has full row rank at x, i.e.,
rank(∇h(x)) ¼ p. In this minimum point, the gradient of the Lagrange function must be zero.
Therefore, the required solution (x, λ) is a stationary point of the Lagrangian. In general, we cannot
find x by minimizing L(x, λ) as a function of x, even if we set λ ¼ λ. The second-order optimality
condition for (x, λ) to be an isolated local minimizer is that the Hessian of the Lagrange function
∇2L(x, λ) should be positive definite within the null space of ∇h(x). That is, the Hessian should
satisfy the condition zT∇2L(x, λ)z > 0 for all nonzero vectors z satisfying ∇h(x)Tz ¼ 0. The
following result, proved by Debreu (1952) on quadratic forms, is relevant.
488 14 Penalty and Augmented Lagrangian Methods

Theorem 14.7 (Debreu). Let H be an n n symmetric matrix and J an m n matrix with m  n. If

zTHz > 0 for every nonzero z satisfying Jz ¼ 0, then for all σ sufficiently large H þ σJTJ is positive
definite. ♦

This result suggests that we should add to the Lagrangian function a term whose Hessian is
σ ∇ h(x)T ∇ h(x). It is exactly what we already did when we introduced the augmented Lagrangian
1
LA ðx, λ, σ Þ ¼ f ðxÞ  λT hðxÞ þ σ khðxÞk2 : ð14:48Þ
2
This may be looked upon as a modification of the Lagrangian or as a shifted quadratic penalty
function. For a given λ and σ, the derivatives of LA(.) are

∇LA ðx, λ, σ Þ ¼ ∇f ðxÞ  ∇hðxÞTb λ, ð14:49aÞ

X
∇2 LA ðx, λ, σ Þ ¼ ∇2 f ðxÞ  b
λi ∇2 hi ðxÞ þ σ∇hðxÞT ∇hðxÞ, ð14:49bÞ
i2E

where b λ ¼ λ þ σhðxÞ: Observe that (14.49b) illustrates Debreu’s theorem.

As already seen, the augmented Lagrangian method for solving (14.39) proceeds by choosing λ
and σ as judiciously as possible and then by minimizing LA(x, λ, σ) with respect to x. The resulting x,
solution of this minimizing problem is used to choose a new λ and σ and the process repeats itself. The
auxiliary vector b λ simplifies the above process and proves to be useful in its own right.
Observe that if σ is reasonably large, minimizing LA will tend to make kh(x)k small for any values of
λ. Also, the Hessian ∇2LA(x, λ, σ) will tend to have positive curvature in the null space of ∇h(x).
Therefore, the problem minLA(x, λ, σ) has a solution. On the other hand, since the minimization of LA
makes k∇LAk small if λ is close to λ, it follows that (x, λ)  (x, λ) almost satisfies (14.40a). Moreover,
if kh(x)k also happens to be small (because σ is large enough), then (x, λ) will almost satisfy (14.40b) as
well. Therefore, the strategy is to check that kh(x)k is suitably small after each approximate minimization
of LA. If so, λ is updated as λ ¼ λ  σh(x). If not, σ is judiciously increased and λ remains the same. Under
favorable conditions, we hope that (x, λ) ! (x, λ) before σ becomes too large (Saunders, 2015a).
This weakness of the augmented Lagrangian method (judicious selection of σ, favorable conditions,
etc.) determined some modifications of it. Two algorithms based on the modifications of the augmented
Lagrangian are presented in this chapter. One is SPENBAR (Breitfeld, & Shanno, 1994a, b, c; Andrei,
1996a, b, c, 1998a, 2017c). In it, the inequality constraints are embedded into a modified logarithmic
penalty function. The other one is MINOS (Murtagh, & Saunders, 1978, 1980, 1982, 1987, 1995). It uses
a combination of successive linear programming with a modification of the augmented Lagrangian.

14.5 A Penalty-Barrier Algorithm (SPENBAR)

Let us consider the following general nonlinear optimization problem:

min f ðxÞ
subject to: ð14:50Þ
ci ðxÞ  0, i 2 I c ,
ci ðxÞ ¼ 0, i 2 E,
xj  l j , j 2 I l ,
x j  uj , j 2 I u ,
14.5 A Penalty-Barrier Algorithm (SPENBAR) 489

where x 2 ℝn, E ≜ {1, . . ., me} and Ic ≜ {1, . . ., m}. The functions ci : ℝn ! ℝ, i 2 Ic [ E, are
assumed to be twice continuously differentiable on ℝn. Il, Iu ⊆ {1, . . ., n}. To simplify the
presentation of the algorithm, the simple bounds on the variables are also denoted ci(x). Define Isb
as the set of indices, such that for all j 2 Il [ Iu there is an i 2 Isb with the property

ci ðxÞ ¼ xj  lj  0, for j 2 I l , or ð14:51aÞ

ci ðxÞ ¼ uj  xj  0, for j 2 I u : ð14:51bÞ

The set of all the inequality constraints including the simple bounds is denoted by I ¼ Ic [ Isb.
Let |I| ¼ p, i.e., I has p elements. Suppose that

(i) The domain XB ¼ {x 2 ℝn : lj  xj, j 2 Il, xj  uj, j 2 Iu}, which is a compact set, has a nonempty
interior.
(ii) The functions f(x), ci(x), i 2 E [ Ic are twice continuously differentiable on XB.

The main idea of the penalty-barrier method to find a solution for the problem (14.50) is to solve a
sequence of unconstrained minimization subproblems where the solution xk of the subproblem at
iteration k is considered as an initial point for the subproblem at the next iteration. The suggested
algorithm combines the augmented Lagrangian with a penalty function (a composite function which
includes the augmented Lagrangian and the log-barrier function) in which the parameters are updated
in such a way as to obtain a KKT point for problem (14.50). The constraints are treated separately as
follows.
The equality constraints ci(x) ¼ 0, i 2 E, are embedded into the augmented Lagrangian. Therefore,
to minimize f(x) subject to ci(x) ¼ 0, i 2 E, the augmented Lagrangian is defined as
X
me
1 X
me
Lðx, σ, λÞ≜f ðxÞ þ λi ci ðxÞ þ cðxÞ2 , ð14:52Þ
i¼1
2σ i¼1

where λi, i ¼ 1, . . ., me, are the estimates of the Lagrange multipliers and σ > 0 is the penalty
parameter. The reason for introducing the augmented Lagrangian function is to make sure that the
penalty parameter σ does not need to be enlarged too much as to avoid the ill-conditioning of the
augmented Lagrangian.
The inequality constraints ci(x)  0, i 2 Ic, are embedded into a logarithmic penalty function.
Therefore, to minimize f(x) subject to ci(x)  0, i 2 Ic, we consider the log-barrier function
X
m
Bðx, σ Þ≜f ðxÞ  σ log ðci ðxÞÞ, ð14:53Þ
i¼1

where σ > 0 is the penalty parameter. For solving the problem min{f(x) : ci(x)  0, i ¼ 1, . . ., m}, the
following algorithm can be presented.

Algorithm 14.4 General barrier

1. Select an initial feasible point x0 and a value for the penalty parameter σ 0 > 0. Set k ¼ 0
2. Solve the subproblem min x2ℝn Bðx, σ k Þ obtaining a local solution xkþ1
3. If σ k is small enough, then stop. Otherwise, select σ kþ1 < σ k, set k ¼ k þ 1, and go to step 2 ♦
490 14 Penalty and Augmented Lagrangian Methods

Observe that when σ k ! 0, the Hessian of B(xk, σ k) is ill-conditioned. On the other hand, it is
difficult to select a good value for σ kþ1. The first-order minimizing condition for B(x, σ k) is
Xm
σk
∇f ðxÞ  ∇ci ðxÞ ¼ 0: ð14:54Þ
c
i¼1 i
ðxÞ

If σ k ! 0, then σ k/ci(x) ! λi, which is the Lagrange multiplier associated to the i-th inequality
constraint. Therefore, the estimates of the Lagrange multipliers associated to the active constraints are
computed as the ratio of two quantities, both of them tending to zero. More exactly, observe that when
1 > ci(x) > 0, i 2 Ic, then log(ci(x)) < 0. Hence, the second term on the right-hand side of (14.53)
implies B(x, σ)
f(x) when any of the constraint functions is small and positive. Suppose that the
inequality constraint problem min{f(x) : ci(x)  0, i 2 Ic} has a solution (x, λ). Also suppose that σ is
a positive constant such that for all σ k < σ, the Hessian matrix ∇2B(x, σ k) of the barrier function
(14.53) is positive definite for all feasible x. If xk denotes the solution of the unconstrained problem
min{B(x, σ k)}, then xk ! x as σ k ! 0. Moreover, from (14.54), σ k =ci ðxk Þ ! λi , i 2 Ic, as σ k ! 0.
Jittorntrum and Osborne (1980) modified the log-barrier function (14.53) by introducing the
estimates of the Lagrange multipliers as
X
m
J ðx, σ, λÞ≜f ðxÞ  σ λi log ðci ðxÞÞ: ð14:55Þ
i¼1

However, this modified log-barrier function presents the same numerical difficulties as those of
function B(x, σ).
Polyak (1992) suggested the following modified log-barrier function:
X  
c ðxÞ
Mðx, σ, λÞ≜f ðxÞ  σ λi log 1 þ i , ð14:56Þ
i2I
σ

where λi, i ¼ 1, . . ., p, are nonnegative estimates of the Lagrange multipliers associated to the
inequality constraints and σ > 0 is the barrier parameter. For this modified log-barrier function, Polyak
(1992) established the convergence properties similar to those given by Bertsekas (1982b) for the
multiplier method. Under reasonably mild conditions, Polyak showed that there exists a threshold
value σ > 0 of the barrier parameter, such that for any fixed σ < σ, i.e., by only updating the
Lagrange multipliers λi, i ¼ 1, . . ., p, the sequence (xk, λk) converges, at least at a linear rate, to a local
solution (x, λ) of the problem (14.50). Very encouraging computational results based on the
log-barrier function (14.53) were reported in Breitfeld and Shanno (1994a); Ben-Tal, Yuzefovich,
and Zibulevsky (1992); and Nash, Polyak, and Sofer (1994).
The simple bounds xj  lj, j 2 Il, xj  uj, j 2 Iu, are handled by means of the classical logarithmic
barrier terms.
Therefore, the classical log-barrier function for inequality constraints is defined by
X
Cðx, σ Þ≜f ðxÞ  σ log ðci ðxÞÞ, ð14:57Þ
i2I

where σ > 0 is the barrier parameter. The log-barrier function (14.57) was introduced by Frisch (1955)
and developed by Fiacco and McCormick (1968).
Our approach considers a combination of all the above functions which define a general penalty-
barrier function. This is embedded into the general scheme of the penalty method for which the
14.5 A Penalty-Barrier Algorithm (SPENBAR) 491

global convergence was proved. Mainly, the algorithm considers a sequence of simple bounded
minimization subproblems which are solved by means of the simple bound minimization techniques
(see Chap. 12).

The Penalty-Barrier Method

Having in view the above developments, for the general nonlinear optimization problem (14.50), the
following penalty-barrier function is defined:
  P
F x, σ k , λk , sk , βk ¼ f ðxÞ  σ k λki Ψk ðci ðxÞÞ
i2I
P
me
1 X
me ð14:58Þ
þ λki ci ðxÞ þ c ðxÞ2 ,
i¼1 2σ k i¼1 i

where k  0 and
 
c ðxÞ
Ψk ðci ðxÞÞ ¼ log ski þ i , for ci ðxÞ  βk σ k ski and i 2 I c , ð14:59aÞ
σk

Ψk ðci ðxÞÞ ¼ Qk ðci ðxÞÞ,for ci ðxÞ < βk σ k ski and i 2 I c , ð14:59bÞ
 
ci ðxÞ
Ψ ðci ðxÞÞ ¼ log
k
, for i 2 I sb , ð14:59cÞ
σk

where Qk(ci(x)) is a quadratic extrapolation function defined as

1
Qðci ðxÞÞ ¼ qai ðci ðxÞÞ2 þ qbi ci ðxÞ þ qci : ð14:60Þ
2
The coefficients qai , qbi , and qci are determined in such a way so that the function value as well as
the first and the second derivatives of Q(ci(x)) match the corresponding values of the logarithmic
terms (14.59a) at ci ðxbÞ ¼ βk σ k ski : A simple computation shows that
1
qai ¼   2 , ð14:61Þ
ski σ k 1  βk

1  2βk
qbi ¼  2 , ð14:62Þ
ski σ k 1  βk
 
βk 2  3βk  k 
qci ¼   þ log si 1  β :
k
ð14:63Þ
k 2
2 1β

The values of the penalty parameter σ k are positive scalars which tend to zero as k goes to infinity.
For the shifts ski and the extrapolation parameter βk, we impose

0  βk  βu < 1, ð14:64aÞ

1  ski  su < þ1, i 2 Ic , ð14:64bÞ

where βu  0 and su  1 for all i 2 Ic and k  0.

492 14 Penalty and Augmented Lagrangian Methods

The algorithm finds an approximate local minimum of (14.50) by repeatedly solving the k-th
subproblem
 
min x2XB F x, σ k , λk , sk , βk , ð14:65Þ

such that
  
∇x F xkþ1 , σ k , λk , sk , βk   εk , ð14:66Þ
2

where εk is the convergence tolerance corresponding to the k-th subproblem and εk ! 0 as k ! þ 1 .

Observe that (14.65) is a simple bounded constrained optimization.
Extrapolating the modified log-barrier terms was suggested by Ben-Tal, Yuzefovich, and
Zibulevsky (1992). The quadratic extrapolating terms are defined beyond the singularities of the
logarithmic terms (14.59a). This allows an efficient line-search for the unconstrained minimization
and the reduction of the penalty-barrier parameter σ without restrictions. To ensure that the logarith-
mic terms (14.59a) are well defined in case the quadratic extrapolation was not used, the shifts sk are
selected such that
ci ðxÞ
 sk > 0 ð14:67Þ
σk

for all i 2 I, k  0. x ¼ xk is the starting point for the k-th subproblem.

Updating the multiplier estimates λk and the penalty-barrier parameters σ k is made in such a way so
that the iterates are convergent to a stationary point x of the problem (14.50), i.e., there exists λ such
that (x, λ) satisfies the first-order optimality conditions
X
∇f ðx Þ  λi ∇ci ðx Þ ¼ 0, ð14:68aÞ
i2E[I
X
λi ci ðx Þ ¼ 0, ð14:68bÞ
i2E[I

ci ðx Þ  0 and λi  0, i 2 I, ð14:68cÞ

ci ðx Þ ¼ 0, i 2 E: ð14:68dÞ

The initialization of the estimates of the Lagrange multipliers is given by a vector λ0 computed as
solution of the problem
 
 X 
 
min λI 0 ∇f ðx0 Þ  λi ∇ci ðx0 Þ , ð14:69Þ
 i2E[I

2

where λI ¼ (λi)i2I and x0 is the initial estimation of x. Having in view the combinatorial nature of
solving (14.69), a procedure is to compute a solution bλ of (14.69) by neglecting the restrictions λI  0
and then setting
8 n n o o
>
< min max λl , b λi , λu , if i 2 I or i ¼ 1, . . . , me , b λi  0,
λ0i ¼ n n o o ð14:70Þ
>
: max min λl , b λi , λu , if i ¼ 1, . . . , me , b
λi < 0,

where λl and λu are the lower and upper bounds for the initial estimates of the multipliers, which are
imposed by numerical reasons. Usually, λl ¼ 1 and λu ¼ 100.
14.5 A Penalty-Barrier Algorithm (SPENBAR) 493

The first derivative of F(x, σ k, λk, sk, βk) is

  P
∇x F x, σ k , λk , sk , βk ¼ ∇f ðxÞ  σ k λki Ψ0 ðci Þ∇ci ðxÞ
 i2I ð14:71Þ
Pme ci ðxÞ
 λi 
k
∇ci ðxÞ,
i¼1 σk

where Ψ0(ci) is the first derivative of Ψk with respect to ci, i.e.,

8
>
> 1
, ci ðxÞ  βk σ k ski , i 2 Ic ,
>
>
> σ
< i k sk
þ c i ð x Þ
Ψ0 ðci ðxÞÞ ¼ qai ci ðxÞ þ qbi , ci ðxÞ < βk σ k ski , i 2 Ic , ð14:72Þ
>
>
>
> 1
>
: , i 2 I sb ,
ci ðxÞ

where qai and qbi are given by (14.61) and (14.62), respectively. The first-order optimality conditions
(14.68) and (14.71) suggest the following scheme for updating the Lagrange multiplier estimates:
8
< λk  ci ðxkþ1 Þ , i 2 E,
λkþ1 ¼ i σk ð14:73Þ
i
: k 0
σ k λi Ψ ðci ðxkþ1 ÞÞ, i 2 I,

where xkþ1 is the solution of the k-th subproblem (14.65) satisfying (14.66). Observe that (14.73)
requires that the components λ0i , i 2 I, should be positive; otherwise, if there is a j 2 I with λ0j ¼ 0,
then for all k  1, λkj ¼ 0:
The penalty-barrier parameter σ is initialized by a positive scalar σ 0 and then updated by
σk
σ kþ1 ¼ , ð14:74Þ
γ

for k  0, where γ > 1 ensures that σ k ! 0 as k ! þ 1 . (σ 0 ¼ 101 and γ ¼ 10.)

The parameter β defines how close to singularities the logarithmic terms (14.59a) are extrapolated.
Breitfeld and Shanno (1994c) show that if β is too small (0  βk < 0.5), then the minimum of an
individual penalty term (14.59a) or (14.59b) might be very far from the feasibility domain of (14.50),
and therefore, this term does not impose a penalty for being infeasible. On the other hand, if β is too
close to one (β ¼ 0.99), then the big function values of the logarithmic terms are no longer avoided,
and therefore, the ill-conditioning is introduced into the penalty-barrier function. Hence, β is
initialized by β0 and updated as

βkþ1 ¼ βk θk , ð14:75Þ

for k  0, where β0 and θk are scalars such that 0  βk  βu for all k  0 with βu < 1. (β0 ¼ 0.9 and
θk ¼ 1, for all k  0.)
The shifting parameters s allow the scaling of the constraints in order to avoid the ill-conditioning
of the terms (14.59b). We can show it by introducing the general shifts s into the modified barrier
terms. The scaling of the inequality constraints, such that kc(x)k ¼ 1, can be achieved without
reformulating the problem (14.50). To see this, consider x for which kc(x)k > 0 and let s ¼ kc(x)k > 0.
Then, the scaled constraint is defined as cbðxÞ ¼ cðxÞ=s  0: Clearly, kcbðxÞk ¼ 1: With this, Polyak’s
modified log-barrier term (14.56) for cbðxÞ is
494 14 Penalty and Augmented Lagrangian Methods

     
cbðxÞ cðxÞ cðxÞ
log 1 þ ¼ log 1 þ ¼ log s þ  log ðsÞ:
σ sσ σ

The last term, log(s), is constant with respect to x, so it can be ignored in the minimization of the
penalty-barrier function. Therefore, the scaling of the inequality constraints can be readily achieved
by simply choosing the appropriate shifts s.
On the other hand, in order to avoid the ill-conditioning of the penalty-barrier function introduced
by the quadratic extrapolation, we set

ski ¼ min f max f1, ci ðxk Þg, su g, ð14:76Þ

for all i 2 Ic and k  0, which basically scales the quadratic terms with respect to the constraint values.
Usually, su ¼ 103.
Concerning the stopping criteria, we take the advantage of estimating the Lagrange multipliers
along the iterations for solving (14.65). Therefore, a KKT point can be identified, and we terminate
the penalty-barrier method for a k  0 for which

vk1  τ, ð14:77Þ

or

vk2  τ and vk3  τf , ð14:78Þ

where τ > 0 is the convergence tolerance, τf ¼ 102τ and

vk1 ¼ max f max i¼1,...,me fjci ðxk Þjg,  min i2I fci ðxk Þg,
  9
 P k 
P  k    >
λ i ci ðxk Þ ∇f ðxk Þ  λi ∇ci ðxk Þ >>
i2E[I
 i2E[I  >=
1
, ,
1 þ kxk k2 1 þ kx k k2 >
>
>
>
;

vk2 ¼ max f max i2E jci ðxk Þj,  min i2I ci ðxk Þg,

jf ðxk Þ  f ðxk1 Þj
vk3 ¼ :
1 þ jf ðxk Þj

Observe that if vk1  τ, then the feasibility, the scaled complementarity, and the scaled first-order
optimality conditions are satisfied, i.e., the current point is a KKT point with accuracy τ.
Finally, for having the penalty-barrier algorithm, we must specify the optimization method used
for solving (14.65) for each set of parameters (σ k, λk, sk, βk). Since (14.65) is a simple bounded
constrained optimization problem, we can apply any method for solving this type of problems (the
spectral projected gradient algorithm by Birgin, Martı́nez, & Raydan, 2000, 2001); the limited-
memory BFGS with gradient projection by Byrd, Lu, and Nocedal (1995a); or the truncated Newton
with simple bounds (Nash, 1984a, b, 1985). In our implementation of the algorithm, we have used the
truncated Newton with simple bounds (TNBC) (Nash, 1984a, b, 1985). The approximate solution xk is
used as the starting point for the next subproblem. Now, the following penalty-barrier algorithm with
quadratic extrapolation of the inequality constraints can be presented (Andrei, 1996a, b, c, 1998a).
14.5 A Penalty-Barrier Algorithm (SPENBAR) 495

Algorithm 14.5 Penalty-barrier—SPENBAR—Andrei

1. Choose: x0 2 int (XB), τ > 0 and the sequences {εk} and {θk}. Determine λ0 as in (14.70) and
s0i ¼ min f max f1, ci ðx0 Þg, su g, i 2 I. Set k ¼ 0
2. Test of convergence. If vk1  τ or if vk2  τ and vk3  τf , stop. Otherwise, go to step 3
3. Using xk as the initial point, determine xkþ1 2 int (XB) for which
k∇x Fðx, σk , λk , sk , βk Þk2
1þkxk  εk
2

4. Update the parameters

8
< λk  ci ðxkþ1 Þ , i 2 E,
λkþ1 ¼ i σk
i
:
σ k λki Ψ0 ðci ðxk ÞÞ, i 2 I,
σ kþ1 ¼ σ k/γ,
βk þ 1 ¼ βkθk,
ski ¼ min f max f1, ci ðxkþ1 Þg, su g, i 2 I.
Set k ¼ k þ 1 and go to step 2 ♦

We notice that SPENBAR has two types of iterates. The first one is the so-called major (external)
iteration, in which the test of convergence in step 2 is checked. The second is the minor (internal)
iteration, in which the simple bounded optimization subproblem (14.65) in step 3 is solved.

Global Convergence

Let {xk} be the sequence generated by the penalty-barrier algorithm. The global convergence is
proved under the assumption that the sequence {xk} converges. To prove the global convergence of
{xk}, suppose that the following three additional assumptions hold:

(i) The set XB is bounded.

(ii) The gradients of the active constraints at the limit point of the sequence {xk} generated by the
algorithm are linearly independent.
(iii) The initial estimates of the Lagrange multipliers corresponding to the inequalities and to the
simple bounds are positive.

The following two propositions proved by Breitfeld and Shanno (1994c) refer to the estimates of
the Lagrange multipliers that correspond to the inequality constraints including the simple bounds,
under the updating formula (14.73).

Proposition 14.1 (Positivity of the Lagrange Multipliers). Suppose that (i), (ii), and (v) hold. Under
the updating scheme (14.73), the estimates of the Lagrange multipliers λki are positive for all i 2 I
and k  0.

Proof For all i 2 I, by assumption, λ0i > 0: The proposition is proved by induction, and suppose
that λki > 0 for all i 2 I and some k  0. For the case (14.59a) and for all i 2 Ic and k  0, from (14.73),
we get
496 14 Penalty and Augmented Lagrangian Methods

λki
λkþ1 ¼ > 0:
i
ski þ ci ðxkþ1 Þ=σ k

The positivity follows from ci ðxkþ1 Þ  βk σ k ski and from

ci ðxkþ1 Þ  
ski þ  ski 1  βk  1  βu > 0,
σk

where βu < 1 is the upper bound of β. For the case (14.59b), from (14.61), since ci ðxkþ1 Þ < βk σ k ski, it
follows that
!
c i ð x kþ1 Þ 1  2β k
λkþ1
i ¼ σ k λki   2 þ k  2
ski σ k 1  βk si σ k ! 1  βk
βk 1  2βk λk λk
> λki   þ   ¼ k i k  i > 0
sk 1  β k
i
2
sk 1  β k
i
2
si 1  β su

for all i 2 Ic and k  0, where su is the upper bound of the shifts s. For the simple bounds, since σ k > 0
and xkþ1 is strictly feasible with respect to the simple bounds, it follows that
σk
λkþ1 ¼ λki > 0,
i
ci ðxkþ1 Þ

for all i 2 Isb and k  0. ♦

Proposition 14.2 (Complementarity Slackness). Suppose that (i), (ii), and (v) hold and the sequence
{xk} generated by the algorithm converges to the point x: Let I in ðxÞ ¼ fi : ci ðxÞ > 0, i 2 I g be the
set of indices of the inactive inequality constraints at x: Then,
 k
λi k0 ! λi ¼ 0

for all i 2 I in ðxÞ:

Proof For every i 2 I in ðxÞ, there is a δ > 0 such that ci ðxÞ  δ: Therefore, for all k  k and i 2 I in ðxÞ,
there exists a k  0 such that
δ
ci ðx k Þ  > 0: ð14:79Þ
2
Observe that for the positive general constraints, the logarithmic terms (14.59a) are utilized. For
estimating the multipliers of the general constraints, from (14.73), (14.79), and ski  1, it follows that

λki
λkþ1 ¼
i
ski þ ci ðxkþ1 Þ=σ k ð14:80Þ
σk σk
 λki  λki ,
σ k þ ci ðxkþ1 Þ σ k þ δ=2

for all i 2 I in ðxÞ \ I c and k  k: From (14.80), we see that λkþ1

i < λki for k  k: Therefore,
σk
λkþ1 ¼ λki , ð14:81Þ
i
σ k þ δ=2

for all i 2 I in ðxÞ \ I c and k  k: On the other hand, for the simple bounds,
14.5 A Penalty-Barrier Algorithm (SPENBAR) 497

σk σ
λkþ1 ¼ λki  λki k ð14:82Þ
i
ci ðxkþ1 Þ δ=2

for all i 2 I in ðxÞ \ I sb and k  k: Since σ k ! 0, there is a kb  k such that σ k  δ/2 for all k  k,
b and
therefore, (14.82) implies λkþ1 b Using this together with (14.81), we get
< λki for k  k:
i

bσ
λkþ1  λki k ð14:83Þ
i
δ=2

for all k  kband i 2 I in ðxÞ: Since σ k ! 0 as k ! þ 1 , λki is positive from Proposition 14.1 and since
b  
λki is finite, it follows that λki k0 ! 0 for all i 2 I in ðxÞ, which proves the proposition. ♦

The following result establishes the convergence of the sequence {xk} and of the estimates
sequence of the Lagrange multipliers to a KKT point which satisfies the first-order optimality
conditions (14.68). We follow the proof given by Breitfeld and Shanno (1994c).

Theorem 14.8 (KKT Theorem). Suppose that {xk} converges to a point x and the assumptions (i)–
(v) hold. Then, the sequence of the multiplier estimates {λk} converges to a limit point λ such that
 
x, λ satisfies the first-order optimality conditions (14.68), i.e., it is a KKT point of the problem
(14.50).

Proof The assumptions (i) and (ii) guarantee that F(x, σ k, λk, sk, βk) is well defined for all x 2 int (XB),
k  0, and there is an xb 2 intðXB Þ for which
 
F xb, σ k , λk , sk , βk < 1: ð14:84Þ

Now, let us define

 
F x, σ k , λk , sk , βk ¼ 1 for x 2 XB ∖intðXB Þ: ð14:85Þ

Observe that F(x, σ k, λk, sk, βk) is continuous and XB is a compact set because it is bounded and
closed. Therefore, by Theorem 11.1 (Weierstrass), F(x, σ k, λk, sk, βk) attains its minimum in XB,
which together with (14.84) and (14.85) imply that the approximate minimum xkþ1 2 int (XB), for all
k  0. Therefore x 2 XB :
Now, let us show that the estimates of the Lagrange multipliers corresponding to the active
constraints are convergent. For simplicity, let us denote F(xk, σ k1, λk1, sk1, βk1) by Fk. There-
fore, from (14.71) and (14.73), we have
X X
∇x Fk ¼ ∇f ðxk Þ  λki ∇ci ðxk Þ  λki ∇ci ðxk Þ, ð14:86Þ
i2I i2E

for all k  1. In the following, we consider the multiplier estimates of the inactive constraints at x
separately from the others. The vectors or the matrices corresponding to the constraints that are
inactive at x are denoted by the subscript in. For example, the vector of the multiplier estimates of the
inactive constraints at x is denoted by λin with λin ¼ ðλi Þi2Iin ðxÞ : On the other hand, the vectors and the
matrices corresponding to the remaining constraints are denoted by the subscript ac. These are all
equalities and inequalities that are active at x, i.e., all the indices i 2 E [ I∖I in ðxÞ: With this, (14.86)
can be rewritten as
498 14 Penalty and Augmented Lagrangian Methods

∇x Fk ¼ ∇f ðxk Þ  ∇Cac ðxk ÞT λkac  ∇Cin ðxk ÞT λkin , ð14:87Þ

where ∇Cac(.) is the Jacobian matrix whose rows are the gradients of the constraints ci,
i 2 E [ I∖I in ðxÞ and ∇Cin(.) is the Jacobian matrix corresponding to the inactive constraints at x:
Observe that (iv) implies that ∇Cac ðxÞ has full rank. Therefore, there is a k  0 such that ∇Cac(xk) has
full rank for all k  k: Thus, (14.87) can be rewritten as
h i1 h i
λkac ¼ ∇Cac ðxk Þ∇Cac ðxk ÞT ∇Cac ðxk Þ ∇f ðxk Þ  ∇Cin ðxk ÞT λkin  ∇x Fk

for all k  k: Since x 2 int (XB), it results that ∇Cin(x) is a bounded matrix (by assumption (ii)),
λki ! 0 for i 2 I in ðxÞ (by Proposition 14.2), and k∇xFkk ! 0, so it follows that if k ! þ 1 , then
h i1
λkac ! ∇Cac ðxÞ∇Cac ðxÞT ∇Cac ðxÞ∇f ðxÞ: ð14:88Þ

Now, let us define

h i1
λac ¼ ∇Cac ðxÞ∇Cac ðxÞT ∇Cac ðxÞ∇f ðxÞ: ð14:89Þ

Observe that by (ii), the components of λac are finite. From (14.87) and (14.89), since k∇xFkk ! 0
and λki ¼ 0 for all i 2 I in ðxÞ, we get
X
∇f ðxÞ  λi ∇ci ðxÞ ¼ 0:
i2E[I

From Proposition 14.2 and from (14.88), we can see that there exist δ > 0 and k  0 with
 
λi  δ  λki  λi þ δ for all i 2 E [ I and k  k, which means that the sequence λk kk is bounded.
From (14.73), we get
ci ðxkþ1 Þ
λkþ1 ¼ λki  , ð14:90Þ
i σk

for all i 2 E and k  0. This, together with the boundedness of λk, implies that the sequence
fci ðxkþ1 Þ=σ k gkk is bounded for all i 2 E. Since σ k ! 0, we must have ci ðxÞ ¼ 0 for all i 2 E.
Therefore, the feasibility of x with respect to the equality constraints is established.
Now, let us show that x is feasible with respect to the inequality constraints, i.e., ci ðxÞ  0 for all
i 2 I. Observe that it is necessary to consider the general constraints ci, i 2 Ic, because the simple
bounds are always strictly feasible. Let us assume that there is an index j 2 Ic with cj ðxÞ ¼ δ < 0:
Then, there exists a k  0 such that
δ
cj ðx k Þ   ð14:91Þ
2

for all k  k: Now, from (14.64), it follows that βk σ k skj ! 0 when σ k ! 0. Thus, there is a kb  k
with
14.5 A Penalty-Barrier Algorithm (SPENBAR) 499

δ
cj ðxk Þ   < βk σ k skj ð14:92Þ
2
b and hence, the quadratic extrapolation is used. Consequently, (14.64) and (14.91)
for all k  k,
imply that
0 1
kB cj ðxkþ1 Þ 1  2βk C
λkþ1
j ¼ σ k λ j @
 
 2 þ  2 A
skj σ k 1  βk skj σ k 1  βk
0 1
!
B δ=2 1 C k δ=2 1
 λkj @
  2  k   2 A  λ j σ s2  ð14:93Þ
σ k skj 1  βk sj 1  β k k u ð1  βu Þ2

b where kb is chosen as in (14.92) with the additional requirement that

for all k  k,
δ
σb 
:
k
2s2u 1 þ 1=ð1  βu Þ2

Then, from (14.93), it follows that λkþ1 b and therefore,

 λkj for k  k,
j
!
k δ=2
b 1
λj  λj
kþ1
 ð14:94Þ
σ k s2u ð1  βu Þ2

b As σ k ! 0, (14.94) implies that λk ! þ1, which contradicts the finiteness of the

for k  k: j
multipliers λj , j 2 Ic (see (14.89) and Proposition 14.2). Therefore, x is feasible.
 
Finally, let us argue that the complementary slackness holds at x, λ , i.e.,

λi ci ðxÞ ¼ 0 ð14:95Þ

for all i 2 E [ I. If ci ðxÞ ¼ 0 for some i 2 E [ I, it is clear that (14.95) is satisfied. Otherwise,
ci ðxÞ ¼ δ > 0, from Proposition 14.2, it follows that λi ¼ 0: Thus, (14.95) holds. Furthermore, from
Proposition 14.1, it follows that λi  0 for all i 2 I.
 
In conclusion, we have proved that (xk, λk) converges to a point x, λ satisfying (14.68).
Hence, the limit point is a KKT point of the problem (14.50), which completes the proof of the
theorem. ♦

The convergence of the penalty-barrier algorithm without assuming that the sequence of iterates
{xk} is convergent is proved in Breitfeld and Shanno (1994c). The convergence of the algorithm when
{xk} is not convergent is proved under the assumptions that the gradients of the active constraints at a
limit point of {xk}k2K are linearly independent, where K is a subset of {k : k  0} and the sequence of
the Lagrange multiplier estimates {λk}k0 stays bounded. The proof is quite technical and it is not
presented here.
For the convex programming, the convergence results under mild conditions are given by Ben-Tal
and Zibulevsky (1993). For the modified log-barrier method, under the convexity assumption, the
convergence is given in Jensen and Polyak (1992). Moreover, Polyak (1992) proves the convergence for
the modified log-barrier method applied to non-convex nonlinear programs under nondegeneracy and
second-order sufficiency assumptions. The above theoretical results assume the exactness of the
solutions to the unconstrained subproblems. For an augmented Lagrangian algorithm, Conn, Gould,
500 14 Penalty and Augmented Lagrangian Methods

and Toint (1991a) establish the global convergence, allowing the inexact minimization of the aug-
mented Lagrangian without the assumption that the sequence of the Lagrange multiplier estimates stays
bounded. A similar approach used in the SPENBAR algorithm is considered by Conn, Gould, and Toint
(1992a). The Lagrangian barrier function comprises similar modified barrier terms, as (14.59a). Conn,
Gould, and Toint (1992a) remark that the convergence theory extends to a composite Lagrangian
barrier, i.e., the augmented Lagrangian algorithm. In this algorithm, the update strategy also allows to
show that the penalty parameter is bounded away from zero, and therefore, the potential ill-conditioning
of the augmented Lagrangian function can be avoided. Also, Powell (1969) suggests updating the
parameters in a similar way, depending on the reduction in the infeasibility of the constrained problem.
In the following, let us present some examples illustrating the numerical performances of
SPENBAR, in which the subproblems (14.65) are solved by the truncated Newton with simple
bounds (TNBC).

Example 14.2 (Schittkowski, 1987, p. 186)

5x5
min ½5ðx1 þ x2 Þ  4x3  x1 x3  6x4 
1 þ x5
8x6
  10ð1  2 exp ðx7 Þ þ exp ð2x7 Þ
1 þ x6
subject to
2x4 þ x5 þ 0:8x6 þ x7  5 ¼ 0,
x22 þ x23 þ x25 þ x26  5 ¼ 0,
10  x1  x2  x3  x4  x5  x6  x7  0,
5  x1  x2  x3  x4  0,
5  x1  x3  x5  x26  x27  0,
xi  0, i ¼ 1, . . . , 7:

Consider the initial point x0 ¼ [0.1, 0.1, . . ., 0.1], for which f(x0) ¼  3.282377. Table 14.3
presents the results given by SPENBAR.

Table 14.3 Iterations generated by SPENBAR. s0 ¼ 0.1, b ¼ 0.9, t ¼ 108

k 1 2 3 4 5
σk 0.1 0.01 0.001 0.0001 0.00001
#itin 40 41 39 8 2
#nf 200 162 175 42 9
x1 1.460992 1.46663 1.468803 1.468808 1.468808
x2 1.940633 1.983322 1.983970 1.983971 1.983971
x3 0.464774 0.354626 0.351885 0.351878 0.351878
x4 1.202136 1.195411 1.195342 1.195342 1.195342
x5 0.577817 0.570095 0.569399 0.569396 0.569396
x6 0.774228 0.784720 0.784745 0.784746 0.784746
x7 1.379950 1.411455 1.412121 1.412122 1.412122
f ( x) −37.68229 −37.41242 −37.41296 −37.41296 −37.41296

In this table, #itin and #nf represent the number of iterations and the number of function evaluations, respectively, in the
truncated Newton method for solving subproblem (14.65).
14.5 A Penalty-Barrier Algorithm (SPENBAR) 501

Note that towards the end of the solving process, the number of iterations itin is smaller and
smaller. This is a characteristic of SPENBAR (Andrei, 1996a, b, c). Observe that SPENBAR takes
5 major iterations, at each of them one subproblem (14.65) being solved, and a total of 130 minor
iterations for solving (14.65) by the truncated Newton with simple bounds. The total number of
function evaluations is 588. Observe that along the iterations, the values of the penalty parameter σ k
are quite reasonable. In SPENBAR, the Jacobian matrices ∇c(x) are considered as sparse matrices.
Therefore, it is able to solve large-scale nonlinear optimization problems.

Example 14.3 (Optimization of a heavy body) (Brown & Bartholomew-Biggs, 1987; Andrei, 2003,
Application A10, p. 356).

min ðx1 þ 0:5x3 þ x4 Þ

subject to
x1  x26  0:1 ¼ 0,
x3  x27 ¼ 0,
x4  x28 ¼ 0,
z2 þ y2  4 ¼ 0,
zu þ vy ¼ 0,
uy  vz  0:7 ¼ 0,

where
 
z ¼ 0:1ðx1 þ x3 þ x4 Þ þ 0:01 x21 þ 2x1 x3 þ 2x1 x4 cos x2 þ 0:01x24 cos x5 ,
 
y ¼ 1 þ 0:1ðx1 þ x3 þ x4 Þ þ 0:01 x21 þ 2x1 x3 þ 2x1 x4 sin x2 þ 0:01x24 sin x5 ,
u ¼ 0.1 þ 0.02(x1 cos x2 þ x4 cos x5),
v ¼ 0.1 þ 0.02(x1 sin x2 þ x4 sin x5).

Considering the initial point x0 ¼ [5.67, 5.23, 11.96, 23.88, 2.61, 2.34, 3.45, 4.89], for which
f(x0) ¼ 35.53, Table 14.4 presents the results of the optimization process, as well as the solution.

Table 14.4 Iterations generated by SPENBAR. s0 ¼ 0.1, b ¼ 0.9, t ¼ 108. Optimization of a heavy body

k 1 2 3 4 5 6 7
σk 0.1 0.01 0.001 0.0001 1.e−5 1.e−6 1.e−7
#itin 44 117 190 290 5 4 1
#nf 264 665 1018 1412 20 17 6
x1 0.203143 0.007214 0.305438 0.39424 0.39417 0.39417 0.39417
x2 7.237269 5.48815 7.430341 7.42649 7.4266 7.4266 7.4266
x3 7.695956 2.232681 13.48405 13.70247 13.703 13.703 13.703
x4 0.081001 9.94434 23.58007 24.03176 24.0325 24.0325 24.0325
x5 −0.90282 −1.23978 −1.91034 −1.91831 −1.9183 −1.9183 1.9183
x6 0.305193 −0.18e−8 0.543286 0.619871 0.61981 0.61981 0.61981
x7 −2.75607 1.49421 3.67206 3.701685 3.70175 3.70175 3.70175
x8 −0.32e−9 −3.15316 −4.85593 −4.90222 −4.9023 −4.9023 −4.9023
f ( x) 4.13212 11.06789 30.62753 31.27724 31.2782 31.2782 31.2782
502 14 Penalty and Augmented Lagrangian Methods

Table 14.5 Performances of SPENBAR. s0 ¼ 0.1, b ¼ 0.9, t ¼ 108. Application DES. Thomson problem

n me mc #itM #itm #nf #qe cpu vfo

150 50 0 6 402 4095 0 2.23 1055.182314
225 75 0 6 638 6600 0 7.47 2454.369689
300 100 0 6 838 9663 0 18.78 4448.350634
450 150 0 6 1138 1,4508 0 61.30 10236.26400
525 175 0 7 1356 18,206 0 103.73 14034.95857
600 200 0 7 1680 25,854 0 190.41 18438.975151
900 300 0 7 2679 37,903 0 618.82 42131.673280
1200 400 0 7 3876 52,568 0 1006.89 75583.037847

In this table, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the number of
inequality constraints, #itM ¼ the number of the major iterations; #itm ¼ the number of the minor iterations in the
truncated Newton method for solving (14.65), #nf ¼ the total number of evaluations of functions in the truncated
Newton method for solving (14.65), #qe ¼ the total number of quadratic extrapolations, cpu ¼ the cpu time (seconds) to
get a local solution to the problem, and vfo ¼ the optimal value of the objective function.

Table 14.6 Performances of SPENBAR. s0 ¼ 0.1, b ¼ 0.9, t ¼ 108. Application HANG. Dog curve

n me mc #itM #itm #nf #qe cpu vfo

50 1 0 3 1338 11,341 0 0.50 5.068577
100 1 0 3 3191 29,213 0 2.42 5.068505
200 1 0 3 3539 32,715 0 12.37 5.068486
400 1 0 4 4873 43,951 0 13.06 5.068481
800 1 0 3 15,896 147,351 0 86.43 5.068480
1000 1 0 3 21,465 196,275 0 144.29 5.068480
2000 1 0 5 102,285 780,770 0 1084.85 5.088175

For solving this problem, SPENBAR needs 7 major iterations and a total of 651 minor iterations
used by the truncated Newton method. The total number of function evaluations is 3402. The final
value of the objective at the local optimum is f(x) ¼ 31.27823435. ♦

Application DES (Distribution of Electrons on a Sphere). (Dolan, Moré, & Munson, 2004; Andrei,
2015a). This is application L10 form the LACOP collection, known as the Thomson problem, as
described in Appendix C. In Table 14.5, we present the performances of SPENBAR for solving this
application for different numbers of electrons on the sphere.

Application HANG (Hanging Chain). This is application L11 from the LACOP collection
presented in Appendix C. Table 14.6 contains the performances of SPENBAR for solving this
application for different numbers of discretization points, with a ¼ 1, b ¼ 3, and L ¼ 4.

Numerical Study—SPENBAR: Solving Applications from the LACOP Collection

In Appendix C, we have presented the LACOP collection, which contains a number of 18 real
nonlinear optimization applications. In Table 14.7 of this numerical study, we can see the numerical
performances of SPENBAR for solving nine applications.
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 503

Table 14.7 Performances of SPENBAR for solving nine applications from the LACOP collection. s0 ¼ 0.1,
b ¼ 0.9, t ¼ 108

n me mc #itM #itm #nf #qe cpu vfo

ELCH 10 3 0 6 215 1109 0 0.04 −47.761090
ALKI 10 3 8 8 419 2064 1290 0.06 −1768.8070
PREC 8 0 6 5 87 403 39 0.01 3.9511635
PPSE 9 6 0 9 137 643 0 0.03 5055.0118
MSP3 13 0 15 9 1434 9687 8128 0.33 97.587531
MSP5 16 0 21 9 1256 9189 8413 0.5 174.787136
POOL 34 20 0 8 1282 11,620 0 1.0 2785.8000
TRAFO 6 0 2 6 171 744 173 0.02 135.075955
LATHE 10 1 14 10 781 5079 2296 0.19 −4430.5793

14.6 The Linearly Constrained Augmented Lagrangian (MINOS)

This section presents one of the most respectable algorithms and software for solving general
nonlinear optimization problems, elaborated by Murtagh and Saunders (1978, 1980, 1982, 1995).
The main idea behind this method is to generate a step by minimizing the Lagrangian or the
augmented Lagrangian, subject to the linearizations of the constraints.
The origins of the linearly constrained augmented Lagrangian method can be found in the papers
of Robinson (1972) and Rosen and Kreuser (1972). Let us consider the general nonlinear optimization
problem with the equality constraints
min ff ðxÞ : hðxÞ ¼ 0, l  x  ug ð14:96Þ

For a given point (xk, λk), Robinson’s method defines the following:

• The linear approximation to h(x): hk ðxÞ ¼ hðxk Þ þ ∇hðxk Þðx  xk Þ,

• The departure from linearity: dk ðxÞ ¼ hðxÞ  hk ðxÞ,
• The modified Lagrangian: Mk ðxÞ ¼ f ðxÞ  λTk dk ðxÞ:

Thus, the next iteration of Robinson’s method is obtained as solution of the following subproblem:
 
min x2ℝn Mk ðxÞ : hk ðxÞ ¼ 0, l  x  u : ð14:97Þ

Under suitable conditions, Robinson (1972) proves that the sequence of the subproblem solutions
{(xk, λk)} converges quadratically to a solution of (14.96).
On the other hand, in MINOS, as described in Murtagh and Saunders (1982), the penalty term of
the augmented Lagrangian is included in the subproblem objective in an attempt to improve the
convergence from arbitrary starting points. Therefore, a modified augmented Lagrangian subproblem
is obtained:
n o
1 
min x2ℝn f ðxÞ  λTk d k ðxÞ þ σ k dk ðxÞ2 
2 ð14:98Þ
subject to
hk ðxÞ ¼ 0, l  x  u:
504 14 Penalty and Augmented Lagrangian Methods

Observe that this is equivalent to the normal augmented Lagrangian because dk(x) and h(x) are the
 
same when the linearized constraints are satisfied hk ðxÞ ¼ 0 : An important benefit is the following:
if h(x) involves only some of the variables nonlinearly, i.e., it is not hard nonlinear, then Mk(x) has the
same property, whereas kh(x)k2 appears to be more nonlinear.
MINOS uses the simplex or the reduced-gradient iterations to satisfy the linearized constraints for
each subproblem. It limits the number of “minor iterations” performed on (14.98) in a heuristic way in
order to avoid excessive optimization within a wrong subspace. MINOS also monitors kxkþ1  xkk
and kλkþ1  λkk, and if they seem to be large, then the step towards (xkþ1, λkþ1) is heuristically
shortened. Only if kh(x)k has increased substantially should σ k be increased.

MINOS for Linear Constraints

As described in Murtagh and Saunders (1982), MINOS is a particular implementation of Wolfe’s

reduced gradient algorithm (1967) (see Chap. 16). It is designed to solve large-scale problems with
nonlinear objective functions, expressed in the following standard format:

min f ðxN Þ þ cT x, ð14:99aÞ

subject to

Ax ¼ b, ð14:99bÞ

l  x  u, ð14:99cÞ

where A is an m n matrix with m  n. The vector x is partitioned into the linear portion xL and the
nonlinear portion xN:

xN
x¼ :
xL

The components of xN are called the nonlinear variables. On the other hand, the components of xL
are the linear variables. Observe that in the objective function, the nonlinear variables are
incorporated in a nonlinear part of it. Possibly, the part of cTx involving xN may be included into
f(xN). It is quite clear that if f(xN) ¼ 0, then (14.99) is a linear programming problem. MINOS is an
extension of the revised simplex method (Dantzig, 1963; Vanderbei, 2001) and is able to deal with
nonlinear terms by using quasi-Newton procedures.
As known, in the simplex method, a basic solution is characterized by having at most m basic
variables lying between their bounds, while the rest of n  m nonbasic variables are equal to one
bound or to another. An associated nonsingular square basis matrix B is extracted from the columns of
the constraint matrix A, and along the iterations of the simplex algorithm, the columns of B are
replaced one at a time.
In the case of nonlinear optimization problems, we cannot expect an optimal point to be a basic
solution. However, if the number of nonlinear variables is small, then it seems reasonable to suppose
that an optimal solution will be nearly basic. Thus, as described in Murtagh and Saunders (1978), as a
generalization of the simplex algorithm, the concept of superbasic variables is introduced. Therefore,
partitioning the vector x as x ¼ [xB xS xN]T, the linear constraints (14.99b) may be written as
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 505

2 3
xB
6 7
Ax ¼ ½B S N 4 xS 5 ¼ b: ð14:100Þ
N
x

The matrix B is square and nonsingular, as in the simplex algorithm. Therefore, it is invertible. The
matrix S is m s with 0  s  n  m and N is given by the remaining columns of A. Clearly, the
associated variables xB, xS, and xN are called the basics, superbasics, and nonbasics, respectively. As
in the simplex algorithm, both the basics and superbasics are free to vary between their bounds. The
superbasic variables may be moved in any direction in order to improve the objective value. On the
other hand, the basic variables are obliged to change in a definite way to maintain the feasibility of the
constraints Ax ¼ b.
The following theorem shows that the optimal solutions of a nonlinear optimization problem are
often near basic (Murtagh & Saunders, 1978).

Theorem 14.9 Let us suppose that a nonlinear optimization problem has t variables occurring
nonlinearly in either the objective function or in the constraints. Then, there is an optimal solution in
which the number of superbasic variables s satisfies s  t.

Proof Let us assume that the nonlinear variables are fixed at their optimal values. The remaining
problem is a linear programming one for which a basic solution exists (s ¼ 0). The conclusion of the
theorem follows immediately if the nonlinear variables are now regarded as superbasics in the
original problem. Observe that at the very beginning s ¼ t, but if any of the nonlinear variables are
actually on a bound, then we can label them as nonbasic, i.e., s < t. ♦

Let us now detail the method of MINOS for linear constraints. Suppose that f is twice continuously
differentiable. Therefore, it can be expanded in Taylor’s series with remainder of second-order
1
f ðx þ ΔxÞ ¼ f ðxÞ þ gðxÞT Δx þ ΔxT H ðx þ γΔxÞΔx, ð14:101Þ
2
where 0  γ  1, g(x) ¼ ∇ f(x), and H(x þ γΔx) is the Hessian of f evaluated at some point
between x and x þ Δx. Observe that if f(x) is a quadratic function, then H is a constant matrix. Now,
let us partition Δx and g(x) corresponding to the partition of A: Δx ¼ [ΔxB ΔxS ΔxN]T and
g ¼ [gB gS gN]T.
If f(x) is quadratic, then a constrained stationary point at x þ Δx is obtained by requiring that the
following two properties of the step Δx should hold:

Property 1 (Feasibility)
2 3
 ΔxB
B S N 6 S7
4 Δx 5 ¼ 0: ð14:102Þ
0 0 I
ΔxN

This property says that the step remains on the surface given by the intersection of the active
constraints, i.e., the new point x þ Δx is feasible. From (14.102), observe that ΔxN ¼ 0.
506 14 Penalty and Augmented Lagrangian Methods

Property 2 (Optimality)
2 3 23 2 T 3
gB ΔxB B 0 
6 7 6 S7 6 T 7 μ
4 S5
g þ H ¼
4 Δx 5 4 S 05 , ð14:103Þ
λ
gN ΔxN NT I

i.e., the gradient at x þ Δx, given by the left-hand side of (14.103), is orthogonal to the surface of the
active constraints and thus is expressed as a linear combination of the active constraint normals.
Moreover, for the optimality of x þ Δx, the negative gradient must be orthogonal outside of the
feasibility domain. For (14.99), we impose that λj  0 if xNj ¼ uj , or λj  0 if xNj ¼ lj , j ¼ 1, . . .,
n  m  s. μ and λ are the Lagrange multipliers.
For a general function f(x), the step Δx may not lead directly to a stationary point, but the
Properties 1 and 2 may be used to get a feasible descent direction. Now, from (14.102), we get

BΔxB þ SΔxS ¼ 0,
ð14:104Þ
ΔxN ¼ 0:

Therefore,

ΔxB ¼ WΔxS , ð14:105Þ

where

W ¼ B1 S: ð14:106Þ

Thus,
2 3
W
6 7
Δx ¼ 4 I 5ΔxS :
0

So, we can work only with ΔxS. The matrix W is not explicitly computed because B1 is
represented as the product form of the inverse or as the elimination form of the inverse (Andrei,
2011d).
Observe that (14.103) may be simplified when it is multiplied to the left by the matrix
2 3
I 0 0
6 7
4 W T I 0 5: ð14:107Þ
0 0 I

With this, from the first row of (14.103) multiplied by (14.107), we get an expression for the
estimates of the Lagrange multipliers for the general constraints
2 3
W
6 7
gB þ ½ I 0 0 H 4 I 5ΔxS ¼ BT μ: ð14:108Þ
0

Observe that if ΔxS ¼ 0, which means that x is stationary, we have

14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 507

BT μ ¼ gB : ð14:109Þ

Let u be the solution of (14.109), which, as we can see, is analogous to the pricing vector in the
revised simplex algorithm.
Considering now the third row of (14.103) multiplied by (14.107), we get
2 3
W
6 7
gN þ ½ 0 0 I H 4 I 5ΔxS ¼ N T μ þ λ: ð14:110Þ
0

Again, if ΔxS ¼ 0, then this equation reduces to

λ ¼ gN  N T μ, ð14:111Þ

which is analogous to the vector of the reduced costs in linear programming.

Finally, considering the second row of (14.103) multiplied by (14.107), we obtain
2 3
W
 6 7
W T I 0 H 4 I 5ΔxS ¼ h, ð14:112Þ
0

where

h ¼ W T I 0 g ¼ gS  ST u: ð14:113Þ

The form of the equation (14.112) suggests that

2 3
W
 6 7
W T I 0 H4 I 5 ð14:114Þ
0

can be regarded as the reduced Hessian and h ¼ W T I 0 g as the reduced gradient. Observe
that (14.112) gives a Newton step in the independent variables ΔxS. Therefore, khk ¼ 0 becomes a
necessary condition for a stationary point on the current set of the active constraints, which, if the
reduced Hessian is nonsingular, implies that kΔxSk ¼ 0.
Practically, an RTR factorization of the reduced Hessian (14.114) is computed, where R is an upper
triangular matrix, such that the step ΔxS is obtained from the system
 T  S
R R Δx ¼ h ð14:115Þ

by forward and backward substitution, like in the simplex algorithm with the LU factorization of the
basis.
Gill and Murray (1974b) considered a class of algorithms in which the search direction along the
surface of the active constraints is characterized as being in the range of a matrix Z which is
b ¼ bb is the current set of n  s active
orthogonal to the matrix of the constraint normals. Thus, if Ax
constraints, then Z is an n s matrix, such that
b ¼ 0:
AZ ð14:116Þ
508 14 Penalty and Augmented Lagrangian Methods

Using this idea, the main steps to be executed at each iteration to get a feasible direction p are as
follows:

(i) Compute the reduced gradient g ¼ Z T g:

(ii) Compute an approximation to the reduced Hessian H ¼ ZT HZ:
(iii) Compute an approximate solution to the system Hp ¼ g:
(iv) Compute the search direction p ¼ Zp:
(v) Perform a line-search to find an approximation to the stepsize α

f ðx þ α pÞ ¼ min ff ðx þ αpÞ : x þ αp feasibleg:

α

This algorithm is very general. Two conditions are imposed on the matrix Z: to be of full rank on
columns and to verify (14.116). Therefore, Z may have any form which verifies the above conditions.
Particularly, in MINOS, Z has the following form:
2 3 2 3
W B1 S
6 7 6 7
Z ¼ 4 I 5 ¼ 4 I 5: ð14:117Þ
0 0

This is a very convenient representation which will be used both in MINOS with linear constraints
and in MINOS with nonlinear constraints. Note that this matrix in not computed. The algorithm works
only with S and the triangular (LU) factorization of B.
Now, let us present the MINOS algorithm with linear constraints. For this, suppose that we have
the following elements:

1. A feasible vector x satisfying ½ B S N x ¼ b and l  x  u

2. The function value f(x) and the gradient gðxÞ ¼ ½ gB gS gN T
3. The number of superbasic variables s, where 0  s  n  m
4. An LU factorization of the m m-basis matrix B
5. An RTR factorization of the quasi-Newton approximation to the matrix ZTHZ
6. A vector u, solution of the system BTu ¼ gB
7. The reduced gradient h ¼ gS  STu
8. Small positive convergence tolerances εh and ελ

With these developments, the MINOS algorithm for solving nonlinear optimization problems with
linear constraints is as follows.

Algorithm 14.6 MINOS–linear constraints–Murtagh and Saunders

1. Initialization of the algorithm with the above described elements. Set k ¼ 1, xk ¼ x, f(xk) ¼ fk, and
g(xk) ¼ gk
2. Test for convergence. If khk > εh, then go to step 4; otherwise, continue with step 3
3. Compute an estimation of the Lagrange multipliers. Modify the number of superbasics:
(a) Compute λ ¼ gN  NTu
(b) Since for the verification of the KKT conditions all the variables fixed at the upper (lower) bound
must have the Lagrange multipliers negative (positive), determine:
λ1 ¼ max {λj : λj > 0} for xNj fixed at its upper bound uj,
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 509

λ2 ¼ min {λj : λj < 0} for xNj fixed at its lower bound lj.
If λ1  ελ and λ2  ελ, stop, the current solution is optimal
(c) Otherwise, choose xNi corresponding to |λi| ¼ max {λ1, |λ2|} as a new superbasic variable
(1) Augment the matrix S with the column corresponding to the variable xNi
(2) Augment the reduced gradient h with λi
(3) Add a suitable new column to R
(d) Update s ¼ s þ 1
4. Compute the search direction p:
(a) Solve the system (RTR)pS ¼  h
(b) Solve the system (LU)pB ¼  SpS
(c) Set p ¼ ½ pB pS 0 T
5. Determine the maximum value of the stepsize:
(a) Find αmax  0 as the greatest positive value of α for which one component of the vector
 B T
x þ αpB xS þ αpS has reached one of its bounds
(b) If αmax ¼ 0, then go to step 10; otherwise, continue with step 6
6. Line-search. Find α such that
f(xk þ αp) ¼ min {f(xk þ αp) : 0 < α  αmax}
7. Update the solution as xkþ1 ¼ xk þ αp and set k ¼ k þ 1
8. Compute f(xk) ¼ fk and g(xk) ¼ ∇ f(xk) ¼ gk
9. Compute the reduced gradient:
(a) Solve the system (UTLT)u ¼ gB
(b) Compute the new reduced gradient h ¼ gS  ST u
(c) Using α, p and the change in the reduced gradient h  h, modify R corresponding to a quasi-Newton
update of RTR
(d) Set h ¼ h
(e) If α ¼ αmax, that is one component of xB or xS has reached one of their bounds, then continue with
step 10; otherwise (α < αmax) go to step 2
10. Here αmax ¼ 0. At this step of the algorithm, a basic variable or a superbasic variable, let us say with
index i, has reached one of its bounds
(a) If the variable i is basic, then:
(1) Interchange the i-th basic variable xBi (i.e. the column corresponding to xBi from B) with the j-th
superbasic variable xSj (i.e. the column corresponding to xSj from S). The index j is chosen to
keep B nonsingular
(2) Update the L and U factors, the matrix R and the vector u
(3) Compute the new reduced gradient h ¼ gS  STu and go to step 10.c)
(b) If the i variable is superbasic, then define j ¼ i  m
(c) Make the j-th variable in S nonbasic at the appropriate bound:
(1) Delete column j from S and place it in N
(2) Delete column j from R
(3) Delete the j-th component from the vectors xS and h
(4) Restore R to the triangular form
(d) Set s ¼ s  1 and go to step 2 ♦

An iteration of the MINOS algorithm for linear constraints is roughly equivalent to an iteration of
the revised simplex algorithm on an m n linear programming problem, plus an iteration of a quasi-
Newton algorithm on an unconstrained optimization problem with s variables. MINOS with linear
constraints is an advanced software, written in Fortran, which takes the sparsity of A and uses an
efficient LU factorization of the basis matrix B, based on the “bump and spike” algorithm (Hellerman
& Rarick, 1971, 1972). Murtagh and Saunders (1978) present plenty of details on the following:
update of the sparse matrix factorizations, quasi-Newton updates, basis change, removal of one
superbasic variable, addition of one superbasic variable, convergence tests, use of the first and of the
second derivatives, forward and backward transformations of vectors, quadratic programs, etc.
510 14 Penalty and Augmented Lagrangian Methods

Table 14.8 MINOS for solving some large-scale linear programs

Characteristics Optimization
m n nz #iter z cpu
cq9 9451 13,778 157,598 104,150 0.5055445e6 449.47
cre-b 9649 72,447 328,542 183,473 0.2312964e8 678.23
cre-d 8927 69,980 312,626 240,460 0.2445497e8 829.64
ge 10,339 11,098 53,763 15,524 0.5581281e7 51.39
ken-11 14,694 21,349 70,354 15,307 −0.69723e10 71.21
ken-13 28,633 42,659 139,834 45,461 −0.10257e10 440.56
pds-10 16,559 48,763 140,063 74,814 0.267270e11 404.45
stocfor3 16,675 15,695 74,004 14,082 −0.399767e5 72.55
osa-07 1118 23,949 167,643 1915 0.5357225e6 2.02
osa-14 2338 52,460 367,220 4116 0.1106462e7 7.91
co5 5878 7993 92,788 30,938 0.7144723e6 81.28
cq5 5149 7530 83,564 37,280 0.4001338e6 85.06
p05 5090 9500 68,455 1675 0.5560002e6 3.82
r05 5190 9500 113,455 1712 0.5578318e6 5.37

Table extracted from Andrei (2011d, p. 325))

In this table, we have: m ¼ the number of linear constraints, n ¼ the number of variables, nz ¼ the number of
nonzeros in the matrix of linear constraints, #iter ¼ the number of iterations necessary to get the optimal solution, z ¼
the value of the objective, and cpu ¼ the CPU computing time (seconds) for solving the problem. The numerical
experiments were run on a Workstation Intel Pentium 4 with 1.8 GHz.

Numerical Study: MINOS for Linear Programming

To see the performance of MINOS for solving linear programming problems, in Table 14.8, we
present the results of solving some large-scale linear programming problems (the terms f(xN) in
(14.99a) is zero). These are taken from the Netlib collection (Gay, 1985), or the collection of linear
programs described in Carolan, Hill, Kennington, Niemi, and Wochmann (1990) (see Andrei, 2011d,
p. 325).

MINOS for Nonlinear Constraints

In the following, we shall extend MINOS for solving general nonlinear optimization problems of the
form

min f 0 ðxÞ þ cT x þ d T y ð14:118aÞ

subject to
f ð x Þ þ A1 y ¼ b1 , ð14:118bÞ

A2 x þ A3 y ¼ b 2 , ð14:118cÞ

x
l  u, ð14:118dÞ
y

where f ðxÞ ¼ f 1 ðxÞ ⋯ f m1 ðxÞ , b1 2 ℝm1 , b2 2 ℝm2 , which emphasizes the linear part as well as
the nonlinear part of the problem. Define m ¼ m1 þ m2. The first n1 variables x are called the
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 511

nonlinear variables. They occur nonlinearly in either the objective function or in the first m1
constraints. The variables y represent the linear variables. They include the slack variables. The
problem may have purely linear constraints (14.118c). Suppose that the functions f i(x), i ¼ 0, . . ., m1,
are twice continuously differentiable with the gradients gi(x), i ¼ 0, . . ., m1, and there exists a local
minimum x with the corresponding Lagrange multipliers λ, such that the first- and second-order
KKT conditions hold.
As described by Murtagh and Saunders (1982), the solution process of this problem consists of a
sequence of major iterations, each one involving a linearization of the nonlinear constraints at some
current point xk corresponding to a first-order Taylor’s series approximation


f i ð x Þ ¼ f i ð x k Þ þ gi ð x k Þ T ð x  x k Þ þ O kx  x k k2 :

Define

feðx, xk Þ≜f ðxk Þ þ J ðxk Þðx  xk Þ, ð14:119Þ

or fe ¼ f k þ J k ðx  xk Þ, where J(x) is the m1 n1-Jacobian matrix of function f(x). Observe that

f  fe ¼ ðf  f k Þ  J k ðx  xk Þ ð14:120Þ

contains the high-order nonlinear terms in the Taylor’s expansion of f(x) about the current point xk.
At the k-th major iteration, the following linearly constrained subproblem is formed:
 
min L x, y, xk , λk , ρ ¼
x, y


T

1
f 0 ðxÞ þ cT x þ dT y  λTk f  fe þ ρ f  fe f  fe ð14:121aÞ
2
subject to
fe þ A1 y ¼ b1 , ð14:121bÞ

A2 x þ A3 y ¼ b 2 , ð14:121cÞ

x
l  u: ð14:121dÞ
y

The objective function L(.) given by (14.121a) is a modified augmented Lagrangian in which f  fe
is used instead of the conventional constraint violation f þ A1y  b1. The partial derivatives of L(.) are

∂Lðx, yÞ h
i
¼ g0 ðxÞ þ c  ðJ  J k ÞT λk  ρ f  fe ,
∂x
ð14:122Þ
∂Lðx, yÞ
¼ d:
∂y

Observe that the nonlinearities in L(.) involve x but not y, which means that the subproblem has the
same nonlinear variables as the original one. The modified Lagrangian was used by Robinson (1972)
with ρ ¼ 0. The use of a penalty term ensures that the augmented Lagrangian maintains a positive
definite Hessian in the appropriate subspace. It was suggested by Arrow and Sollow (1958) and later
considered by Hestenes (1969), Powell (1969), and Sargent and Murtagh (1973).
In the following, in order to specify the MINOS algorithm for nonlinear constraints, it is necessary
to define the procedures for the choice of the Lagrange multipliers λk and of the penalty parameter ρ.
512 14 Penalty and Augmented Lagrangian Methods

Choice of λk The best choice is λk ¼ λ, but the optimal value of the multipliers is not known.
Therefore, the simplest choice is λk ¼ b λ, where bλ is the vector of the multipliers corresponding to the
linearized constraints at the solution of the previous subproblem. Let us assume that the problem does
not have linear constraints. Then, b λ is the solution of the system BTb λ ¼ gB from the end of the
b Tb
previous iteration. Moreover, λ also verifies the system S λ ¼ gS : As we know, g is zero for all the
slack variables, and it follows that b λi ¼ 0 if the i-th linearized constraint is inactive. Therefore, the
choice λk ¼ b λ ensures that an inactive nonlinear constraint will be excluded from the Lagrangian term
λT ðf  feÞ in the next subproblem.
k

Choice of ρ As known, x need not be a local minimum of the Lagrangian function. If we assume that
J(x) is of full rank, then λ exists, so that

Lðx, λÞ ¼ f 0 ðxÞ þ cT x þ dT y  λT ðf þ A1 y  b1 Þ,

is stationary at (x, λ), but L(x, λ) may have a negative curvature in x at x. If we consider that the
constraints are satisfied at x as equalities and ignore the inactive constraints, then the necessary
(sufficient) conditions for x to be a local minimum are

∂Lðx , λ Þ
Z ðx ÞT ¼0
∂x
and

∂ Lðx , λ Þ
2
Z ðx  ÞT Z ðx  Þ
∂x2
to be positive semidefinite (positive definite), where Z(x) is defined in (14.117). Therefore, if the
search is restricted to the linearly constrained subspace defined by Z(x), then we do seek a minimum
of the Lagrangian. We may expect that when xk is sufficiently close to x for J(xk) to be close to J(x),
we may minimize (14.121a) with ρ ¼ 0. As it is discussed in Murtagh and Saunders (1982), the
difficulty arises when xk is far away from x, since, in this case, the linearized constraints may define a
subspace where perhaps a saddle point would be closer to x than a minimum would. Successive
minima of (14.121) with ρ ¼ 0 may therefore fail to converge to x. The addition of a penalty term
ρðf  feÞ ðf  feÞ imposes the correct curvature properties on (14.121a) for a sufficiently large ρ > 0.
T

To illustrate the importance of the penalty term in (14.121), let us consider the following problem
with equality constraints:

min f 0 ðxÞ
subject to ð14:123Þ
f ðxÞ ¼ 0,

where the functions are twice continuously differentiable with bounded Hessians. Assume that at
some point x the Jacobian J(x) is of full rank, that there is a λ such that ∂f0/∂x ¼ J(x)Tλ, and that
the reduced Hessian
h i
Zðx ÞT ∂ Lðx , λ Þ=∂x2 Zðx Þ
2

is positive definite, i.e., the sufficient conditions of optimality are satisfied for x to be a local
minimum. As in Murtagh and Saunders (1982), the following theorems can be presented.
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 513


Theorem 14.10 Let (xk, λk) be an approximate solution to (14.123) and let xb, b
λ be a solution to the
linearized subproblem


T

1
min f 0 ðxÞ  λTk f  fe þ ρ f  fe f  fe
2 ð14:124Þ
subject to
e
f ðx, xk Þ ¼ 0,


where feðx, xk Þ is given by (14.119). If b λ  λk ¼ ε1 and f ðxbÞ ¼ ε2 , then xb, b
λ is also a solution to the
perturbed problem


min f 0 ðxÞ þ ðε1 þ ρε2 ÞT f  fe
subject to ð14:125Þ
f ð x Þ ¼ ε2 ,

for ε1 and ε2 sufficiently small.


Proof If xb, b
λ is a solution of (14.124), then we must have fe ¼ 0 and

T
T

g0 ðxbÞ  Jb  J k λk þ ρ Jb  J k f  fe ¼ J Tk b
λ,

T
b f, and feare evaluated at xb: Now, adding Jb  J k b
where Jk is the Jacobian at xk, but J, λ to both sides
of the above equality and inserting the expression for ε1 and ε2, we get

T
T
g0 ðxbÞ  Jb  J k ε1 þ ρ Jb  J k ε2 ¼ Jb b
T
λ,


which shows that xb, b
λ also satisfies the conditions for a stationary point of (14.125). Observe that
the Hessians for the Lagrangian function of (14.124) and (14.125) differ only in the amount

T

ρ Jb  J k Jb  J k at the solution of (14.125), which is of order ρkΔxkk, where Δxk ¼ xb  xk :
Hence, for sufficiently small ε1, ε2, and Δxk, if the reduced Hessian of (14.124) is positive definite at
xb, then, by continuity, the reduced Hessian of (14.125) will also be positive definite, thus satisfying
the sufficient conditions for a local minimum of (14.125) at xb: ♦


Theorem 14.11 Let (xk,λk) be an approximate solution of (14.123) and let xb, b
λ be a solution of the
linearized subproblem


1
min f 0 ðxÞ  λTk f  fe þ ρf T f
2 ð14:126Þ
subject to
feðx, xk Þ ¼ 0:


λ  λk ¼ ε1 and f ðxbÞ ¼ ε2 , then xb, b
If b λ is also a solution to the perturbed problem
514 14 Penalty and Augmented Lagrangian Methods


min f 0 ðxÞ þ εT1 f  fe þ ρεT2 f
subject to ð14:127Þ
f ðxÞ ¼ ε2 :

Proof The proof is similar to the proof of Theorem 14.10. ♦

If ε1 and ε2 are sufficiently small, then ρ can safely be reduced to zero (Murtagh & Saunders,
1982). Apparently, problem (14.125) is less sensitive to deviations from its optimum than (14.127).
Thus, let Δx be an arbitrary small change to the solution xbof (14.125). Then, the objective function for


(14.125) differs from the true objective f 0(x) by an amount δ1 ¼ ðε1 þ ρε2 ÞT f  fe : Clearly,
|δ1|  (kε1k þ ρkε2k)O(kΔxk2). On the other hand, for (14.127), the analogous deviation is





δ2 ¼ εT1 f  fe þ ρεT2 f ¼ εT1 f  fe þ ρεT2 fb þ JΔx
b þ O kΔxk2 :

  T 
 
In this case, jδ2 j  ðkε1 k þ ρkε2 kÞO kΔxk2 þ ρkε2 k2 þ ρJb ε2 kΔxk: Since δ1 is of order
kΔxk2 while δ2 is of order kΔxk, it follows that the modified penalty term in (14.124) has a theoretical
advantage over the conventional penalty term of (14.126).
Having in view the above theoretical developments, the MINOS algorithm for solving the general
nonlinear optimization problems is as follows.

Algorithm 14.7 MINOS–nonlinear constraints–Murtagh and Saunders

1. Consider k ¼ 0. Choose the initial estimates xk, yk, and λk. Select a value for the penalty parameter ρ > 0 as
well as a convergence tolerance εc > 0
2. Solve the linear subproblem:
(a) Given xk, yk, λk, and ρ, using the Algorithm 14.6 (MINOS for linear constraints), solve the linear
constrained subproblem (14.121) obtaining xkþ1, ykþ1 and u, where u is the vector of the
Lagrange multipliers associated to the constraints from (14.121)
(b) Form the vector λkþ1 where its first m1 components are those of u
3. Test for convergence. If (xk, yk) satisfies the KKT optimality conditions, then stop
4. If
kf ðxkþ1 ÞþA1 ykþ1 b1 k λkþ1 λk k
1þk½xkþ1 ykþ1 k
 εc , and k1þ kλkþ1 k  εc ,
then set ρ ¼ 0; otherwise, set ρ ¼ 10ρ
5. Consider a re-linearization of the constraints in xkþ1, set k ¼ k þ 1, and go to step 2 ♦

In Murtagh and Saunders (1982), we find some details on the algorithm concerning sparse matrices
techniques for solving linear algebraic systems, infeasible subproblems, user options, verification of
the gradients option, Jacobian options (dense or sparse), evaluation options (the constraints and the
gradients are evaluated only once per major iteration or they are evaluated as often as the objective),
convergence conditions (major iterations, minor iterations, radius of convergence, row tolerance),
etc. Let us present some numerical examples.

Example 14.4 Consider the nonlinear optimization problem (Wright (No.4), 1976; Murtagh &
Saunders, 1982)
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 515

min ðx1  1Þ2 þ ðx1  x2 Þ2 þ ðx2  x3 Þ3 þ ðx3  x4 Þ4 þ ðx4  x5 Þ4

subject to
pffiffiffi
x1 þ x22 þ x33 ¼ 2 þ 3 2,
pffiffiffi
x2  x23 þ x4 ¼ 2 þ 2 2,
x1 x5 ¼ 2:

Considering the initial points

A ¼ ½1 1 1 1 1T , B ¼ ½2 2 2 2 2T , C ¼ ½1 3  0:5  2  3T ,

D ¼ ½1 2 1  2  2T , E ¼ ½2  2  2  2  2T ,

Algorithm 14.7 (MINOS with nonlinear constraints) gives the following results:

A, B ! x1 ¼ ½ 1:11663 1:22043 1:53779 1:97277 1:79110 T ,

 
f x1 ¼ 0:0029307,

C ! x2 ¼ ½ 0:703393 2:63570 0:0963618 1:79799 2:84336 T ,

 
f x2 ¼ 44:022089,

D! x3 ¼ ½ 1:27305 2:41035 1:19486 0:154239 1:57103 T ,

 
f x3 ¼ 27:8718714,

E! x4 ¼ ½ 2:79087 3:00414 0:20538 3:87474 0:71662 T ,

 
f x4 ¼ 607:03036:

In Table 14.9, we can see the characteristics of the optimization process, where #itM is the number
of the major iterations, #itm is the number of minor iterations necessary for solving the subproblems
with linear constraints, and #evf is the number of evaluations of the objective and of the constraints,
including their gradients.

Example 14.5 Consider the problem (Wright (No.9), 1976; Murtagh & Saunders, 1982)

min 10x1 x4  6x22 x3 þ x31 x2 þ 9 sin ðx5  x3 Þ þ x32 x24 x45

subject to
x21 þ x22 þ x23 þ x24 þ x25  20,
x21 x3 þ x4 x5  2,
x22 x4 þ 10x1 x5  5:

Table 14.9 MINOS with different initializations. Example 14.4

A B C D E
#itM 8 9 7 6 8
#itm 30 21 14 17 21
#evf 64 49 35 35 55
516 14 Penalty and Augmented Lagrangian Methods

Table 14.10 MINOS with different initializations. Example 14.5

A B C D
#itM 10 8 20 9
#itm 51 25 105 46
#evf 136 64 195 97

For the initial points

A ¼ ½1 1 1 1 1T , B ¼ ½1  3 1  1 2T , C ¼ ½5  5 1  3 1T ,

D ¼ ½10  1 1 10 2T ,

Algorithm 14.7 gives the following results:

A ! x1 ¼ ½ 0:08145 3:69238 2:48741 0:37713 0:17398 T ,

 
f x1 ¼ 210:407817,

B, C ! x2 ¼ ½ 1:47963 2:63661 1:05467 1:61151 2:67388 T ,

 
f x2 ¼ 2500:584488,

D ! x3 ¼ ½ 0:0774 2:58139 0:01143 2:10766 2:98229 T ,

 
f x3 ¼ 6043:539113:

Table 14.10 contains the characteristics of the optimization process initialized in different points.

Numerical Study—MINOS: Solving Applications from the LACOP Collection

In Appendix C, we have presented the LACOP collection, which includes a number of 18 real
nonlinear optimization applications. In Tables 14.11 and 14.12 of this numerical study, we can see the
numerical performances of MINOS for solving 12 small-scale nonlinear optimization applications
and 6 large-scale nonlinear optimization applications of different dimensions, respectively.

Table 14.11 Performances of MINOS for solving 12 applications from the LACOP collection. Small-scale nonlinear
application

n me mc #itM #itm #nf #nr #s cpu vfo

ELCH 10 3 0 1 29 84 1 7 0.02 −47.761090
ALKI 10 3 8 12 29 74 74 1 0.02 −1768.8070
PREC 8 0 6 17 71 171 171 3 0.02 4.286758
PPSE 9 6 0 6 13 17 17 1 0.01 5055.0118
MSP3 13 0 15 7 60 95 95 1 0.01 97.587531
MSP5 16 0 21 10 89 139 139 0 0.01 174.787136
POOL 34 20 0 5 60 133 133 3 0.01 2569.800
TRAFO 6 0 2 8 41 83 83 4 0.00 135.075962
LATHE 10 1 14 23 150 327 327 0 0.02 −4430.08793
DES 150 50 0 66 2344 5838 5838 100 2.75 1055.182314
CSTC 303 200 0 6 160 320 320 100 0.19 −3.480074
DIFF 396 324 0 1 48 0 0 0 0.02 0.00
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 517

Table 14.12 Performances of MINOS for solving six applications from the LACOP collection. Large-scale nonlinear
applications

n me mc #itM #itm #nf #s cpu vfo

HANG 1002 501 0 94 3681 8642 496 4.46 5.06855
2002 1001 0 209 8252 19712 987 27.49 5.06851
FLOW 1182 734 0 1 997 14 0 0.19 0.00
FLOWO 1556 1005 0 1 709 2 0 0.25 0.00
POL 4004 3000 0 28 1002 3173 0 5.29 14.14484
6004 4500 0 40 1502 4821 0 12.87 14.14341
8004 6000 0 53 2000 6524 0 22.45 14.12985
10,004 7500 0 66 2499 8489 0 36.52 14.14179
CAT 3003 2000 0 11 272 1183 0 2.18 −0.04805
6003 4000 0 18 536 2430 0 9.06 −0.04805
9003 6000 0 23 796 3703 0 20.73 −0.04804
CONT 2505 2000 0 1 506 663 1 0.09 1.013239
5005 4000 0 1 1005 1314 2 0.20 1.005922
7500 6000 0 1 1509 1978 1 0.69 1.004561
10,005 8000 0 1 2005 2627 0 1.33 1.004072

Table 14.13 Performances of MINOS for solving the HANG application from the LACOP collection. Large-scale
nonlinear applications

n me mc #itM #itm #nf #s cpu vfo

HANG 4002 2001 0 447 17,791 42,955 1950 205.64 5.06849
8002 4001 0 704 28,153 68,784 3385 1024.63 5.06893

In Tables 14.11–14.13 we have: n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the
number of inequality constraints, #itM ¼ the number of major iterations, #itm ¼ the number of minor iterations, #nf ¼
the number of evaluations of the objective function, #nr ¼ the number of evaluations of constraints, #s ¼ the number of
superbasic variables, cpu ¼ the CPU computing time for solving the problem (seconds), and vfo ¼ the value of the
objective function.

Table 14.13 shows the performances of MINOS for solving the application HANG with a large
number of variables.

Notes and References

The penalty and the augmented Lagrangian are two very important concepts in nonlinear optimiza-
tion. The content of this chapter is based on the books of Nocedal and Wright (2006) and Bertsekas
(1982b). The criticism of penalty and of the augmented Lagrangian methods is taken from Saunders
(2015a) and Andrei (2015a). The quadratic penalty function was first proposed by Courant (1943).
The quadratic penalty method is often used in practice for its simplicity, although it has a major
disadvantage, mainly because of the ill-conditioning of the Hessian of the quadratic penalty function.
In the nonsmooth exact penalty methods, a single unconstrained optimization problem rather than a
sequence replaces the original constrained problem. However, nonsmoothness may create
complications. A well-known function of this type is the l1 penalty function. This computational
scheme based on the exact penalty method sometimes works very well in practice, but it can be
inefficient. If the initial penalty parameter σ 0 in Algorithm 14.2 is too small, many cycles may be
needed. Besides, the iterates may move away from the solution in the initial cycles. If σ 0 is large, then
the penalty function will be difficult to minimize by requiring a large number of iterations. A different
518 14 Penalty and Augmented Lagrangian Methods

kind of exact penalty approach is the augmented Lagrangian method, which avoids the
ill-conditioning by using the explicit Lagrange multiplier estimate.
The augmented Lagrangian was independently proposed by Hestenes (1969) and Powell (1969).
The l1 penalty method was developed by Fletcher. All these concepts are implemented in different
combinations in very efficient nonlinear optimization packages. For example, the l1 penalty method
that uses linear programming subproblems has been implemented as part of the KNITRO/ACTIVE
software package (see Chap. 15).
The augmented Lagrangian methods solve the nonlinear constrained optimization problems (14.3)
by a sequence of subproblems that minimize the augmented Lagrangian, either subject to a lineariza-
tion of the constraints or as a bound-constrained problem. The best known software implementing the
penalty or the augmented Lagrangian are as follows:
ALGENCAN (Andreani, Birgin, Martı́nez, & Schuverdt, 2007, 2008; Birgin & Martı́nez, 2014) is
a general algorithm of the augmented Lagrangian type, in which the subproblems are solved by using
GENCAN. GENCAN (included in ALGENCAN) is a Fortran code for minimizing a smooth function
with a potentially large number of variables and box constraints. The subproblem is solved by using a
quasi-Newton method. The penalty on each constraint is increased if the previous iteration does not
yield a better point.
GALAHAD (Gould, Orban, & Toint, 2004) contains a range of solvers for the large-scale
nonlinear optimization. It also contains a presolve routine for quadratic programming (Gould &
Toint, 2004), as well as some other support routines. It includes LANCELOT B, an augmented
Lagrangian method with a nonmonotone descent condition; FILTRANE, a solver for feasibility
problems based on a multidimensional filter; and an interior point and active-set methods for solving
large-scale quadratic programs.
LANCELOT (Conn, Gould, & Toint, 1992b) is a large-scale implementation of a bound-
constrained augmented Lagrangian method. In LANCELOT, the simple bounds are treated explicitly,
and all the other constraints are converted to equations and incorporated into an augmented Lagrang-
ian function. It approximately solves a sequence of bound-constrained augmented Lagrangian
subproblems by using a trust-region approach. In LANCELOT, each trust-region subproblem is
solved approximately, firstly by identifying a Cauchy point to ensure the global convergence and then
by applying conjugate gradient steps to accelerate the local convergence. Although this approach
proves to be robust (Conn, Gould, & Toint, 1996a), it has a number of drawbacks. One of them is that
the augmentation may not be the ideal way to treat the linear constraints. A more attractive approach
is to handle all the linear constraints explicitly (Conn, Gould, Sartenaer, & Toint, 1996b). The
methods that use the linearly constrained subproblems are described by Rosen and Kreuser (1972),
Robinson (1972, 1974), and Murtagh and Saunders (1978).
MINOS (Murtagh & Saunders, 1987, 1995) uses a projected augmented Lagrangian for solving
general nonlinear problems. At each “major iteration,” a linearly constrained nonlinear subproblem is
solved, where the linear constraints constitute all the linear constraints of (14.3) and the linearizations
of the nonlinear constraints. The objective function of this subproblem is an augmented Lagrangian
which contains the departure from linearity of the nonlinear constraints. The subproblems are solved
by using a reduced gradient algorithm along with a quasi-Newton algorithm. The quasi-Newton
algorithm provides a search direction along which a line-search is performed to improve the value of
the objective function and to reduce the infeasibilities. MINOS integrates a lot of computational
ingredients from linear algebra. The most important is LUSOL, a set of procedures for computing and
updating the LU factors of a general sparse matrix A. The original LUSOL procedures are described
in Gill, Murray, Saunders, and Wright (1987). The main factorization uses a traditional Markowitz
strategy with threshold partial pivoting (Markowitz, 1957). It is designed to allow A to be square or
14.6 The Linearly Constrained Augmented Lagrangian (MINOS) 519

rectangular with arbitrary rank, to factorize A ¼ LU directly by finding suitable row and column
orderings, to replace a column or a row of A, to add or delete a column or a row of A, etc. LUSOL
continues to evolve with the addition of rank-revealing LU factorizations for sparse matrices, using
either threshold rook pivoting or threshold complete pivoting (Saunders, 2015a, b). MINOS is
embedded in the GAMS technology (Andrei, 2017c). Both LANCELOT and MINOS are suitable
for solving large-scale nonlinear optimization problems. However, they differ significantly in the
formulation of the step-computation subproblems and in the techniques used to solve these
subproblems. MINOS implements the linearly constrained Lagrangian, followed by a reduced-
space approach to handle the linearized constraints, and it employs a dense quasi-Newton approxi-
mation to the Hessian of the Langrangian. On the other hand, LANCELOT implements the bound-
constrained Lagrangian method. Numerical experiments have shown an advantage of MINOS over
LANCELOT for problems with linear constraints. Our description of MINOS is based on the papers
of Murtagh and Saunders (1978, 1980, 1982, 1995).
SPENBAR is another approach, based on the logarithmic penalty function. In this approach, the
logarithmic terms prevent the feasible iterates from moving too close to the boundary of the feasible
region of the problem. SPENBAR is designed for solving large-scale constrained optimization
problems. Only the nonzero elements of the Jacobian of the equalities and of the inequalities are
stored separately. In SPENBAR, the subproblem is solved by the truncated Newton with simple
bounds (TNBC). The SPENBAR algorithm, presented in Andrei (1996a, b, c, 1998a, 2001), combines
the augmented Lagrangian with a log-barrier function (a composite function which includes the
augmented Lagrangian and the classical log-barrier function) in which the parameters are updated in
such a way as to obtain a KKT point for the considered nonlinear optimization problem. The equality
and inequality constraints are treated separately. In this variant, the penalty-barrier function includes
only the equality and the inequality constraints. Thus, the algorithm reduces to minimizing a
sequence of simple bound optimization subproblems. The simple bounds on variables can also be
introduced in the composite function, thus obtaining another variant of the penalty-barrier algorithm,
in which a sequence of unconstrained minimization subproblems is used. For both variants, the
convergence theory is the same (Ben-Tal & Zibulevsky, 1993; Conn, Gould, & Toint, 1991a, 1992a).
However, the numerical performances could be different. SPENBAR illustrates the importance of
modifying the classical augmented Lagrangian in order to avoid the ill-conditioning. Also see
Vassiliadis and Floudas (1997).
PENNON (Kocvara & Stingl, 2003) is an augmented Lagrangian penalty-barrier method. It solves
a sequence of unconstrained optimization subproblems in which the inequality constraints are
included in the barrier functions and the equality constraints in the penalty functions. Every uncon-
strained minimization subproblem is solved with Newton’s method.
A new penalty method for solving the general nonlinear optimization problems subject to
inequality constraints was presented by Xavier (2001).
 Sequential Quadratic Programming
15

Sequential quadratic programming (SQP) methods are very effective for solving optimization
problems with significant nonlinearities in constraints. These are active-set methods and generate
steps by solving quadratic programming subproblems at every iteration. These methods are used both
in the line-search and in the trust-region paradigm. In this chapter, we consider both the equality
quadratic programming and the inequality quadratic programming. There are some differences
between these approaches. In the inequality quadratic programming, at every iteration, a general
inequality-constrained quadratic programming problem is solved for computing a step and for
generating an estimate of the optimal active-set. On the other hand, in the equality quadratic
programming, these are separated. Firstly, they compute an estimate of the optimal active-set, then
they solve an equality-constrained quadratic programming problem to find the step. Our presentation
proceeds to develop the theory of the sequential quadratic programming approach for the step
computation, followed by practical line-search and trust-region methods that achieve the convergence
from remote starting points. At the same time, a number of three implementations of these methods
are discussed, which are representative of the following: an SQP algorithm for large-scale-
constrained optimization (SNOPT), an SQP algorithm with successive error restoration (NLPQLP),
and the active-set sequential linear-quadratic programming (KNITRO/ACTIVE). Their performances
are illustrated in solving large-scale problems from the LACOP collection.

Equality-Constrained Problems
Let us consider the equality-constrained nonlinear optimization problem
min f ðxÞ ð15:1aÞ
subject to
ð15:1bÞ
hðxÞ ¼ 0,

where f : ℝn ! ℝ and h : ℝn ! ℝm are smooth functions, where h(x) ¼ [h1(x), . . ., hm(x)]T. The idea
of the SQP method is to model (15.1) at the current point xk by a quadratic programming subproblem
and then to use the solution of this subproblem to define a new iterate xkþ1. The most challenging is
how to design the quadratic subproblem so that it yields a good approximation of (15.1). One of the
simplest derivations of the SQP methods is to apply Newton’s method to the KKT optimality
conditions for the nonlinear problem (15.1).

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 521
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_15
522 15 Sequential Quadratic Programming

The Lagrange function for (15.1) is

Lðx, λÞ ¼ f ðxÞ  λT hðxÞ, ð15:2Þ

where λ 2 ℝm is the vector of the Lagrange multipliers associated to the constraints of the problem.
Let us define A(x) as the Jacobian matrix of the constraints, that is

AðxÞT ¼ ½∇h1 ðxÞ, . . . , ∇hm ðxÞ 2 ℝnm , ð15:3Þ

where hi(x), i ¼ 1, . . ., m, are the components of the vector h(x). The first-order optimality conditions
of the equality-constrained problem can be written as a system of n þ m equations with n þ m
unknowns x and λ:
" #
∇f ðxÞ  AðxÞT λ
Fðx, λÞ ¼ ¼ 0: ð15:4Þ
hð x Þ

Observe that any solution (x, λ) of (15.1) for which A(x) has full rank satisfies (15.4). To find
such a point, the solution of the nonlinear system (15.4), the Newton method is the most suitable. The
Jacobian of (15.4) with respect to x and λ is given by
" #
0 ∇2xx Lðx, λÞ AðxÞT
F ðx, λÞ ¼ , ð15:5Þ
Að x Þ 0

P
m
where ∇2xx Lðx, λÞ ¼ ∇2 f ðxÞ  λi ∇2 hi ðxÞ: The Newton step from the iterate (xk, λk) is given by
i¼1
     
xkþ1 xk dx
¼ þ , ð15:6Þ
λkþ1 λk dλ

where dx and dλ are solutions of the Newton system

" #  " #
∇2xx Lðxk , λk Þ Aðxk ÞT dx ∇f ðxk Þ þ Aðxk ÞT λk
¼ : ð15:7Þ
Aðxk Þ 0 dλ hðxk Þ

The Newton iteration (15.6) is well defined when the KKT matrix in (15.7) is nonsingular. As we
know (see Chap. 11), this matrix is nonsingular if the following two assumptions hold:

C1. The Jacobian A(x) of the constraints has full row rank.
C2. The matrix ∇2xx Lðx, λÞ is positive definite on the tangent space of the constraints, that is,
dT ∇2xx Lðx, λÞd > 0 for all d 6¼ 0 for whichA(x)d ¼ 0.

Observe that the first assumption is exactly the linear independence constraint qualification
(LICQ) discussed in Chap. 11 (see Remark 11.2). The second assumption holds whenever (x, λ) is
close to the optimum point (x, λ) and the second-order sufficient condition is satisfied at the solution
(see Theorem 11.13).
This approach, which theoretically is very well justified, has an alternative that illustrates the
motivation of the sequential quadratic programming. Let us focus on the Newton iteration given by
(15.6) and (15.7) and let us assume that at the iteration (xk, λk) the problem (15.1) is modeled as a
quadratic program
 15 Sequential Quadratic Programming 523

1
min d2ℝn f ðxk Þ þ ∇f ðxk ÞT d þ d T ∇2xx Lðxk , λk Þd, ð15:8aÞ
2
subject to
Aðxk Þd þ hðxk Þ ¼ 0: ð15:8bÞ

If the assumptions (C1) and (C2) hold, then this problem has a unique solution (dk, lk) that satisfies

∇2xx Lðxk , λk Þdk þ ∇f ðxk Þ  Aðxk ÞT lk ¼ 0, ð15:9aÞ

Aðxk Þdk þ hðxk Þ ¼ 0: ð15:9bÞ

But the vectors dk and lk can be immediately identified as the solution of the Newton system (15.7).
Now, if we subtract A(xk)Tλk, from both sides of the first equation in (15.7), we get
" #   
∇2xx Lðxk , λk Þ Aðxk ÞT dk ∇f ðxk Þ
¼ : ð15:10Þ
Að x k Þ 0 λkþ1 hðxk Þ

Therefore, having in view the nonsingularity of the coefficient matrix of (15.10), it follows that
λkþ1 ¼ lk and dk represents a solution of the quadratic program (15.8) and of the system (15.7).
In conclusion, the new iterate (xkþ1, λkþ1) can be defined either as the solution of the quadratic
programming problem (15.8) or as the iterate generated by the Newton method (15.6) and (15.7)
applied to the optimality conditions of the problem (15.1). Both these approaches are useful. The
Newton point of view is suitable for analysis, whereas the sequential quadratic programming enables
us to derive practical algorithms and to extend the technique to the inequality-constrained case.
Based on these theoretical developments, the following sequential quadratic programming algo-
rithm for equality constraints can be presented.

Algorithm 15.1 Sequential quadratic programming—equality constraints

1. Choose an initial point (x0, λ0) and set k ¼ 0

2. If a test for stopping the iterations is satisfied, stop; otherwise, go to step 3
3. Evaluate: f(xk), ∇f(xk), ∇2xx Lðxk , λk Þ, h(xk), and A(xk)
4. Solve the quadratic subproblem (15.8) to obtain a solution (dk, lk)
5. Set xkþ1 ¼ xk þ dk and λkþ1 ¼ lk. Set k ¼ k þ 1 and go to step 2 ♦

It is obvious that in the objective function (15.8a), the linear term ∇f(xk)Td can be replaced by
∇xL(xk, λk)Td, since the constraints (15.8b) make the two choices equivalent. In this case, (15.8a) is
exactly a quadratic approximation of the Lagrangian function. This is the main motivation for the
choice of the quadratic model (15.8): first replace the nonlinear optimization problem (15.1) by the
problem of minimizing the Lagrangian subject to the equality constraints (15.1b), then make a
quadratic approximation to the Lagrangian and a linear approximation to the constraints to obtain
the quadratic program (15.8).
Now, let us present the conditions that guarantee the local convergence of Algorithm 15.1.
Consider that the algorithm uses exact second derivatives.
524 15 Sequential Quadratic Programming

Theorem 15.1 Assume that the point x is a local solution of the problem (15.1) at which the
following conditions hold:

(i) The functions f and h are twice continuously differentiable in a neighborhood of x with
Lipschitz continuous second derivatives.
(ii) The linear independence constraint qualification (see Remark 11.2) holds at x. This condition
implies that the KKT conditions (11.21) (see Theorem 11.15) are satisfied for a certain vector of
multipliers λ.
(iii) The second-order sufficient conditions (see Theorem 11.16) hold at (x, λ).

Then, if (x0, λ0) is sufficiently close to (x, λ), the pairs (xk, λk) generated by Algorithm 15.1
quadratically converge to (x, λ).

Proof This result follows directly from the convergence of the Newton method applied to the
nonlinear system (15.4). ♦

Algorithm 15.1 can be specified in some variants. A version of the algorithm uses a quasi-Newton
estimate of ∇2xx Lðxk , λk Þ rather than calculating it from the second derivatives (see Wilson (1963),
Han (1976), and Powell (1978a)). In step 5 of the algorithm, we can use the line-search to determine
xkþ1 ¼ xk þ αdk ensuring P(xkþ1) < P(xk), where P denotes a certain penalty function. Different
penalty functions can be tried. The line-search is important because it forces a reduction in a
composite function, involving both the objective f and the constraints h. It ensures that the new
point xkþ1 is, in a measurable way, an improvement on xk and therefore provides a basis for a proof of
the convergence.

Inequality-Constrained Problems
The above-described sequential quadratic programming approach can be extended to the general
nonlinear optimization problem
min f ðxÞ ð15:11aÞ
subject to
ð15:11bÞ
ci ðxÞ ¼ 0, i 2 E,

ci ðxÞ  0, i 2 I, ð15:11cÞ

where E ≜ {1, . . ., me} and I ≜ {me þ 1, . . ., m} are sets on indices for the equality and inequality
constraints, respectively. Assume that all the functions of the problem (15.11) are twice continuously
differentiable.

As in the case of the equality-constrained problems, at the current point xk, the problem (15.11) is
modeled as a quadratic program by linearizing both the equality and the inequality constraints to
obtain
1
min d2ℝn f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx Lðxk , λk Þd ð15:12aÞ
2
subject to
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 E, ð15:12bÞ

∇ci ðxk ÞT d þ ci ðxk Þ  0, i 2 I: ð15:12cÞ

 15 Sequential Quadratic Programming 525

For solving the subproblem (15.12), we can use the algorithms for quadratic programming
described in Chap. 13. The new iterate is given by (xk þ dk, λkþ1), where dk and λkþ1 are the solution
and the corresponding Lagrange multiplier of (15.12). The algorithm is as follows:

Algorithm 15.2 Sequential quadratic programming—inequality constraints

1. Choose an initial point (x0, λ0) and set k ¼ 0

2. If a test for stopping the iterations is satisfied, stop; otherwise, go to step 3
3. Evaluate: f(xk), ∇f(xk), ∇2xx Lðxk , λk Þ, ci(xk) and ∇ci(xk), i 2 E [ I
4. Solve the quadratic problem (15.12) to obtain a solution (dk, lk)
5. Set xkþ1 ¼ xk þ dk and λkþ1 ¼ lk. Set k ¼ k þ 1 and go to step 2 ♦

In this approach, the set of active constraints at the solution of (15.12) is the guess of the active-set
at the solution of the nonlinear problem (15.11). If the SQP method is able to correctly identify the
optimal active-set (i.e., the active-set at the solution of (15.11)), then the method works like a Newton
method for the equality-constrained optimization. In this case, the convergence will be rapid. The
following result, given by Robinson (1974), presents the conditions under which this behavior of the
algorithm holds.

Theorem 15.2 Assume that x is a local solution of (15.11) at which the KKT conditions are satisfied
for some λ. Also, assume that the linear independence constraint qualification (Remark 11.2), the
strict complementarity (Definition 11.22) and the second-order sufficient conditions (11.13) hold at
(x, λ). Then, if (xk, λk) is sufficiently close to (x, λ), there is a local solution of the subproblem
(15.12) whose active-set is the same as the active-set of the nonlinear program (15.11) at x.

Remark 15.1 To design SQP algorithms for solving the general nonlinear optimization problem
(15.11), two methods are known. The first, which has been described above, solves the quadratic
subproblem (15.12) at each iteration by taking the active-set at the solution of this subproblem as a
guess of the optimal active-set. This approach is called IQP (inequality-constrained QP). It has
proved to be quite effective and successful in practice. Its main disadvantage is the expense of solving
the general quadratic subproblem (15.12), which can be high when the problem is large. However, an
improvement of this situation could be as follows: as the iterates converge to the solution, solving the
quadratic subproblem (15.12) becomes economical if the information from the previous iteration is
used to make a good guess of the optimal solution to the current subproblem. This strategy is called
warm-start. The second approach selects a subset of constraints at each iteration to be the so-called
working set and solves only equality-constrained subproblems of the form (15.8), where the
constraints in the working set are imposed as equalities and all the other constraints are ignored.
This approach is called EQP (equality-constrained QP). In it, the working set is updated at every
iteration by rules based on the Lagrange multiplier estimates or by solving an auxiliary subproblem.
The EQP is more effective because the equality-constrained quadratic subproblems are less expensive
to solve than (15.12), when the problem is large. An example of an EQP method is the sequential
linear-quadratic programming (SLQP) method, which is to be discussed in Sect. 15.6. Another
successful EQP is the gradient projection method in the context of the bound-constrained quadratic
programming.
All these ideas will be developed in the next sections of this chapter. We note that Algorithms 15.1
and 15.2 are successful for solving nonlinear optimization problems. However, these algorithms in
the form in which they were shown represent a simple approach of the sequential quadratic
526 15 Sequential Quadratic Programming

programming method. Some other ingredients must be introduced in their structure, in order to have
efficient and robust SQP algorithms. In the following, we try to be more specific and detail the SQP
for inequality-constrained problems. After that, some important ingredients are introduced in the SQP
approach for having modern, efficient, and robust algorithms, able to solve complex large-scale
nonlinear optimization problems. ♦

15.1 A Simple Approach to SQP

The SQP Algorithms 15.1 and 15.2 have good local properties. However, they represent a rudimen-
tary approach of this method. As a matter of fact, it can be shown that Algorithm 15.2 converges
quadratically if: (i) the initial point is sufficiently close to a local minimizer x, (ii) the coefficient
matrix in (15.7) or in (15.10) for (x0, λ0) is nonsingular, (iii) the second-order sufficient conditions
hold for (x, λ) with rank(A(x)) ¼ m, and (iv) for (x0, λ0) the quadratic programming problem
(15.12) has a unique solution. (See Fletcher 1987, Chap. 12).
To be more specific, let us consider the inequality-constrained problem
min f ðxÞ ð15:13aÞ
subject to
ci ðxÞ  0, i 2 I, ð15:13bÞ

where I ≜ {1, . . ., m} is the set of indices for the inequality constraints. Suppose that the functions
f(x) and ci(x), i ¼ 1, . . ., m, are twice continuously differentiable and the feasible region determined by
the constraints (15.13b) is nonempty. The Lagrange function associated to (15.13) is
X
m
Lðx, λÞ ¼ f ðxÞ  λi ci ðxÞ: ð15:14Þ
i¼1

The KKT conditions are as follows:

∇x Lðx, λÞ ¼ 0, ð15:15aÞ

ci ðxÞ  0, i ¼ 1, . . . , m, ð15:15bÞ

λ  0, ð15:15cÞ

λi ci ðxÞ ¼ 0, i ¼ 1, . . . , m: ð15:15dÞ

Now, for the iterate (xk, λk) an increment (δx, δλ) needs to be found such that the next iterate
(xkþ1, λkþ1) ¼ (xk þ δx, λk þ δλ) approximates the KKT conditions (15.15) in the sense that

∇x Lðxkþ1 , λkþ1 Þ  ∇x Lðxk , λk Þ þ ∇2xx Lðxk , λk Þδx þ ∇2xλ Lðxk , λk Þδλ ¼ 0, ð15:16aÞ

ci ðxk þ δx Þ  ci ðxk Þ þ δTx ∇ci ðxk Þ  0, i ¼ 1, . . . , m, ð15:16bÞ

ðλkþ1 Þi  0, i ¼ 1, . . . , m, ð15:16cÞ
 
ci ðxk Þ þ δTx ∇ci ðxk Þ ðλkþ1 Þi ¼ 0, i ¼ 1, . . . , m: ð15:16dÞ
15.1 A Simple Approach to SQP 527

From (15.14) it follows that

X
m
∇x Lðxk , λk Þ ¼ ∇f ðxk Þ  ðλk Þi ∇ci ðxk Þ  gk  ATk λk , ð15:17aÞ
i¼1

X
m
∇2xx Lðxk , λk Þ ¼ ∇2 f ðxk Þ  ðλk Þi ∇2 ci ðxk Þ  W k , ð15:17bÞ
i¼1

∇2xλ Lðxk , λk Þ ¼ ATk , ð15:17cÞ

where
2 3
∇T c 1 ð x k Þ
6 7
Ak ¼ 4 ⋮ 5: ð15:17dÞ
∇ cm ðxk Þ
T

With this, the approximate KKT conditions can be expressed as

W k δx þ gk  ATk λkþ1 ¼ 0, ð15:18aÞ

Ak δx  ck , ð15:18bÞ

λkþ1  0, ð15:18cÞ

ðλkþ1 Þi ðAk δx þ ck Þi ¼ 0, i ¼ 1, . . . , m, ð15:18dÞ

where

ck ¼ ½c1 ðxk Þ, c2 ðxk Þ, . . . , cm ðxk ÞT ð15:18eÞ

is the vector of the constraints from (15.13).

Given (xk, λk), then (15.18) can be interpreted as the exact KKT conditions of the quadratic
problem
1 T
min δ W δ þ δTx gk
2 x k x
subject to ð15:19Þ
Ak δx  ck :

If δx is a regular solution of (15.19), that is the gradients of the constraints that are active at xk are
linearly independent, then (15.18a) can be written as

W k δx þ gk  ATak lkþ1 ¼ 0,

where the rows of Aak are the rows of Ak satisfying the equality (Akδk þ ck)i ¼ 0 and lkþ1 is the
associated Lagrange multiplier vector. Therefore, lkþ1 can be computed as
 1
lkþ1 ¼ Aak ATak Aak ðW k δx þ gk Þ: ð15:20Þ

Of course, from the complementarity conditions (15.18d) it follows that the Lagrange multiplier
λkþ1 can be obtained by inserting zeros where necessary in lkþ1. With this, the following algorithm
can be presented.
528 15 Sequential Quadratic Programming

Algorithm 15.3 Sequential quadratic programming—inequality constraints. Variant

1. Choose an initial point (x0, λ0) and set k ¼ 0. x0 and λ0 are chosen such that ci(x0)  0, i ¼ 1, . . ., m, and
λ0  0. Choose the sufficiently small convergence tolerance ε > 0
2. Evaluate: gk ¼ ∇ f(xk), c(xk), Ak and Wk
3. Solve the quadratic problem (15.19) to obtain a solution δx and from (15.20) compute the multiplier lkþ1
4. Set xkþ1 ¼ xk þ δk. If kδkk ε, then output x ¼ xkþ1 and stop. Otherwise, set k ¼ k þ 1 and go to step
2 ♦

Algorithm 15.3 is very close to the Algorithms 15.1 and 15.2. The main disadvantage of the
Algorithms 15.1, 15.2, and 15.3 is that, they may fail to converge when the initial point x0 is far away
from x. Another disadvantage is that they need to evaluate the Hessian of the Lagrangian which
combines the Hessian of the minimizing function with the Hessians of the constraints. Therefore,
these algorithms would require a large number of function evaluations, even for problems of
moderate size.
To ameliorate and to save their behavior, two modifications could be operational. The first one
uses a line-search method proposed by Han (1977), which enables the Algorithms 15.1, 15.2, and 15.3
to converge from arbitrary initial points. The second one approximates the Hessian of the Lagrangian
by using the BFGS updating formula. As in the BFGS algorithm for unconstrained optimization,
when starting with an initial positive definite approximation for the Hessian, all the subsequent
approximations will remain positive definite under a certain mild condition.

SQP with Line-Search Step

Consider the problem (15.13). Then a line-search step can be introduced in Algorithm 15.3 by
generating the (k þ 1)-th iteration as

xkþ1 ¼ xk þ αk δx , ð15:21Þ

where δx is the solution of the quadratic problem (15.19) and αk is the stepsize obtained by a line-
search. Han (1977) suggested that the scalar αk should be computed by minimizing the one-variable
exact penalty function
Xm
φðαÞ ¼ f ðxk þ αδx Þ þ r i¼1 ½ci ðxk þ αδx Þ ð15:22Þ

on an interval [0, δ], where the length δ of the interval and the parameter r > 0 are fixed throughout the
minimization process and [ci(xk þ αδx)] ¼ max {0, ci(xk þ αδx)}. Observe that the second term in
the right-hand side of (15.22) is always nonnegative and contains only those constraints that violate
the nonnegativity condition in (15.13). The term “penalty function” used for φ(α) is related to the fact
that the value of φ(α) depends on how many constraints from (15.13) are violated at α and on the
degree of violation. Choosing an appropriate value for r, a line-search algorithm applied to φ(α) will
yield a value αk such that the objective function at xk þ αkdk is reduced with fewer violated constraints
and a reduced degree of violation. The difficulty with this approach is that the penalty function φ(α) is
not differentiable. However, efficient search methods for the minimization of the nondifferentiable
one-dimensional functions are available (see the direct methods from Chap. 9).

An alternative to this method for the determination of αk was suggested by Powell (1978c). In this
method, an inexact line-search is applied to the Lagrangian (15.14) with x ¼ xk þ αδk and λ ¼ λkþ1 to
obtain the following one-variable function
15.1 A Simple Approach to SQP 529

Xm
ψ ðαÞ ¼ f ðxk þ αδk Þ  i¼1
ðλkþ1 Þi ci ðxk þ αδk Þ, ð15:23Þ

where (λkþ1)i is the i-th component of the vector λkþ1. Observe that if λkþ1  0, then the second term
in the right-hand side (15.23) acts as a penalty term since an α for which ci(xk þ αδk) < 0 will increase
the value of ψ(α). Powell suggested a method for determining αk by minimizing ψ(α) on the interval
[0,1]. Firstly, a line-search is performed to find the minimizer α1 of ψ(α) on [0,1], that is

α1 ¼ arg min ψ ðαÞ:

0 α 1

Observe that ψ(α) is differentiable and hence efficient gradient algorithms can be used to
determine α1 : Next, from the transversality condition of the optimality conditions (15.18d), observe
that, if for a particular index i, ci ðxk Þ þ δTk ∇ci ðxk Þ ¼ 0, then (λkþ1)i ¼ 0 and in this case the term
(λkþ1)ici(xk þ αδk) is not present in function ψ(α) from (15.23). In other words, the constraints
ci(x)  0 for which (λkþ1)i ¼ 0 need to be dealt separately. Now, let us define the index set
I ¼ {i : (λkþ1)i ¼ 0} and evaluate
α2 ¼ min max fα : ci ðxk þ αδk Þ  0g:
i2I

Therefore, the value of αk in (15.21) is computed as


αk ¼ 0:95 min α1 , α2 :

With this stepsize αk, in order to compute lkþ1 and λkþ1 the increment δx in (15.20) needs to be
modified as αkδx. In addition, in step 4 of Algorithm 15.3, xkþ1 ¼ xk þ δk is modified as xkþ1 ¼ xk þ αkδx.
With this line-search included, the modified SQP algorithm turns out to be more robust in the sense
that it converges from arbitrary initial points.

SQP with Approximate Hessian

The BFGS updating formula applied to the Lagrangian (15.14) is given by

W k δx δTx W k yk yTk
W kþ1 ¼ W k  þ T : ð15:24Þ
δTx W k δx yk δx

where δx ¼ xkþ1  xk and

 
yk ¼ ∇x Lðxkþ1 , λkþ1 Þ  ∇x Lðxk , λk Þ ¼ gkþ1  gk  ðAkþ1  Ak ÞT λkþ1 : ð15:25Þ

If Wk is positive definite, then Wkþ1 obtained from (15.24) is also positive definite if and only if
yTk δx> 0: Obviously, this condition does not hold when the Lagrangian has a negative curvature for
the iterate (xkþ1, λkþ1). Powell proposed a method for overcoming this difficulty by replacing yk in
(15.24) by

ηk ¼ θyk þ ð1  θÞW k δx ,

where yk is given by (15.25) and

8
>
< 1, if δTx yk  0:2δTx W k δx ,
θ¼ 0:8δx W k δx
T
>
: T , otherwise:
δx W k δx  δTx yk
530 15 Sequential Quadratic Programming

Algorithm 15.3 can be easily adapted to include both the search step (15.21) and the BFGS
approximate to the Hessian (15.24).

Equality Constraints
Now, let us consider the nonlinear optimization problems with equality constraints
min f ðxÞ
subject to
hi ðxÞ ¼ 0, i 2 E,

where E ≜ {1, . . ., me} is the set of indices for the equality constraints. Suppose that the functions
f(x) and hi(x), i ¼ 1, . . ., me, are twice continuously differentiable and the feasible region determined
by the constraints is nonempty. The Lagrange function associated to this problem is
X
me
Lðx, λÞ ¼ f ðxÞ  λi hi ðxÞ:
i¼1

The first-order optimality condition is ∇L(x, λ) ¼ 0. Then the formula for the Newton method is
xkþ1 ¼ xk þ dk ,
λkþ1 ¼ λk þ vk ,

where dk and vk are solutions to the linear system

" #   
∇2xx Lðxk , λk Þ ∇hðxk Þ dk ∇x Lðxk , λk Þ
¼ ,
∇hðxk ÞT 0 vk hð x k Þ

where vk is the vector of the Lagrange multipliers. Observe that this system of linear equations
represents the first-order optimality conditions for the following optimization problem:
1  
min dT ∇2xx Lðxk , λk Þ d þ dT ½∇x Lðxk , λk Þ
2
subject to
½∇hðxk ÞT d þ hðxk Þ ¼ 0:

But this is a quadratic program which is the minimization of a quadratic function subject to linear
constraints. In a sequential quadratic programming method, at each iteration, a quadratic program is
solved to get (dk, vk). This solution is used to update (xk, λk), and the process is repeated at the new
point.
Computing the updates dk and vk by solving the quadratic program corresponds to applying the
Newton method to the optimality conditions associated to the original problem. If the initial point is
close enough to the solution, then the method is quadratic convergent, provided that ∇2xx Lðx , λ Þis
nonsingular. The Hessian of the Lagrangian will be nonsingular if the regularity conditions (see
Remark 11.4) and the second-order sufficient conditions (see Theorem 11.13) for the original
optimization problem are satisfied, that is if ∇h(x) is of full rank and if Z T ∇2xx Lðx , λ ÞZ is positive
definite, where Z is a basis matrix for the null space of ∇h(x)T.
As in the case of nonlinear optimization problems with inequality constraints, for problems with
equality constraints, it is very easy to introduce the SQP with line-search step and the SQP with
approximate Hessian. The convergence results for the sequential quadratic programming method are
15.2 Reduced-Hessian Quasi-Newton Approximations 531

based on the idea that the approximation (xkþ1, λkþ1) is a better estimate of the solution than (xk, λk).
To determine the progress of the algorithm along the iterations, a merit function φ(.) is used. Usually,
a merit function is the sum of two terms: the objective function and the amount of infeasibility of the
constraints. If the new point reduces both the objective function and the infeasibility, then the value of
the merit function will decrease. However, for many problems, improvements in the objective value
come to increase the infeasibility and vice versa, so the merit function must balance these two goals.
Ideally, the merit function would be chosen in such a way that the solution (x, λ) would be a local
minimizer of the merit function if and only if it was a local solution of the optimization problem. If
this is the case, then a line-search with respect to the merit function could be performed as
xkþ1 ¼ xk þ αdk ,
λkþ1 ¼ λk þ αvk ,

where α is chosen such that φ(xkþ1, λkþ1) < φ(xk, λk). To be successful, the search direction from the
quadratic program would have to be a descent direction for the merit function. Unfortunately, it is
rarely possible to guarantee that the local minimizers of the merit function coincide with the local
solutions of the optimization problem.
Another point is when a quasi-Newton approximation to the Hessian is used to define the quadratic
program. Then the positive definiteness of the reduced Hessian is often guaranteed by the choice of
the quasi-Newton update formula. If the Newton method is used so that the Hessian in the quadratic
program is exactly ∇2xx Lðxk , λk Þ, then it is necessary to test if the Hessian is positive definite and, if
the case, to modify it by using, for example, the methods presented in Sect. 4.5 (Gill and Murray
modification of the Newton method). In the constrained nonlinear optimization, testing whether the
Hessian is positive definite is more complicated than in the unconstrained case.
Normally, near the solution, we would like to take a step α ¼ 1 in the line-search so that the
quadratic convergence rate of the Newton method could be achieved. Therefore, the merit function
should be chosen so that a step of α ¼ 1 is guaranteed to be accepted in the limit as the solution is
approached. However, this is not true for some merit functions.
Therefore, as we can see this naive or rudimentary approach of the sequential quadratic program-
ming has a number of deficiencies which preclude its applicability for solving large-scale problems.
Along the years, there has been considerable interest in understanding and improving the design
and implementation of algorithms based on the idea of sequential quadratic programming. Some of
these improvements, which are to be discussed in the next sections of this chapter, refer to enforcing
the convergence from remote starting points, handling inconsistent linearizations, full quasi-Newton
approximations, damped BFGS updating, reduced-Hessian quasi-Newton approximations, the merit
function in sequential quadratic programming using nonsmooth penalty functions or augmented
Lagrangians, second-order correction to overcome the Maratos effect, etc. In the following, we
describe some of these points needed to produce practical SQP algorithms able to solve large-scale
nonlinear optimization problems.

15.2 Reduced-Hessian Quasi-Newton Approximations

Let us consider the equality-constrained problem (15.1). By examining the KKT system (15.10), we
can observe that the part of step dk in the range space of A(xk)T is completely determined by
A(xk)dk ¼  h(xk). The Lagrangian Hessian∇2xx Lðxk , λk Þ affects only the part of dk in the orthogonal
subspace, i.e., the null space of Ak. Therefore, it seems reasonable to consider the quasi-Newton
methods that find approximations to only that part of ∇2xx Lðxk , λk Þ which affects the component of dk
532 15 Sequential Quadratic Programming

in the null space of Ak. This is the idea of the reduced-Hessian quasi-Newton approximations, which
we are to be detailed in this section. For this, we consider the solution of the step equations (15.10) by
means of the null-space approach presented in Sect. 13.1.
Let us define the matrices Yk and Zk whose columns span the range space of A(xk)T and the null
space of A(xk), respectively. Using these matrices, dk can be written as
dk ¼ Y k dY þ Zk dZ , ð15:26Þ

where dY is the normal component and dZ is the tangential component of dk. By substituting dk from
(15.26) in (15.10) the following system is obtained:
½Aðxk ÞY k dY ¼ hðxk Þ, ð15:27aÞ
 T 2 
Z k ∇xx Lðxk , λk ÞZ k d Z ¼ Z Tk ∇2xx Lðxk , λk ÞY k dY  Z Tk ∇f ðxk Þ, ð15:27bÞ

to get the components dY and dZ, in this order. Observe that from the first line of (15.10) the Lagrange
multipliers λkþ1 can be computed by solving the linear system
 
½Aðxk ÞY k T λkþ1 ¼ Y Tk ∇f ðxk Þ þ ∇2xx Lðxk , λk Þd k : ð15:28Þ

In practical implementations, some simplifications are often considered. The idea is to avoid the
computation of ∇2xx Lðxk , λk Þ by introducing several approximations in the null-space approach.
One simplification in the null-space approach is to remove the cross term ZTk ∇2xx Lðxk , λk ÞY k d Y , in
(15.27b), thus obtaining the more simple system
 T 2 
Z k ∇xx Lðxk , λk ÞZk d Z ¼ Z Tk ∇f ðxk Þ: ð15:29Þ

Dropping the cross term is motivated when ZTk ∇2xx Lðxk , λk ÞZ k is replaced by a quasi-Newton
approximation because the normal component dY usually converges to zero faster than the tangential
component dZ, thereby making (15.29) a good approximation of (15.27b).
Another simplification consists in deleting the term involving dk from the right-hand side of
(15.28), thus decoupling the computations of dk and λkþ1. A motivation of this simplification is that
dk converges to zero as we approach the solution, whereas {∇f(xk)} does not. If we choose Yk ¼ A(xk)T
(when A(xk) has full row rank), we obtain
h i1
b
λkþ1 ¼ Aðxk ÞAðxk ÞT Aðxk Þ∇f ðxk Þ: ð15:30Þ

15.3 Merit Functions

Often, the SQP methods use a merit function to see whether a trial step should be accepted. In the line-
search methods, the merit function controls the size of the step. In the trust-region methods, it
determines whether the step is accepted or rejected and whether the trust-region radius should be
adjusted. A variety of the merit functions was proposed in literature, including the nonsmooth penalty
functions (l1 merit function) and the augmented Lagrangians. For the step computation and evaluation
of a merit function, the inequality constraints are converted to equalities by introducing the slack
variables. Therefore, in the following, we consider only the nonlinear equality-constrained problem
(15.1).
15.3 Merit Functions 533

The l1 merit function for (15.1) is defined by

Pðx, σ Þ ¼ f ðxÞ þ σ khðxÞk1 : ð15:31Þ

Another merit function was proposed by Wright (1976) and Schittkowski (1981, 1983) as
augmented Lagrangian
1
Lðx, σ, λÞ ¼ f ðxÞ  λT hðxÞ þ σhðxÞT hðxÞ, ð15:32Þ
2
where λ is an estimation of the Lagrange multipliers and σ > 0 is the penalty parameter.
In a line-search method, a step αkdk is accepted if the following sufficient decrease condition holds:

Pðxk þ αk d k , σ k Þ Pðxk , σ k Þ þ ηαk P0dk ðxk , σ k Þ, ð15:33Þ

where P0dk ðxk , σ k Þ is the directional derivative of P(.) in the direction dk and η is a parameter
sufficiently small, η 2 (0, 1). If dk is a descent direction, that is P0dk ðxk , σ k Þ < 0, then the reduction
(15.33) resembles the rule of Armijo used in the unconstrained optimization. The following theorem
shows that if σ is chosen sufficiently large, then the descent condition holds.

Theorem 15.3 Let dk and λkþ1 be generated by the system (15.10). Then, the directional derivative of
P(.) in the direction dk,satisfies

P0dk ðxk , σ k Þ ¼ ∇f ðxk ÞT dk  σ khðxk Þk1 : ð15:34Þ

Moreover, we have
 
P0dk ðxk , σ k Þ d Tk ∇2xx Lðxk , λk Þdk  σ  kλkþ1 k1 khðxk Þk1 : ð15:35Þ

Proof By applying Taylor’s theorem we obtain

Pðxk þ αd, σ Þ  Pðxk , σ Þ ¼ f ðxk þ αd Þ  f ðxk Þ þ σ khðxk þ αdÞk1  σ khðxk Þk1
α∇f ðxk ÞT d þ γα2 kdk2 þ σ khðxk Þ þ αAðxk Þdk1  σ khðxk Þk1 ,

where γ is a positive constant which bounds the second derivative terms. If d ¼ dk is given by (15.10),
then A(xk)dk ¼  h(xk), so that for α 1 it follows that
h i
Pðxk þ αdk , σ Þ  Pðxk , σ Þ α ∇f ðxk ÞT dk  σ khðxk Þk1 þ α2 γ kdk k2 :

Similarly, we obtain the following lower bound:

h i
Pðxk þ αdk , σ Þ  Pðxk , σ Þ  α ∇f ðxk ÞT dk  σ khðxk Þk1  α2 γ kdk k2 :

Taking limits, we conclude that

P0dk ðxk , σ k Þ ¼ ∇f ðxk ÞT dk  σ khðxk Þk1 ,

which proves (15.34).

534 15 Sequential Quadratic Programming

Now, since dk satisfies the first equation from the system (15.10), then

P0dk ðxk , σ k Þ ¼ d Tk ∇2xx Lðxk , λk Þdk þ dTk Aðxk ÞT λkþ1  σ khðxk Þk1 : ð15:36Þ

But from the second equation of the system (15.10), the term d Tk Aðxk ÞT λkþ1 from (15.36) can be
replaced by h(xk)Tλkþ1. Having in view the H€ older inequality (see Appendix A), we have
h(xk) λkþ1 kh(xk)k1kλkþ1k1, then, from (15.36), we obtain (15.35).
T
♦

A strategy for selecting σ, which is appropriate for both line-search and trust-region algorithms,
has in view the effect of the step on a model of the merit function. In (Nocedal, & Wright, 2006) a
quadratic model for function P(.) is defined as
μ
qσ ðdÞ ¼ f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx Lðxk , λk Þd þ σmðdÞ, ð15:37Þ
2
where m(d ) ¼ kh(xk) þ A(xk)dk1 and μ is a parameter to be defined. The strategy is as follows. After
computing a step dk, the penalty parameter σ is chosen large enough such that
qσ ð0Þ  qσ ðdk Þ  ρσ ½mð0Þ  mðdk Þ, ð15:38Þ

for some value of the parameter ρ 2 (0, 1). Now, from (15.37) and (15.8b) it follows that the
inequality (15.38) is satisfied for

∇f ðxk ÞT dk þ ðμ=2ÞdTk ∇2xx Lðxk , λk Þdk

σ : ð15:39Þ
ð 1  ρÞ khð x k Þ k1

If the value of σ from the previous iteration of the SQP method satisfies (15.39), then it is left
unchanged. Otherwise, σ is increased so that it satisfies this inequality with some margins. The
constant μ is used to handle the situations in which the Hessian ∇2xx Lðxk , λk Þ is not positive definite. A
simple strategy for μ is
(
1, if dTk ∇2xx Lðxk , λk Þd k > 0,
μ¼ ð15:40Þ
0, otherwise:

15.4 Second-Order Correction (Maratos Effect)

In the constrained nonlinear optimization, we practically face two purposes. One is to minimize the
objective function and another one is to satisfy the constraints. A way to balance these competing
goals is to use the merit functions or the filters. The idea is that a step generated by an algorithm will
be accepted only if it leads to a sufficient reduction in the merit function or if it is acceptable to the
filter. However, the following situation may appear. Suppose that an algorithm for solving nonlinear
optimization problems generates a step that reduces the objective function value, but increases the
violation of the constraints, a phenomenon known as the Maratos effect (1978). The following
example due to Powell (1986b) illustrates this situation.
15.4 Second-Order Correction (Maratos Effect) 535

Exampe 15.1 (Maratos effect) Let us consider the problem

 
min f ðx1 , x2 Þ ¼ 2 x21 þ x22  1  x1
subject to ð15:41Þ
hðx1 , x2 Þ ¼ x21 þ x22  1 ¼ 0:

The solution of the problem is x1 ¼ 1, x2 ¼ 0 and the Lagrange multiplier is λ ¼ 3/2. Moreover,
∇2xx Lðx , λ Þ ¼ I:
Consider an iterate xk as xk ¼ [cosθ sin θ]T. We immediately see that xk is feasible for any value of
θ. Suppose that our algorithm computes the following step:
" #
sin 2 θ
dk ¼ , ð15:42Þ
 sin θ cos θ

which determines the next point (iterate)

" #
cos θ þ sin 2 θ
xk þ dk ¼ :
ð1  cos θÞ sin θ

Now, we can see that

kxk þ d k  x k2 ¼ 2 sin 2 ðθ=2Þ and kxk  x k2 ¼ 2j sin ðθ=2Þj:

Hence,

kx k þ d k  x  k2 1
¼ ,
kxk  x k22 2

i.e., this step tends Q-quadratically to the solution of the problem. However, if we insist and compute,
we find

f ðxk þ dk Þ ¼ sin 2 θ  cos θ >  cos θ ¼ f ðxk Þ,

 
h xk þ dk ¼ sin 2 θ > hðxk Þ ¼ 0:

Therefore, even if the considered step determines a Q-quadratic convergence, the values of the
objective function increase, and the constraints are more and more unsatisfied. This behavior occurs
for any value of θ although the initial point is arbitrarily close to the solution.
For this example, let us consider the sequential quadratic programming algorithm in which dk is
determined as solution of the quadratic program (15.8), in which ∇2xx Lðxk , λk Þ is replaced by
∇2xx Lðx , λ Þ ¼ I: Hence, consider the current point xk ¼ ½ cos θ sin θ T : Since
   
4 cos θ  1 2 cos θ
f ðxk Þ ¼  cos θ, ∇f ðxk Þ ¼ , Að x k Þ T ¼ ,
4 sin θ 2 sin θ

the quadratic subproblem (15.8) is

536 15 Sequential Quadratic Programming

1 1
min d ð4 cos θ  1Þd1 þ 4d2 sin θ þ d21 þ d22
2 2
subject to
d2 þ d1 ctg θ ¼ 0:

By solving this subproblem, we get the direction

" #
sin 2 θ
dk ¼ ,
 sin θ cos θ

which coincides with (15.42), showing the failure of the sequential quadratic programming
algorithm. ♦

The Maratos effect shows that for many merit functions a superlinear convergent step may not be
accepted, thus preventing the algorithm from a fast convergence. There are three ways to overcome
the Maratos effect. The first is to relax the line-search conditions. The second is to use a second-order


correction step dbk which satisfies P xk þ dk þ dbk < Pðxk Þ: In this way, dk þ dbk is an acceptable step
and it is still a superlinear convergent step. The third is to use the smooth exact penalty functions as
merit functions.
As we said, the technique for avoiding the Maratos effect is to introduce the second-order
corrections, as shown by Coleman and Conn (1982a, 1982b), Fletcher (1982), Mayne and Polak
(1982), Fukushima (1986), etc.
Suppose that the SQP algorithm has computed a step dk as solution of the subproblem (15.12). If
this step yields an increase in the merit function P(xk, σ), then a possible cause is that the linear
approximation of the constraints given by (15.12b) and (15.12c) is not accurate enough. To overcome
this situation, a natural solution is to re-solve (15.12) with the linear terms ci(xk) þ ∇ ci(xk)Td replaced
by the quadratic approximations
1
ci ðxk Þ þ ∇ci ðxk ÞT d þ d T ∇2 ci ðxk Þd: ð15:43Þ
2
However, the resulting subproblem with quadratic constraints is too difficult to solve. Instead, the
following procedure is used. The constraints are evaluated in the new point xk þ dk and then we make
use of their approximation by Taylor’s theorem
1
ci ðxk þ d k Þ ffi ci ðxk Þ þ ∇ci ðxk ÞT dk þ dTk ∇2 ci ðxk Þdk : ð15:44Þ
2
Assuming that the second-order step d, which is not known, will not be too different from dk, then
the last term in (15.43) can be approximated as

dT ∇2 ci ðxk Þd ¼ dTk ∇2 ci ðxk Þdk : ð15:45Þ

Therefore, by making this substitution in (15.43) and by using the approximation (15.44), the
following second-order correction subproblem is obtained:
1
min d2ℝn f ðxk Þ þ ∇f ðxk ÞT d þ d T ∇2xx Lðxk , λk Þd ð15:46aÞ
2
15.5 The Line-Search SQP Algorithm 537

subject to

∇ci ðxk ÞT d þ ci ðxk þ dk Þ  ∇ci ðxk ÞT d k ¼ 0, i 2 E, ð15:46bÞ


∇ci ðxk ÞT d þ ci ðxk þ dk Þ  ∇ci ðxk ÞT d k  0, i 2 I: ð15:46cÞ

Let dbk be the solution of (15.46). Observe that the second-order correction step implies the
evaluation of the constraints in the point xk þ dk, i.e., ci(xk þ dk), i 2 E [ I, which can be a laborious
activity. Therefore, a strategy is to use this correction step only if the increase in the merit function is
concomitant with an increase in the constraint norm. When the step dk is generated by the SQP
algorithm based on (15.12), then, near a solution satisfying the second-order sufficient conditions, the
algorithm based on (15.46) takes either the full step dk or the corrected step dk þ dbk

15.5 The Line-Search SQP Algorithm

Taking into consideration the above developments, we can see that there is a large variety of line-
search SQP algorithms that differ in many respects, such as the way in which the Hessian approxi-
mation is computed (limited-memory BFGS that is suitable for large-scale problems), the step
acceptance mechanism using different techniques to avoid the Maratos effect, and the use of the
merit functions or of the filters. In the following, we present a practical line-search SQP algorithm for
solving the general nonlinear optimization problem (15.11). To keep the description as simple as
possible, the algorithm below does not include details to ensure the feasibility of the subproblem or
the second-order correction mechanisms. Instead, the algorithm is simply obtained by solving the
subproblem (15.12). Assume that the quadratic subproblem (15.12) is convex. Therefore, it can be
solved by means of the active-set methods for quadratic programming described in Chap. 13.

Algorithm 15.4 Line-search SQP algorithm

1. Choose an initial pair (x0, λ0), as well as the parameters η 2 (0, 1/2) and τ 2 (0, 1). Evaluate f(x0), ∇f(x0),
c(x0), and A(x0). If a quasi-Newton approximation of the Hessian of the Lagrange function is used, then
choose an initial n  n symmetric and positive definite approximation B0; otherwise, compute
∇2xx Lðx0 , λ0 Þ: Set k ¼ 0
2. Test a criterion for stopping the iterations
3. Compute the search direction dk as solution of the quadratic subproblem (15.12). Let bλ be the
corresponding Lagrange multipliers
4. Set d λ ¼ b λ  λk
5. Choose σ k to satisfy (15.39) with μ ¼ 1. Set αk ¼ 1
6. Inner while loop. Select η 2 (0, 1) and τα 2 (0, τ]. While
Pðxk þ αk dk , σ k Þ > Pðxk , σ k Þ þ ηαk P0dk ðxk , σ k Þ,
reset αk ¼ τααk; otherwise, go to step 7
7. Set xkþ1 ¼ xk þ αkdk and λkþ1 ¼ λk þ αkdλ
8. Evaluate f(xkþ1), ∇f(xkþ1), c(xkþ1), and A(xkþ1). If the case, evaluate ∇2xx Lðxkþ1 , λkþ1 Þ
9. If a quasi-Newton approximation method is used, then set
sk ¼ αkdk and yk ¼ ∇xL(xkþ1, λkþ1)  ∇xL(xk, λkþ1)
and obtain Bkþ1 by updating Bk using a quasi-Newton method (BFGS)
10. Set k ¼ k þ 1 and go to step 2 ♦
538 15 Sequential Quadratic Programming

An efficient implementation of Algorithm 15.4 is to use the warm start procedure. The working set
for each quadratic programming subproblem is initialized with the working set of the previous SQP
iteration. In step 6 of the algorithm, where the inner while loop is implemented instead of a merit
function, a filter may be used. In step 9, the limited memory BFGS updating approach can be used,
especially for large-scale problems.

15.6 The Trust-Region SQP Algorithm

The easiest way to formulate a trust-region SQP method is to add a trust-region constraint to the
subproblem (15.12), as
1
min d2ℝn f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx Lðxk , λk Þd ð15:47aÞ
2
subject to
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 E, ð15:47bÞ

∇ci ðxk ÞT d þ ci ðxk Þ  0, i 2 I: ð15:47cÞ

kdk Δk : ð15:47dÞ

Even if the constraints (15.47b) and (15.47c) are compatible, it is often possible that the subprob-
lem (15.47) may not always have a solution because of the trust-region constraint (15.47d). To solve
this possible conflict between the linear constraints (15.47b), (15.47c), and (15.47d), it is not
appropriate to increase Δk until the set of the steps d satisfying the linear constraints has intersected
the trust-region. A more appropriate viewpoint is not to exactly satisfy the linearized constraints at
every step; instead, we try to improve the feasibility of these constraints at each step and to exactly
satisfy them only if the trust-region constraint permits it. This point of view can be implemented in
three ways: by relaxation methods, penalty methods, and filter methods.

A Relaxation Method for the Equality-Constrained Optimization

Consider the equality-constrained nonlinear optimization problem (15.1). At the iterate xk, the SQP
step is computed by solving the subproblem
1
min d2ℝn f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx Lðxk , λk Þd ð15:48aÞ
2
subject to
Aðxk Þd þ hðxk Þ ¼ r k , ð15:48bÞ

kdk2 Δk , ð15:48cÞ

where rk 2 ℝm is the relaxation vector. The relaxation vector is selected as the smallest vector such
that (15.48b) and (15.48c) are consistent for some reduced value of the trust-region radius Δk. To
achieve this, first solve the subproblem

min y kAðxk Þy þ hðxk Þk22 ð15:49aÞ

subject to
kyk2 0:8Δk : ð15:49bÞ
15.6 The Trust-Region SQP Algorithm 539

Let yk be the solution of (15.49). Define the relaxation vector as

r k ¼ Aðxk Þyk þ hðxk Þ: ð15:50Þ

In the following, the step dk is computed by (15.48) and then the new iterate is defined as
xkþ1 ¼ xk þ dk. The new multiplier estimate λkþ1 is computed as
h i1
λkþ1 ¼ Aðxk ÞAðxk ÞT Aðxk Þ∇f ðxk Þ: ð15:51Þ

Now, for computing an approximate solution dk of (15.48), one method is the projected conjugate
gradient. The idea is to apply this algorithm to (15.48a, and 15.48b), monitoring the satisfaction of
the trust-region constraint (15.48c) and stopping the algorithm if the boundary of this region has been
reached or if a negative curvature has been detected. A merit function that takes into consideration
all these aspects is the nonsmooth l2 function P2(x, σ) ¼ f(x) þ σkh(x)k2. This merit function is
modeled by
1  
qσ ðd Þ ¼ f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx L xk , λk d þ σmðd Þ, ð15:52Þ
2
where
mðdÞ ¼ khðxk Þ þ Aðxk Þdk2 : ð15:53Þ

The penalty parameter σ is selected large enough so as to satisfy (15.38). The acceptability of the
step dk is monitored by the ratio

ared k P2 ðxk , σ Þ  P2 ðxk þ dk , σ Þ

ρk ¼ ¼ : ð15:54Þ
pred k qσ ð0Þ  qσ ðd k Þ

A description of the trust-region SQP algorithm for solving the equality-constrained nonlinear
optimization problem (15.1), min{f(x) : h(x) ¼ 0}, is as follows Omojokun (1989), Lalee, Nocedal,
and Plantenga (1998), Nocedal and Wright (2006).

Algorithm 15.5 Trust-region SQP algorithm

1. Choose the initial point x0 and the initial trust-region radius Δ0 > 0. Choose the constants ε > 0 and
η, γ 2 (0, 1). Set k ¼ 0
2. Evaluate f(xk), ∇f(xk), h(xk), and A(xk)
3. Compute the Lagrange multipliers: λkþ1 ¼ [A(xk)A(xk)T]1A(xk) ∇ f(xk)
4. If k∇f(xk)  A(xk)Tλkk1 < ε and kh(xk)k1 < ε, then stop. xk is the solution of the problem
5. Solve the subproblem (15.49) with respect to yk
6. Compute the relaxation parameter rk ¼ A(xk)yk þ h(xk)
7. Compute ∇2xx Lðxk , λk Þ or a quasi-Newton approximation of it
8. Compute dk as solution of the quadratic programming subproblem (15.48)
9. Choose σ, large enough to satisfy (15.39)
10. Compute the ratio ρk ¼ aredk/predk
11. If ρk > η, then set xkþ1 ¼ xk þ dk, choose Δkþ1 to satisfy Δkþ1  Δk and go to step 12.
If ρk η, then set xkþ1 ¼ xk, compute Δkþ1 ¼ γkdkk and go to step 12
12. Set k ¼ k þ 1 and continue with step 2 ♦
540 15 Sequential Quadratic Programming

We can simply note that a second-order correction can be introduced to avoid the Maratos effect.
The main computational cost of this algorithm is given by the projected conjugate gradient iteration
for solving the quadratic programming subproblem (15.48).

Sequential l1 Quadratic Programming for the Equality and Inequality Constraints

In this method, the linearized constraints (15.47b) and (15.47c) are placed into the objective of the
quadratic program as an l1 penalty term. Thus, the following subproblem is obtained:
1
min d qσ ðdÞ≜f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx Lðxk , λk Þd
2
X  Xh i
þσ  ci ðxk Þ þ ∇ci ðxk ÞT d þ σ ci ðxk Þ þ ∇ci ðxk ÞT d
i2E i2I
subject to ð15:55Þ
kd k1 Δ k ,

where σ is the penalty parameter and [y] ¼ max {0,  y}. When introducing the slack variables,
this problem is reformulated as
1 X X
min d,v,w,t f ðxk Þ þ ∇f ðxk ÞT d þ dT ∇2xx Lðxk , λk Þd þ σ ðvi þ wi Þ þ σ ti
2 i2E i2I
subject to ð15:56Þ
T
∇ci ðxk Þ d þ ci ðxk Þ ¼ vi  wi , i 2 E,
∇ci ðxk ÞT d þ ci ðxk Þ  ti , i 2 I,
v, w, t  0,
k d k1 Δ k :

The constraints of this problem are always consistent. Moreover, since the trust-region has been
defined by using the l1 norm, (15.56) is a smooth quadratic program which can be solved by means of
a quadratic programming algorithm.
To determine the acceptance of the step, the algorithm uses the l1 merit function
X X
Pðx, σ Þ ¼ f ðxÞ þ σ jci ðxÞj þ σ ½ci ðxÞ : ð15:57Þ
i2E i2I

Observe that qσ defined in (15.55) can be viewed as a model of the merit function (15.57) at xk, in
which we approximate each constraint function ci by its linearization and in which f is replaced by a
quadratic function whose curvature term includes information from both the objective and the
constraints.
After computing the step dk as solution of (15.56), from (15.54) the ratio ρk is computed by using
this time the above merit function and defining qσ by (15.55). The step is accepted or rejected
according to the standard trust-region rules, as described in Algorithm 15.5. Clearly, a second-order
correction step can be introduced in order to prevent the Maratos effect.
15.7 Sequential Linear-Quadratic Programming (SLQP) 541

15.7 Sequential Linear-Quadratic Programming (SLQP)

As we have already seen, the SQP methods presented above require the solution of a general quadratic
programming problem with equality and inequality constraints at each iteration. However, this is a
difficult task, especially when the second derivative information is incorporated into the algorithm.
This approach imposes a limit on the size of the problem that can be solved in practice by these
methods.
The sequential linear-quadratic programming method, which we are going to briefly describe, tries
to overcome this situation by computing the step in two phases. In the first one, a linear programming
problem (LP) is formed and solved to identify a working set W. The second one consists of an
equality-constrained quadratic programming phase in which the constraints in the working set W are
imposed as equalities. The total step is a combination of the steps obtained in the linear programming
and equality-constrained phases.
In the LP phase, the following problem is solved:

min d2ℝn f ðxk Þ þ ∇f ðxk ÞT d ð15:58aÞ

subject to
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 E, ð15:58bÞ

∇ci ðxk ÞT d þ ci ðxk Þ  0, i 2 I, ð15:58cÞ

kdk1 ΔLP
k : ð15:58dÞ

Observe that this linear programming problem differs from the standard SQP subproblem (15.47)
only in the fact that the second-order term dT ∇2xx Lðxk , λk Þd has been omitted and that a l1 norm has
been used to define the trust-region. Since the constraints of (15.58) may be inconsistent, instead of
(15.58), a l1 penalty reformulation is defined as

min d lσ ðdÞ≜f ðxk Þ þ ∇f ðxk ÞT d

X  Xh i
þσ ci ðxk Þ þ ∇ci ðxk ÞT d þ σ ci ðxk Þ þ ∇ci ðxk ÞT d
i2E i2I
subject to ð15:59Þ
kd k1 ΔLP
k :

Introducing the slack variables, (15.59) can be reformulated as a linear programming problem. Let
dLP be the solution of (15.59). From this solution, the following explicit estimate of the optimal
active-set can be obtained:
  n o n o
Ak d LP ¼ i 2 E : ci ðxk Þ þ ∇ci ðxk ÞT d LP ¼ 0 [ i 2 I : ci ðxk Þ þ ∇ci ðxk ÞT dLP ¼ 0 :

Similarly, the set Vk of the violated constrained is defined as

  n o n o
V k dLP ¼ i 2 E : ci ðxk Þ þ ∇ci ðxk ÞT dLP 6¼ 0 [ i 2 I : ci ðxk Þ þ ∇ci ðxk ÞT dLP < 0 :

The working set Wk is defined as some linearly independent subset of the active-set Ak(dLP). To
ensure the progress of the algorithm on the penalty function (15.57), the Cauchy step is defined as
542 15 Sequential Quadratic Programming

dC ¼ αLP d LP , ð15:60Þ

where αLP 2 [0, 1] is the stepsize that provides sufficient decrease in the model qσ defined in (15.55).
With the working set Wk an equality-constrained quadratic programming is solved, where the
constraints in Wk are considered as equalities, ignoring all the others. Thus, the following subproblem
is obtained:
!T
1 T 2 X
min d f ðxk Þ þ d ∇xx Lðxk , λk Þd þ ∇f ðxk Þ þ σ k γ i ∇ci ðxk Þ d
2 i2V k
subject to ð15:61Þ
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 E \ Wk,
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 I \ Wk ,
kd k2 Δ k ,

where the scalars γ i represent the algebraic sign of the i-th violated constraint. Observe that the trust-
region constraint in (15.61) is spherical and Δk is different from the trust-region radius ΔLP k used in
(15.59). Let dQ be the solution of (15.61). The total step dk of the SLQP method is computed as
 
d k ¼ d C þ αQ dQ  dC ,

where αQ 2 [0, 1] is the step size that approximately minimizes the model qσ defined in (15.55). Byrd,
Gould, Nocedal, and Waltz (2004a) argue that the choice of the radius ΔLP kþ1 for the LP phase is more
delicate, since it influences our guess of the optimal active-set. The value of ΔLP kþ1 should be set to be a
little larger than the total step dk, subject to some other restrictions. The Lagrange multiplier estimates
λk used in the evaluation of the Hessian ∇2xx Lðxk , λk Þ are computed as in (15.51) using the working set
Wk and modified so that λi  0, i 2 I. The advantage of using SLQP is that well-established algorithms
for solving large-scale versions of linear programming and equality quadratic subproblems are
readily available.

15.8 A SQP Algorithm for Large-Scale-Constrained Optimization (SNOPT)

The algorithm described in this section, implements the above discussions as well as some other
ideas. It was elaborated by Gill, Murray, and Saunders (2002, 2005) to solve nonlinear optimization
problems of the following form:
min x2ℝn f ðxÞ
subject 2to 3 ð15:62Þ
x
6 7
l 4 cðxÞ 5 u,
Ax

where f is the objective function (linear or nonlinear), c(x) is the vector of nonlinear constraints, A is
the matrix corresponding to the linear constraints, and l and u are bounds on variables and on
constraints. Suppose that the nonlinear functions are smooth, their first derivatives are available
and the Jacobian of the constraints is a sparse matrix.
SNOPT is the implementation of a particular SQP algorithm that exploits the sparsity in the
constraint Jacobian and maintains a limited-memory quasi-Newton approximation Bk to the Hessian
15.8 A SQP Algorithm for Large-Scale-Constrained Optimization (SNOPT) 543

of the Lagrange function. To update Bk in the presence of the negative curvature, a new method is
used. The quadratic programming subproblems are solved using an inertia-controlling reduced-
Hessian active-set method (SQOPT). Other features include the treatment of infeasible nonlinear
constraints using elastic programming, the use of a well-conditioned nonorthogonal basis for the null
space of the quadratic programming working set, early termination of the quadratic programming
subproblems, and the finite-difference estimates of the missing gradients. The method used by the
quadratic programming solver SQOPT is based on solving a sequence of linear systems involving the
reduced Hessian ZTBkZ, where Z is implicitly defined by using the sparse LU factorization.
In the following, we present a technique for treating the infeasible constraints and then we show
how SNOPT is working for solving the general nonlinear optimization problem with the inequality
constraints min{f(x) : c(x)  0}. Finally, some particularizations of these developments will be
applied to problems with linear and nonlinear constraints (including simple bounds) given in (15.62).

Infeasible Constraints
In SNOPT, the infeasible constraints are treated by means of the l1 penalty functions. At the very
beginning, the infeasible linear constraints are identified by solving a problem of the following form:

min x,v,w eT ðv þ wÞ
subject to  ð15:63Þ
x
l u, v  0, w  0,
Ax  v þ w

where e is a vector with all the components equal to one. The problem (15.63) is to minimize the
one-norm of the general linear constraint violations subject to the simple bounds. This problem is
often called elastic programming (Conn, 1976), (Bartels, 1980). If the linear constraints are infeasible
(v 6¼ 0 and w 6¼ 0), then SNOPT stops without computing the nonlinear functions. Otherwise, all the
subsequent iterates satisfy the linear constraints. SNOPT then solves (15.62) by using the quadratic
programming subproblem based on the linearization of the nonlinear constraints. If a quadratic
programming subproblem is infeasible or unbounded (or if the Lagrange multiplier estimates for
the nonlinear constraints become large), SNOPT enters the nonlinear elastic mode and solves the
problem

min x,v,w f ðxÞ þ γeT ðv þ wÞ

subject to ð15:64Þ
2 3
x
6 7
l 6 7
4 cðxÞ  v þ w 5 u,
Ax
v  0, w  0,

where f(x) þ γeT(v þ w) is known as the composite objective function and the penalty parameter
γ (γ  0) may take a finite sequence of increasing values. If (15.62) has a feasible solution and
γ is large enough, then the solutions of (15.62) and (15.64) are identical. On the other hand, if
(15.62) has no feasible solution, then (15.64) will tend to determine an infeasible point if γ is
sufficiently large.
544 15 Sequential Quadratic Programming

The SQP Iteration for the General Inequality Nonlinear Optimization

The SQP approach was popularized mainly by Biggs (1972), Han (1976), Powell (1977, 1978b), and
further developed by Schittkowski (1986) (NLPQL), Gill, Murray, Saunders and Wright (1986)
(NPSOL), and Spellucci (1998) (DONLP). Under mild conditions, these solvers typically find a
local optimum for the continuous general nonlinear optimization problems from arbitrary starting
points, and they require relatively few evaluations of the functions defining the problem and their
gradients compared, for example, to MINOS (Murtagh, & Saunders, 1978, 1995). For large-scale
problems with equality constraints, the method of Lalee, Nocedal, and Plantenga (1998) uses either
the exact Hessian of the Lagrangian or a limited-memory quasi-Newton approximation. The methods
by Biegler, Nocedal, and Schmid (1995) use the reduced Hessian method. They maintain a dense
approximation to the reduced Hessian based on quasi-Newton updates. For large problems with
general inequality constraints as in (15.49), the SQP methods were proposed by Eldersveld (1991)
(LSSQP), Fletcher, and Leyffer (1998, 2002) (filterSQP). In the following, we shall present the
SNOPT algorithm based on the works of Gill, Murray, and Saunders (2005). Consider the problem
min x f ðxÞ
subject to ð15:65Þ
cðxÞ  0,

where x 2 ℝn, f(x) : ℝn ! ℝ, c(x) : ℝn ! ℝm are twice continuously differentiable. Suppose that a
KKT point (x, λ) exists, where the first-order optimality conditions are satisfied

cðx Þ  0, λ  0, cðx ÞT λ ¼ 0, ∇cðx ÞT λ ¼ ∇f ðx Þ: ð15:66Þ

The sequential quadratic programming method involves major and minor iterations. The major
iterations generate a sequence of iterates (xk, λk) that converge to (x, λ). At each iteration, a
quadratic programming subproblem is solved to generate a search direction towards the next iteration
(xkþ1, λkþ1). Solving the quadratic subproblems is based on an iterative procedure involving the
so-called minor iterations.
Let xk and λk be estimates of x and λ. Gill, Murray, and Saunders (2005) use the following
modified Lagrange function associated to (15.65):

Lðx, xk , λk Þ ¼ f ðxÞ  λTk d L ðx, xk Þ, ð15:67Þ

which is defined in terms of constraint linearization and departure from linearity

cL ðx, xk Þ ¼ cðxk Þ þ ∇cðxk Þðx  xk Þ, ð15:68aÞ

d L ðx, xk Þ ¼ cðxÞ  cL ðx, xk Þ: ð15:68bÞ

This idea is taken from the MINOS algorithm. In fact, SNOPT is an improved extension of
MINOS (Saunders, 2015a, 2015b). The first and the second derivatives of the modified Lagrangian
are as follows:

∇Lðx, xk , λk Þ ¼ ∇f ðxÞ  ð∇cðxÞ  ∇cðxk ÞÞT λk , ð15:69aÞ

X
m
∇2 Lðx, xk , λk Þ ¼ ∇2 f ðxÞ  ðλk Þi ∇2 ci ðxÞ: ð15:69bÞ
i¼1

Observe that for x ¼ xk we have L(xk, xk, λk) ¼ f(xk) and ∇L(xk, xk, λk) ¼ ∇ f(xk).
15.8 A SQP Algorithm for Large-Scale-Constrained Optimization (SNOPT) 545

Now, let
1
Lq ðx, xk , λk Þ ¼ f ðxk Þ þ ∇f ðxk ÞT ðx  xk Þ þ ðx  xk ÞT ∇2 Lðxk , xk , λk Þðx  xk Þ
2
be the quadratic approximation of L at xk. If (xk, λk) ¼ (x, λ), then the optimality conditions for the
quadratic program
min x Lq ðx, xk , λk Þ
subject to ð15:70Þ
cL ðx, xk Þ  0

are identical to those of the original problem (15.65). This suggests that if Bk is an approximation to


∇2L at the point (xk, λk), then an improved estimate of the solution may be obtained from xbk , b
λk as
the solution of the following quadratic programming subproblem:
1
min x f ðxk Þ þ ∇f ðxk ÞT ðx  xk Þ þ ðx  xk ÞT Bk ðx  xk Þ
2
subject to ð15:71Þ
cðxk Þ þ ∇cðxk Þðx  xk Þ  0:

The optimality conditions of (15.71) are

cðxk Þ þ ∇cðxk Þðxbk  xk Þ ¼ b

sk , bλk  0, b
sk  0,
∇cðxk ÞTb bT
∇f ðxk Þ þ Bk ðxbk  xk Þ ¼ λk , λk b
sk ¼ 0,


sk is the vector of the slack variables for the linearized constraints. The triplet xbk , b
where b λk , b
sk is an
estimate of (x, λ, s), where the slack variables s satisfy c(x)  s ¼ 0, s  0. The vector b
sk is
needed in the line-search procedure.

The working set matrix The working set is important for both major and minor iterations and
represents the set of constraints that are binding at a solution (see Definition 11.12). If the subproblem
(15.71) has been solved, then the algorithm which solves this problem returns an independent set of
constraints that are active at the solution. This is the optimal working set for the subproblem (15.71).
The same constraint indices define the working set of (15.65) and of the next subproblem (15.71) at
the iteration k þ 1. The corresponding gradients of these constraints form the rows of the working set
matrix Wk, an nY  n full rank submatrix of the Jacobian ∇c(xk). Let Zk be an n  nZ full rank matrix
which is a basis of the null space of Wk. Then WkZk ¼ 0 and nZ ¼ n  nY. In SNOPT, Zk is defined by
the LU factorization of Wk. The matrix Zk is useful both in theoretical developments and in the
construction of the reduced Hessian Z Tk Bk Z k and of the reduced gradient ZTk ∇f ðxk Þ:

The merit function and the line-search Let (xk, λk, sk) be the current solution of (15.71). As soon as


we have a solution xbk , b
λk , b
sk for (15.71), a new estimation for the solution of (15.65) can be
computed by means of a linear search from (xk, λk, sk) towards the optimal solution of (15.71). The
purpose of the linear search is a sufficient decrease of a merit function based on the augmented
Lagrangian
546 15 Sequential Quadratic Programming

1X
m
Mσ ðx, λ, sÞ ¼ f ðxÞ  λT ðcðxÞ  sÞ þ σ ðc ðxÞ  si Þ2 , ð15:72Þ
2 i¼1 i i

where σ ¼ [σ 1, . . ., σ m]T 2 ℝm is the vector of the penalty parameters. For the stepsize α 2 (0, 1], let
v(α) be the following line:
2 3 2 3
xk xbk  xk
6 7 6 7
vðαÞ ¼ 4 λk 5 þ α4 bλk  λk 5
sk b
sk  sk

and φσ (α) ¼ Mσ (v(α)), which represents Mσ as a function by α. Observe that φ0σ ð0Þ is the directional
derivative of the merit function at the base point α ¼ 0 for a given σ. To determine the values of the
penalty parameters, Gill, Murray, and Saunders (2005) recommend the following procedure. Let σ 
be the solution of the following least-squares problem:

min kσ k2
subject to ð15:73Þ
1
φ0σ ð0Þ ¼  ðxbk  xk ÞT Bk ðxbk  xk Þ, σ  0:
2
The solution of this problem can be obtained analytically and it can be shown that for any σ  σ ,
φ0σ ð0Þ  12 ðxbk  xk ÞT Bk ðxbk  xk Þ, (Eldersveld, 1991), (Gill, Murray, Saunders, and Wright, 1992).

It is important to allow the penalty parameters to decrease during the early major iterations. The
reduction scheme involves a parameter Δσ  1. Let b σ be the value of the penalty parameter at the start
of the iterate k. Then define the new parameter σ i as the geometric mean of σ i and σ i as long as this
mean is sufficiently positive and not too close to σ i:

σ i ¼ max σ i , b
σi , ð15:74Þ

where
8  
< σi, if σ i < 4 σ i þ Δσ ,
σ i ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
b   
: σ i σ i þ Δσ , otherwise:

Initially, for k ¼ 0, Δσ ¼ 1. This value of Δσ can be increased by a factor of two in order to ensure a
positive value for σ i and not too close to σ i.
Now, considering σ ¼ σ in the merit the function (15.72), a safeguarded line-search is used to find
αkþ1 that reduces Mσ to give the next solution estimate v(αkþ1) ¼ (xkþ1, λkþ1, skþ1). With this, prior to
the solution of (15.71) at the iteration k þ 1, skþ1 is redefined to minimize the merit function as a
function of s, (Gill, Murray, Saunders, and Wright, 1986), (Eldersveld, 1991).

Bounding the constraint violation In the line-search, the following condition is enforced for some
vector b > 0:
cðxk þ αk ðxbk  xk ÞÞ  b:

SNOPT uses bi ¼ τv max {1, ci(x0)}, where τv is a constant, for example, τv ¼ 10. This defines a
region in which the objective function is expected to be defined and bounded below. If αk ¼ 0, then
15.8 A SQP Algorithm for Large-Scale-Constrained Optimization (SNOPT) 547

the objective is considered to be “unbounded” in the expanded region. In this case, the elastic program
is introduced.

The approximate Hessian As soon as the line-search has been finished, the change in x and the
gradient of the modified Lagrangian are defined as
sk ¼ xkþ1  xk , yk ¼ ∇Lðxkþ1 , xk , λÞ  ∇Lðxk , xk , λÞ,

for some vector λ. Then, an estimate of the curvature of the modified Lagrangian along sk is
incorporated in the BFGS quasi-Newton update

yk yTk Bk sk sTk Bk
Bkþ1 ¼ Bk þ  T : ð15:75Þ
yTk sk sk Bk sk

When Bk is positive definite, then Bkþ1 is positive definite if and only if the curvature yTk sk is
positive. From (15.67) observe that
yk ¼ ∇Lðxkþ1 , xk , λkþ1 Þ  ∇Lðxk , xk , λkþ1 Þ
¼ ∇f ðxkþ1 Þ  ∇f ðxk Þ  ð∇cðxkþ1 Þ  ∇cðxk ÞÞT λkþ1 :

Maintaining the positive definiteness of the Hessian Since the Hessian of the modified Lagrangian
need not be positive definite at a local minimizer, the approximate curvature yTk sk can be negative or
very small at points arbitrarily close to (x, λ). The curvature is considered not sufficiently positive if

yTk sk < ρk , ρk ¼ αk ð1  ηÞdTk Bk dk , ð15:76Þ

where η 2 (0, 1) is a given constant (0 < η < 1) and dk ¼ xbk  xk is the search direction defined by the
quadratic programming subproblem (15.71). In such cases, if the problem has nonlinear constraints,
two attempts are made to modify the update: the first is modifying sk and yk, the second is modifying
only yk. If neither modification provides a sufficiently positive approximate curvature, then no update
is made.

The first modification Define a new point zk and evaluate the nonlinear functions in order to obtain
the new values for sk ¼ xkþ1  zk and yk ¼ ∇ L(xkþ1, xk, λkþ1)  ∇ L(zk, xk, λkþ1). The point zk is
chosen as the first feasible point xk of (15.71). The search direction may be written as
d k ¼ ðxk  xk Þ þ ðxbk  xk Þ  dR þ dN :

Consider zk ¼ xk þ αkdR, thus obtaining sk ¼ αkdN and

yTk sk ¼ αk yTk d N ffi α2k dTN ∇2 Lðxk , xk , λk ÞdN ,

so that yTk sk approximates the curvature along dN. If Wk is the final working set of (15.71) at xk, it is
also the working set at xk , then WkdN ¼ 0. Therefore, yTk sk approximates the curvature for the reduced
Hessian, which must be positive semi-definite at the minimizer of (15.65).

The second modification If (xk, λk) is not close to (x, λ), then the modified approximate curvature
yTk sk may not be sufficiently positive. Hence, the second modification may be necessary as follows.
Choose Δyk so that (yk þ Δyk)Tsk ¼ ρk and redefine yk as yk þ Δyk. This approach was suggested by
Powell (1978b), who proposed to redefine yk as a linear combination of yk and Bksk.
548 15 Sequential Quadratic Programming

To get Δyk, consider the augmented modified Lagrangian

1
LA ðx, xk , λk Þ ¼ f ðxÞ  λTk dL ðx, xk Þ þ dL ðx, xk ÞT Ωd L ðx, xk Þ, ð15:77Þ
2
where Ω ¼ diag (ωi), ωi  0, i ¼ 1, . . ., m, is a matrix of the parameters to be determined. The
perturbation

Δyk ¼ ð∇cðxkþ1 Þ  ∇cðxk ÞÞT Ωd L ðxkþ1 , xk Þ

is equivalent to redefining the gradient difference as

yk ¼ ∇LA ðxkþ1 , xk , λkþ1 Þ  ∇LA ðxk , xk , λkþ1 Þ: ð15:78Þ

The elements ωi, i ¼ 1, . . ., m, are determined by solving the linearly constrained least-squares
problem

min ω kωk22
subject to ð15:79Þ
aT ω ¼ β, ω  0,

where β ¼ ρk  yTk sk and ai ¼ viwi, i ¼ 1, . . ., m, with v ¼ (∇c(xkþ1)  ∇ c(xk))Tsk and w ¼ dL(xkþ1, xk).
The idea is to choose the smallest ωi that increases yTk sk to ρk (see (15.76)). If there is no solution for
the problem (15.79) or if kωk is very large, no update of the Hessian is made. It is worth mentioning
that the second modification is not required in the neighborhood of a solution because ∇2LA converges
to ∇2L when x ! x and the first modification will have already been successful. The second
modification is related to updating an approximation of the Hessian of the Lagrangian suggested by
Han (1976) and Tapia (1974).

Convergence tests A point (x, λ) is a satisfactory solution for (15.65) if it satisfies the first-order
optimality conditions (15.66) within a certain tolerance. In SNOPT, two sufficiently small constants
τP and τD are used, which define the quantities τx ¼ τP(1 þ kxk1) and τλ ¼ τD(1 þ kλk1). The SQP
algorithm terminates if
 
ci ðxÞ  τx , λi  τλ , ci ðxÞλi τλ , d j  τλ , ð15:80Þ

where d ¼ ∇ f(x)  ∇ c(x)Tλ.

These conditions cannot be satisfied if (15.65) is infeasible. However, in these situations, the
problem (15.65) can be transformed in its elastic variant

min x,v f ðxÞ þ γeT v

subject to ð15:81Þ
cðxÞ þ v  0, v  0,

where γ takes the increasing values {γ l} up to some maximum value. The first-order optimality
conditions for (15.81) include
0 λi γ, ðci ðxÞ þ vi Þλi ¼ 0, vi ðγ  λi Þ ¼ 0: ð15:82Þ

The fact that kλk1 γ at the solution of (15.81) leads us to initiate the elastic mode if kλkk1
exceeds some value, let us say γ 1, or if (15.71) is infeasible. SNOPT uses
15.8 A SQP Algorithm for Large-Scale-Constrained Optimization (SNOPT) 549

γ 1 ≜γ 0 k∇f ðxk1 Þk1 , γ l ¼ γ 1 10lðl1Þ=2 , l ¼ 2, 3, . . . , ð15:83Þ

where γ 0 ¼ 104 and xk1 is the iterate at which γ is first needed.

The Quadratic Programming Solver SQOPT

We now return to our problem given by (15.62) considered by SNOPT. The form of this problem
requires solving (at every major iteration) a quadratic programming subproblem with f(x) replaced by
a convex quadratic function and with c(x) replaced by its current linearization in the current point
1
min x f ðxk Þ þ ∇f ðxk ÞT ðx  xk Þ þ ðx  xk ÞT Bk ðx  xk Þ
2
subject 2to 3 ð15:84Þ
x
6 7
l 4 cðxk Þ þ ∇cðxk Þðx  xk Þ 5 u:
Ax

In SNOPT, this subproblem is solved by the package SQOPT (Gill, Murray, and Saunders, 1997),
which employs a two-phase active-set algorithm and implicitly implements the elastic programming
when necessary. The Hessian Bk may be positive semi-definite and is defined by a routine for forming
the products Bkv for a given vector v.

At each minor iteration, when (15.84) is solved, the active-set method determines a search
direction d satisfying the KKT system
" #   
Bk W T d g
¼ q , ð15:85Þ
W 0 y 0

where W is the current working set matrix and gq is the gradient of the objective function from
(15.84). SQOPT implements several null-space methods, as described in (Gill, Murray, and Saunders,
2005).

The null-space approach One way to obtain a solution for the linear system (15.85) is to solve the
reduced Hessian system

ZT Bk Zd Z ¼ ZT gq , d ¼ Zd Z , ð15:86Þ

where Z is a basis for the null space of W. SQOPT maintains Z in the reduced-gradient form as in
MINOS. The idea is to use the sparse LU factors of a square nonsingular matrix B, called basis, whose
columns change as the working set Wchanges
2 3
  B1 S
B S N 6 7
W¼ P, Z ¼ PT 4 I 5 , ð15:87Þ
0 0 I
0

where P is a permutation matrix such that the selected basis B is nonsingular. The variables associated
with B and S are called basic and superbasic, respectively. The other variables are called nonbasic.
The number of the superbasic variables nZ is exactly the column dimension of S and Z. The products
Zv and ZTg are obtained by solving the corresponding systems with B or BT, respectively. If nZ is
550 15 Sequential Quadratic Programming

small, then SQOPT uses the dense Cholesky factorization ZTBkZ ¼ RTR. As the major iterations
converge, the quadratic programming subproblems require fewer changes to their working set, and
with a warm start they are eventually solved in one or two minor iterations. Hence, the work required
by SQOPT becomes dominant by the computation of the reduced Hessian ZTBkZ and its factor R. For
this reason, SQOPT can optionally maintain a quasi-Newton approximation ZTBkZ ffi RTR, as in
MINOS.

The conjugate gradient approach By construction, the Hessians Bk of (15.84) are positive definite or
positive semi-definite. Therefore, the conjugate gradient method is a natural tool for solving the very
large systems (15.85). SQOPT includes a conjugate gradient option for finding approximate solutions
to
 T 
Z Bk Z þ δ2 I dZ ¼ ZT gq , ð15:88Þ

where δ ffi 103 is a small regularization parameter to allow for the singular ZTBkZ. When Z has many
columns, then it is possible for the conjugate gradient methods to require many iterations to get a
useful approximation to dZ. The conjugate gradient methods require some sort of preconditioning,
which is much dependent on the system to be solved. However, in SQOPT no preconditioner is used.
The explanation is that, when looking at (15.88), we see that both Bk and ZTZ have similar structures:
“a diagonal matrix plus a matrix of small rank.”

The initial point To use a good starting point x0, the algorithm SQOPT is applied to one of the
proximal-point problems for initialization
   
x
min x kx  x0 k1 : l u ð15:89Þ
Ax

or
   
x
min x kx  x0 k22 : l u ð15:90Þ
Ax

where l and u are the corresponding bounds for the linear constraints and the simple bounds from
(15.84). x and x0 correspond to the nonlinear variables in x and x0. In practice, (15.89) is preferred
because it is linear and can use SQOPT’s implicit elastic bounds. The solution of (15.89) (or (15.90))
defines a new starting point x0 for the SQP iteration.
SNOPT, as described by Gill, Murray, and Saunders (2005), is a complex algorithm with plenty of
details which are not presented here. They refer to the following: null-space computation, choices for
multipliers, large-scale Hessians in case of problems with few nonlinear variables, dense Hessians,
limited-memory procedure to the update Hessians, elastic bounds, inertia control, unbounded qua-
dratic programming subproblems, basis handling in SQOPT, threshold pivoting in the LU factoriza-
tion, basis repair in the singular case, basis repair in the rectangular case, undefined functions, early
termination of quadratic programming subproblems, and linearly constrained problems. Suppose that
a starting point (x0, λ0) is available and that the reduced-Hessian quadratic programming solver
SQOPT is being used. The main steps of SNOPT are as follows.
15.8 A SQP Algorithm for Large-Scale-Constrained Optimization (SNOPT) 551

Algorithm 15.6 SNOPT—Gill, Murray, and Saunders

1. Apply the quadratic programming solver to the problem (15.89) (or (15.90)) to find a point close to x0
satisfying the linear constraints and the simple bounds on variables. If (15.89) is infeasible, then the
problem (15.62) is infeasible. Otherwise, a working set matrix W0 is obtained. Set k ¼ 0 and evaluate the
functions and the gradients at x0. Initialize the penalty parameters σ i ¼ 0, i ¼ 1, . . ., m
2. Factorize Wk
3. Define sk to minimize the merit function (15.72) as a function of the slacks s
4. Determine xk as a feasible solution for (15.84). This is an intermediate point for the SQOPT solver,
which also provides a working set matrix W k as well as its null space Z k : If no feasible point exists,
initiate the elastic mode and restart the SQOPT solver
5. T
Compute the reduced Hessian Z k Bk Z k and compute its Cholesky factor


6. Using SQOPT, continue solving the quadratic programming subproblem (15.84) to find xbk , bλk :
Observe that in step 4, only a feasible solution was obtained for (15.84). SQOPT provides a working set
b k as well as its null-space matrix Zbk
W
 
 
If the elastic mode has not been initiated, but bλk  is “large,” then enter the elastic mode and restart the
1
SQOPT solver
If the subproblem (15.84) is unbounded and xk satisfies the nonlinear constraints, the problem
(15.62) is unbounded ( fis unbounded below in the feasible region). Otherwise, if (15.84) is unbounded,
go to step 8
7. If the solution (xk, λk) satisfies the convergence tests for (15.62) analogously to (15.80), then this is the
optimal solution of the problem; stop
If the convergence tests similar to (15.80) are satisfied for the subproblem (15.81), then go to step
8. Otherwise, go to step 9
8. If the elastic mode of (15.84) has not been initiated, then consider the elastic variant of the problem and
repeat step 6. Otherwise, if γ has not reached its maximum value, increase γ and repeat step
6. Otherwise, declare the problem (15.62) infeasible
9. Update the penalty parameters as in (15.74)
10. Find a stepsize αk that gives a sufficient

reduction
 in the merit function (15.72). Set xkþ1 ¼
xk þ αk ðxbk  xk Þ and λkþ1 ¼ λk þ αk bλk  λk : Evaluate the functions and the gradients at the new
point xkþ1
11. Compute sk ¼ xkþ1  xk, yk ¼ ∇ L(xkþ1, xk, λkþ1)  ∇ L(xk, xk, λkþ1). If yTk sk < ρk , where ρk is
computed as in (15.76), then re-compute sk and yk with xk redefined as xk þ αk ðxk  xk Þ: (This requires
an extra evaluation of the derivatives.) If necessary, increase yTk sk (if possible) by increasing yk with Δyk
and by redefining the difference of the gradients of the augmented Lagrangian (see (15.77))
12. If yTk sk  ρk , then apply the BFGS formula to update Bk by using the pair (Bksk, yk)
13. Using W b k , redefine Wk, set k ¼ k þ 1 and go to step 2 ♦

For both linearly and nonlinearly constrained problems, SNOPT applies a sparse sequential
quadratic programming method (Gill, Murray, and Saunders, 2005), using limited-memory quasi-
Newton approximations to the Hessian of the Lagrangian. The merit function for the stepsize
computation is an augmented Lagrangian, as in the dense SQP solver NPSOL (Gill, Murray,
Saunders, & Wright, 1986). The computational effort is in steps 2 and 5, where the matrix Wk is
T
factorized, and the reduced Hessian Z k Bk Zk is formed, and Cholesky factorized.

Numerical study—SNOPT: Solving Applications from the LACOP Collection

Table 15.1 contains the performances of SNOPT for solving 12 applications of nonlinear optimiza-
tion from the LACOP collection presented in Appendix C.
552 15 Sequential Quadratic Programming

Table 15.1 Performances of SNOPT for solving 12 applications from the LACOP collection. Small-scale nonlinear
optimization applications

n me mc #itM #itm #nf #nr #s cpu vfo

ELCH 10 3 0 19 49 54 0 7 0.01 −47.761090
ALKI 10 3 8 16 47 28 28 1 0.01 −1768.8070
PREC 8 0 6 53 119 202 202 4 0.01 3.951163
PPSE 9 6 0 9 17 20 20 1 0.01 5055.01167
MSP3 13 0 15 71 141 367 367 1 0.02 97.5875581
MSP5 16 0 21 54 152 106 106 2 0.02 174.910868
POOL 34 20 0 5 36 8 8 4 0.01 2569.8000
TRAFO 6 0 2 30 38 36 36 4 0.01 135.075962
LATHE 10 1 14 33 86 55 55 0 0.01 −4430.0882
DES 150 50 0 104 204 130 130 100 0.16 1055.18231
CSTC 303 200 0 10 111 15 15 100 0.03 3.48007467
DIFF 396 324 0 0 60 0 0 0 0.01 0.0

Table 15.2 Performances of SNOPT for solving 6 applications from the LACOP collection. Large-scale nonlinear
optimization applications

n me mc #itM #itm #nf #s cpu vfo

HANG 1002 501 0 59 1032 93 499 3.37 5.0685783
2002 1001 0 60 2028 125 999 18.80 5.0685126
FLOW 1163 735 0 0 457 2 0 0.01 0.0
FLOWO 1556 1005 0 0 329 2 0 0.01 0.0
POL 4004 3000 0 57 1562 114 0 0.92 13.480213
6004 4500 0 63 2272 128 0 1.86 13.469429
8004 6000 0 63 2992 125 0 3.06 13.638381
10004 7500 0 73 3392 145 0 4.51 13.210731
CAT 3003 2000 0 11 272 1183 0 2.28 −0.048055
6003 4000 0 18 536 2430 0 8.75 −0.048052
9003 6000 0 23 796 3703 0 20.47 −0.048047
CONT 2505 2000 0 3 696 5 0 0.05 1.01323
5005 4000 0 3 1394 5 2 0.11 1.00592
7505 6000 0 3 2094 5 0 0.27 1.00456
10005 8000 0 3 1424 5 0 0.44 1.00407

In Tables 15.1 and 15.2, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the
number of inequality constraints, #itM ¼ the number of major iterations, #itm ¼ the number of minor iterations, #nf ¼
the number of evaluations of the objective function, #nr ¼ the number of evaluations of the constraints, #s ¼ the number
of superbasic variables, cpu ¼ the CPU computing time to solve the problem (seconds), and vfo ¼ the value of the
objective function at the optimal point.

Table 15.2 presents the performances of SNOPT for solving 6 large-scale nonlinear optimization
applications from the LACOP collection of different dimensions.
SNOPT uses many concepts defined in MINOS. A comparison between these two algorithms
shows the importance of the sequential quadratic programming implemented in SNOPT versus the
minimization of the augmented Lagrangian subject to the linearization of the constraints used in
MINOS. For all these 15 large-scale numerical experiments shown in Tables 14.12 (MINOS) and
15.2 (SNOPT), respectively, Table 15.3 shows the total number of major iterations (#itMt), the total
15.9 A SQP Algorithm with Successive Error Restoration (NLPQLP) 553

Table 15.3 Comparison: MINOS versus SNOPT for solving 15 large-scale applications from the LACOP collection.
Large-scale nonlinear optimization applications

#itMt #itmt cput

MINOS 548 27271 143.80
SNOPT 439 21276 64.91

number of minor iterations (#itmt), and the total CPU computing time (cput) in seconds, to get a
solution for all these applications.

15.9 A SQP Algorithm with Successive Error Restoration (NLPQLP)

Let us consider the general nonlinear optimization problem with equality and inequality constraints

min f ðxÞ
subject to ð15:91Þ
ci ðxÞ ¼ 0, i ¼ 1, . . . , me ,
ci ðxÞ  0, i ¼ me þ 1, . . . , m,

where it is assumed that the functions f : ℝn ! ℝ, and ci : ℝn ! ℝ, i ¼ 1, . . ., m, are twice

continuously differentiable. Also, assume that
X ¼ fx 2 ℝn : ci ðxÞ ¼ 0, i ¼ 1, . . . , me , ci ðxÞ  0, i ¼ me þ 1, . . . , mg 6¼ ∅:

For solving this problem, we shall present the sequential quadratic programming with the
successive error restoration algorithm NLPQLP, elaborated by Schittkowski (1986, 2002, 2005,
2009, and 2010). Since the sequential quadratic programming is sensitive to the accuracy by which
the partial derivatives are provided, NLPQLP is stabilized by a non-monotone line-search and by
internal and external restarts in case of errors when computing the search direction due to inaccurate
derivatives.

Search Direction
The basic idea is to formulate and solve a sequence of quadratic programming subproblems obtained
by the linearization of the constraints and by approximating the Lagrangian function
X
m
Lðx, uÞ ¼ f ðxÞ  ui c i ð x Þ ð15:92Þ
i¼1

quadratically, where x 2 ℝn is the vector of the primal variables and u ¼ [u1, . . ., um]T 2 ℝm is the
vector of the dual variables or the vector of the Lagrange multipliers.

Let vk 2 ℝm be an approximation of the Lagrange multipliers. To formulate the quadratic

programming subproblem at the current iterate xk, the gradients ∇f(xk) and ∇ci(xk), i ¼ 1, . . ., m, as
well as an approximation Bk 2 ℝnn of the Hessian to the Lagrangian are computed. Then the
quadratic programming subproblem is formed and solved
554 15 Sequential Quadratic Programming

1
min ∇f ðxk ÞT d þ dT Bk d
2
subject to ð15:93Þ
T
∇ci ðxk Þ d þ ci ðxk Þ ¼ 0, i ¼ 1, . . . , me ,
∇ci ðxk ÞT d þ ci ðxk Þ  0, i ¼ me þ 1, . . . , m:

Let dk be the optimal solution and uk the corresponding Lagrange multipliers of (15.93). A new
iteration is computed as
     
xkþ1 xk dk
¼ þ αk , ð15:94Þ
vkþ1 vk uk  v k

where αk 2 (0, 1] is the stepsize.

Although we can guarantee that the matrix Bk is positive definite, it is quite possible for (15.93) not
to be solvable because its constraints are inconsistent. To remedy this situation, Schittkowski (1986)
suggested introducing an additional variable δ 2 ℝ, thus transforming (15.93) into a modified
quadratic subproblem with (n þ 1) variables, with consistent constraints.
Another numerical difficulty with (15.93) is that the gradients of all the constraints must be
re-evaluated at each iteration. But if xk is close to the solution, then the computation of the gradients
of the inactive constraints is redundant. To avoid this redundant computational effort, define the set
I ≜ {i : me < i m}, and, given a constant ε > 0 small enough, define the sets
n o
ðk Þ ðiÞ ðk Þ ðkÞ
I 1 ¼ i 2 I : ci ðxk Þ ε or vk > 0 , I 2 ¼ I∖I 1 , ð15:95Þ
h iT
ð1Þ ðmÞ
where vk ¼ vk , . . . , vk : With these developments, at each iteration the following modified
quadratic programming subproblem is solved:
1 1
min d2ℝn , δ2½0,1 ∇f ðxk ÞT d þ dT Bk d þ ρk δ2
2 2
subject to ð15:96Þ
∇ci ðxk ÞT d þ ð1  δÞci ðxk Þ ¼ 0, i ¼ 1, . . . , me ,
T ðk Þ
∇ci ðxk Þ d þ ð1  δÞci ðxk Þ  0, i 2 I1 ,
 T ðk Þ
∇ci xκðk,iÞ d þ ci ðxk Þ  0, i 2 I2 :

In (15.96), κ(k, i) k represents the previous iterations where the corresponding gradient has been
ð0Þ ð0Þ
last evaluated. We start with I 1 ¼ I and I 2 ¼ ∅: At the subsequent iterations, only the constraint
ðk Þ
gradients belonging to the active-set I 1 are reevaluated. The remaining rows of the Jacobian matrix
remain filled with the previously computed gradients. In (15.96), ρk > 0 is a parameter sufficiently
small.
Let (dk, δk, uk) be the solution of (15.96), where uk is the vector of the multipliers and δk is the
additional variable introduced to prevent the inconsistency of the linear constraints. If the linear
independency of the constraints qualification (LICQ) is satisfied (see Remark 11.2), then δk < 1. The
matrix Bk is a positive definite approximation of the Hessian to the Lagrange function. For the global
convergence analysis, any choice of Bk is appropriate as long as its eigenvalues are bounded away
from zero. However, to guarantee a superlinear convergence rate, Bk is updated by the BFGS quasi-
Newton method modified by a stabilization procedure to ensure the positive definiteness. The penalty
parameter ρk is required to reduce the perturbation of the search direction by the additional variable δ
as much as possible. A suitable choice is given in Schittkowski (1983).
15.9 A SQP Algorithm with Successive Error Restoration (NLPQLP) 555

Stepsize Computation
The stepsize parameter αk is required in (15.94) in order to enforce the global convergence of the SQP
method when the initial point x0 is arbitrarily selected and v0 ¼ 0, B0 ¼ I. The stepsize αk should
satisfy at least a sufficient decrease condition of a merit function Φσ (α) given by
   
x d
Φσ ðαÞ≜Pσ þα , ð15:97Þ
v uv

where Pσ (x, v) is a suitable penalty function, for example, the augmented Lagrange function
X
1

1 X v2i
Pσ ðx, vÞ ¼ f ðxÞ  vi ci ðxÞ  σ i ci ðxÞ2  , ð15:98Þ
i2J
2 2 i2K σ i

where the sets J and K are defined as (Schittkowski, 1983)

J ¼ f1, . . . , me g [ fi : me < i m, ci ðxÞ vi =σ i g and K ¼ f1, . . . , mg∖J:

The objective function is penalized as soon as an iterate has left the feasible region. The penalty
parameters σ i, i ¼ 1, . . ., m, control the degree of the constraint violation. To guarantee a descent
direction of the merit function, σ i are chosen such that
 
0 T dk
Φσ k ð0Þ ¼ ∇Pσk ðxk , vk Þ < 0: ð15:99Þ
uk  v k

Usually, the stepsize αk is computed by means of the Armijo rule, i.e., a sufficient descent
condition of the merit function (15.98) which guarantees the convergence to a stationary point.
However, to take into consideration the curvature of the merit function, we need some kind of
compromise between a polynomial interpolation, typically a quadratic one, and a reduction of the
stepsize by a given factor until a stopping criterion has been reached (Schittkowski, 2010). The
determination of the stepsize αk is very important in the economy of the NLPQLP algorithm. The
purpose is not to have too many function calls in the procedure for the αk computation. Moreover, the
behavior of the merit function becomes irregular in the case of the constrained optimization because
of the very steep slopes at the border caused by the large penalty terms. The implementation of a
procedure for the αk computation is more complex if the linear constraints and the simple bounds on
variables are to be satisfied during the line-search. Since Φσ (0), Φ0σ ð0Þ and Φσ (αi) are known (αi is the
current value of the stepsize), then a procedure based on the quadratic interpolation for the αkþ1
computation is as in the following fragment of the algorithm:

Algorithm 15.7 Linear search in NLPQLP

1. Choose the parameters 0 < β < 1 and 0 < μ < 1/2. Consider σ ¼ σ k. Set α0 ¼ 1 and i ¼ 0
2. If Φσ ðαi Þ < Φσ ð0Þ þ μαi Φ0σ ð0Þ, then stop; otherwise, go to step 3
3. α2i Φ0σ ð0Þ
Compute: αi ¼ 12 αi Φ0σ ð0ÞΦσ ðαi ÞþΦσ ð0Þ
4. Let αi ¼ max fβαi , αi g: Set i ¼ i þ 1 and go to step 2 ♦

The algorithm is well known, see for example (Powell, 1978b). αi is the minimizer of the quadratic
interpolation polynomial. For checking the termination, the Armijo descent property is used. The
computation of αi in step 4 of the algorithm as αi ¼ max fβαi , αi g is required to avoid the irregular
556 15 Sequential Quadratic Programming

values, since the minimizer of the quadratic interpolation polynomial can be outside the feasible
region [0, 1]. Schittkowski mentions that Algorithm 15.7 is equipped with some safeguards to
prevent, for example, the violation of the bounds. In Algorithm 15.7, it is supposed that Φσ (1) is
known before calling it. Moreover, the algorithm stops if sufficient descent is not observed after a
certain number of iterations. If the tested stepsize falls below the machine precision, then the merit
function cannot decrease further on. Algorithm 15.7 could break down because of too many
iterations. In this case, Schittkowski recommends proceeding from a descent direction of the merit
function when Φ0σ ð0Þ is extremely small. In order to avoid the interruption of the whole iteration
process of NLPQLP, another criterion is used instead of the criterion in step 2 of Algorithm 15.7. The
idea is to repeat the line-search with another stopping criterion. Instead of testing (15.98), the stepsize
αk is accepted as soon as the inequality

Φσk ðαk Þ max ktðkÞ j k Φσj ð0Þ þ μαk Φ0σ k ð0Þ, ð15:100Þ

has been satisfied, where tðkÞ ¼ min fk, tg, and t is a given parameter. Thus, an increase of the
reference value Φσ j is allowed, i.e., an increase of the merit function value. To implement (15.100), it
is necessary to store the previous values of the merit function from the first t(k) iterations. The
nonmonotone search goes back to Grippo, Lampariello, and Lucidi (1986) and was extended to the
trust-region methods for the constrained optimization by Bonnans, Panier, Tits, and Zhou (1992),
Deng, Xiao, and Zhou (1993), Grippo, Lampariello, and Lucidi (1989, 1991), Raydan (1993, 1997),
Panier and Tits (1991), Toint (1996, 1997).

Updating the Matrix Bk

To avoid the calculation of the second derivatives and to obtain a final superlinear convergence rate,
the standard approach used in NLPQLP is to update Bk by the quasi-Newton BFGS formula

qk qTk Bk sk sTk Bk
Bkþ1 ¼ Bk þ  T , ð15:101Þ
sTk qk s k Bk s k

where qk ¼ ∇xL(xkþ1, uk)  ∇xL(xk, uk) and sk ¼ xkþ1  xk. The algorithm is equipped with special
safeguards guarantee that ensures the curvature condition sTk qk > 0: In this case, if B0 is positive
definite, then all the matrices Bk generated by (15.101) remain positive definite along the iterations.
The algorithm contains a scaling and a restart procedure to replace an actual Bk by γ kI before
performing the update (15.101), where γ k ¼ sTk qk =sTk sk , (Liu, & Nocedal, 1989).
The main steps of the NLPQLP algorithm are as follows.

Algorithm 15.8 NLPQLP—Schittkowski

1. Choose an initial point x0, an initial approximation to the Hessian of the Lagrange function B0 and the
tolerances ε1 and ε2. Set k ¼ 0
2. Compute f(xk), c(xk), ∇f(xk), and ∇c(xk)
3. Compute dk as solution of the quadratic programming subproblem (15.96)
4. Compute the stepsize by means of Algorithm 15.7
5. If the KKT optimality conditions are satisfied with the tolerance ε1 and the norm of the gradient of the
Lagrange function is smaller than the tolerance ε2, then stop; otherwise, go to step 6
6. Using (15.101), update Bk
7. Compute a new Cholesky factorization of Bk
8. Set k ¼ k þ 1 and continue with step 2 ♦
15.10 Active-Set Sequential Linear-Quadratic Programming (KNITRO/ACTIVE) 557

Table 15.4 Performances of NLPQLP for solving 8 applications from the LACOP collection. Small-scale nonlinear
optimization applications

n me mc #iter #nf #ng #nq KKT cpu vfo

ELCH 10 3 0 21 30 21 21 0.11e-8 0.03 −47.76109
ALKI 10 3 8 53 69 53 53 0.13e-11 0.03 −1768.807
PREC 8 0 6 19 20 19 19 0.36e-8 0.02 3.9511635
PPSE 9 6 0 9 10 9 9 0.21e-8 0.03 5055.0118
MSP3 13 0 15 146 264 146 146 0.62e-8 0.04 79.631357
POOL 34 20 0 22 22 22 22 0.64e-13 0.07 2785.800
TRAFO 6 0 2 19 19 19 19 0.19e-10 0.03 135.0759
LATHE 10 1 14 19 19 19 19 0.27e-9 0.03 −4430.087

In Table 15.4, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the number of
inequality constraints, #iter ¼ the number of iterations to get a local optimal solution, #nf ¼ the number of evaluations
of the functions of the problem, #ng ¼ the number of the evaluations of the gradients of the functions, #nq ¼ the number
of the quadratic programming subproblems solved to get a local optimal solution of the problem, KKT ¼ the norm of the
KKT conditions, cpu ¼ the CPU computing time for solving the problem (seconds), and vfo ¼ the value of the objective
function at the solution.

Table 15.5 Performances of NLPQLP for solving the ETA-MACRO application

n me mc #iter #nf #ng #nq KKT cpu vfo

60 1 41 31 31 31 31 0.74e-8 0.17 5.3700593
120 1 81 103 103 103 13 0.83e-8 3.69 7.9532086
300 1 201 823 823 823 823 0.86e-8 454.90 9.3301684

Numerical Study—NLPQLP: Solving Applications from the LACOP Collection

Table 15.4 presents the performances of NLPQLP for solving 8 nonlinear optimization applications
from the LACOP collection, described in Appendix C.

Application P1: ETA-MACRO

ETA-MACRO simulates a market economy through a dynamic nonlinear optimization process. To
describe the production relationships within this economy, two dynamic submodels are incorporated:
(a) ETA, a process analysis for the energy technology assessment and (b) a macroeconomic growth
model providing for the substitution between capital, labor, and energy inputs. The application is
described in (Manne, 1977) (see also (Bergman, 1988), (Murtagh, & Saunders, 1995, pp. 98)).
Table 15.5 contains the performances of NLPQLP for solving the application ETA-MACRO for a
different number n of variables.

15.10 Active-Set Sequential Linear-Quadratic Programming

(KNITRO/ACTIVE)

KNITRO represents one of the most elaborated algorithms (and Fortran packages) for solving general
large-scale nonlinear optimization problems (Byrd, Gould, Nocedal, & Waltz, 2004a). This is
characterized by great flexibility and robustness, integrating two very powerful and complementary
algorithmic approaches for nonlinear optimization: the active-set sequential linear-quadratic
approach and the interior-point approach. KNITRO includes a number of much-studied algorithms
for linear algebra, very carefully implemented in computing programs, able to solve a large variety of
558 15 Sequential Quadratic Programming

nonlinear optimization problems like special cases of unconstrained optimization, systems of nonlin-
ear equations, least square problems, and linear and nonlinear programming problems.
KNITRO has two algorithmic options. The first one, known as KNITRO/ACTIVE, uses a new
active-set method based on the sequential linear-quadratic programming (SLQP) and the projected
conjugate gradient iteration. The second, known as KNITRO/INTERIOR, uses the interior-point
methods in two implementations: KNITRO/INTERIOR-CG, in which the algorithmic step is
computed by means of an iterative conjugate gradient method, and KNITRO/INTERIOR-DIRECT,
in which the step is (usually) computed via a direct factorization of the corresponding linear systems.
These two approaches, KNITRO/ACTIVE and KNITRO/INTERIOR, communicate by the so-called
crossover technique. This crossover procedure, used for the first time by Megiddo (1989) in linear
programming, is implemented internally by switching to the ACTIVE algorithm after the
INTERIOR-DIRECT or the INTERIOR-CG algorithm has solved the problem to a requested
tolerance.
The difference between these two approaches is as follows. The active-set sequential linear-
quadratic programming algorithm is similar in nature to a sequential quadratic programming method,
but it uses linear programming subproblems to estimate the active-set at each iteration. This active-set
approach may be useful when a good initial point can be provided. On the other hand, in the interior-
point methods, also known as barrier methods, the nonlinear programming problem is replaced by a
sequence of barrier subproblems controlled by a barrier parameter. The algorithm uses the trust-
region and a merit function to promote convergence. The algorithm performs one or more minimiza-
tion steps on each barrier problem, then decreases the barrier parameter and repeats the process until
the problem has been solved to the desired accuracy.
In this chapter, we discuss only KNITRO/ACTIVE. This algorithm is described in (Byrd, Gould,
Nocedal, & Waltz, 2004a), and its global convergence is presented in (Byrd, Gould, Nocedal, &
Waltz, 2002).
KNITRO/INTERIOR will be described in Chap. 17, where the crossover technique is also
presented.
Let us consider the general nonlinear optimization problem
min f ðxÞ
subject to ð15:102Þ
cE ðxÞ ¼ 0,
cI ðxÞ  0,

where f : ℝn ! ℝ, cE : ℝn ! ℝl and cI : ℝn ! ℝm are twice continuously differentiable functions.

Define: E ≜ {1, . . ., l}and I ≜ {1, . . ., m}.
The active-set method implemented in KNITRO/ACTIVE does not follow a SQP approach
because the cost of solving linear-constrained quadratic programming subproblems generally
imposes limitations on the size of problems that can be solved in practice. In addition, the
incorporation of the second derivative information in the SQP methods has proved to be difficult in
practice. Instead, in KNITRO/ACTIVE a sequential linear-quadratic programming (SLQP) method is
used (see Sect. 15.7). SLQP computes a step in two phases. Firstly, a linear programming problem
(LP) is solved in order to identify a working set W. This is followed by an equality-constrained
quadratic programming phase, in which the constraints in the working set W are imposed as
equalities. The total step of the algorithm is a combination of the previous steps obtained in linear
programming and equality-constrained quadratic programming phases.
15.10 Active-Set Sequential Linear-Quadratic Programming (KNITRO/ACTIVE) 559

KNITRO/ACTIVE Algorithm
To achieve progress on both feasibility and optimality, the algorithm is designed to reduce the l1
penalty function
X X
Pðx, σ Þ≜f ðxÞ þ σ jci ðxÞj þ σ max f0, ci ðxÞg, ð15:103Þ
i2E i2I

where ci, i 2 E, represents the components of vector cE, and ci, i 2 I, represents the components of cI.
The penalty parameter σ is chosen by an adaptive procedure described later.
In the LP phase, given an estimate xk of the solution of the nonlinear optimization problem
(15.102), the following linear programming problem is formed and solved:

min d ∇f ðxk ÞT d ð15:104aÞ

subject to
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 E, ð15:104bÞ

∇ci ðxk ÞT d þ ci ðxk Þ  0, i 2 I, ð15:104cÞ

kd k1 ΔLP
k , ð15:104dÞ

where ΔLPk > 0 is the trust-region radius. Observe that (15.104) differs from the classical subproblem
used in the SQP methods only in that (15.104) does not contain the quadratic term 12 d T Bk d, where Bk
is an approximation to the Hessian of the Lagrangian of (15.102). Now, since the constraints of
(15.104) may be inconsistent, instead of it we solve the l1 penalty reformulation of (15.104) as
X 
min d lσ ðdÞ≜∇f ðxk ÞT d þ σ k ∇ci ðxk ÞT d þ ci ðxk Þ
i2E
X n o
þσ k max 0, ∇ci ðxk ÞT d  ci ðxk Þ
i2I
subject to ð15:105Þ
kdk1 ΔLP
k :

The solution of this linear program, denoted by dLP, is computed by the simplex algorithm. Based
on this solution, we define the working set W as a linearly independent subset of the active-set at the
LP solution pLP
  n o
A dLP ¼ i 2 E : ∇ci ðxk ÞT dLP þ ci ðxk Þ ¼ 0
n o
[ i 2 I : ∇ci ðxk ÞT d LP þ ci ðxk Þ ¼ 0 : ð15:106Þ

Similarly, define the set V of the violated constraints as

  n o n o
V dLP ¼ i 2 E : ∇ci ðxk ÞT dLP þ ci ðxk Þ 6¼ 0 [ i 2 I : ∇ci ðxk ÞT dLP þ ci ðxk Þ < 0 : ð15:107Þ

To ensure the progress of the algorithm subject to the penalty function P(x, σ) defined in (15.103),
define the Cauchy step

dC ¼ αLP d LP , ð15:108Þ

where αLP 2 [0, 1] is the stepsize that provides sufficient decrease in the following piece-wise
quadratic model of the penalty function P(x, σ)
560 15 Sequential Quadratic Programming

1
qk ðdÞ ¼ lσ ðdÞ þ d T Bðxk , λk Þd: ð15:109Þ
2
In (15.109), lσ (d) is defined by (15.105) and B(xk, λk) is the Hessian of the Lagrange function or an
approximation of it.

Given the working set Wk, the following equality-constrained quadratic program in the variable
d is solved, treating the constraints in Wk as equalities and ignoring all the other constraints
!T
1 T X
min d Bðxk , λk Þd þ ∇f ðxk Þ þ σ k γ i ∇ci ðxk Þ d ð15:110aÞ
2 i2V
subject to
∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 E \ Wk, ð15:110bÞ

∇ci ðxk ÞT d þ ci ðxk Þ ¼ 0, i 2 I \ Wk, ð15:110cÞ

kd k2 Δ k , ð15:110dÞ

where γ i is the algebraic sign of the i-th constraint violated in xk. Observe that (15.110d) is spherical
and is distinct from the trust-region radius ΔLPk used in (15.104d). The solution of (15.110), denoted
by dQ, is obtained by the projected conjugate gradient algorithm described later.
The total step d of the SLQP method is computed as
 
d ¼ d C þ αQ d Q  d C , ð15:111Þ

where αQ 2 [0, 1] is the stepsize that approximately minimizes the model function (15.109). Now, we
can present the KNITRO/ACTIVE algorithm as follows:

Algorithm 15.9 KNITRO/ACTIVE—Byrd, Gould, Nocedal, and Waltz

1. Choose an initial point x0 2 ℝn, as well as the parameters Δ0 > 0, ΔLP 0 > 0, 0 < η < 1. Set k ¼ 0
2. If a stopping test for the nonlinear problem (15.102) is satisfied, stop; otherwise, continue with step 3
3. Update the penalty parameter σ k and solve the linear program (15.105) to obtain the step d LPk and the
working set Wk
 LP 
4. Compute αLP k 2 ½0, 1 as an approximate minimizer of the quadratic model q αd k given by (15.109)
such that αLP d  Δk :Set d ¼ αLP d
LP C LP
k k k k k
5. Solve the equality quadratic programming subproblem (15.110) to obtain the solution d Q
k
Q
k ¼ d k  d k as the segment line from the Cauchy point to the solution of the equality
6. Compute dCE C

quadratic programming subproblem

 C 
7. Compute αQ k 2 ½0, 1 as an approximate minimizer of qk d k þ αd k
CE
with respect to α
8. Compute dk ¼ d Ck þ αQ k d k and set xT ¼ xk þ dk
CE

9. Compute the standard ratio

ρk ¼ Pðxqk , ðσ0kÞq
ÞPðxT , σ k Þ
:
k k ðd k Þ
If ρk  η, then set xkþ1 ¼ xT; otherwise, set xkþ1 ¼ xk and go to step 10
10. Update ΔLPkþ1 and Δkþ1. Set k ¼ k þ 1 and go to step 2 ♦
15.10 Active-Set Sequential Linear-Quadratic Programming (KNITRO/ACTIVE) 561

The trust-region radius Δk is updated based on the standard ratio ρk. The choice of ΔLP
k is important
because this determines the working set. The estimates of the multipliers λk used in the Hessian are the
least square estimates using the working set Wk and modified so that λi  0 for i 2 I.

Strategy for Penalty Parameter Update

The algorithm KNITRO/ACTIVE in step 3 requires an updating of the penalty parameter σ k. This is
done in the first phase, where the linear program (15.104) is solved. For this, a piecewise linear model
of the constraint violation at the current point xk is defined as
X  X n o
mk ðdÞ ¼ ∇ci ðxk ÞT d þ ci ðxk Þ þ max 0, ∇ci ðxk ÞT d  ci ðxk Þ , ð15:112Þ
i2E i2I

so that the objective (15.105) of the LP subproblem can be written as

lσ ðd Þ ¼ ∇f ðxk ÞT d þ σ k mk ðd Þ: ð15:113Þ

Given a value σ k, then the solution of the LP subproblem (15.105) is denoted as dLP(σ k) to
emphasize its dependence on the penalty parameter. Let dLP(σ 1) be the minimizer of mk(d ) subject
to the trust-region constraint kdk1 ΔLPk from (15.105). Then, the following algorithm describes the
computation of the LP step dLP k and the updating of the penalty parameter σ k.

Algorithm 15.10 Penalty update algorithm

1. Initialize the data: xk, σ k  1 > 0 and ΔLPk > 0: Choose the parameters ε1, ε2 2 (0, 1]
2. Solve the subproblem (15.105) with σ ¼ σ k  1 to obtain dLP(σ k  1)
3. If mk(dLP(σ k  1)) ¼ 0, then set σ + ¼ σ k  1; otherwise, compute dLP(σ 1)
4. If mk(dLP(σ 1)) ¼ 0, then find σ + > σ k  1 such that mk(dLP(σ +)) ¼ 0. Otherwise, find σ +  σ k  1 such that
mk(0)  mk(dLP(σ +))  ε1[mk(0)  mk(dLP(σ 1))]
5. If necessary, increase the value of σ + to satisfy    
lσþ ð0Þ  lσþ d LP ðσ þ Þ  ε2 σ þ mk ð0Þ  mk dLP ðσ þ Þ :
LP þ
Set σ k ¼ σ and dk ¼ d ðσ Þ
+ LP
♦

The selection of σ + > σ k1 is achieved by the successive increase of the current trial value by
10, for example, and by resolving the LP problem. The algorithm is simple and easy to be
implemented when the worm start strategy is used. The penalty update algorithm above guarantees
that σ is chosen large enough to ensure the convergence to a stationary point (Byrd, Gould, Nocedal,
& Waltz, 2002).
Since Algorithm 15.10 is a penalty method, it can naturally deal with infeasibility. If a problem is
infeasible, then the penalty parameter will be driven to infinity. Moreover, if the algorithm is
converging to a stationary point, for our infeasibility measure we have
 
mk ð0Þ  mk d LP ðσ 1 Þ ! 0

during the penalty update procedure, providing a clear indication of local infeasibility.

Iteration of Projected Conjugate Gradient Algorithm

One of the main modules of Algorithm 15.9 is solving the equality quadratic programming subprob-
lem (15.110) in step 5. This module is shared by all the algorithms implemented in KNITRO (with
active-set or interior-point). In KNITRO, the equality quadratic programming subproblems (15.110)
562 15 Sequential Quadratic Programming

are solved by using the projected conjugate gradient method (Gould, Hribar, & Nocedal, 2001),
(Keller, Gould, & Wathen, 2000), which is well suited for large-scale problems. This is also suitable
in case of large-scale subproblems and can treat the situations in which the negative curvature appears
without modifications of the Hessian of the Lagrange function. A variant of this iteration in the
context of the sequential linear-quadratic programming is as follows.
Let us consider the following quadratic program
1 T
min x Gx þ hT x ð15:114aÞ
2
subject to
Ax ¼ b, ð15:114bÞ

where G is supposed to be symmetric and positive definite on the null space of A 2 ℝn  l. One way to
solve (15.114) is to apply a special form of the conjugate gradient (CG) iteration to the KKT system
associated to (15.114), which is given by
" #   
G AT x h
¼ : ð15:115Þ
A 0 v b

Although the coefficient matrix is not positive definite, we can apply the CG method to (15.115),
provided that we precondition and project the CG method so that it effectively solves the positive
definite reduced problem within the feasible manifold (15.114b). Denote the preconditioning matrix
by P. The following algorithm gives the iterations of the projected conjugate gradient.

Algorithm 15.11 Preconditioned projected conjugate gradient algorithm

1. Choose an initial point x0 satisfying Ax0 ¼ b, as well as the tolerance ε > 0. Set x ¼ x0 and compute
r ¼ Gx þ h. Set z ¼ Pr and d ¼  z
2. If kzk ε, then stop; otherwise, go to step 3
3. Compute (in this order):
α ¼ r T z=dT Gd, x ¼ x þ αd,
r þ ¼ r þ αGd, zþ ¼ Pr þ ,
β ¼ ðr þ Þ zþ =r T z, d ¼ zþ þ βd,
T

þ
z¼z , r ¼ rþ
and continue with step 2 ♦

Algorithm 15.11 has exactly the same form as the standard preconditioned CG algorithm for
solving symmetric and positive definite systems of linear equations (Golub, & Van Loan, 1996). The
difference is that P is normally a symmetric and positive definite matrix, whereas in our case it
represents a projection and preconditioning matrix defined as follows.
Given a vector r, compute z ¼ Pr as solution of the system
" #   
D AT z r
¼ , ð15:116Þ
A 0 w 0

where D is a symmetric and positive definite matrix on the null space of A and w is an auxiliary vector.
To be a practical preconditioning matrix, D must be a sparse matrix so that solving (15.116) is
15.10 Active-Set Sequential Linear-Quadratic Programming (KNITRO/ACTIVE) 563

significantly less costly than solving (15.115). By construction, z ¼ Pr is in the null space of A and so
are all the search directions generated by Algorithm 15.11. Since Ax0 ¼ b, all the subsequent iterates
x also satisfy these linear constraints. Let Z 2 ℝn(nl ) be a basis for the null space of A from
(15.114b). Then, in the null space of A the solution of (15.114) may be expressed as x ¼ x0 þ Zu,
where u 2 ℝnl is the solution of the linear system
 T 
Z GZ u ¼ ZT ðGx0 þ hÞ: ð15:117Þ

The iterations generated by Algorithm 15.11 are given by x ¼ x0 þ Zu, where u are the iterations of
the preconditioned CG method of the system (15.117) using the matrix ZTDZ as a preconditioner.
Therefore, Algorithm 15.11 is a standard preconditioned CG algorithm as soon as G and D are
positive definite on the null space ofA (Gould, Hribar, & Nocedal, 2001).
Algorithm 15.11 has two advantages over the reduced conjugate gradient approach. The first is that
there is no need to compute a null space basis and consequently no risk for the ill-conditioning in Z to
deteriorate the rate of convergence of the conjugate gradient iteration. The second benefit is that the
projection matrix in (15.116) can also be used to compute the normal step and the Lagrange
multipliers, and thus the extra cost of each of these computations is only one back solve involving
the factors of this projection matrix (Byrd, Nocedal, & Waltz, 2006).
Algorithm 15.11 assumes that an initial feasible point x0 is provided (see step 1). The factorization
of the system (15.116) allows us to compute such a point by solving
" #   
D AT w 0
¼ ,
A 0 x 0 b

which is in fact the minimum norm solution in the norm weighted by D.

Hessian Options
KNITRO contains some options for the first and second derivatives. The user can supply these
elements, which generally results in the greatest level of efficiency and robustness of the algorithms
implemented in KNITRO. However, if these elements are not possible to be supplied, then KNITRO
has other options on the second derivative: the dense quasi-Newton BFGS, the dense quasi-Newton
SR1, the finite-difference Hessian-vector product, and the limited-memory BFGS Hessian. The dense
quasi-Newton BFGS option uses the gradient information to compute a symmetric, positive definite
approximation to the Hessian matrix. Typically, this method requires more iterations to converge than
the exact Hessian version. The dense quasi-Newton SR1 approach builds an approximate Hessian by
using the gradient information, but in this case, the SR1 Hessian approximation is not restricted to be
positive definite. Therefore, the SR1 approximation may be a better approach than BFGS if there is a
lot of negative curvature in the problem. In the case of large-scale applications, the Hessian of the
Lagrangian cannot be computed or it is too large to store. In KNITRO, the Hessian-vector product can
be obtained by the finite differences of the gradients of the Lagrangian. Each Hessian-vector product
requires one additional gradient evaluation.
In the following, let us present the performances of KNITRO/ACTIVE for solving some
applications from the LACOP collection. KNITRO/ACTIVE implements an active-set method
based on the sequential linear-quadratic programming (SLQP) and the projected conjugate gradient
iteration, known as KNITRO/ACTIVE Option 3.
564 15 Sequential Quadratic Programming

Numerical Study—KNITRO/ACTIVE: Solving Applications from the LACOP Collection

Table 15.6 contains the performances of KNITRO/ACTIVE for solving 12 nonlinear optimization
applications from the LACOP collection presented in Appendix C.

In Table 15.7 we can see the performances of KNITRO/ACTIVE for solving 6 large-scale
nonlinear optimization applications from the LACOP collection.
Table 15.8 shows the performances of KNITRO/ACTIVE (option 3) for solving the applications
HANG with a large number of variables.
All SNOPT, NLPQLP, and KNITRO/ACTIVE are reliable algorithms for solving small or large-
scale problems. SNOPT evolved from MINOS and performs better than MINOS. Our numerical
experiments show that for solving 12 small-scale applications from the LACOP collection
(Table 15.1) SNOPT needs 0.31 seconds, while KNITRO/ACTIVE (option 3) (Table 15.6) needs

Table 15.6 Performances of KNITRO/ACTIVE for solving 12 applications from the LACOP collection. Option
3. Small-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng #nh cpu vfo

ELCH 10 3 0 18 80 19 19 18 0.019 −47.761090
ALKI 10 3 8 22 33 58 23 31 0.022 −1768.8069
PREC 8 0 6 45 116 117 46 68 0.028 3.95116344
PPSE 9 6 0 6 5 11 7 7 0.016 5055.01174
MSP3 13 0 15 7 6 9 8 7 0.015 97.5910347
MSP5 16 0 21 21 34 54 22 32 0.032 174.786994
POOL 34 20 0 15 20 33 16 22 0.025 2569.800
TRAFO 6 0 2 30 88 100 31 60 0.026 135.07596
LATHE 10 1 14 83 127 242 84 126 0.056 −4429.3999
DES 150 50 0 24 378 79 25 41 0.369 1055.18231
CSTC 303 200 0 6 16 7 7 6 0.084 3.4800747
DIFF 396 324 0 1 0 2 1 0 0.090 0.00

Table 15.7 Performances of KNITRO/ACTIVE for solving 6 applications from the LACOP collection. Option
3. Large-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng #nh cpu vfo

HANG 1002 501 0 10 226 24 11 13 0.343 5.0685777
2002 1001 0 10 226 24 11 13 0.982 5.0685101
FLOW 1182 754 0 10 9 11 11 10 3.456 0.15e-11
FLOWO 1556 1005 0 17 16 18 18 17 7.017 0.79e-31
POL 4004 3000 0 193 255 363 194 257 46.793 9.8214896
6004 4500 0 219 310 464 220 308 79.314 10.545559
8004 6000 0 74 105 164 75 105 81.193 10.621358
10004 7500 0 150 216 337 151 214 134.66 10.291229
CAT 3003 2000 0 117 261 429 118 233 11.138 −0.048055
6003 4000 0 155 333 551 156 299 33.039 −0.048055
9003 6000 0 112 190 266 113 183 37.491 −0.048045
CONT 2505 2000 0 14 10 15 15 14 1.893 1.0132389
5005 4000 0 14 11 15 15 14 4.947 1.0059224
7505 6000 0 12 12 13 13 12 8.573 1.0045614
10005 8000 0 16 17 17 17 16 15.245 1.0040718
15.10 Active-Set Sequential Linear-Quadratic Programming (KNITRO/ACTIVE) 565

Table 15.8 Performances of KNITRO/ACTIVE for solving the HANG application from the LACOP collection.
Option 3. Large-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng #nh cpu vfo

HANG 4002 2001 0 10 226 24 11 13 2.964 5.0684889
8002 4001 0 10 226 24 11 13 8.288 5.068482

In Tables 15.6, 15.7, and 15.8, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the
number of inequality constraints, #it ¼ the number of iterations for solving the application, #itcg ¼ the number of
conjugate gradient iterations, #nf ¼ the number of evaluations of the functions, #ng ¼ the number of evaluations of the
gradients, #nh ¼ the number of evaluations of the Hessians, cpu ¼ the CPU computing time to get a solution (seconds),
and vfo ¼ the value of the objective function at the optimal solution.

0.782 seconds. For solving 15 large-scale optimization problems from the LACOP collection,
SNOPT (Table 15.2) needs 64.91 seconds, while KNITRO/ACTIVE (option 3) (Table 15.7) needs
466.084 seconds. Both SNOPT and KNITRO/ACTIVE include mechanisms which ensure that the
subproblems are always feasible and which guard against the rank-deficient Jacobian of the
constraints. SNOPT uses the penalty (or elastic) mode which is used if the sequential quadratic
problem is infeasible or if the Lagrange multiplier estimates become very large in norm. Both the
BFGS and the limited-memory BFGS approximations are implemented in SNOPT and in KNITRO/
ACTIVE. KNITRO/ACTIVE also implements the SR1 approximation that may be more efficient
than the BFGS approximation. Many other ingredients and practical details concerning the redundant
constraints, discontinuities, inaccurate gradient estimation, and scaling are implemented in these
algorithms.

Notes and References

The sequential quadratic programming (SQP) is one of the most effective methods for solving
nonlinear optimization problems of different structures and complexities. It relies on a profound
theoretical foundation and provides powerful algorithmic tools for the solution of large-scale relevant
problems. This approach can be used both in the line-search and in the trust-region framework. SQP is
a very important component in many algorithms for nonlinear optimization. The motivation for which
this approach is suitable for solving nonlinear optimization problems is based on the fact that, locally,
any optimization problem can be very well approximated by a quadratic model.
SQP needs to solve at each iteration a quadratic approximation of the original problem, trying to
estimate which constraints need to be kept (active) and which can be ignored. Therefore, a practical
consequence is that the algorithm partly walks along the boundary of the feasible region given by the
constraints. The iterates are feasible with respect to all the linear constraints and to a local lineariza-
tion of the nonlinear constraints, feasibility which is preserved along the iterations. The complemen-
tarity condition is satisfied by default. Once the active-set of the constraints has been correctly
determined and the optimality conditions are in a given tolerance, the solver terminates with the
optimal solution. The number of iterations might be high, but each iteration is relatively cheap.
SQP can be used both in the line-search and in the trust-region frameworks. The algorithms
NLPQLP, KNITRO, SNOPT, and CONOPT (see Chap. 16) described in this book use SQP as a
crucial component in different computational structures. If the penalty and the augmented Lagrangian
methods (SPENBAR and MINOS) are effective when most of the constraints are linear, the SQP
methods (NLPQLP, KNITRO, SNOPT, and CONOPT) show their strength when the problems have
significant nonlinearities in constraints. A competing approach for handling nonlinear optimization
problems is represented by the interior-point methods, implemented for example in KNITRO and
IPOPT. The content of this chapter is based on the theoretical developments from Nocedal and
566 15 Sequential Quadratic Programming

Wright (2006), Schittkowski (1986, 2002, 2005, 2009, 2010), and Gill, Murray, and Saunders (2002,
2005, 2006).
There are two types of SQP methods. In the inequality quadratic programming approach, at each
iteration a general inequality-constrained quadratic programming subproblem is solved in order to
compute a step and to generate an estimate of the optimal active-set. On the other hand, the equality
quadratic programming methods decouple these computations and operate in two phases: in the first
one, an estimate of the optimal active-set is determined, and in the second phase an equality-
constrained quadratic programming subproblem is solved to find the step.
The main advantages of SQP are as follows: the iterates are feasible with respect to the linear
constraints for most of the iterations; they are very efficient for solving highly constrained problems;
they give better results on pathological problems; in general; they require fewer functions
evaluations; i.e., they are most efficient for problems with expensive functions evaluations; they
allow worm starts, permit infeasibility detection, and can capitalize on a good initial point
There is a vast literature on SQP. These methods were proposed by Wilson (1963) in his Ph.D.
thesis and developed inter alia by Han (1976, 1977), Powell (1977, 1978a, 1978b), Byrd, Schnabel,
and Shultz (1987b), Byrd, Tapia, and Zhang (1990), etc. For literature surveys see the papers by
Boggs and Tolle (1989, 1995), Herskovits (1995), and Gould, Orban, and Toint (2005a).
The SLQP approach was proposed by Fletcher and Sainz de la Maza (1989) and was further
developed by Byrd, Gould, Nocedal, and Waltz (2004a). The second-order corrections were
suggested by Coleman and Conn (1982a), Mayne and Polak (1982), and Gabay (1982). The filter
SQP was developed by Fletcher and Leyffer (2002) and Fletcher, Leyffer, and Toint (1999).
The most important software implementing sequential linear/quadratic solvers are as follows.
The KNITRO/ACTIVE is based on the works described in Byrd, Gould, Nocedal, and Waltz
(2002, 2004a), Byrd, Nocedal, and Waltz (2003, 2006), Nocedal and Wright (2006), Waltz (2004).
KNITRO includes both sequential linear-quadratic programming and interior-point methods. The
active-set sequential linear-quadratic programming method described in this chapter to determine an
estimate of the active-set at each iteration solves a linear programming subproblem that approximates
the l1 exact penalty subproblem. The linear programming subproblem has an additional infinity-norm
trust-region constraint. The constraints in this linear programming subproblem that are satisfied as
equalities are marked as active, and they are used to set up an equality-constrained quadratic
programming subproblem whose objective is a quadratic approximation of the Lagrangian of the
problem at the current iterate. This quadratic programming subproblem includes an l2-norm trust-
region constraint. The equality quadratic programming subproblem is solved by using a projected
conjugate gradient algorithm. The penalty parameter is updated to ensure sufficient decrease towards
feasibility.
NLPQLP is an extension of the sequential quadratic programming solver NLPQL (Schittkowski,
1985) that implements a nonmonotone line-search to ensure global convergence. Our description of
NLPQLP is based on the papers of Schittkowski (1986, 2002, 2005, 2009, 2010). It uses a quasi-
Newton approximation to the Hessian of the Lagrangian, which is updated with the BFGS formula.
To calculate the stepsize that minimizes an augmented Lagrangian merit function, a nonmonotone
line-search is used. NLPQLP illustrates the sequential quadratic programming paradigm, also
referred to as active-set methods, because it provides an estimate of the active-set at every iteration.
In NLPQLP, the quadratic programming subproblems are modified to avoid the redundant computa-
tional effort of evaluating the gradients corresponding to the inactive constraints. The derivatives, if
unavailable, can be estimated by using finite differences. If we compare NLPQLP (Table 15.4) versus
15.10 Active-Set Sequential Linear-Quadratic Programming (KNITRO/ACTIVE) 567

KNITRO/ACTIVE (Table 15.6) we can see that, for solving 8 nonlinear optimization applications
from the LACOP collection, NLPQLP needs 308 iterations, while KNITRO/ACTIVE needs only
226.
NPSOL (Gill, Murray, Saunders, & Wright, 1986) solves general nonlinear optimization problems
by using a sequential quadratic programming algorithm with a line-search on the augmented
Lagrangian. The Hessian of the quadratic programming subproblem is a quasi-Newton approximation
of the Hessian of the Lagrangian. The quadratic programming subproblem is solved by using a dense
BFGS update. If the gradients of the functions defining the problem are not available, NPSOL can
estimate them by using finite differences. All the matrices in NPSOL are dense and therefore it is not
efficient for solving large-scale problems.
SNOPT is a pure sequential quadratic programming algorithm. The description of SNOPT follows
the paper of Gill, Murray, and Saunders (2005). SNOPT solves the quadratic programming
subproblems by using SQOPT (Gill, Murray, & Saunders, 2006), which is a reduced-Hessian
active-set method. SNOPT uses a positive semi-definite quasi-Newton Hessian approximation Bk.
If the number of the nonlinear variables is moderate, then Bk is stored as a dense matrix. Otherwise,
the Hessian of the Lagrangian is updated by using the limited-memory quasi-Newton BFGS updates.
The quadratic programming solver SQOPT works with a sequence of the reduced Hessian systems of
the form ZTBkZd ¼  ZTg, where Z is a basis for the null space of W, a rectangular matrix with nz
columns. SQOPT can deal with the reduced Hessian systems in various ways, depending on the size
of nz. If the constraints are currently active in the quadratic subproblem and nz are not excessively
large, then SQOPT uses the dense Cholesky factorization ZTBkZ ¼ RTR. Otherwise, SQOPT can
maintain a dense quasi-Newton approximation to avoid the cost of forming and of factorizing the
reduced Hessian. SNOPT includes an option for using a projected conjugate gradient method instead
of the factorization of the reduced Hessian. If a quadratic subproblem is found to be infeasible or
unbounded, then SNOPT tries to solve an elastic subproblem that corresponds to a smooth reformu-
lation of the l1 exact penalty function. The solution of the quadratic subproblem solved at the major
iteration is used to obtain a search direction along which an augmented Lagrangian merit function is
minimized. SNOPT uses a lot of procedures from MINOS. When the gradients are not available,
SNOPT uses finite differences to estimate them. SNOPT is embedded in the GAMS technology
(Andrei 2017c).
SQPlab (Gilbert, 2009) is a laboratory for testing different options of the sequential quadratic
programming methods (Bonnans, Gilbert, Lemaréchal, & Sagastizábal, 2006). It implements a line-
search sequential quadratic programming method that can use either the exact Hessian or a BFGS
approximation of the Lagrangian.
CONOPT (Drud, 1996, 2011), described in Chap. 16 of this book, implements three active-set
methods. The first one is a gradient projection method that projects the gradient of the minimizing
function onto a linearization of the constraints. The second variant is a sequential linear programming
method and the third one is a sequential quadratic programming method. CONOPT is a line-search
method and includes algorithmic switches that automatically detect which method is most suitable.
Other software implementing SQP are filterSQP (Fletcher, & Leyffer, 2002) presented in Chap. 18
of this book, and RSQP (Bartlett, & Biegler, 2003), which maintains a quasi-Newton approximation
to the reduced Hessian.
 Primal Methods: The Generalized
Reduced Gradient with Sequential 16
Linearization

By primal method, we understand a search method that works directly on the original problem by
searching the optimal solution taking a path through the feasible region. Every iteration in these
methods is feasible and along the iterations, the values of the minimizing function constantly
decrease. The most important primal methods are the feasible direction method of Zoutendijk
(1960), the gradient projection method of Rosen (1960, 1961), the reduced gradient method of
Wolfe (1967), the convex simplex method of Zangwill (1967), and the generalized reduced gradient
method of Abadie and Carpentier (1969). The last four methods can be embedded into the class of the
active set methods. The idea of the active set methods is to partition the inequality constraints into two
groups: those that can be treated as active (satisfied with equality in the current point) and those that
have to be treated as inactive. The constraints treated as inactive are essentially ignored. Of course,
the fundamental component of an active set method is the algorithm for solving an optimization
problem with equality constraints only.
The primal methods have three main advantages. Firstly, they generate feasible points. Thus, if the
optimization process terminates before reaching the solution, then the terminating point is feasible
and possibly near the optimal solution of the problem. The second attractive advantage is that, if the
optimization process is convergent, then the limit point of this sequence has to be at least a local
constrained minimum. Finally, the third advantage is that the primal methods do not use the special
structure of the problem such as convexity, for example. Therefore, these methods can be used for
solving general constrained nonlinear optimization problems.
The purpose of this chapter is to give a short description of the primal methods, followed by a more
detailed presentation of the modern generalized reduced gradient method, which is representative for
the primal methods.

16.1 Feasible Direction Methods

For solving the inequality constrained problem

min f ðxÞ
subject to ð16:1Þ
cj ðxÞ  0, j ¼ 1, . . . , m,

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 569
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_16
570 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

where f : ℝn ! ℝ and cj : ℝn ! ℝ, j ¼ 1, . . ., m, are continuously differentiable functions, a feasible

direction method takes steps through the feasible region of the known form
xkþ1 ¼ xk þ αk dk , ð16:2Þ

where dk is the search direction, and αk is the stepsize. The crucial aspects of this method are that the
scalar αk is chosen in such a way so as to minimize the function f under the constraint that the point
xkþ1 given by (16.2), and the segment of the line connecting xk and xkþ1 are all feasible. Therefore, the
problem of minimizing f with respect to α is nontrivial if the ray xkþ1 ¼ xk þ αdk, α > 0, is entirely
contained in the feasible region. This is the main motivation of using the feasible directions for
solving (16.1). A vector dk is a feasible direction in the point xk if there is an α > 0 such that
xkþ1 ¼ xk þ αdk is feasible for all α with 0  α  α: The concept of feasible direction is a natural
extension of the concept of descent direction from the unconstrained optimization. The following two
methods put in a concrete form the feasible direction methods.

The Frank-Wolfe Method For the problem (16.1), in which all the constraints are linear cTj x  bj ,
j ¼ 1, . . ., m, one of the earliest feasible direction methods is the one of Frank and Wolfe (1956). In
this method, given the current point xk, the search direction is computed as dk ¼ xk  xk , where xk is
the solution of the linear programming problem

min ∇f ðxk ÞT x
subject to ð16:3Þ
cTj x  bj , j ¼ 1, . . . , m:

If function f is continuous differentiable and the polyhedron defined by the linear constraints
cTj x  bj , j ¼ 1, . . ., m, is bounded, or f(x) ! þ 1 when kxk ! 1 , then, for any feasible initial
point the Frank-Wolfe method converges to a local minimizer of the problem
n o
min f ðxÞ : cTj x  bj , j ¼ 1, . . . , m :

The Zoutendijk Method Referring to the problem (16.1) with nonlinear constraints, this method
solves a sequence of linear programming subproblems as follows (Zoutendijk, 1960). Given a
feasible point xk, let Ik be the set of indices representing the active constraints, that is the constraints
cj(xk) ¼ 0, j 2 Ik. Then, the search direction dk is chosen as solution of the following problem:

min ∇f ðxk ÞT d
subject to ð16:4Þ
T
cj ðxk Þ þ ∇cj ðxk Þ d  0, j 2 Ik ,
Xn
jd j ¼ 1:
i¼1 i

The last equation in (16.4) is a normalizing equation that ensures a bounded solution. The other
constraints ensure that the vectors of the form xk þ αdk will be feasible for sufficiently small α > 0.
Moreover, solving the problem (16.4) determines that d is chosen to line up as closely as possible to
the negative gradient of f in the current point. The main difficulties with this simplified Zoutendijk
method (16.4) are that for general problems, there may not exists any feasible directions, and in this
form the simplified Zoutendijk method is not globally convergent.
16.2 Active Set Methods 571

Topkis and Veinot (1967) improved the simplified Zoutendijk method and suggested another
procedure for the search direction computation as solution of the following linear programming
problem:

min δ
subject to ð16:5Þ
∇f ðxk ÞT d  δ  0,
cj ðxk Þ þ ∇cj ðxk ÞT d  θj δ  0, j ¼ 1, . . . , m,
1  d i  þ1, i ¼ 1, . . . , n,

where the unknowns are d 2 ℝn and δ 2 ℝ, while θj, j ¼ 1, . . ., m, are positive parameters. (e.g.,
θj ¼ 1, j ¼ 1, . . ., m.). Obviously, if the minimum of (16.5) corresponds to a value δ < 0, then
∇f(xk)Td < 0, that is d is a descent direction. Besides, cj(xk) þ ∇ cj(xk)Td  θjδ < 0, proving that all the
constraints are satisfied for small values in the direction d. Details on this method are given in
(Andrei, 1999a, b, pp 597–608).

16.2 Active Set Methods

These methods make a partition of the inequality constraints of the problem in two groups: those
which are active at the current point, namely, they are satisfied as equalities, and those that are
inactive. The inactive constraints are essentially ignored. Consider the problem
min f ðxÞ
subject to ð16:6Þ
cðxÞ  0,

where all the functions f : ℝn ! ℝ and c : ℝn ! ℝm are continuously differentiable. For this problem,
the necessary optimality conditions are

∇f ðxÞ þ μT ∇cðxÞ ¼ 0, ð16:7aÞ

cðxÞ  0, ð16:7bÞ

μT cðxÞ ¼ 0, ð16:7cÞ

μ  0, ð16:7dÞ

(see Definition 11.14) where μ 2 ℝm is the vector of the Lagrange multipliers. Denote A as the set of
the indices of the active constraints, that is, A is the set of j such that cj(x) ¼ 0. In terms of active
constraints, these optimality conditions can be expressed in a more simple form as
X
∇f ðxÞ þ μj ∇cj ðxÞ ¼ 0, ð16:8aÞ
j2A

cj ðxÞ ¼ 0, j 2 A, ð16:8bÞ

cj ðxÞ < 0, 2A,

j= ð16:8cÞ
572 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

μj  0, j 2 A, ð16:8dÞ

μj ¼ 0, 2A,
j= ð16:8eÞ

Observe that (16.8a) and (16.8b) correspond to the necessary optimality conditions of the equality
constrained problem (see Theorem 11.11). (16.8c) guarantees that the inactive constraints are
satisfied. The sign requirement of the Lagrange multipliers given by (16.8d) and (16.8e) guarantees
that every constraint that is active should be active.
Obviously, if the active set of constraints were known, then the original problem could be replaced
by the corresponding problem having only the equality constraints. Now, suppose that an active set of
constraints was guessed and the corresponding equality constrained problem solved. Then, if the
other constraints were satisfied and the Lagrange multipliers turned out to be nonnegative, then the
solution of the equality constrained problem would be exactly the solution to the original problem.
At each iteration of an algorithm, the active set methods define a set of constraints denoted by W,
termed working set, treated as the active set. The working set is chosen to be a subset of the
constraints that are actually active at the current point. In other words, the current point is feasible
for the working set. The algorithm tries to move on the surface defined by the working set of the
constraints, the so-called working surface, in order to get an improved point, i.e., a point at which the
minimizing function is reduced. At this new point, the working set may be changed and all the process
is repeated. There are several methods for moving on the working surface. The most important ones
(the gradient projection, the reduced gradient, the convex simplex, and the generalized reduced
gradient methods) are discussed in the following sections. The asymptotic convergence properties
of the active set methods depend on the procedure for moving on the working surface, since near the
solution, the working set of constraints is generally equal to the correct active set.
Now, let us discuss the strategy of choosing the working set and of moving on the working surface
corresponding to the active set methods. Suppose that, for a given working set W the problem with
equality constraints is
min f ðxÞ
subject to ð16:9Þ
cj ðxÞ ¼ 0, j 2 W:

Let x be the solution of (16.9), that is x satisfies cj ðxÞ < 0, j 2

= W. Of course, this point satisfies the
necessary optimality conditions
X
∇f ðxÞ þ μj ∇cj ðxÞ ¼ 0: ð16:10Þ
j2W

If μj  0 for all j 2 W, then the point x is a local solution to the original problem (16.6). On the other
hand, if there is an index j 2 W such that μj < 0, then the objective function can be decreased by
relaxing the constraint j. This follows from the sensitivity interpretation of the Lagrange multipliers
(see Theorem 11.18): a small decrease in the constraint value from 0 to b will lead to a change in the
value of the minimizing function of μjb, which is clearly negative. Thus, dropping the constraint
j from the working set, an improved solution can be obtained. Therefore, the Lagrange multiplier
indicates which constraint should be dropped from the working set. Now, since all the points
determined by the algorithm must be feasible, it is necessary to monitor the values of the other
constraints to ensure that they are not violated. Obviously, moving on the working surface, a new
constraint boundary is attained. Therefore, this constraint is added to the working set, thus obtaining a
new working set with which the process can be repeated.
16.3 The Gradient Projection Method 573

In conclusion, the active set strategy proceeds by systematically dropping and adding constraints,
that is starting with a given working set and minimizing the function f over the corresponding working
surface. If new constraint boundaries are attained, they may be added to the working set, but no
constraints are dropped from the working set. Hence, a point is obtained that minimizes f with respect
to the current working set of constraints. The corresponding Lagrange multipliers are determined and
if they are all nonnegative, it follows that the solution is optimal. Otherwise, one or more constraints
with negative Lagrange multipliers are dropped from the working set. With this new working set, the
optimization process is restarted until an optimal solution is obtained.
If for every working set W the problem min{f(x) : cj(x) ¼ 0, j 2 W} is well defined with a unique
nondegenerate solution (that is for all j 2 W, μj 6¼ 0), then the sequence of points generated by the
above described active set strategy converges to the solution of the inequality constrained problem
(16.6).
The difficulty with the active set method is that several problems with incorrect active sets must be
solved. Besides, the solutions of these intermediate problems must be the exact global minimum
points in order to determine the correct sign of the Lagrange multipliers and to ensure that along the
iterations the current working set is not encountered again. The convergence of these methods cannot
be guaranteed. They are subject to jamming, where the working set changes an infinite number of
times. However, numerical experiments proved that the phenomenon of zigzagging (jamming) is very
rare and the algorithms based on the active set strategy work very well in practice. The combinatorial
difficulty of the inequality constrained optimization solved by the active set methods is presented in
Sect. 11.3.

16.3 The Gradient Projection Method

In this method, the search direction is chosen as the projection of the negative gradient onto the
working surface. Consider the problem
min f ðxÞ
subject to ð16:11Þ
aTj x  bj , j ¼ 1, . . . , m,

with linear inequality constraints, where x 2 ℝn. The constraints from (16.11) define the matrix
A 2 ℝm  n, where the rows of A are defined by the constraints aTj x  bj , j ¼ 1, . . ., m. Assume that
the constraints from (16.11) define a nonempty region X. If it exists, a feasible solution to this problem
can be obtained by applying the simplex algorithm. At a given feasible point x there will be a certain
number q of active constraints aTj x ¼ bj and some inactive constraints aTj x < bj : The set of active
constraints is W(x). Now, at the feasible point x we are interested in finding a feasible direction
d which is descent, that is it satisfies ∇f(x)Td < 0 so that the movement in the direction d will
determine a decrease of the values of the function f. Initially, consider the directions d satisfying
aTi x ¼ 0, i 2 W(x), so that all the working constraints remain active. Therefore, the search direction
d is in the tangent subspace T defined by the working set of constraints. The search direction used in
this method is exactly the projection of the negative gradient onto this subspace. To compute this
projection, consider Aq 2 ℝqn as the matrix composed by the rows corresponding to the working
constraints. Assume that x is a regular point, i.e., the gradient vectors of the active constraints in x are
linear independent (see Definition 11.13). In this case, q < n and rank(Aq) ¼ q < n. The tangent
subspace T in which d must lie is the subspace of the vectors which satisfy Aqd ¼ 0. Therefore, the
574 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

range subspace N of ATq , i.e., the set of all the vectors of the form ATq μ for μ 2 ℝq, is orthogonal to T.
Any vector can be written as a sum of vectors from each of these two complementary subspaces. In
particular,  ∇ f(xk) can be written as

∇f ðxk Þ ¼ dk þ ATq μk ,

where dk 2 T and μk 2 ℝq. Therefore, having in view that Aqdk ¼ 0, it follows that
 
Aq ∇f ðxk Þ  Aq ATq μk ¼ 0,

which determines
 1
μk ¼  Aq ATq Aq ∇f ðxk Þ:

Therefore,
  1 
dk ¼  I  ATq Aq ATq Aq ∇f ðxk Þ  Pk ∇f ðxk Þ: ð16:12Þ

The matrix Pk from (16.12) is called the projection matrix corresponding to the subspace T.
Observe that, since Aq is of full rank, it follows that Aq A1
q is nonsingular. From (16.12) we can write
 1
dk ¼ ∇f ðxk Þ þ ATq Aq ATq Aq ∇f ðxk Þ:

Since ∇f(xk) þ dk is orthogonal to dk, if dk 6¼ 0, it follows that

 
∇f ðxk ÞT dk ¼ ∇f ðxk ÞT þ dTk  dTk dk ¼ kdk k2 < 0,

Therefore, if dk is computed as in (16.12) and is nonzero, then it is a feasible descent direction on

the working set.
The problem we face now is how to compute the stepsize along the direction dk.Obviously, when α
is increased from zero, the point xk þ αdk will remain feasible and the corresponding value of f will
decrease. The maximum stepsize can be computed as
α max ¼ max fα : α > 0, xk þ αdk 2 Xg:

The stepsize is computed as the minimum of f(xk þ αdk), where α is on the segment [0, αmax].
When the projected negative gradient is zero, it follows that

∇f ðxk Þ þ μTk Aq ¼ 0, ð16:13Þ

that is the point xk satisfies the necessary conditions for a minimum on the working surface. If the
components of μk corresponding to the active inequalities are all nonnegative, then this fact together
with (16.13) implies that the KKT conditions for the problem (16.11) are satisfied at xk and the
optimization process terminates. In this case, μk is the Lagrange multiplier vector for the problem
(16.11). If, on the other hand, at least one component of μk corresponding to the active inequalities is
negative, by relaxing the corresponding inequality it is possible to move in a new direction to get an
improved point. Suppose that the component μjk of μk is negative and corresponds to the inequality
aTj x  bj : The new search direction is determined by relaxing the j-th constraint and by projecting the
16.3 The Gradient Projection Method 575

negative gradient onto the subspace defined by the remaining q  1 active constraints. Let Aq be the
matrix Aq with row aj deleted. Then, for some μk we have

∇f ðxk Þ ¼ ATq μk , ð16:14Þ

∇f ðxk Þ ¼ dk þ ATq μk , ð16:15Þ

where d k is the projection of  ∇ f(xk) using Aq : Obviously, dk 6¼ 0, otherwise (16.15) would be a

special case of (16.14) with μjk ¼ 0, which is impossible, since the rows of Aq are linearly indepen-
dent. But we know that ∇f ðxk ÞT dk < 0: By multiplying the transpose of (16.14) by d k and by using
Aq dk ¼ 0 we get

0 > ∇f ðxk ÞT d k ¼ μjk aTj dk :

Since μjk < 0 it follows that aTj d k < 0: Therefore, since aTi dk ¼ 0 for i 2 W(xk), i 6¼ j and aTj dk < 0,
it follows that j can be dropped from W(xk). Thus, a new working set of active constraints is obtained,
with which a new iteration is started.
The gradient projection method can be extended for solving nonlinear problems of the form
min f ðxÞ
subject to ð16:16Þ
cðxÞ ¼ 0,

where all the functions are continuously differentiable. The idea is that at a feasible point xk the active
constraints are determined and the negative gradient is projected onto the subspace tangent to the
surface determined by the active constraints in xk. If this vector is nonzero, it determines the searching
direction for the next step. However, since the surface is curved, this vector is not a feasible direction
in general. In this case, additional computations must be executed to get a new feasible point. Firstly,
a move is made along the projected negative gradient to a point y. Then another move is made from
y in the direction perpendicular to the tangent plane at the point xk to get a feasible point xkþ1 on the
working surface. In this case, the projection matrix at xk is computed as
h i1
Pk ¼ I  ∇cðxk ÞT ∇cðxk Þ∇cðxk ÞT ∇cðxk Þ:

The search direction dk is computed as dk ¼  Pk ∇ f(xk). Observe that, when moving from xk to
xkþ1, the Jacobian ∇c will change and the new projection matrix cannot be found from the old one.
Therefore, it must be recomputed at each iteration of the algorithm.
The computation of the projection matrix might not be so delicate, but the most important and the
finest feature of the method is the problem of returning to the feasible region from points outside this
region. This is a common problem in nonlinear optimization, including the interior-point methods for
linear and nonlinear optimization, i.e., from any point near xk, the difficulty is to move back to the
constraint surface of (16.16) in a direction orthogonal to the tangent plane at xk. Thus, from a point y a
new point of the form y þ ∇ c(xk)Tv ¼ y is determined such that c(y) ¼ 0. To find a suitable first
approximation to v and hence to y, the equation at xk is linearized, thus obtaining
 
c y þ ∇cðxk ÞT v ffi cðyÞ þ ∇cðxk Þ∇cðxk ÞT v:
576 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

Obviously, this approximate is accurate when kvk and ky  xkk are small. Therefore, this
motivates the first approximate
h i1
y1 ¼ y  ∇cðxk ÞT ∇cðxk Þ∇cðxk ÞT cðyÞ:

Thus, by successively repeating this process, we obtain the sequence {yj} generated as
h i1  
yjþ1 ¼ yj  ∇cðxk ÞT ∇cðxk Þ∇cðxk ÞT c yj ,

which, initialized close enough to xk, will converge to a solution y.

Let x be a local solution of the problem min{f(x) : c(x) ¼ 0} and suppose that λmax and λmin are the
largest and the smallest eigenvalues of the Hessian ∇2L(x) of the Lagrange function L(x) ¼ f(x) þ μTc(x)
restricted to the tangent subspace T(x).If {xk} is a sequence generated by the gradient projection
method that converges to x, then the sequence {f(xk)} converges to f(x) linearly with a ratio no
greater than ((λmax  λmin)/(λmax þ λmin))2. This result on the rate of the convergence of the gradient
projection method is proved by Luenberger, and Ye, (2016, pp. 375–377). Details on this method are
given in (Andrei, 1999a, b, pp. 611–624).

16.4 The Reduced Gradient Method

The reduced gradient method elaborated by Wolfe (1967) is closely related to the simplex method for
linear programming. The idea of the method is to partition the variables of the problem into two
groups as the basic and the nonbasic variables. Using specific rules, the basic variables are changed
into nonbasic variables and vice-versa until a criterion for stopping the iterations has been satisfied.
Consider the problem
min f ðxÞ
subject to ð16:17Þ
Ax ¼ b, x  0,

where x 2 ℝn, b 2 ℝm, A 2 ℝmn, m < n and f is twice continuously differentiable. Suppose that a
nondegeneracy assumption is satisfied, that is every collection of the m columns from A is linear
independent and every basic solution has exactly m strictly positive variables. Under this assumption,
any feasible solution of the problem (16.17) will have at most n  m variables assigned to zero. A
given feasible vector x for (16.17) is partitioned as x ¼ [y, z]T, where y 2 ℝm and z 2 ℝn  m. This
partition is formed in such a way so that all the components of y are strictly positive. With this, the
problem (16.17) can be written as
min f ðxÞ ð16:18aÞ

subject to
By þ Cz ¼ b, ð16:18bÞ

y  0, z  0, ð16:18cÞ

where B and C collect the columns from A corresponding to the variables from y and from z,
respectively. Since B is nonsingular, if z is specified, then from (16.18b) y can be uniquely deter-
mined. Therefore, z represents the independent variables, while y is the vector of the dependent
16.4 The Reduced Gradient Method 577

variables (basic variables) computed as y ¼ B1(b  Cz). Observe that a small change Δz of the
independent variables which leaves z þ Δz nonnegative will yield by (16.18b) another feasible
solution. Since y was originally taken to be strictly positive, y þ Δy will be also positive for small Δy.
Therefore, we move from one feasible solution to another one by choosing a Δz and then by moving
z on the line z þ αΔz, α  0. Similarly, y will move along the corresponding line y þ αΔy. If in this
moving a certain variable becomes zero, a new inequality constraint becomes active. Thus, if an
independent variable becomes zero, a new Δz must be chosen. On the other hand, if a dependent
variable becomes zero, then the partition must be modified. The basic variable is declared indepen-
dent, and one of the strictly positive independent variables is made dependent.
The reduced gradient method represents the problem in terms of independent variables. Obviously,
since y is determined from z as y ¼ B1(b  Cz), it follows that the minimizing function can be
considered as a function depending only on z. The gradient with respect to the independent variables
z is obtained by computing the gradient of f(B1b  B1Cz, z) as

r T ¼ ∇z f ðy, zÞ  ∇y f ðy, zÞB1 C, ð16:19Þ

which is called the reduced gradient. A point (y, z) satisfies the first-order necessary optimality
conditions if and only if
ri ¼ 0 for all zi > 0,
ri  0 for all zi ¼ 0:

In the reduced gradient method, the vector z of the independent variables is moved in the direction
of the reduced gradient on the working surface. Thus, at each iteration a search direction

r i , i=2W ðzÞ,
Δzi ¼ ð16:20Þ
0, i 2 W ðzÞ

is determined, and Δy ¼  B1CΔz is computed. With this, compute the stepsizes as follows:
α1 ¼ max fα : y þ αΔy  0g,
α2 ¼ max fα : z þ αΔz  0g,
α3 ¼ min ff ðx þ αΔxÞ : 0  α  α1 , 0  α  α2 g,

with which a new iteration is computed as x ¼ x þ α3Δx.

The working set corresponding to the inequality constraints is augmented whenever a new variable
is zeroed. If it is a dependent variable, then a new partition of the vector x is formed, that is B and
C are updated. If a point is found where ri ¼ 0 for all i 2= W(z), but ri < 0 for some j 2 W(z), then j is
deleted from W(z).
Even if the reduced gradient method borrows many ideas from the simplex algorithm of linear
programming, it is subject to jamming. However, the global convergence can be ensured by a trivial
modification of the search direction, i.e., instead of (16.20), the search direction is computed as

r i , r i  0,
Δzi ¼
xi r i , r i > 0:

Details on the reduced gradient method are given in Luenberger and Ye (2016, pp. 378–381) and in
Andrei (1999a, b, pp. 624–629).
578 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

16.5 The Convex Simplex Method

This method elaborated by Zangwill (1967) is a modification of the reduced gradient method. The
major difference between this method and the reduced gradient method is that, instead of moving all
or several independent variables in the direction of the negative reduced gradient, only one indepen-
dent variable is changed at a time. The selection of that independent variable to change is made as in
the simplex algorithm.
For the problem (16.17) at a given feasible point, consider the partition of the vector x as x ¼ [y, z]T,
where y is the vector of the dependent (basic) variables and z is the vector of the independent variables
(nonbasic). Assume that the bounds on x are x  0, as in (16.17). The reduced gradient is computed as
in (16.19). Then, the component zi to be changed is determined according to the following procedure:

1. Let r i1 ¼ min fr i g,
i
2. Let r i2 zi2 ¼ max fr i zi g,
i
• If r i1 ¼ r i2 zi2 ¼ 0, then terminate. Otherwise
• If r i1  jr i2 zi2 j, increase zi1 ,
• If r i1  jr i2 zi2 j, decrease zi2 :

Once a particular component of z has been selected for changing according to the above procedure,
then the corresponding y is recomputed as y ¼ B1(b  Cz), so that the constraints are satisfied. The
components of z are changed until either a local minimum with respect to a component has been
attained or the boundary of one nonnegativity constraint has been reached. The details of this method
are identical to those of the reduced gradient method.

Remark 16.1 The optimization methods presented so far have a lot in common. The concept of
feasible direction is directly obtained from the optimality conditions of these methods. The methods
based on this concept are susceptible to jamming. Some methods (the gradient projection, the reduced
gradient, and the convex simplex) are based on the idea of partitioning the vector of variables into two
groups, the dependent (basic) variables and the independent (nonbasic) variables, in such a way that
the Jacobian corresponding to the dependent variables should be nonsingular. For solving large-scale
problems, the dependent variables are selected in such a way that the inverse of the corresponding
Jacobian is easy to be computed. By means of this partition, the constraints can be used to express the
minimizing objective as a function depending only on the independent variables. Thus, the concept of
the reduced gradient of the minimizing function may be introduced. This is the key element to spread
the variables in and out of the basis by using similar rules as in the simplex algorithm from linear
porogramming adapted for the nonlinear case. The rate of convergence of these methods is deter-
mined by the eigenvalues of the Hessian of the Lagrangian restricted to the subspace tangent to the
active constraints. The limitations of all these methods are that, for determining the search direction
they use only the information given by the reduced gradient, and they do not use the approximation of
the minimizing problem by linear or by quadratic programming in the current point. This is the reason
why these methods are considered as simple or naive approaches for solving nonlinear optimization
problems, and we gave them a short presentation. In the next section, we present the generalized
reduced gradient method and an improvement of it based on sequential linear or sequential quadratic
programming.
16.6 The Generalized Reduced Gradient Method (GRG) 579

16.6 The Generalized Reduced Gradient Method (GRG)

One of the most elaborate methods for solving general large-scale problems of nonlinear optimization
is the generalized reduced gradient method (GRG). Its efficiency comes from the fact that it includes
a multitude of fundamental concepts, both from the simplex algorithm of linear programming and
from the optimization techniques and methods based on the Newton method. Actually, the GRG
method is an extension of Wolfe’s reduced gradient method due to Abadie and Carpentier (1969) and
the improvements by Abadie (1978, 1979), Abadie and Guigou (1970), Abadie and Haggag (1979),
Lasdon, Fox and Ratner (1974), Lasdon and Waren (1978, 1980, 1982), Lasdon, Waren, Jain and
Ratner (1978), Gabriele and Ragsdell (1980), Drud (1976, 1983, 1984, 1985, 1995, 1996), Andrei
(1985, 1987), who consider both the objective function and the constraints of the problem as
sufficiently smooth functions.
At the beginning, we present the general form of the GRG method for solving nonlinear optimiza-
tion problems with equality constraints and simple bounds on variables. After that, in the next section,
a particular implementation of GRG with sequential linear or sequential quadratic programming,
known as CONOPT, is described.
Consider the general nonlinear optimization problem
min f ðXÞ ð16:21aÞ
subject to
ci ðXÞ ¼ 0, i ¼ 1, . . . , m, ð16:21bÞ

l j  X j  uj , j ¼ 1, . . . , n, ð16:21cÞ

where X 2 ℝn, lj and uj are the lower and the upper bounds on the variables Xj, j ¼ 1, . . ., n. Assume
that lj < uj, j ¼ 1, . . ., n. Suppose that m < n, since in most cases, m  n involves an infeasible problem
or one with a single feasible point. Moreover, suppose that f and ci, i ¼ 1, . . ., m, are twice
continuously differentiable. Denote V as the set of feasible points (manifold) which satisfy the
constraints ci(X) ¼ 0, i ¼ 1, . . ., m, and P as the parallelepiped lj  X  uj, j ¼ 1, . . ., n.
The main idea of the GRG method is to use the equality constraints ci(X) ¼ 0, i ¼ 1, . . ., m, in
order to express m variables, called basic variables, as functions of the rest of n  m variables, called
nonbasic variables. Let X be a feasible point and consider y 2 ℝm as the subvector of the basic
variables and x 2 ℝn  m as the subvector of the nonbasic variables. Therefore, X and X are partitioned
as X ¼ [x y]T and X ¼ ½x y T , respectively. With this, the equality constraints (16.21b) can be
rewritten as
cðy, xÞ ¼ 0, ð16:22Þ

where c ¼ [c1 . . . cm]T.

Suppose that the m  m-dimensional Jacobian matrix ∂c/∂y evaluated in X, the basic matrix, is
nonsingular. Then, from the implicit function theory (see Theorem A2.6), it follows that in a
neighborhood of x the nonlinear algebraic system (16.22) can be solved with respect to the basic
variable y in order to obtain the solution y(x). Therefore, the objective function (16.21a) can be
expressed as a function which depends only on the nonbasic variables
FðxÞ ¼ f ðyðxÞ, xÞ: ð16:23Þ

Hence, in a neighborhood of x a reduced problem is obtained which contains only the simple
bound on the nonbasic variables x
580 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

min FðxÞ
subject to ð16:24Þ
l N  x  uN ,

where obviously lN and uN are n  m-dimensional vectors with the lower and upper bounds of the
nonbasic variables. The GRG method solves the original problem (16.21) by solving a sequence of
subproblems (16.24) which are nonlinear optimization problems with simple bounds. The algorithms
for solving the simple bounded nonlinear optimization are presented in Chap. 12.
Obviously, for the reduced problem (16.24) to have a solution, the nonbasic variables x should be
free around the current point x: Of course, the presence of the simple bounds limits the evolution of
the nonbasic variables, but the algorithms which maintain the simple bound satisfied are relatively
simple to design. The evolution of the basic variables is more complicated. Indeed, if some
components of y are fixed on their lower or upper bounds, then a small modification of x from x
may determine the violation of the simple bounds on y. In order to avoid such situations the following
nondegeneracy hypothesis is introduced.
”For any feasible point X which satisfies the constraints of (16.21) there is a partition of it in m
basic variables y and n  m nonbasic variables x, such that
lB < y < uB , ð16:25Þ

where lB, uB 2 ℝm are the lower and the upper bounds of the basic variables and the Jacobian matrix
B ¼ ∂c/∂y evaluated in X is nonsingular.”
For the problem (16.21) let X be a feasible point with y as the basic variables and x as the nonbasic
variables. To evaluate the objective F(x) of (16.24), we have to know the basic variables y(x). Except
for the linear functions or for some particular nonlinear functions, y(x) cannot be explicitly deter-
mined. However, for a given x this can be computed by solving the algebraic nonlinear system
(16.22).

The Generalized Reduced Gradient

Therefore, for solving (16.21) its constraints are used in order to express the set of dependent (basic)
variables y as functions of the independent (nonbasic) variables x. It is not possible to establish this
dependence for the general nonlinear functions. However, using the method of the variations of the
constraints, it is possible to find y as functions of x, thus obtaining a reduced problem and introducing
the reduced gradient as an optimality criterion.
The differential of f is

df ðXÞ ¼ ∇y f ðXÞT dy þ ∇x f ðXÞT dx, ð16:26Þ

where ∇yf(X) and ∇xf(X) are the gradients of f subject to the basic variables and the nonbasic variables,
respectively. Therefore,
df ðXÞ dy
¼ ∇x f ðXÞT þ ∇y f ðXÞT : ð16:27Þ
dx dx
To eliminate the matrix dy/dx in (16.27), observe that

dci ðXÞ ¼ ∇y ci ðXÞT dy þ ∇x ci ðXÞT dx ¼ 0, ð16:28Þ

i ¼ 1, . . ., m, or in the matriceal form

16.6 The Generalized Reduced Gradient Method (GRG) 581

dcðXÞ ∂c ∂c dy
¼ þ ¼ 0, ð16:29Þ
dx ∂x ∂y dx

where ∂c/∂y and ∂c/∂x are the Jacobians of c subject to the basic and the nonbasic variables,
respectively. From the nondegeneracy hypothesis, it follows that the matrix ∂c/∂y is nonsingular.
Therefore,
 1
dy ∂c ∂c
¼ : ð16:30Þ
dx ∂y ∂x

Using this in (16.27), we get

"  1  #
T ∂c T ∂c
df ¼ ∇x f ðXÞ  ∇y f ðXÞ dx≜rdx, ð16:31Þ
∂y ∂x

where
 1  
∂c ∂c
r ¼ ∇ x f ðX ÞT  ∇ y f ðX ÞT ð16:32Þ
∂y ∂x

is the generalized reduced gradient of function f.

Define
 1
∂c
u ¼ ∇y f ðXÞT : ð16:33Þ
∂y

With this, the generalized reduced gradient is

 
∂c
r ¼ ∇ x f ðX ÞT þ u : ð16:34Þ
∂x

Remark 16.2 If the functions of the problem f and ci, i ¼ 1, . . ., m, are all linear, (16.34) are
exactly the reduced cost factors from linear programming, and u is the vector of the
simplex multipliers. ♦

It is interesting to see the relation between the generalized reduced gradient and the KKT
optimality conditions.

Teorema 16.1 If X is an optimal solution of (16.21) and the gradients of all the active constraints in
X are linear independent, then in X the KKT conditions holds.

Proof Let u be the vector of the Lagrange multipliers associated to the equality constraints (16.21b)
and λ and μ the corresponding multipliers associated to the lower and the upper bounds from (16.21c),
respectively. The Lagrange function of the problem (16.21) is

L ¼ f ðx, yÞ þ uT cðx, yÞ þ λTx ðx  lx Þ þ λTy y  ly þ μTx ðux  xÞ þ μTy uy  y ,

582 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

where λy and μy are the subvectors from λ and μ corresponding to the basic variables, respectively, and
similarly for λx, μx for the nonbasic variables.
The KKT conditions written in terms of the x and y variables are as follows:
∂f ∂c
þ uT þ λy  μy ¼ 0, ð16:35Þ
∂y ∂y

∂f ∂c
þ uT þ λx  μx ¼ 0, ð16:36Þ
∂x ∂x
λ  0, μ  0, ð16:37Þ

λTx ðx  lx Þ ¼ λTy y  ly ¼ 0,

ð16:38Þ
μTx ðux  xÞ ¼ μTy uy  y ¼ 0:

If X is an optimal solution of (16.21), then there exist the vectors u, λ, and μ, which together with
X verify the relations (16.35, 16.36, 16.37, and 16.38). Since y lies between its bounds, from (16.38) it
follows that λy ¼ μy ¼ 0, that is from (16.35) we have
 1
∂f ∂c
u¼ :
∂y ∂y

Therefore, u defined by (16.33) is exactly the vector of the Lagrange multipliers associated to the
equalities constraints (16.21).
The condition (16.36) can be written as
∂f ∂c
þ uT ¼ μx  λx : ð16:39Þ
∂x ∂x
Observe that the left-hand side of (16.39) is r, the generalized reduced gradient of the problem
(16.21). With this, from (16.38) it follows that in the minimizing point the generalized reduced
gradient verifies
8
< r i ¼ λxi  0,
> if xi ¼ lxi ,
ri ¼ μxi  0, if xi ¼ uxi , ð16:40Þ
>
:
ri ¼ 0, if lxi < xi < uxi :

In order to see the connection between (16.39) and the reduced problem (16.24), we note that, if xi
is strictly between its bounds, then λxi ¼ μxi ¼ 0 and therefore from (16.39), we have

∂F
 r i ¼ 0: ð16:41Þ
∂xi

If xi ¼ lxi , then μxi ¼ 0, and therefore

∂F
 r i ¼ λxi  0: ð16:42Þ
∂xi

On the other hand, if xi ¼ uxi , then λxi ¼ 0, that is

∂F
 r i ¼ μxi  0: ð16:43Þ
∂xi
16.6 The Generalized Reduced Gradient Method (GRG) 583

But (16.41, 16.42, and 16.43) are exactly the optimality conditions associated to the reduced
problem (16.24) expressed in terms of the generalized reduced gradient. Therefore, the KKT
optimality conditions for the problem (16.21) can be viewed as the optimality conditions of the
reduced problem (16.24), where u from (16.34) is the vector of the multipliers associated to the
constraints from (16.21) ♦

Computation of the Search Direction and of the Stepsize

The KKT conditions for the optimality of X0 ¼ [x0 y0]T are given by the existence of the vectors
u 2 ℝm and r 2 ℝnm such that
∂f ∂c
þ uT ¼ r, ð16:44Þ
∂x0 ∂x0

∂f ∂c
þ uT ¼ 0, ð16:45Þ
∂y0 ∂y0
8
<  0,
> if x0i ¼ lxi ,
r i  0, if x0i ¼ uxi , ð16:46Þ
>
:
¼ 0, if lxi < x0i < uxi :

From (16.44) and (16.45) it follows that

 1
∂f ∂c
u¼ ,
∂y0 ∂y0
 1
∂f ∂f ∂c ∂c
r¼  :
∂x0 ∂y0 ∂y0 ∂x0

If r is computed in such a way as to satisfy (16.46), then X0 ¼ [x0 y0]T is a stationary point (or even
a minimizer if the problem has some convexity properties). On the other hand, if r does not satisfy
(16.46), then the vector of the nonbasic variables x is modified as
x ¼ x0 þ αd, ð16:47Þ

where d is the search direction and α is the stepsize.

The search direction d ¼ [d1 . . . dn  m]T for modifying the nonbasic variables is defined in the
terms of the generalized reduced gradient as

if x0i ¼ lxi and r i < 0,
di ¼ 0 ð16:48aÞ
if x0i ¼ uxi and r i > 0,

d i ¼ r i , if lxi < x0i < uxi : ð16:48bÞ

On the manifold V, in a neighborhood of [x0 y0]T a curve Γ can be defined as

x ¼ x0 þ αd, α  0,
y ¼ yðαÞ, y ð 0Þ ¼ y 0 ,
cðx0 þ αd, yðαÞÞ ¼ 0:
584 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

The tangent L at Γ in the point [x0 y0]T is the semiline

x ¼ x0 þ αd, α  0,
y ¼ y0 þ αh,
∂c ∂c
d þ h ¼ 0:
∂x ∂y

Notice that the stepsize α has to be limited so that

lx  x0 þ αd  ux ,
ly  y0 þ αh  uy ,

that is, α is defined in a nonempty interval, possibly infinite, 0  α  α: With this, a step can be taken
along the line L to arrive in the point Xe ¼ ½x1 ye1 T , where
1

x1 ¼ x0 þ α1 d,
ye1 ¼ y0 þ α1 h,

is definite by the stepsize α1 computed as solution of the problem

min f ðx0 þ αd, y0 þ αhÞ: ð16:49Þ
0αα

In general, the point Xe does not belong to the variety V. However, from Xe a new point
1 1

X1 ¼ [x1 y1]T 2 V belonging to the curve Γ can be computed. This new point can be
obtained by solving the nonlinear algebraic system

cðx1 , yÞ ¼ 0: ð16:50Þ

with respect to the basic variables y.

Obviously, any method for solving the nonlinear algebraic system can be selected for solving
(16.50), particularly the Newton method. For example, starting from ye1 , an iteration of the Newton
method is defined by
 1
∂c
Δyt ¼  cðx1 , yet Þ,
∂y ð16:51Þ
yetþ1 ¼ yet þ Δyt , t ¼ 1, 2, . . . :

Illustration of a Particular Case with m = 1 (A Constraint), n = 2, l = 0 and u = 1 1

Figure 16.1 shows the points Xe1 and X1. Suppose that the method selected for solving (16.50)
converges to the point X1 2 V \ P. Two cases have to be considered now:

A. If y1 obtained as solution of (16.50) satisfies ly < y1 < uy, then, from X1 we can continue the
iterations in the same way as from X0. This case is illustrated in Fig. 16.1.

B. On the other hand, if ðy1 Þr ¼ lyr or ðy1 Þr ¼ uyr for a certain index r, then the variable yr has to leave
the set of the basic variables and a nonbasic variable must enter the basis. This operation of
changing the basis is executed in the same way as in the simplex algorithm from linear program-
ming. This situation is illustrated in Figs. 16.2 and 16.3.
16.6 The Generalized Reduced Gradient Method (GRG) 585

Fig. 16.1 Computation of

X1 ¼ [x1 y1]T 2 V. The
point Xe 2 L corresponds
1

to α ¼ α1, while point X

corresponds to α ¼ α

Fig. 16.2 The iterations

for solving the system
(16.50) starting from the
point Xe lead to the point
1

Xb 2
2
=V

Fig. 16.3 The iterations

for solving the system
(16.50) starting from the
point Xe lead to the points
1

Xb , Xb , and then to X1
2 3
586 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

Obviously, for different reasons, the method for solving the system (16.50) in combination with
the algorithm for solving the optimization problem (16.21) can fail. The following situations may
happen:

1. The method is not convergent. For example, after a certain number of iterations, let us say 20, the
point ½x1 ye20 T is not on Γ or in a neighborhood of Γ. Or, after a few iterations, let us say
5, kcðx1 ye5 Þk  kcðx1 ye1 Þk:
2. For a certain iteration, yet determines f ðx1 yet Þ > f ðx0 y0 Þ:
 T
3. At a certain iteration, yetþ1 leads to a point x1 yetþ1 that is not in the parallelepiped P.

In the situations (1) and (2) above, the remedy is to reduce α1 (e.g., set α1/2 or α1/10) and from the
new point on L continue the iterations for solving the nonlinear system (16.50). In situation (3), let
 T
½x1 yet T 2 P: Then the segment of the line connecting the point ½x1 yet T with x1 yetþ1 crosses the
 T

frontier of P in a point x1 ybtþ1 such that for a certain index r, yetþ1 r ¼ lyr or yetþ1 r ¼ uyr : In this
situation, a changing of the basis is executed and with this new partition of X, continue the iterations
for solving the nonlinear system (16.50).
Figure 16.2 illustrates the situation in which, starting from the point Xe 2 L a new point
1

 T
Xb ¼ x1 ybtþ1 is obtained. Assume that a changing of the basis was executed, in which the variable
2

y is replaced by x2. Then, by solving (16.50) a new point X1 2 V is obtained, which is considered as a
starting point for the next iteration.
Another possible situation which can appear is illustrated in Fig. 16.3. In this case, starting from
the point Xe ¼ ½x11 x12 ye1 T , the method for solving the system (16.50) determines the point
1

Xb ¼ ½x11 x12 0 T from which, by a changing of the basis, the point Xb ¼ ½x11 0 0 T is obtained
2 3

and then the point X1 2 V is determined.

Algorithm 16.1 (Generalized reduced gradient)

1. Determine a feasible initial point X0 and set k ¼ 0

2. Compute the Jacobian matrix ∂c/∂Xk and Xk is partitioned as Xk ¼ [xk yk]T such that xk 2 ℝn  m,
yk 2 ℝm, ly < yk < uy and rank[∂c/∂yk] ¼ m
3. Compute the inverse of the matrix B ¼ [∂c/∂yk]
h i1
4. ∂f ∂c ∂f ∂c
Compute the multipliers u ¼  ∂y ∂y
and the generalized reduced gradient r ¼ ∂x k
þ u ∂xk
k k

5. Compute the search direction8from the space of the nonbasic variables d ¼ [d1 . . . dn  m]T, where
> 0,
< if ðxk Þi ¼ lxi and r i < 0,
d i ¼ 0, if ðxk Þi ¼ uxi and r i > 0,
>
:
r i , if lxi < ðxk Þi < uxi ,
for i ¼ 1, . . ., n  m
6. If d ¼ 0, stop; otherwise, compute the search direction in the space of the basic variables
h i1 h i
∂c ∂c
h ¼  ∂y ∂xk
d
k

7. Determine the maximum value α of α such that

lx  xk þ αd  ux ,
ly  yk þ αh  uy :
8. Compute α1 as solution of the problem: min f ðxk þ αd, yk þ αhÞ:
0αα

Compute the point Xe ¼ ½x1 ye1 where x1 ¼ xk þ α1d, ye1 ¼ yk þ α1 h

9. 1
16.7 GRG with Sequential Linear or Sequential Quadratic Programming (CONOPT) 587

10. Starting from ye1 compute a new feasible point Xk þ 1 by solving the nonlinear algebraic system
c(x1, y) ¼ 0 with respect to the variables y. Let yej be the iterations generated by a method for solving this
system (e.g., the Newton method), j ¼ 1, 2, . . . .
p as a fixed
Consider  arbitrary integer.

(a) If c x1 , yej  ε is not achieved for any j  p, then set α1 ¼ α1/2 and continue with step 9
 
(b) If f ðx1 , ye1 Þ < f x1 , yej for a certain j, that is the function f is not reduced along the iterations, then
set α1 ¼ α1/2 and continue with step 9
(c) Let y1 be the solution of the system c(x1, y) ¼ 0. If ly < y1 < uy, then set Xk þ 1 ¼ [x1 y1]T, k ¼ k þ 1
and continue with step 2; otherwise, execute step 11
11. If ðy1 Þi ¼ lyi or ðy1 Þi ¼ uyi for a certain index i, then make a change of variables in which the variable yi
leaves the basis and a nonbasic variable, not fixed at its bounds, enters the basis
12. Update the partition of X and the inverse of the basis B1 and continue with step 4. ♦

Some comments are as follows:

1. The criterion for stopping the iterations from step 6 of the algorithm can be implemented in a more
convenient way as |di| < εi, i ¼ 1, . . ., n  m, where εi > 0 are tolerances which depend on the size
of the components xi of the vector x.
2. If after two or three reductions of α1 the iterations of the Newton method implemented in step 10 of
the algorithm for solving the system c(x1, y) ¼ 0 with respect to the basic variables y fails, then a
change of variables is executed.
3. One of the most important operations is the computation of the inverse of the basic matrix
B ¼ [∂c/∂yk] in step 3 of Algorithm 16.1. The advanced procedures for the inverse of the basis
computation make a partition of the vector Xk in step 2 of Algorithm 16.1 in such a way that the
basic matrix B is nonsingular and as sparse as possible.

16.7 GRG with Sequential Linear or Sequential Quadratic Programming

(CONOPT)

CONOPT is a very interesting and profitable combination of the generalized reduced gradient with
sequential linear-programming and with sequential quadratic-programming. All these algorithms are
embedded into the generalized reduced gradient (GRG) scheme as described in (Drud, 1976, 1983,
1985, 1994, 1995, 1996, 2005, 2011). CONOPT implements some algorithmic novelties regarding
selection and factorization of the basis, line-search and the Newton iterations, sequential linear or
quadratic programming, computation of the generalized reduced gradient of the minimizing function,
advanced procedures for the inverse representation of a nonsingular matrix, etc.
CONOPT is a particularization of the generalized reduced gradient Algorithm 16.1, which
considers the general nonlinear optimization problems of the following form
min f ðxÞ
subject to ð16:52Þ
cðxÞ ¼ b,
l  x  u,

where x 2 ℝn, f : ℝn ! ℝ and c : ℝn ! ℝm are twice continuously differentiable functions, b 2 ℝm

and l, u 2 ℝn are simple bounds on variables. Suppose that n  m. Assume that li  ui, i ¼ 1, . . ., n.
588 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

Some of the bound values may be minus or plus infinity. If the problem has inequality constraints,
then these are transformed into equalities by using the slack variables. Moreover, in CONOPT it is
assumed that x includes slack and artificial variables for all the constraints. This implies that the
Jacobian of the constraints includes a unit matrix and therefore, it has full row rank.
CONOPT works as follows. The set of m equations c(x) ¼ b defining the (nonlinear) constraints of
the problem can be used to eliminate m variables. For the simplicity of the presentation, consider that
the vector x is partitioned in the sub-vectors xb 2 ℝm (basic) and xn 2 ℝnm (nonbasic). Therefore, the
system of the nonlinear constraints c(x) ¼ b can be rewritten as xb ¼ c1(xn).It is obvious that it is not
possible to find an analytic expression for the function c1, except for some very particular cases, like
the linear constraints. However, from the theory of the implicit functions (see Theorem A2.6), we

have the following result. If x0b , x0n satisfies the system c x0b , x0n ¼ 0 and the Jacobian ∂c/∂xb has

rank m, i.e., it is nonsingular, then in a neighborhood of x0b , x0n it is possible to transform c(xb, xn) ¼ 0
into xb ¼ c1(xn). The function c1 is differentiable and its Jacobian is found by implicit differentiation
as
∂c ∂c ∂c
dx ¼ dxb þ dxn ¼ 0: ð16:53Þ
∂x ∂xb ∂xn

Hence,
 1
∂c1 ∂xb ∂c ∂c
¼ ¼ : ð16:54Þ
∂xn ∂xn ∂xb ∂xn

In practice, the function xb ¼ c1(xn) is computed by solving the nonlinear system c(xb, xn) ¼ 0 with
respect to xb, xn being known.
The equation xb ¼ c1(xn) is introduced in the objective function to get
f ðxb , xn Þ ¼ f ðc1 ðxn Þ, xn Þ≜Fðxn Þ,

where F(xn) is differentiable with the derivative

 1
∂F ∂f ∂xb ∂f ∂f ∂f ∂c ∂c
¼ þ ¼  : ð16:55Þ
∂xn ∂xb ∂xn ∂xn ∂xn ∂xb ∂xb ∂xn

With these developments, (16.52) can be reformulated as

min Fðxn Þ
subject to
ð16:56Þ
xb ¼ c1 ðxn Þ,
lb < xb < ub , l n  x n  un :

Observe that the transformations that have been used are very similar to those from linear
programming. xb contains the basic variables. They are used to compensate for the changes in the
nonbasic variables xn so that the constraints c(x) ¼ b are satisfied. The derivative ∂F/∂xn from (16.55)
is similar to the vector of the reduced costs from linear programming. It shows the influence on the
objective function of changes in the nonbasic variables, taking into consideration the corresponding
changes in the basic variables. Usually, ∂F/∂xn is called the reduced gradient. The matrix ∂c/∂xb is
similar to the basis matrix from linear programming. However, the major difference between the
nonlinear problem and the linear problem is that in the optimal solution, the nonlinear problem can
have more than m variables strictly between the lower and the upper bounds. Thus, an optimization
16.7 GRG with Sequential Linear or Sequential Quadratic Programming (CONOPT) 589

method like the simplex algorithm that works only with basic solutions cannot be used in the
nonlinear case. The main steps of GRG in the implementation of CONOPT are as follows:

Algorithm 16.2 (CONOPT—Drud)

1. Find an initial feasible solution x0

2. Compute the Jacobian matrix ∂c/∂x. Partition x into xb 2 ℝm and xn 2 ℝn  m, so that lb < xb < ub and the
submatrix ∂c/∂xb is nonsingular
3. Compute the inverse (∂c/∂xb)1
4. Compute the Lagrange multipliers as solution
 of the system
∂c ∂f
∂xb
u ¼  ∂x
 
b

5. ∂f ∂c
Compute the reduced gradient r ¼ ∂x n
þu ∂xn
6. Compute the projection h of the reduced gradient on space defined by the simple bounds of the nonbasic
variables: 8
> 0, if ri < 0 and x0ni ¼ lni ,
<
hi ¼ 0, if ri > 0 and x0ni ¼ uni ,
>
:
ri , otherwise
7. If h ¼ 0, then stop

8. One-dimensional search. Choose α > 0 to minimize F x0n þ αh according to the following steps for
8 of α:
different values
>
< lni , if x0ni þ αhi < lni ,
(i) xni ¼ uni , if x0ni þ αhi > uni ,
>
: 0
xni þ αhi , otherwise
(ii) Compute xb ¼ c1(xn),
i.e., solve the system c(xb, xn) ¼ b subject to xb
(iii) Compute F x0n þ αh ¼ f ðxb , xn Þ
9. Store the best solution in x0 and go to step 2. ♦

The GRG algorithm is based on the same principle as the simplex algorithm for linear program-
ming: elimination of the basic variables. Locally, the problem is therefore reduced to an optimization
problem in nonbasic variables, which are lower and upper bounded. The main aspect of CONOPT is
that it uses the sequential linearization. Sequential linearization methods are defined as methods
based on solving a sequence of subproblems with linearized constraints and an augmented Lagrang-
ian as objective function.
Every step of the GRG algorithm contains more details from the linear programming and the
nonlinear optimization. The optimizing steps are specialized in several versions according to whether
the model appears to be almost linear or not. For almost linear models, some of the linear algebra
work involving Jacobian and basis matrices can be avoided or can be done by using cheap linear
programming updating techniques. The second-order information is not needed. The line-search can
be improved by observing that, like in linear programming, the optimal step will almost always be
determined by the first variable that reaches a bound. In this case, the algorithm uses the sequential
linear-quadratic programming. If the model is fairly nonlinear, other aspects can be optimized: the
set of basic variables will often remain constant over several iterations and other parts of the sparse
matrix algebra will take advantage of this. In this case, an improved search direction can be computed
by using the specialized sequential quadratic programming.
590 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

Some Details on the Algorithm

Firstly, in step 1 of Algorithm 16.2, as described by Drud (1994), CONOPT identifies equations that
involve only one variable that is not fixed at its bounds. It solves the equations for these variable and
removes the corresponding constraints. This procedure is applied repeatedly by removing the
constraints. Secondly, CONOPT tries to guess a set of basic variables that are away from the bounds
and uses this basis in some cheap Newton iterations. If the process does not converge quickly, then the
constraints with large residuals or the constraints that deviate from the linear model are taken out of
the process. In this way, CONOPT usually gets a point where the most constraints are satisfied.
Finally, a procedure is used in which the sum of the slacks in the remaining violated constraints is
minimized. This means that once a constraint has become satisfied, it remains satisfied for the rest of
the optimization. Thus, by using this approach CONOPT finds a good initial basis that will lead to
very few infeasible slacks and, therefore, involves a few iterations. In step 2 of the algorithm, the
Jacobian matrix is computed at the points where the nonlinear constraints are satisfied. Moreover, this
evaluation of the Jacobian is made only at the beginning of each line-search. This reduces the
computing effort. Selection of the variables involves many tactical decisions and has crucial influence
on the Jacobian matrix factorization is step 3. Step 4, where the Lagrange multipliers are determined,
involves the solving of a linear system which uses the factorization of the basic matrix from step
3. For each step in the line-search, in step 8 CONOPT adjusts the basic variables to get a feasible
solution. This involves repeated evaluations of the constraints residuals and solving with the basis
factors. In addition, if one or more of the basic variables approach a bound, extra work is needed to
find the exact stepsize at which the critical variable reaches the bound. To achieve feasibility,
CONOPT uses a Newton process. If it makes a large step and the Newton procedure fails to converge,
then it reduces the step and tries again. Since the derivatives are evaluated at a feasible point from
which the search starts, the Newton process will always converge for small steps, ignoring degener-
acy problems, taking only a few steps. In the following, let us describe some details on the steps of
CONOPT.

Selection and Factorization of the Basis

This is a very important step in the economy of the algorithm. During the optimization, the values of
the estimated basic variables are calculated and checked against their bounds. This process works
when the values of the basic variables are within the bounds for small changes in the nonbasic
variables. Therefore, the basic variables should preferably be away from their bounds and the basis
matrix should be in such a way that small changes in the numerical values corresponding to small
movements do not change the solution of the system c(xb, xn) ¼ 0 too much. In other words, the basis
matrix should be well conditioned. The basis must be factored, and the factors must be updated when
the matrix changes.
Intensive numerical tests referring to the selection and to the factorization of the basis have given
the following results.

• The condition of the basis is more important than its sparsity. A well-conditioned basis will
generally lead to fewer iterations because the Hessian of the reduced objective function also
becomes better conditioned.
• The set of the basic variables should not vary too much. In other words, from one iteration to
another one, the second-order information in the reduced Hessian approximation can be preserved
if the basis differs only by one variable before and after a change of the basis.
16.7 GRG with Sequential Linear or Sequential Quadratic Programming (CONOPT) 591

• At degenerate points, the basis must be very carefully controlled to avoid cycling. Nonlinear
models seem to behave like linear models: degeneracy becomes more accentuated as the models
grow in size.
• For poorly scaled problems, the factorization procedure by Hellerman and Rarick (1971, 1972) is
slow as many proposed pivots appear to be too small relative to other elements in the column, thus
resulting in many expensive column swaps.

Initially, in CONOPT the basis is LU factorized using a Markowitz (1957) pivot selection
procedure, similar to the one suggested by Suhl and Suhl (1990, 1991), which is an improvement
of the Hellerman and Rarick, where only updated spike columns are separately stored. The remaining
columns of the basis factors are represented by the original columns in the Jacobian, as described in
(Kalan, 1971). The main difference between the implementation given by Suhl and Suhl (1990) and
the one given by Drud (1994) is that in CONOPT, a block decomposition of the bump is performed
and the Markowitz criterion is applied to each individual block.
The Jacobian is not changed in the degenerate iterations. Therefore, a cheap simplex-like update of
the basis factors similar to the one suggested by Reid (1982) is applied. This will reduce the length of
the spikes during factorization.
The Jacobian will change during the nondegenerate iterations. To prevent unnecessary
calculations of the Jacobian and expensive full factorizations after small steps, an estimation of the
quality of the Jacobian is performed in CONOPT. This estimation is based on the speed of
convergence of the Newton iterations in the line-search and on the change in the objective relative
to the change predicted by a linear model (see Drud (1994)).
CONOPT keeps the old set of basic variables after the iterations in which no basic variable reached
a bound. If the step was limited by a superbasic variable reaching a bound, CONOPT estimates the
quality of the Jacobian as mentioned above. If the Jacobian is good, then CONOPT keeps the
factorization of the basis. If it is poor or if the step was determined by nonlinearities, then the
Jacobian is recomputed. The subsequent factorization uses the same pivot sequence and sparsity
pattern determined during the last full factorization.
When the line-search behaves poorly because the Newton iterations do not converge even for
small steps, CONOPT tries to find a better conditioned basis. The procedure is to replace a basic
variable with a better superbasic. All the superbasic columns are updated with the active eta-vectors,
and the largest possible pivot is selected. To avoid cycling, the basis is changed only if the pivot is
greater than one. The heuristic is based on the assumption that a large determinant indicates a well-
conditioned basis, a reasonable assumption for well-scaled problems.

Line-Search and Newton Iteration

The selection of the search direction in CONOPT involves two parts:

• The selection of the set of the superbasic variables to be changed in the coming iteration
• The determination of a direction for the superbasic variables, given by the reduced gradient

Concerning the first part, the variables between their bounds are always superbasics. A variable at
a bound is only made superbasic if its reduced gradient has the right sign, it is greater than an absolute
tolerance and it is large relative to other reduced gradients. In the second part, two modes are
identified to determine a direction for the superbasics: the linear mode and the nonlinear mode.
In the linear mode, it is expected that a linear approximation to the problem is a good one and that
the stepsize in the coming line-search will be determined by bounds. The superbasic variables are
592 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

moved in the direction of the negative reduced gradient, i.e., in the steepest descent direction.
CONOPT immediately tries to go as far as possible until a basic or a superbasic variable has reached
a bound. This point is used as the initial point in the Newton iterations. The linear mode has two
advantages. It helps save the overhead of the quasi-Newton procedure in areas where it is not needed
and it can accelerate the line-search significantly by going directly to the bound.
In the nonlinear mode, the line-search is expected to be limited by the nonlinearity of the problem,
i.e., it is expected that no variable will reach a bound. The search direction for the superbasic variables
is determined from a quasi-Newton method that also determines the initial stepsize suggested to the
line-search. For generating search directions in the nonlinear mode, a quasi-Newton method based on
BFGS is used. With the growing size of the problems, some kind of conjugate gradient method is used
to handle problems with a very large number of superbasics.
Firstly, CONOPT uses the linear mode and switches from the linear mode to the nonlinear mode
after a number of consecutive line-searches in which no variable has reached a bound. After that, it
switches back from the nonlinear to the linear mode after a number of consecutive line-searches in
which a variable has reached a bound.

The Newton Iterations

In CONOPT, the line-search has the following three components: (i) the Newton algorithm that
restores the feasibility for the given values of the superbasic variables and for the given estimate of
the basic variables, (ii) the computation of the initial estimates for the basic variables and the
management of the bounds on the superbasic and basic variables, and (iii) the computation of the
stepsize and the implementation of the termination criteria.
For each step in the one-dimensional search, the nonlinear system c(xb, xn) ¼ b must be solved
subject to xb for xn fixed. Since this system must be solved several times, this is the most expensive
component of CONOPT. It is important not to solve this system more accurately than necessary and
not to spend too much time trying to solve it. Finally, the output of a line-search procedure consists of
the value of the objective function and of the values of the basic variables. At all the intermediate
points, only the objective function value is used to determine a new stepsize. The values of the basic
variables are used to determine new initial values for the next Newton iteration. Therefore, the main
criterion for stopping the iteration is based on the error df, where df is the change that would occur
in the objective function if the residuals of the system c(xb, xn) ¼ b were all reduced to zero. This can
be approximated by df ¼ Δxj ¼ eTj Δxb ¼ eTj J 1 b z ¼ u z, where u is the vector of the Lagrange
T

multipliers (computed in step 4 of CONOPT) and z ¼ b  c(xb, xn) is the current vector of the
residuals. The convergence rate of the Newton algorithm is monitored as follows. If kzk1  0.001n or
kzk1(kzk1/kzoldk1)p < 0.001n, then the Newton iterates are convergent and the line-search continues
ahead. Otherwise, the Newton algorithm terminates. (zold is the residual from the previous iteration
and p is an integer parameter which is modified along the iterations.) If the residuals are small, then
the approximation of df is supposed to be good enough and therefore the test of the rapid convergence
on df can be applied: if |df|  rtobj, then the Newton iterations are successfully stopped. Otherwise, if
|df|(|df|/|dfold|)p > rtobj, then the Newton iterations are stopped with failure and the initial stepsize
must be decreased. This will reduce the number of the Newton iterations as well as the number of
evaluations of the functions. The basic variables are computed as xb ¼ xb þ Δxb, where Δxb is the
solution of the linear system JbΔxb ¼ z which uses the same inverse as in the computation of the
reduced gradient (see step 5). If this new xb overflows a bound by more than a predefined threshold,
then the Newton algorithm is stopped. No solution is expected to exist within the bounds and
therefore the algorithm returns to the stepsize procedure in which a new and smaller step is chosen.
16.7 GRG with Sequential Linear or Sequential Quadratic Programming (CONOPT) 593

Otherwise, the corresponding basic variable is fixed at its bound. Some details including the Newton
algorithm used in CONOPT are given in (Drud, 1985).

The Linear Mode—Sequential Linear Programming

When the problem appears linear, CONOPT will often take many small steps, each determined by a
new variable reaching a bound. Although the line-search is fast in the linear mode, each step requires
one or more evaluations of the nonlinear constraint and the overall cost may become high relative to
the progress. In order to avoid the many nonlinear constraints evaluations, CONOPT may replace the
steepest descent direction with a sequential linear programming (SLP) technique, to find a search
direction that anticipates the bounds on all the variables and therefore gives a larger expected change
in the objective function in each line-search. The search direction and the last basis from the SLP
procedure are used in an ordinary GRG-type line-search in which the solution is made feasible at each
step. The SLP procedure is only used to generate good directions. CONOPT uses a test which
determines whether SLP is used or not. This test is based on the progress of the algorithm to the
solution. In other words, in CONOPT the feasibility is conserved at each iteration.
When the optimization is in the linear mode, then a steepest descent algorithm is used to determine
the search direction. As an alternative, CONOPT implements the steepest edge algorithm (Reid,
1975; Goldfarb, 1976; Goldfarb, & Reid, 1977; Andrei, 1999a). The idea, taken from linear program-
ming, is to scale the nonbasic variables according to the Euclidian norm of the updated column, the
so-called edge length. A unit step for a nonbasic variable will give rise to changes in the basic
variables proportional to the edge length. A unit step for a nonbasic variable with a large edge length
will therefore give large changes in the basic variables. This step has two adverse effects relative to a
unit step for a nonbasic variable with a small edge length: a basic variable is more likely to reach a
bound after a very short step length and the large change in basic variables is more likely to give rise
to larger nonlinear terms. The steepest edge algorithm has been very successful in linear program-
ming (Goldfarb, & Reid, 1977). Also, this technique is profitable in nonlinear optimization, leading to
fewer iterations for most nonlinear problems (Drud, 2005). However, the cost for maintaining the
edge lengths can be more expensive in the nonlinear case. Therefore, the steepest edge procedure is
mainly useful during the linear mode iterations.

The Nonlinear Mode—Sequential Quadratic Programming

As we have already seen in the GRG algorithm, when the progress toward the solution is given by
nonlinearities (and not by bounds), then CONOPT executes many small steps which determine small
variations of the superbasic variables and also small variations of the reduced gradient. CONOPT can
use the second-order information to make good progress and to determine if the bounds should be
active or not. The second-order information is used in the sequential quadratic programming (SQP)
procedure that, like in the SLP procedure, finds a good search direction and a good basis while
maintaining the feasibility of the steps. Therefore, CONOPT is an algorithm which generates feasible
iterations. Like in the SLP case, CONOPT uses a test which determines if it should use the SQP
procedure or not. The SQP procedure can be inhibited by assigning a value to a special parameter. If
the matrix of the second-order derivatives becomes too dense, then CONOPT skips over the SQP
procedure. A dense matrix of the second derivatives will need more memory and will make the SQP
iterations so slow that the possible saving in the number of iterations is wasted in the computation and
manipulation of the Hessian. As in the linear mode where the sequential linear programming is used,
in the nonlinear mode the sequential quadratic programming generates a scaled search direction and
the expected step length in the following line-search is therefore 1. However, the step length may be
less than 1 for several reasons: the line-search is ill-behaved, a basic variable reaches a bound before
594 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

predicted by the linear constraints, the objective is much more nonlinear along the search direction
than expected, and the optimal step is not 1. For the superbasic variables, the search direction is
determined by using the quasi-Newton BFGS updating.
In (Drud, 2005, 2011), plenty of aspects on CONOPT are presented and discussed, namely, hints
on good model formulation, algebraic information on initial point determination, preprocessing,
scaling, finding a feasible solution, linear and nonlinear modes, SLP procedure, SQP procedure,
etc. A detailed comparison between sequential linearization methods (e.g., MINOS) versus CONOPT
is presented in (Drud, 1994) with respect to finding the first feasible solution, maintaining the
feasibility of iterations, computing the Jacobian matrix, the choice and factorization of the basis,
the Lagrange multipliers computation, the line-search and the Newton iterations, etc. CONOPT
represents a very advanced combination of three algorithms which integrate sequential linear
programming and sequential quadratic programming, both of them being embedded in the
generalized reduced gradient technology. The algorithm is improved with many computational
procedures based on numerical observations obtained in the process of solving large classes of
different nonlinear optimization problems of different dimensions and complexities.

Numerical Study—CONOPT: Solving Applications from the LACOP Collection

Table 16.1 shows the performances of CONOPT in case of small-scale nonlinear optimization
applications included in the LACOP collection (see Appendix C). Table 16.2 presents the
performances of CONOPT for solving the application PENICI from the LACOP collection, while
Table 16.3 shows the performances of CONOPT for large-scale nonlinear optimization applications.

Table 16.1 Performances of CONOPT for solving 12 applications from the LACOP collection. Small-scale nonlinear
optimization applications

n me mc #nj #njn #nhd #nhs #it cpu vfo

ELCH 10 3 0 25 10 10 45 27 0.043 −47.7610909
ALKI 10 3 8 38 14 5 7 14 0.062 −1768.80696
PREC 8 0 6 18 14 8 8 27 0.037 3.9511634
PPSE 9 6 0 37 32 7 10 13 0.040 5055.011803
MSP3 13 0 15 50 31 3 12 36 0.038 97.5875096
MSP5 16 0 21 89 71 5 19 21 0.040 174.7869944
POOL 34 20 0 115 78 0 63 12 0.037 2569.80
TRAFO 6 0 2 19 18 4 15 16 0.038 135.0759628
LATHE 10 1 14 53 37 8 10 36 0.049 −4430.08793
DES 150 50 0 301 300 150 11175 37 0.169 1055.182314
CSTC 303 200 0 1304 1303 303 202 28 0.211 3.4800747
DIFF 396 324 0 1621 0 0 0 5 0.036 0.0

In Table 16.1, we have the following: n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the
number of inequality constraints, #nj ¼ the number of nonzeros in the Jacobian matrix, #njn ¼ the number of nonlinear
nonzeros in the Jacobian matrix, #nhd ¼ the number of nonzero diagonal elements of the Hessian to the Lagrangian,
#nhs ¼ the number of nonzero sub-diagonal elements of the Hessian to the Lagrangian, #it ¼ the number of iterations,
cpu ¼ the CPU computing time for solving the problem (seconds), vfo ¼ the value of the objective function at the
solution.

Table 16.2 Performances of CONOPT for solving the PENICI application from the LACOP collection

n me mc #nj #njn #nhd #nhs #it cpu vfo

PENICI 707 602 0 3908 3305 303 1010 2243 20.07 113.990
16.7 GRG with Sequential Linear or Sequential Quadratic Programming (CONOPT) 595

Table 16.3 Performances of CONOPT for solving 6 applications from the LACOP collection. Large-scale nonlinear
optimization applications

n me mc #nj #njn #nhs #it cpu vfo

HANG 1002 501 0 3504 1503 501 22 0.27 5.0685777
2002 1001 0 7004 3003 1001 19 0.379 5.0685101
FLOW 1163 735 0 4979 312 0 5 0.058 0.0
FLOWO 1556 1005 0 6714 400 0 7 0.078 0.0
POL 4004 3000 0 18002 8000 0 33 0.232 14.139480
6004 4500 0 27002 12000 0 38 0.525 14.142097
8004 6000 0 36002 16000 0 41 0.760 14.143172
10004 7500 0 45002 20000 0 45 1.055 14.141714
CAT 3003 2000 0 12003 12000 2002 82 0.763 −0.048055
6003 4000 0 24003 24000 4002 81 2.136 −0.048055
9003 6000 0 36003 36000 6002 32 1.242 −0.048053
CONT 2505 2000 0 10005 4 0 21 0.109 1.013238
5005 4000 0 20005 4 0 30 0.270 1.005922
7505 6000 0 30005 4 0 31 0.480 1.004561
10005 8000 0 40005 4 0 33 0.767 1.004071

Table 16.4 Comparison between KNITRO/ACTIVE and CONOPT

#itt cput
KNITRO/ACTIVE 1123 466.084
CONOPT 520 9.124

It is important to make comparisons. The numerical experiments presented in Table 15.7

(KNITRO/ACTIVE) (option 3) and Table 16.3 (CONOPT) for solving all the 15 large-scale nonlin-
ear optimization applications are used in Table 16.4, which shows the total number of iterations (#itt)
and the total CPU computing time (cput), (seconds) needed by KNITRO/ACTIVE and CONOPT,
respectively, to get a solution of these applications.
It is worth noting the behavior of CONOPT when the procedure of the search direction computed
by the sequential quadratic programming is inhibited. In other words, the search direction is
determined by means of the reduced gradient. Table 16.5 contains the performances of CONOPT
with the sequential quadratic programming inhibited.
A comparison between CONOPT with sequential quadratic programming (Table 16.3) versus
CONOPT without sequential quadratic programming (Table 16.5) is given in Table 16.6, where #itt is
the total number of iterations and cput is the total CPU computing time (seconds) for solving all the
15 large-scale nonlinear optimization applications.
Observe that CONOPT equipped with sequential quadratic programming for the line-search
determination is much better than its variant without sequential quadratic programming. A simple
analysis of Tables 16.3 and 16.5 shows that the nonlinear optimization applications in the linear mode
(POL and CONT) are indifferent subject to the second-order information used by the SQP procedure.
Table 16.7 includes the performances of CONOPT with SQP for solving the application HANG
with a large number of variables.
Table 16.8 shows the performances of CONOPT with SQP inhibited for solving the application
HANG with a large number of variables.
596 16 Primal Methods: The Generalized Reduced Gradient with Sequential Linearization

Table 16.5 Performances of CONOPT for solving 6 applications from the LACOP collection with SQP inhibited.
Large-scale nonlinear optimization applications

n me mc #it cpu vfo

HANG 1002 501 0 81 0.269 5.6857779
2002 1001 0 151 0.838 5.0685101
FLOW 1163 735 0 5 0.058 0.0
FLOWO 1556 1005 0 7 0.066 0.0
POL 4004 3000 0 33 0.250 14.139480
6004 4500 0 38 0.507 14.142096
8004 6000 0 41 0.759 14.143172
10004 7500 0 45 1.060 14.141714
CAT 3003 2000 0 283 4.965 -0.048055
6003 4000 0 157 3.108 -0.048055
9003 6000 0 75 3.441 -0.048053
CONT 2505 2000 0 21 0.113 1.013238
5005 4000 0 30 0.275 1.005922
7505 6000 0 31 0.465 1.004561
10005 8000 0 33 0.787 1.004071

Table 16.6 CONOPT with SQP versus CONOPT without SQP

#itt cput
CONOPT with SQP 520 9.124
CONOPT without SQP 1031 16.961

Table 16.7 Performances of CONOPT with SQP for solving the HANG application from the LACOP collection

n me mc #it cpu vfo

HANG 4002 2001 0 28 1.366 5.0684889
8002 4001 0 31 4.766 5.0684827

Table 16.8 Performances of CONOPT without SQP for solving the HANG application from the LACOP collection

n me mc #it cpu vfo

HANG 4002 2001 0 769 6.311 5.0684889
8002 4001 0 1658 24.901 5.0684828

Table 16.9 Comparison: MINOS, KNITRO/ACTIVE, SNOPT and CONOPT. CPU computing time (seconds)

MINOS KNITRO/ACTIVE SNOPT CONOPT

143.80 466.084 64.91 9.124

As a conclusion to these numerical studies, let us compare MINOS (Table 14.12), KNITRO/
ACTIVE (Table 15.7), SNOPT (Table 15.2), and CONOPT with SQP (Table 16.3) subject to the total
CPU computing time. Table 16.9 presents the performances of these algorithms for solving all the
15 large-scale nonlinear optimization applications with respect to the total CPU computing time.
16.7 GRG with Sequential Linear or Sequential Quadratic Programming (CONOPT) 597

CONOPT is one of the most elaborated algorithms for solving large-scale nonlinear optimization
problems and applications. It is based on the generalized reduced gradient implemented in the sparse
matrix technology by using sequential linear programming and sequential quadratic programming
algorithms. The best performances are obtained with its variant which implements the sequential
quadratic programming. CONOPT has one of the most advanced techniques for solving linear
algebraic systems of equations based on permuting the rows and columns of the matrix to the
“bump and spike” structure in order to minimize the number of the newly created nonzeros during
the factorization and to maintain the numerical stability.

Notes and References

CONOPT is a line-search algorithm described in a number of papers by Arne Drud (1976, 1983,
1985, 1994, 1995, 1996, 2005, 2011)—ARKI Consulting and Development A/S. Each of them gives
different computational details on the numerical linear algebra used in CONOPT. The algorithm
implements three active-set methods. The first one is a gradient projection method in the frame of the
generalized reduced gradient method that projects the gradient of the objective function onto a
linearization of the constraints. The second is a sequential linear programming algorithm and the
third is a sequential quadratic programming algorithm. CONOPT includes algorithmic switches that
automatically detect which method is the best. The search direction is computed in two modes. In the
linear mode, the progress toward solution is given by bounds. In the nonlinear mode, this progress is
given by nonlinearities. CONOPT is embedded in the GAMS technology (Andrei, 2017c).
For solving constrained nonlinear optimization problems, some approaches based on the gradient
have been designed. Among them, the most well known are the projected gradient method proposed
by J.B. Rosen (1960, 1961), the feasible direction method originally suggested and developed by
Guus Zoutendijk (1960), the reduced gradient method originally proposed by Philip Wolfe (1967) for
problems with linear constraints, the convex simplex method for problems with linear constraints
proposed by Willard Zangwill (1967), and the generalized reduced gradient method by Abadie and
Carpentier (1969). All these methods borrow some ideas from the simplex algorithm from linear
programming by changing the basis in specific ways and by taking into account the nonlinear
character of the objective and of the constraints.
Thus, the projected gradient method determines an initial basis by using the simplex algorithm. At
every iteration, this matrix is changed by eliminating and by introducing only one row from the
matrix of constraints. The search direction is obtained by projecting the negative gradient on the null
subspace of the basis. The convex simplex method changes the basis by modifying only one variable.
The search direction is computed by projection in three distinct cases. On the other hand, the reduced
gradient method changes the basis by simultaneously modifying more variables. In this method, the
search direction is obtained as the reduced gradient of the minimizing function at the subspace of the
nonbasic variables. The generalized reduced gradient method solves the original problem by solving a
sequence of nonlinear optimization subproblems with simple bounds specified in the subspace of the
nonbasic variables defined by the nonlinear constraints. The weakness of all these methods is that
they use only the information given by the projection of the negative gradient onto the null space of
the basis or the reduced gradient making no attempt to use the approximation of the minimizing
problem by sequential linear or by sequential quadratic programming in the current point. The most
advanced is the generalized reduced gradient method with linear or sequential quadratic program-
ming. The main components included only in this generalized reduced gradient method are an
advanced inverse of the basis representation under the product form of the inverse, advanced Newton
methods for solving nonlinear algebraic systems of equations, and the use of the sequential linear
programming or the sequential quadratic programming according to the situations in which the model
appears to be almost linear or not. This is one of the modern approaches for solving large-scale
nonlinear optimization problems.
 Interior-Point Methods
17

One of the most powerful methods for solving nonlinear optimization problems known as the interior-
point method is to be presented in this chapter. It is related to the barrier functions. The terms “interior-
point methods” and “barrier methods” have the same significance and may be used interchangeably.
The idea is to keep the current points in the interior of the feasible region. A method for remaining in
the interior of the feasible region is to add a component to the objective function, which penalizes close
approaches to the boundary. This method was first suggested by Frisch (1955) and developed both in
theoretical and computational details by Fiacco and McCormick (1964, 1966, and 1968).
Interior-point methods have two implementations. The first one uses the line-search technique and
the second one uses filters. The interior-point methods based on line-search are also classified into two
methods. The first one is a direct extension of the interior-point methods from linear programming.
They use line-searches to enforce convergence. The second one uses a quadratic model to define the
step and incorporates a trust-region constraint to provide stability. For these methods, we follow the
main developments from Nocedal and Wright (2006). The interior-point method with filter uses a
filter for selecting the stepsize in the direction of the calculated step (Fletcher, & Leyffer, 2002).
In this chapter, we consider the interior-point method for general nonlinear optimization problems
in the line-search and in the trust-region paradigms, respectively. At the same time, a variant of the
interior-point algorithm is presented, which illustrates a methodology for the theoretical development
of interior-point methods and of their convergence analysis.
Let us consider the problem

min f ðxÞ ð17:1aÞ

subject to
cE ðxÞ ¼ 0, ð17:1bÞ

cI ðxÞ  s ¼ 0, ð17:1cÞ

s  0, ð17:1dÞ

where x 2 ℝn, cE : ℝn ! ℝl, cI : ℝn ! ℝm, are twice continuously differentiable functions, and s 2 ℝm
are slack variables introduced to transform the inequality constraints cI(x)  0 into equalities. Here,
E ≜ {1, . . ., l} and I ≜ {1, . . ., m}.
As described in Nocedal and Wright (2006), in the following we present two derivations of
interior-point methods, which emphasize the main idea and the terminology for solving general

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 599
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_17
600 17 Interior-Point Methods

nonlinear optimization problems (17.1). The first derivation is based on continuation or homotopy
methods. The second one is based on barrier methods.
Homotopy methods. By homotopy method, we understand a continuous transformation of a
mathematical object into another one, in the sense that a simple (an easy) problem can be continu-
ously deformed into the given (hard) problem (Henri Poincaré). The solutions to the deformed
problems are related and can be tracked as the deformation proceeds. The function describing the
deformation is called a homotopy map. The basic idea of the homotopy approach is to replace the
original problem with a parameterized problem. At one end of the parameter range, there is a problem
which is easy to solve, while the original problem is at the other end.
The KKT optimality conditions for the problem (17.1) are as follows:

∇f ðxÞ  ∇cE ðxÞT y  ∇cI ðxÞT z ¼ 0, ð17:2aÞ

Sz  μe ¼ 0, ð17:2bÞ

cE ðxÞ ¼ 0, ð17:2cÞ

cI ðxÞ  s ¼ 0, ð17:2dÞ

with μ ¼ 0, together with

s  0, z  0: ð17:2eÞ

In (17.2), ∇cE(x) and ∇cI(x) are the Jacobian matrices of the equality and inequality constraints,
respectively. The vectors y and z are the Lagrange multipliers associated to these constraints. Let us
define S and Z as diagonal matrices whose diagonal elements are the components of the vectors s and
z, respectively. Observe that (17.2) with μ strictly positive are the perturbed KKT conditions. The
Eq. (17.2b) with μ ¼ 0 and the conditions (17.2e) introduce a combinatorial aspect of determining the
optimal active-set of (17.1). This is the main difficulty of this approach based on the KKT conditions.
The homotopy method consists in approximately solving the KKT conditions (17.2) for a sequence of
positive parameters {μk} that converges to zero, while maintaining s, z > 0. In the limit, we hope to
obtain a point that satisfies the KKT conditions (17.2). Moreover, asking for the iterates to decrease a
merit function or to be acceptable to a filter, we hope that the iterations converge to a minimizer of
(17.1) and not simply to a KKT point. The justification of this method is as follows. In a neighborhood
of (x, s, y, z) which satisfies the linear independence of the constraint qualification (Remark
11.2), the strict complementarity condition (see Definition 11.23), and the second-order sufficient
conditions (see Theorem 11.16), then, for all sufficiently small positive values of μ, the system (17.2)
has a locally unique solution denoted by (x(μ), s(μ), y(μ), z(μ)). All these points describe the so-called
primal-dual central path. When μ ! 0, this converges to (x, s, y, z), solution of the KKT
conditions given by (17.2).
Barrier methods. In this approach of the derivation of the interior-point methods, a barrier problem
is associated to (17.1) as
X
m
min x,s f ðxÞ  μ log ðsi Þ, ð17:3aÞ
i¼1
subject to
cE ðxÞ ¼ 0, ð17:3bÞ

cI ðxÞ  s ¼ 0, ð17:3cÞ
17.1 Prototype of the Interior-Point Algorithm 601

where μ is a positive parameter. Note that the inequality s  0 is not included in (17.3) because the
P
minimization of the barrier term μ m i¼1 log ðsi Þ from (17.3a) determines the components of s not to
become too close to zero. Observe that the reformulation by using (17.3) avoids the combinatorial
aspect of the nonlinear programs, but, as we can see, its solution does not coincide with the solution of
(17.1) for μ > 0. In this context, the barrier approach consists in finding an approximate solution of the
barrier problem (17.3) for a positive sequence {μk} that converges to zero.
It is worth comparing these two approaches based on the KKT conditions and on the barrier
problem. For this, let us write the KKT conditions for (17.3) as

∇f ðxÞ  ∇cE ðxÞT y  ∇cI ðxÞT z ¼ 0, ð17:4aÞ

μS1 e þ z ¼ 0, ð17:4bÞ

cE ðxÞ ¼ 0, ð17:4cÞ

cI ðxÞ  s ¼ 0: ð17:4dÞ

Looking at (17.2) and (17.4), we observe that they differ only in the second equations (17.2b) and
(17.4b). Notice that (17.4b) becomes very nonlinear near the solution as s ! 0. To eliminate this
difficulty, it is advantageous for the Newton method to transform the rational equation (17.4b) into a
quadratic one. This is done by multiplying (17.4b) by S, thus obtaining the quadratic term Sz. This is
always possible because all the diagonal elements of S are positive. With this simple transformation,
the KKT conditions for the barrier problem coincide with the perturbed KKT conditions given by
(17.2). We emphasize that the presence of the quadratic term Sz in (17.4), the simplest possible
nonlinearity introduced in the KKT system, makes the barrier method with all its defects be nontrivial
for solving the general nonlinear optimization problems (17.1).
Both the homotopy and the barrier interpretation of nonlinear optimization problems are useful.
The homotopy gives rise to the definition of the primal-dual directions, whereas the barrier view is
crucial in the design of globally convergent algorithms.
We mention that the interior-point derives from the fact that the early barrier methods introduced
by Fiacco and McCormick (1968) did not use slack variables and assumed that the initial point is
feasible with the inequality constraints. These methods used the barrier function f(x)  ∑i 2 I log (ci(x))
to prevent the iterates from leaving the feasible region defined by inequalities (see the logarithmic
penalty-barrier function (14.53) used in the penalty-barrier algorithm SPENBAR).

17.1 Prototype of the Interior-Point Algorithm

Let us consider the Lagrange function associated to the problem (17.1) as

Lðx, s, y, zÞ ¼ f ðxÞ  yT cE ðxÞ  zT ðcI ðxÞ  sÞ: ð17:5Þ

Now, applying the Newton method to the nonlinear system (17.2), in the variables x, s, y, z, the
following linear algebraic system is obtained:
2 2 32 3 2 3
∇xx L 0 J TE ðxÞ J TI ðxÞ dx ∇f ðxÞ  J TE ðxÞy  J TI ðxÞz
6 0 S 7 6 7 6 Sz  μe 7
6 Z 0 76 d s 7 6 7
6 76 7 ¼ 6 7, ð17:6Þ
4 J E ðxÞ 0 0 0 54 d y 5 4 cE ðxÞ 5
J I ðxÞ I 0 0 dz cI ðxÞ  s
602 17 Interior-Point Methods

where JE(x) ¼ ∇ cE(x) 2 ℝln and JI(x) ¼ ∇ cI(x) 2 ℝmn, are the Jacobian matrices of the equality
and inequality constraints, respectively.
The system (17.6) is called the primal-dual system. If it is solved, we get a solution d ¼ (dx, ds, dy,
dz) with which a new iteration (xþ, sþ, yþ, zþ) can be computed as

xþ ¼ x þ αsmax dx , sþ ¼ s þ αsmax ds , ð17:7aÞ

yþ ¼ y þ αzmax dy , zþ ¼ z þ αzmax dz , ð17:7bÞ

where

αsmax ¼ max fα 2 ð0, 1 : s þ αds  ð1  τÞsg, ð17:8aÞ

αzmax ¼ max fα 2 ð0, 1 : z þ αdz  ð1  τÞzg, ð17:8bÞ

with τ 2 (0, 1). The relations (17.8) for determining the stepsize, called the fraction to the boundary
rule, are designed to prevent the variables s and z from approaching their bounds of zero too quickly.
A typical value of τ is 0.995.
To establish the prototype of the interior-point algorithm an error function based on the perturbed
KKT system (17.2) is introduced
n  o
Eðx, s, y, z, μÞ ¼ max ∇f ðxÞ  J E ðxÞT y  J I ðxÞT z, kSz  μek, kcE ðxÞk, kcI ðxÞ  sk ,
ð17:9Þ

where k.k is an arbitrary vector norm.

This simple procedure described above is the basis of all the modern interior-point methods.
However, in order to get efficient algorithms, various modifications are needed. For example, a major
factor is the procedure for solving the primal-dual linear system (17.6). Another important procedure
is to choose the sequence of the barrier parameters {μk}. Of course, the determination of the initial
point for starting the algorithm is crucial, as is the procedure for the approximation of the Hessian of
the Lagrangian ∇2xx L.

Algorithm 17.1 Prototype of the interior-point algorithm

1. Choose the initial elements x0 and s0 > 0. Compute the initial values of the multipliers y0 and z0 > 0.
Choose an initial value for the barrier parameter μ0 > 0, as well as the parameters σ, τ 2 (0, 1). Set k ¼ 0
2. If a stopping test for the nonlinear optimization problem (17.1) is satisfied, stop; otherwise, go to step 3
3. If E(xk, sk, yk, zk, μk) > μk, then go to step 4; otherwise, go to step 8
4. Solve the primal-dual system (17.6) with respect to the searching direction d ¼ (dx, ds, dy, dz)
5. Using (17.8), compute αsmax and αzmax
6. Using (17.7), compute a new estimation of the solution (xkþ1, skþ1, ykþ1, zkþ1)
7. Set μkþ1 ¼ μk, k ¼ k þ 1 and go to step 3
8. Choose μk 2 (0, σμk) and go to step 2 ♦

Theorem 17.1 Let f(x), cE(x), and cI(x) be continuously differentiable functions. Suppose that the
Algorithm 17.1 generates a sequence of iterates {xk} and that {μk} ! 0. Then all the limit points xb of
the sequence {xk} are feasible. Moreover, if any limit point xb of the sequence {xk} satisfies the linear
independence constraint qualification (LICQ), then the first-order optimality conditions of the
problem (17.1) hold at xb:
17.2 Aspects of the Algorithmic Developments 603

Proof To simplify the proof, as in (Nocedal, & Wright, 2006), suppose that the problem contains
only the inequality constraints cI(x) and denote them as c(x). Let xb be a limit point of the sequence
{xk} and consider a subsequence fxkl g convergent to xb, i.e., fxkl g ! xb: If μk ! 0, then the error
function E given by (17.9) converges to zero, that is ðckl  skl Þ ! 0: By the continuity of c, this
implies that cb≜cðxbÞ  0, i.e., xb is feasible and skl ! b s ¼ cb: Now, suppose that the linear indepen-
dence constraint qualification holds at xb: Let A ¼ fi : cbi ¼ 0g be the set of the active constraints at xb:
= A, it follows that cbi > 0 and b
For any i 2 si > 0: Therefore, by the complementarity condition (17.2b) it
follows that ½zkl i ! 0: Since ∇f ðxkl Þ  J ðxkl ÞT zkl ! 0, we obtain
X
∇f ðxkl Þ  ½zkl i ∇ci ðxkl Þ ! 0: ð17:10Þ
i2A

But, by the constraint qualification hypothesis LICQ, the vectors f∇b ci : i 2 Ag are linear indepen-
dent. Hence, from (17.10) and from the continuity of ∇f(.) and ∇ci(.), i 2 A, it follows that the positive
sequence fzkl g converges to a value b z  0: Taking the limit in (17.10) we get
X
∇f ðxbÞ ¼ zi ∇ci ðxbÞ:
b
i2A

Moreover, we have cbTb

z ¼ 0, thus completing the proof. ♦

Observe that the theoretical basis of the interior-point methods is quite accessible. All that is
required is the continuity of the functions of the problem and of their gradients, as well as the LICQ
hypothesis. This is the reason why these methods have very good behavior in solving large-scale
nonlinear optimization problems.
As we have already said, the interior-point methods are classified into two classes. The first one is
based on the Algorithm 17.1 which is completed by adding a linear search, a strict control of the
reduction of the slack variables s and of the multipliers z, as well as the modification of the primal-
dual system (17.6) when negative curvature is encountered. The second class computes the step by
minimizing a quadratic model of the barrier problem (17.3) with respect to a trust-region constraint.
Both these approaches have many points in common. In the following lines, we describe some aspects
of the algorithmic development. Many other details may be found in (Nocedal, & Wright, 2006),
(Sun, & Yuan, 2006), (Forsgren, Gill & Wright, 2002), or (Wright, 1991).

17.2 Aspects of the Algorithmic Developments

This section presents some modifications and extensions of the Algorithm 17.1 that enable it to solve
large-scale nonconvex problems starting from any initial point.

Solving the Primal-Dual System

The most important issue is the linear algebra used for solving a large, sparse linear system at each
iteration in order to find the search direction. Actually, the sparse linear algebra module is by far the
most difficult component of Algorithm 17.1 (step 4). The system (17.6) can be rewritten in symmetric
form as
604 17 Interior-Point Methods

2 32 3 2 3
∇2xx L 0 J TE ðxÞ J TI ðxÞ dx ∇f ðxÞ  J TE ðxÞy  J TI ðxÞz
6 0 Σ I 7 6 7 6 z  μS1 e 7
6 0 76 ds 7 6 7
6 76 7 ¼ 6 7, ð17:11Þ
4 J E ðxÞ 0 0 0 54 dy 5 4 cE ðxÞ 5
J I ðxÞ I 0 0 dz c I ðxÞ  s

where Σ ¼ S1Z. Clearly, this symmetric form is suitable for using symmetric linear system solvers
which reduce the computational effort at each iteration. For solving the system (17.11), we can use
the direct methods (the factorization of the coefficient matrix, the Schur complement method, or the
null space method), as well as the iterative methods (the conjugate gradient method).
Observe that the system (17.11) can be further reduced by the elimination of the variable
ds ¼  Σ1dz þ Σ1(z þ μS1e) ¼  Σ1dz  Σ1z þ μZ1e, thus obtaining
2 2 32 3 2 3
∇xx L J TE ðxÞ J TI ðxÞ dx ∇f ðxÞ  J TE ðxÞy  J TI ðxÞz
6 76 7 6 7
4 J E ðxÞ 0 0 54 d y 5 ¼ 4 cE ðxÞ 5: ð17:12Þ
1 dz 1 1
J I ðxÞ 0 Σ cI ðxÞ  s þ Σ z  μZ e

This system can be factorized by the symmetric indefinite factorization method (Bunch, & Parlett,
1971), (Bunch, & Kaufman, 1977), (Cheng, 1998), (Cheng, & Higham, 1998). Let us denote the
matrix from (17.12) by M, then this factorization computes PTMP ¼ LBLT, where L is a lower
triangular matrix, B is a block-diagonal matrix with 1  1 or 2  2 blocks and P is a permutation
matrix for conserving the sparsity and for ensuring the numerical stability.
The system (17.12) can be further reduced by the elimination of the variable
dz ¼  ΣJI(x)dx  ΣcI(x) þ Σs  z þ μZ1e, thus obtaining
" 2 #" #
∇xx L þ J TI ðxÞΣ J I ðxÞ J TE ðxÞ dx
J E ðxÞ 0 d y
"  # ð17:13Þ
∇f ðxÞ  J E ðxÞy  J I ðxÞz þ J TI ðxÞ ΣcI ðxÞ  Σs þ z  μZ1 e
T T
¼ :
cE ðxÞ

If the number of the inequality constraints is large, then the dimension of the system (17.13) is
smaller than the dimension of the system (17.12). Moreover, the presence of the matrix J TI ðxÞΣ J I ðxÞ
in (17.13) dramatically modifies the structure of the sparse matrix of the Hessian ∇2xx L, which can be
rather unwanted for large-scale problems. A particular, favorable case is when the matrix J TI ðxÞΣ J I ðxÞ
is diagonal, case which appears when all the constraints are simple bounds.
We emphasize that the symmetric primal-dual system (17.11) or its reduced forms (17.12) or
(17.13) are ill-conditioned. The ill-conditioning is given by the presence of the matrix Σ ¼ S1Z.
When μ ! 0, some elements of Σ tend to þ1, while others tend to zero. However, because of the
special form in which this ill-conditioning arises, the direction computed by a stable factorization
method is usually accurate enough. Errors can appear only when the slacks s or the multipliers
z become very close to zero or when the Hessian ∇2xx L or the Jacobian matrix JE(x) are almost rank
deficient. This is the reason why the direct factorization methods are considered the most efficient and
reliable for computing the steps in the interior-point methods.
Cholesky factorization followed by triangular substitutions is a remarkable stable method to
compute solutions of linear systems with symmetric positive definite coefficient matrix. Usually,
Cholesky factorization of a sparse matrix produces fill-in that is some lower triangular locations in the
17.2 Aspects of the Algorithmic Developments 605

Cholesky factor contain nonzero elements even though the same locations in the original matrix are
zero (Ng, & Peyton, 1993). The problem of finding ordering (permutations) that reduces the fill-in is
too complicated, in fact being NP-complete (Rose, & Tarjan, 1975). Possible the best-known class of
ordering heuristics is the minimum degree (George, & Liu, 1989), (Mészáros, 1995). Other ordering
includes the minimum local fill and nested-dissection orderings (Rothberg, & Hendrickson, 1996).
Iterative techniques for solving the linear systems (17.11), (17.12), or (17.13) can also be used for
the search direction computation. In this case, preconditioners that cluster the eigenvalues of Σ must
be used (Nocedal, & Wright, 2006). For example, by introducing in (17.11) the change of variables
des ¼ S1 d s and multiply the second equation from (17.11) by S, then the term in Σ is transformed in
SΣS. Now, as μ ! 0, since Σ ¼ S1Z, it follows that all the elements of SΣS cluster around μI. Another
pffiffiffi
scaling like des ¼ Σ1=2 d s provides a perfect preconditioner. Indeed, des ¼ μS1 d s transforms Σ to
SΣS/μ, which converges to I when μ ! 0. To be efficient and robust, besides preconditioning that
removes the ill-conditioning determined by the barrier approach, the iterative methods must be
protected from possible ill-conditioning caused by the Hessian ∇2xx L or the Jacobinan matrices
JE(x) and JI(x).

Inertia and Singularity

It is known that the inertia of a matrix is a triplet of integers, which identifies the number of positive,
negative, and zero eigenvalues of the matrix. For the primal-dual system (17.11), the step p is a
descent direction if the matrix
" #
∇2xx L 0
ð17:14Þ
0 Σ

is positive definite on the null space of the matrix

 
J E ðxÞ 0
: ð17:15Þ
J I ðxÞ I

The positive definiteness condition holds if the inertia of the primal-dual matrix (17.11) is given by
(n þ m, l þ m, 0), where n is the number of variables, m is the number of inequality constraints, and
l is the number of equality constraints of the problem (17.1). In other words, if this matrix has exactly
n þ m positive eigenvalues, l þ m negative eigenvalues and no zero eigenvalues, then the positive
definiteness condition holds.

If the primal-dual matrix (17.11) does not satisfy this inertia condition, then this can be modified as
follows. Observe that the diagonal matrix Σ is positive definite by construction, but ∇2xx L can be
indefinite. Therefore, ∇2xx L can be replaced by the matrix ∇2xx L þ δI, where δ > 0 is a sufficiently
large parameter to ensure that the inertia of the modified matrix is exactly (n þ m, l þ m, 0).
Another problem that we face is when the matrix of the primal-dual system (17.11) is singular,
caused by the rank deficiency of the matrix JE. Observe that the matrix [JI  I] is always of full rank.
To overcome this singularity of the Jacobian JE, in the matrix of the primal-dual system (17.11) a
regularization parameter γ > 0 is introduced. Hence, taking into consideration the correction of inertia
and the protection to the singularity of the Jacobian of the equality constraints, the matrix of the
primal-dual system (17.11) can be modified as
606 17 Interior-Point Methods

2 3
∇2xx L þ δI 0 J TE ðxÞ J TI ðxÞ
6 Σ I 7
6 0 0 7
6 7: ð17:16Þ
4 J E ðxÞ 0 γI 0 5
J I ðxÞ I 0 0

Suppose that we have the current value of the barrier parameter μ, the constants η > 0 and β < 1 and
the value of the perturbation parameter δold from the previous iteration. Then, a procedure for
selecting the parameters γ and δ is as follows.

Algorithm 17.2 Inertia correction and regularization

1. In the matrix (17.16), set δ ¼ γ ¼ 0. Factorize the matrix (17.16)

2. If the matrix (17.16) is nonsingular and its inertia is (n þ m, l þ m, 0), then compute the primal-dual step;
stop
3. If the matrix (17.16) has zero eigenvalues, then set γ ¼ 108ημβ
4. If δold ¼ 0, then set δ ¼ 104; otherwise, set δ ¼ δold/2
5. Factorize the modified matrix (17.16)
6. If the inertia of this matrix is (n þ m, l þ m, 0),then set δold ¼ δ. Compute the primal-dual step by solving
the system (17.11) with modified matrix; stop. Otherwise, set δ ¼ 10δ and go to step 5 ♦

The Algorithm 17.2 is a variant established by Nocedal and Wright (2006) of a more elaborated
algorithm given by Wächter and Biegler (2006). The constants used in it are arbitrary. The purpose of
the algorithm is to drastically avoid modifications of the Hessian ∇2xx L while minimizing the number
of factorizations. Severe modifications of the matrix of the primal-dual system (17.11) deteriorate the
performance of the algorithm because the second-order information given by ∇2xx L is modified and, in
this case, the algorithm behaves like a steepest descent. The algorithm for inertia correction and
regularization is used at each iteration of the interior-point algorithm.

Update of the Barrier Parameter

It is important to update the barrier parameter in a proper way. The sequence of {μk} must converge to
zero, so in the limit, we can recover the solution of the problem (17.1). But, if μk is decreased too
slowly, then a large number of iterations is required by the algorithm. On the other hand, if μk is
decreased too quickly, then some slack variables s or multipliers z may approach zero prematurely,
thus slowing the progress of the algorithm to the solution of the problem. In the following, let us
present some strategies for the barrier parameter selection, which prove to be efficient in practice.

(a) The Fiacco-McCormick strategy fixes the values of the barrier parameter until the perturbed
KKT conditions (17.2) have been satisfied with some accuracy. Then the barrier parameter is
decreased as
μkþ1 ¼ σk μk , σk 2 ð0, 1Þ: ð17:17Þ

The parameter σk can be selected as a constant value, for example σk ¼ 0.2. However, it is
preferable to have a strategy for the selection of σk. One idea is to choose smaller and smaller values
for σk when the solution is being approached. If close to the solution, consider that σk ! 0 and the
parameter τ in (17.8) converges to 1. Then the interior-point algorithm has a superlinear convergence
17.2 Aspects of the Algorithmic Developments 607

rate. The Fiacco-McCormic strategy works well on many problems, but it is sensitive to the initial
point selection and to the scaling of the problem.

(b) The adaptive strategies for updating the barrier parameter are more robust in the case of nonlinear
problems. These strategies are based on complementarity exactly as in the linear programming,
and they modify the barrier parameter at each iteration k as

sTk zk
μkþ1 ¼ σk : ð17:18Þ
m
A strategy for selecting σk in (17.18) is based on the interior-point methods in linear programming.
Firstly, with μ ¼ 0 in (17.11) we solve this system to get the predictor direction (Δxaf, Δsaf, Δyaf,
Δzaf). With this, as in (17.8), determine αaf s and αz as the longest step that can be taken along the
af

affine scaling direction before violating the nonnegativity conditions (s, z)  0. These values are used
to compute μaf as the value of the complementarity along the affine scaling step, i.e.,
  
af T

sk þ αaf
s Δs zk þ αaf
z Δz
af
μaf ¼ : ð17:19Þ
m
With this, compute σk as
 3
μaf
σk ¼ : ð17:20Þ
sTk zk =m

Merit Functions and Filters for Step Acceptance

Let us assume that we have the primal-dual direction. The next step is to determine a stepsize along
this direction. For this, the merit functions or filters can be used. Since the interior-point methods can
be considered as methods for solving the barrier problem (17.3), then it is quite natural to define the
merit function Φ or the filter in terms of barrier functions. For example, we may use an exact merit
function of the form
X
m
Φσ ðx, sÞ ¼ f ðxÞ  μ log ðsi Þ þ σðkcE ðxÞk þ kcI ðxÞ  skÞ, ð17:21Þ
i¼1

where the norm in (17.21) can be l1 or l2 and σ > 0 is a penalty parameter updated, as described in
Sect. 15.3.

An interior-point algorithm with line-search is as follows. After the primal-dual step d has been
computed and the maximum step lengths αsmax and αzmax given by (17.8) have been determined,


execute a line-search by backtracking. This line-search determines the stepsizes αs 2 0, αsmax and


αz 2 0, αzmax , which provide a sufficient decrease of the merit function or ensure acceptability by
the filter. The new estimate of the solution is computed as

xþ ¼ x þ αs dx , sþ ¼ s þ αs d s , ð17:22aÞ

yþ ¼ y þ αz d y , zþ ¼ z þ αz dz : ð17:22bÞ
608 17 Interior-Point Methods

In case we use the filter, the pairs of the filter are formed by using the values of the barrier function
P
f ðx Þ  μ mi¼1 log ðsi Þ on one side and the constraint violations k(cE(x), cI(x)  s)k on the other side.
A step is accepted only if it is not dominated by any element in the filter (see Chap. 18).

Quasi-Newton Approximations of the Hessian

In the interior-point algorithm, the Hessian matrix ∇2xx L which appears in (17.11) (or in its reduced
variants (17.12) or (17.13)) can be approximated by means of the quasi-Newton methods BFGS or
SR1. For large-scale problems, the limited-memory BFGS method is recommended (Nocedal, 1980),
(Liu, & Nocedal, 1989). We emphasize that it is the approximation of the Hessian to the Lagrange
function (17.5) which is computed, not the Hessian of the barrier function (17.3), which is highly
ill-conditioned and changes rapidly.

If we apply the BFGS update formula, then we must have the pairs of vectors (Δx, Δl ), where Δx
and Δl are the variations of the solution and of the gradient of the Lagrange function. After computing
a step from (xk, sk, yk, zk) to (xkþ1, skþ1, ykþ1, zkþ1), these variations are defined as
 
Δxk ¼ xkþ1  xk , Δlk ¼ ∇x L xkþ1 , skþ1 , ykþ1 , zkþ1  ∇x Lðxk , sk , yk , zk Þ:

The BFGS update of Bkþ1 is defined as

Bk Δxk ðΔxk ÞT Bk Δlk ðΔlk ÞT

Bkþ1 ¼ Bk  þ : ð17:23Þ
ðΔxk ÞT Bk Δxk ðΔlk ÞT Δxk

Observe that the BFGS update (17.23) generates dense n  n matrices. Therefore, for large-scale
problems it is preferable to use the compact representation of the limited-memory BFGS update. This
representation is based on the following theorem (Nocedal, & Wright, 2006).

Theorem 17.2 Let B0 be a symmetric and positive definite matrix and assume that a number of k
pairs of vectors (Δxi, Δli), i ¼ 0, . . ., k  1, satisfy the condition (Δxi)TΔli > 0, i ¼ 0, . . ., k  1. Let Bk
be the matrix obtained by applying k BFGS updates with these vector pairs to B0, using (17.23). Then
" #1 " #
MTk B0 Mk Lk MTk B0
Bk ¼ B 0  ½ B 0 M k N k  , ð17:24Þ
LTk Dk N Tk

where the n  k matrices Mk and Nk are defined as

Mk ¼ ½Δx0 , . . . , Δxk1  2 ℝnk , ð17:25aÞ

N k ¼ ½Δl0 , . . . , Δlk1  2 ℝnk , ð17:25bÞ

and the k  k matrices Lk and Dk have the following form

(
ðΔxi1 ÞT Δlj1 , if i > j,
ðLk Þi,j ¼ ð17:26aÞ
0, otherwise,
h i
Dk ¼ diag ðΔx0 ÞT Δl0 , . . . , ðΔxk1 ÞT Δlk1 : ð17:26bÞ
17.2 Aspects of the Algorithmic Developments 609

This result is proved by induction. The conditions (Δxi)TΔli > 0, i ¼ 0, . . ., k  1, ensure that the
inverse of the matrix from (17.24) exists, thus this expression being well defined.
Like in L-BFGS, in this case, we keep the r most recent pairs (Δxi, Δli) and, at each iteration, this
set of vectors is updated by eliminating the oldest one and by adding a newly generated pair. During
the first r iterations, the updating procedure described in Theorem 17.2 can be used without any
modification. For the iterations k > r, the updating procedure must be modified to reflect the changing
of the set of the vector pairs (Δxi, Δli), i ¼ k  r, k  r þ 1, . . ., k  1. Now, let us define the
n  rmatrices Mk and Nk, by
Mk ¼ ½Δxkr , . . . , Δxk1  2 ℝnr , ð17:27aÞ

N k ¼ ½Δlkr , . . . , Δlk1  2 ℝnr : ð17:27bÞ

ðkÞ
Then, the matrix Bk is obtained by applying r updates to the basic matrix B0 ¼ δk I, where
δk ¼ 1/γ k and γ k ¼ (Δxk1)TΔlk1/(Δlk1)TΔlk1, to get
" #1 " #
δk MTk Mk Lk δk MTk
Bk ¼ δk I  ½ δk Mk N k  , ð17:28Þ
LTk Dk N Tk

where Lk and Dk are the r  r matrices defined by

(
ðΔxkr1þi ÞT Δlkr1þj , if i > j,
ðLk Þi,j ¼ ð17:29aÞ
0, otherwise,
h i
Dk ¼ diag ðΔxkr ÞT Δlkr , . . . , ðΔxk1 ÞT Δlk1 : ð17:29bÞ

After the new iteration xkþ1 has been computed, the matrix Mkþ1 is obtained by deleting Δxkr and
by adding the new Δxk. The matrix Nkþ1 is updated in a similar way. The advantage of this compact
representation is that the (2r  2r) matrix in the middle of (17.28) and which must be inverted is of
small dimensions. Usually, r, the number of stored pairs (Δxi, Δli), is very small: 3, 5, or 7 (rarely 9).
Therefore, the numerical effort for the inverse computation of this matrix is negligible.
Interior-point algorithms are implemented using the compact representation of the BFGS updating
to the Hessian of the Lagrange function (17.5). Moreover, since B is positive definite, assuming that
JE is of full rank, then the matrix of the primal-dual system (17.11) is nonsingular. Hence, the solution
of (17.11) can be simply obtained by the Sherman-Morrison-Woodbury formula.

Feasible Interior-Point Methods

In many nonlinear optimization applications, we need all the iterates generated by the interior-point
algorithm to be feasible with respect to some or to all the inequality constraints. Interior-point
methods provide a natural framework for designing feasible algorithms. If the current iterate
x satisfies the inequality constraint cI(x) > 0, then it is easy to adapt the primal-dual iteration given
by (17.11) so that the feasibility is conserved. After computing the step d, we let x+ ¼ x þ dx, redefine
the slacks as s+ ¼ cI(x+) and test whether the new point (x+, s+) is acceptable for the merit function. If
so, we define this point as the new iterate. Otherwise, we reject step d and compute a new one, a
shorter trial step. Clearly, in a line-search algorithm backtracking is used, while in a trust-region
method a new step is computed by reducing the trust-region radius.
610 17 Interior-Point Methods

17.3 Line-Search Interior-Point Algorithm

To describe the line-search interior-point algorithm, let us denote by DΦσ(x, s, d) the directional
derivative of the merit function Φσ in the direction d. The algorithm of this method is very close to the
prototype Algorithm 17.1, which contains two imbricate loops, the first being responsible for testing
the KKT optimality conditions and the second one for solving the primal-dual system (17.11). If in
the line-search a quasi-Newton method is used, then it is necessary to choose an initial symmetric and
positive definite matrix B0 which is updated along the iteration by quasi-Newton techniques.

Algorithm 17.3 Line-search interior-point algorithm

1. Choose x0 and s0 > 0. Compute the initial values of the multipliers y0 and z0 > 0. Choose an initial value
of the barrier parameter μ0 > 0, as well as the parameters σ, τ 2 (0, 1) and the tolerances εμ and εTOL.
Select an n  n matrix, symmetric and positive definite B0. Set k ¼ 0
2. If E(xk, sk, yk, zk, 0)  εTOL, then continue with step 3; otherwise, stop. The current point is the optimal
solution of the problem
3. If E(xk, sk, yk, zk, μ)  εμ, then go to step 4; otherwise, continue with step 10
4. Solve the primal-dual system (17.11) for the primal-dual direction d ¼ (dx, ds, dy, dz)
5. Using (17.8), compute αsmax and αzmax : Set dw ¼ [dx, ds]



6. Compute the stepsizes αs and αz satisfying αs 2 0, αsmax and αz 2 0, αzmax , as well as the condition
Φσ(xk þ αsdx, sk þ αsds)  Φσ(xk, sk) þ ηαsDΦσ(xk, sk, dw)
7. Using (17.22), compute a new estimation of the solution (xkþ1, skþ1, ykþ1, zkþ1)
8. If a quasi-Newton approach is used, update Bk
9. Set k ¼ k þ 1. Continue with step 3
10. Set μ ¼ σμ and update εμ. Continue with step 2 ♦

In step 4 of this algorithm, the matrix of the system (17.11) is very likely to be modified as in
(17.16). If the merit function can cause the Maratos effect, then a second-order correction or a
nonmonotone strategy can be implemented. The barrier tolerance εμ can be selected as εμ ¼ μ, as in
the prototype Algorithm 17.1.
To be reliable and to ensure the convergence, the algorithm must be equipped with some
protection mechanisms which substantially modify the line-search in step 6. These mechanisms
refer to a careful control of the decrease in the barrier parameter μ and in the (inner) convergence
tolerance εμ, and they let the parameter τ in (17.8) converge to 1 rapidly enough.
Another specific difficulty in interior-point methods is the lack of coordination between the step
computation and the satisfaction of the bounds on variables. An approach is to monitor the stepsizes
αs and αz in (17.7). If they are smaller than a given threshold, then the primal-dual step is replaced by a
step that guarantees progress in feasibility and improvement in optimality.
An alternative to using the merit function is to use a filter to perform the line-search. In a filter
method, when the stepsizes are very small, the feasibility restoration phase can be started (Fletcher, &
Leyffer, 2002).
17.4 A Variant of the Line-Search Interior-Point Algorithm 611

17.4 A Variant of the Line-Search Interior-Point Algorithm

In the following, we present a variant of a line-search interior-point algorithm for solving general
nonlinear optimization problems that include simple bounds on variables. We also point out some
very important details which emphasize the mathematical technology based on the interior-point
theory (Andrei, 1998b, 1998c).
Let us consider the problem

min f ðxÞ ð17:30aÞ

subject to
cE ðxÞ ¼ 0, ð17:30bÞ

cI ðxÞ  0, ð17:30cÞ

l  x  u, ð17:30dÞ

where f : ℝn ! ℝ, cE : ℝn ! ℝme, cI : ℝn ! ℝm are supposed to be twice continuously differentiable

functions. The bounds l 2 ℝn and u 2 ℝn may have any values with l  u. The idea of the algorithm
for solving (17.30) is to use the first-order KKT optimality conditions in an iterative computational
scheme of the Newton type, in which the inequality constraints are penalized in a barrier term. The
updating of the barrier parameter is given by a merit function. The stepsize is based on the Wolfe line-
search combined with the interior-point centrality and with the condition of limitation of the
convergence of the complementarity conditions to zero compared to the optimality conditions. We
mention that this mathematical technology is also found in (Kortanek, Potra, & Ye, 1991), (Goldfarb,
Liu, & Wang, 1991), (El-Bakry, Tapia, Tsuchiya, & Zhang, 1996), (Vanderbei, & Shanno, 1997,
1999) and (Gay, Overton, & Wright, 1997).

KKT Optimality Conditions

Considering the slack variables s 2 ℝm associated to the functional inequality constraints (17.30c)
and w, v 2 ℝn associated to the simple bounds (17.30d), the problem (17.30) is reformulated as

min f ðxÞ ð17:31aÞ

subject to
cE ðxÞ ¼ 0, ð17:31bÞ

cI ðxÞ  s ¼ 0, ð17:31cÞ

x  w ¼ l, ð17:31dÞ

x þ v ¼ u, ð17:31eÞ

s  0, w  0, v  0: ð17:31fÞ

Now, the inequality constraints (17.31f) are transferred into the objective function by a barrier
term, thus obtaining the problem
612 17 Interior-Point Methods

X
m X
n   X
n  
min f ðxÞ  μ log ðsi Þ  μ log wj  μ log vj ð17:32aÞ
i¼1 j¼1 j¼1
subject to
cE ðxÞ ¼ 0, ð17:32bÞ

cI ðxÞ  s ¼ 0, ð17:32cÞ

x  w  l ¼ 0, ð17:32dÞ

x þ v  u ¼ 0: ð17:32eÞ

The Lagrange function of (17.32) is

P
m P
n   P
n  
Lðx, s, w, v, y, z, p, q, μÞ ¼ f ðxÞ  μ log ðsi Þ  μ log wj  μ log vj
i¼1 j¼1 j¼1

y cE ðxÞ  z ðcI ðxÞ  sÞ  p ðx  w  lÞ  qT ðx þ v  uÞ,

T T T

ð17:33Þ

where y, z, p, and q are the Lagrange multipliers. Hence, the first-order optimality conditions are as
follows:

∇f ðxÞ  J E ðxÞT y  J I ðxÞT z  p þ q ¼ 0, ð17:34aÞ

cE ðxÞ ¼ 0, ð17:34bÞ

cI ðxÞ  s ¼ 0, ð17:34cÞ

x  w  l ¼ 0, ð17:34dÞ

x þ v  u ¼ 0, ð17:34eÞ

SZe  μe ¼ 0, ð17:34fÞ

WPe  μe ¼ 0, ð17:34gÞ

VQe  μe ¼ 0, ð17:34hÞ

where the matrices S, Z, W, P, V, and Q are diagonal with the elements si, zi, wi, pi, vi, and qi,
respectively. e ¼ [1, 1, . . ., 1]T. JE(x)and JI(x) are the Jacobian matrices of the equality cE(x) ¼ 0 and
of the inequality constraints cI(x)  0, respectively.
If in (17.34) the barrier parameter μ is set to zero, then the last three equations are exactly the
complementarity slackness conditions. Usually, these last three equations are called μ complemen-
tarity conditions. Observe that the KKT system (17.34) is a nonlinear algebraic one with 5n þ 2m þ me
equations and a similar number of unknowns, parameterized by the barrier parameter μ. For solving
this system, the Newton algorithm is fundamental, which is known to be very efficient near the
solution. Supposing that the system (17.34) has a solution, then for each μ > 0 we get a solution
(xμ, sμ, wμ, vμ, yμ, zμ, pμ, qμ). The path
17.4 A Variant of the Line-Search Interior-Point Algorithm 613

n  o
x μ , s μ , w μ , v μ , y μ , z μ , pμ , q μ : μ > 0

is called the primal-dual central path. The interior-point algorithm which will be presented in this
section is an iterative procedure. At each iteration, the algorithm attempts to move toward a point on
the central path closer to the optimal point than the current point is. If the barrier parameter μ is set to
zero, then the KKT conditions (17.34) can be partitioned into two classes:
2 3
∇f ðxÞ  J E ðxÞT y  J I ðxÞT z  p þ q
6 7
6 c E ðxÞ 7
6 7
COðtÞ ¼ 66 c I ð x Þ  s 7,
7 ð17:35Þ
6 7
4 xwl 5
xþvu
2 3
SZe
6 7
CT ðs, z, w, p, v, qÞ ¼ 4 WPe 5: ð17:36Þ
VQe

It is obvious that if we could have a point t ¼ [x, y, z, p, q, s, w, v]T to satisfy the system
 
COðtÞ
FðtÞ≜ ¼ 0, ð17:37Þ
CT ðs, z, w, p, v, qÞ

then the component x of this point would be the solution of the problem (17.30). The separation of the
optimality conditions into these two classes is crucial for the elaboration of an efficient algorithm for
solving (17.30) and also for proving its convergence.
For linear programming, the system (17.34) is much simpler. The only nonlinear expressions in
(17.34) are simple multiplications of the slack and dual variables, and the presence of these simple
nonlinearities makes the subject of linear programming nontrivial.

The Newton System

As above, denote t ¼ [x, y, z, p, q, s, w, v]T. Then, the Newton method applied to the system (17.37)
consists in the determination of the direction Δt ¼ [Δx, Δy, Δz, Δp, Δq, Δs, Δw, Δv]T as solution of
the following perturbed system:

F0 ðtÞΔt ¼ FðtÞ þ μb
e, ð17:38Þ

where eb 2 ℝ5nþ2mþme is a vector with zero components, except for the last 2n þ m ones, which are all
equal to one. F0(t) is the Jacobian of the function F(t) computed at the current point t. The new point is
computed as

tþ ¼ t þ αΔt, ð17:39Þ

where α is the stepsize.

Two problems are critical with this algorithm: the choice of α at each iteration and possibly, the
modification of the system (17.38) in order to find a local solution to it. For linear or convex quadratic
programming, the modification of (17.38) is not necessary, and the stepsize at each iteration is
determined by standard ratio tests (Lustig, Marsten, & Shanno, 1990, 1991, 1992, 1994), (Vanderbei,
614 17 Interior-Point Methods

1990, 1994, 2001), (Ye, 1997), (Wright, 1997). In the case of general nonlinear optimization
problems, it is well known that for a poor initial estimate of the solution, the Newton method may
diverge and so the linear system (17.38) needs to be modified once again. Besides, the technique for
choosing the stepsize α is more complex. In order to achieve the convergence to a solution of (17.34),
El-Bakry, Tapia, Tsuchiya, and Zhang (1996) introduced a merit function and showed that for a
proper choice of the barrier parameter μ there is a stepsize α such that the algorithm (17.39) is
convergent to a solution of (17.30) provided that the Jacobian F0(t) from (17.38) remains nonsingular
at each iteration (Shanno, Breitfeld, & Simantiraki, 1996), (Shanno, & Simantiraki, 1997). A
modification of the merit function introduced by El-Bakry, Tapia, Tsuchiya, and Zhang (1996) was
considered by Vanderbei and Shanno (1997). They used the l2 norm of the constraints multiplied by a
parameter β > 0 and proved that there is a value of β such that the direction given by the Newton
system is a descent direction for their merit function. The corresponding algorithm was implemented
as a nonlinear version of LOQO by Vanderbei (1995).

The Newton Direction Determination

The Newton system for direction determination involves the computation of the Jacobian matrix of
the function F(t) defined in (17.37). From (17.34) we get
2 32 3 2 3
K ðx, y, zÞ J E ðxÞT J I ðxÞT I I 0 0 0 Δx χ
6 J ðxÞ 0 07 6 7 6 7
6 E 0 0 0 0 0 76 Δy 7 6 cE ðxÞ 7
6 76 7 6 7
6 J I ðxÞ 0 0 0 0 I 0 0 7 6 Δz 7 6 cI ðxÞ þ s 7
6 76 7 6 7
6 0 0 0 I 0 7 6 7 6 7
6 I 0 0 76 Δp 7 6 l þ w  x 7
6 76 7¼6 7: ð17:40Þ
6 I 0 0 0 0 0 0 I7 6 Δq 7 6 u  v  x 7
6 76 7 6 7
6 76 7 6 7
6 0 0 S 0 0 Z 0 0 76 Δs 7 6 SZe þ μe 7
6 76 7 6 7
4 0 0 0 W 0 0 P 0 54 Δw 5 4 WP þ μe 5
0 0 0 0 V 0 0 Q Δv VQ þ μe

This system is not symmetric, but it is easy to be written in a symmetric form by multiplying the
second and the third equations by 1 and the last three equations by S1, W1, and V1, respectively.
After completing this very simple algebraic operation and after eliminating the variables Δs, Δw, and
Δv as

Δs ¼ μZ1 e  Se  Z1 SΔz, ð17:41aÞ

Δw ¼ μP1 e  We  P1 WΔp, ð17:41bÞ

Δv ¼ μQ1 e  Ve  Q1 VΔq, ð17:41cÞ

we get the following reduced Newton system:

2 3 2
32 3
K ðx, y, zÞ J E ðxÞT J I ðxÞT I I Δx χ
6 76 7 6 7
6 J E ðxÞ 0 0 0 0 76 Δy 7 6 cE ðxÞ 7
6 76 7 6 7
6 J I ðxÞ Z 1
S 0 7 6 Δz 7 ¼ 6 cI ðxÞ  μZ 1 e 7, ð17:42Þ
6 0 0 76 7 6 7
6 76 7 6 7
4 I 0 0 P1 W 0 54 Δp 5 4 x  l  μP1 e 5
I 0 0 0 Q1 V Δq u  x  μQ1 e

where
17.4 A Variant of the Line-Search Interior-Point Algorithm 615

X
me X
m
K ðx, y, zÞ ¼ ∇2 f ðxÞ  yi ∇2 cEi ðxÞ  zi ∇2 cIi ðxÞ, ð17:43Þ
i¼1 i¼1

χ ¼ ∇f ðxÞ þ J E ðxÞT y þ J I ðxÞT z þ p  q: ð17:44Þ

As above, JE(x) ¼ ∇ cE(x) and JI(x) ¼ ∇ cI(x), are the Jacobian matrices of the equality constraints
cE(x) ¼ 0 and of the inequality constraints cI(x)  0, respectively.
Observe that the matrix of the reduced Newton system (17.42) is symmetric of order (3n þ m þ me).
By solving this system and by using (17.41), we get the search direction Δt ¼ [Δx, Δy, Δz, Δp, Δ
q, Δs, Δw, Δv]T. The existence of a solution for (17.42) implies the existence of the inverse of the
diagonal matrices Z, P, and Q, which assumes a certain condition of initialization of the algorithm.
The standard Newton method assumptions are as follows (Dennis, & Schnabel, 1983), (Andrei,
1998c).

(i) There is a solution for the problem (17.30), and the associated dual variables satisfy the KKT
conditions (17.34).
(ii) The Hessian matrices ∇2f(x), ∇2cEi(x), i ¼ 1, . . ., me, ∇2cIi(x), i ¼ 1, . . ., m, exist and are locally
Lipschitz continuous at x.
(iii) The set of vectors {∇cE1(x), . . ., ∇cEme(x)} [ {∇cIi(x), i 2 A(x)} is linearly independent,
where A(x) is the set of the active inequality constraints at x.
(iv) For every vector d 6¼ 0 satisfying ∇cEi(x)Td ¼ 0, i ¼ 1, . . ., me, and ∇cIi(x)Td ¼ 0, i 2 A(x), we
have dTK(x, y, z)d > 0.
   
(v) For i ¼ 1, . . ., m, zicIi(x) > 0 and for j ¼ 1, . . ., n, pj xj  lj > 0 and qj uj  xj > 0:

Proposition 17.1 Suppose that the conditions (i)-(v) hold and s ¼ cI(x), w ¼ x  l and v ¼ u  x.
Then the Jacobian matrix F0(t) of function F(t) given by (17.40) is nonsingular.

Proof Consider the reduced problem in which only the inequality constraints active in the minimizer
point are used. Then, from the theory of the equality constraints optimization it follows that the
matriceal block given by the first five rows and columns of the matrix (17.40) is nonsingular. Hence,
the nonsingularity of the matrix from (17.40) follows from the strict complementarity condition (v)
and from the nonsingularity of the matriceal block from (17.40). ♦

Generally, in current implementations the reduced Newton system (17.42) is not used in the form
in which it appears. Usually, the reduction is continued by the elimination from (17.42) of the
variables Δz, Δp and Δq as

Δz ¼ μS1 e  S1 ZcI ðxÞ  S1 Z J I ðxÞΔx, ð17:45aÞ

Δp ¼ μW 1 e  W 1 Pðx  lÞ  W 1 PΔx, ð17:45bÞ

Δq ¼ μV 1 e  V 1 Qðu  xÞ þ V 1 QΔx, ð17:45cÞ

Hence, the system (17.42) is further reduced as

" #   
K J E ðxÞT Δx r þ μr
¼ , ð17:46Þ
J E ðxÞ 0 Δy cE ðxÞ

where
616 17 Interior-Point Methods

K ¼ K ðx, y, zÞ þ J I ðxÞT S1 Z J I ðxÞ þ W 1 P þ V 1 Q, ð17:47aÞ

r ¼ χ  J I ðxÞT S1 ZcI ðxÞ  W 1 Pðx  lÞ þ V 1 Qðu  xÞ, ð17:47bÞ

r ¼ J I ðxÞT S1 e þ W 1 e  V 1 e: ð17:47cÞ

Now, by solving (17.46) we get the variables Δx and Δy with which the rest of the variables are
immediately obtained from (17.45) and (17.41). Notice that, as in linear programming, we must solve
a symmetric undefined algebraic system, but in this case the Jacobian and the Hessian matrices of the
functions defining the problem are involved, which is much more complicated.

The Merit Function

By solving the system (17.46) and by using (17.45) and (17.41), we get a direction Δt pointing to the
central path. We must now determine the stepsize along this direction. This is done by a merit
function associated to the optimality conditions (17.34). The idea of the merit function is to enable a
progress toward a local minimizer of the problem by conserving the feasibility of the solution. In our
algorithm, the merit function used for the line-search is the squared l2 norm of the KKT conditions
(17.34), that is

ΦðtÞ ¼ kFðtÞk22 ¼ FðtÞT FðtÞ: ð17:48Þ

Denote Φk ¼ Φ(tk) the value of the merit function at the iterate tk and Φk(α) ¼ Φ(tk þ αΔtk), which
illustrates the dependences of the merit function on the stepsize α. Clearly, Φk(0) ¼ Φ(tk) ¼ Φk.
Therefore,

Φk ðαÞ ¼ Fðtk þ αΔtk ÞT Fðtk þ αΔtk Þ: ð17:49Þ

The following proposition shows that the search direction given by the perturbed Newton system
(17.38)
1
Δtk ¼ ½F0 ðtk Þ ½Fðtk Þ þ μb
e ð17:50Þ

is a descent one for the merit function Φ(t).

Proposition 17.2 The direction Δtk given by (17.50), solution of the perturbed Newton system
(17.38), is a descent direction for the merit function (17.48).

Proof Considering the derivative of Φk(α) at α ¼ 0, we get

1
Φ0 ð0Þ ¼ 2Fðtk ÞT F0 ðtk ÞΔtk ¼ 2Fðtk ÞT F0 ðtk Þ½F0 ðtk Þ ½Fðtk Þ þ μb
e
¼ 2Fðtk ÞT ½Fðtk Þ þ μbe ¼ 2kFðtk Þk22 þ 2μFðtk ÞT eb,

that is Φ0(0) < 0 if and only if 2μFðtk ÞT eb  2kFðtk Þk22 : This determines the following estimation of the
barrier parameter μ to be used in the logarithmic barrier function (17.32a):

kFðtk Þk22 kFðtk Þk22

μ ¼ : ð17:51Þ
Fðtk ÞT eb sTk zk þ wTk pk þ vTk qk

Therefore, choosing at every iteration the value of the barrier parameter μ as in (17.51), the
direction Δtk given by (17.50) is indeed a descent direction for the merit function Φ(tk). ♦
17.4 A Variant of the Line-Search Interior-Point Algorithm 617

The Stepsize Determination

As we have learned from linear programming, (see (Wright, 1997)), for the nonlinear optimization
problem (17.30) the distance from centrality is also given by
min i fsi zi , wi pi , vi qi g
ξk ¼ : ð17:52Þ
sTk zk þ wTk pk þ vTk qk
2n þ m

Clearly, 0 < ξk  1 and ξk ¼ 1 if and only if sizi, wipi and viqi are equal to a constant for all i. To
specify a value for α, the following function is firstly introduced:
min i fsi ðαÞzi ðαÞ, wi ðαÞpi ðαÞ, vi ðαÞqi ðαÞg
Θ I ðα Þ ¼  γτ1 , ð17:53Þ
sðαÞT zðαÞ þ wðαÞT pðαÞ þ vðαÞT qðαÞ
2n þ m
where τ1 is the initial distance from centrality

min i s0i z0i , w0i p0i , v0i q0i
τ1 ¼ 0T 0 , ð17:54Þ
s z þ w0T p0 þ v0T q0
2n þ m
and γ 2 (0, 1) is a constant by which we can modify the distance from centrality. Observe that for
t ¼ t0 and γ ¼ 1, it follows that ΘI(0) ¼ 0. Besides, ΘI(α) is a piecewise quadratic function. To choose
the stepsize αk at each iteration, αk must satisfy ΘI(α)  0 for all α 2 [0, αk], i.e.,
min i fsi ðαÞzi ðαÞ, wi ðαÞpi ðαÞ, vi ðαÞqi ðαÞg
 γ k τ1 , ð17:55Þ
sðαÞT zðαÞ þ wðαÞT pðαÞ þ vðαÞT qðαÞ
2n þ m
where the variables of the problem are considered at the iteration k and γ k 2 (0, 1). Since ΘI(α) is a
piecewise quadratic function, from (17.55) it follows that αk can be easily computed.
Considering the merit function (17.48), the Wolfe conditions for its minimization are

Φðtkþ1 Þ  Φðtk Þ þ βαk ∇Φðtk ÞT Δtk , ð17:56Þ

∇Φðtkþ1 ÞT Δtk  δ∇Φðtk ÞT Δtk , ð17:57Þ

where 0 < β < δ < 1 are parameters responsible for the reduction of both the merit function and of the
rate of decreasing this function along the direction Δtk. Having in view that Φ(tkþ1) ¼ Φk(0), the first
Wolfe condition (17.56) is equivalent to
Φk ðαÞ  Φk ð0Þ þ βαk Φ0k ð0Þ: ð17:58Þ

Proposition 17.3 For the merit function (17.48), taking

sTk zk þ wTk pk þ vTk qk

μk ¼ σk ð17:59Þ
2n þ m
where σk 2 (0, 1), we have
618 17 Interior-Point Methods

h 2 i
σk  T
Φ0k ð0Þ ¼ 2 Φk ð0Þ  sk zk þ wTk pk þ vTk qk : ð17:60Þ
2n þ m

Proof By direct computation we get



Φ0k ð0Þ ¼ 2Φk ð0Þ þ 2μFðtk ÞT eb ¼ 2Φk ð0Þ þ 2μ sTk zk þ wTk pk þ vTk qk
sT zk þ wTk pk þ vTk qk  T

¼ 2Φk ð0Þ þ 2σk k sk zk þ wTk pk þ vTk qk

h þ
2 i
2n m
σk  T
¼ 2 Φk ð0Þ  sk zk þ wTk pk þ vTk qk : ♦
2n þ m

Proposition 17.4 At every iteration we have

T 2
sk zk þ wTk pk þ vTk qk
 Φk ð0Þ: ð17:61Þ
2n þ m

Proof By simple algebraic manipulation, we obtain

T 2
sk zk þ wTk pk þ vTk qk
 kSZek22 þ kWPek22 þ kVQek22
2n þ m
 kSZek22 þ kWPek22 þ kVQek22 þ kCOðtk Þk22 ¼ Φk ð0Þ: ♦

It is easy to show that the estimation of the barrier parameter μk given by (17.59) with σk 2 (0, 1) is
smaller than the estimation recommended by (17.51). Therefore, μk given by (17.59) ensures the
descent character of the search direction Δtk. Moreover, the following proposition gives an estimation
of the reduction of the values of the merit function.

Proposition 17.5 The direction Δtk, solution of the perturbed Newton system (17.38) with μ given by
(17.59), is a descent direction for the merit function Φ(t) at every tk. Moreover, if the first Wolfe
condition (17.56) is satisfied, then
Φk ðαk Þ  ½1  2αk βð1  σk ÞΦk ð0Þ: ð17:62Þ

Proof As we know,
 
Φ0k ð0Þ ¼ 2kFðtk Þk22 þ 2μk sTk zk þ wTk pk þ vTk qk :

Taking μk as in (17.59), it follows that

 2
sTk zk þ wTk pk þ vTk qk
Φ0k ð0Þ ¼ 2Φk ð0Þ þ 2σk
2n þ m
 2Φk ð0Þ þ 2σk Φk ð0Þ ¼ 2Φk ð0Þð1  σk Þ < 0,

proving the descent character of the direction given by the perturbed Newton system. Moreover,
taking into consideration the above propositions (17.3 and 17.4), from (17.58) we have
17.4 A Variant of the Line-Search Interior-Point Algorithm 619

0
Φk ðαÞ  Φk ð0Þ þ βαk Φ
" k ð 0Þ T 2 ! #
sk zk þ wTk pk þ vTk qk
¼ Φk ð0Þ þ βαk 2 Φk ð0Þ  σk
2n þ m
 Φk ð0Þ  2βαk Φk ð0Þ þ 2βαk σk Φk ð0Þ
¼ ½1  2αk βð1  σk ÞΦk ð0Þ,

which proves the proposition. ♦

This proposition shows that the sequence {Φk} is monotonous and nonincreasing. Therefore, for
all k, Φk  Φ0. Moreover, if the sequence of the stepsize {αk} is bounded away from zero and the
parameter σk is bounded away from one at every iteration, then the merit function is linearly
convergent to zero. Hence, the above inequality (17.62) is equivalent to
kFðtkþ1 Þk2
 ½1  2αk βð1  σk Þ1=2 : ð17:63Þ
kFðtk Þk2

Some numerical examples illustrate that a problem that may lead to the nonconvergence of the
algorithm is the case in which the sequence {kCT(sk, zk, wk, pk, vk, qk)k} converges to zero faster
than the sequence {Φ(tk)}. In such a case, the sequence of the stepsizes {αk} is decreasing to zero,
thus determining the nonconvergence of the algorithm. To avoid this situation, let us introduce the
following function:

ΘII ðαÞ ¼ sðαÞT zðαÞ þ wðαÞT pðαÞ þ vðαÞT qðαÞ  γτ2 kCOðtðαÞÞk2 , ð17:64Þ

where

sT0 z0 þ wT0 p0 þ vT0 q0

τ2 ¼ , ð17:65Þ
kCOðt0 Þk2

and γ 2 (0, 1) is a constant, the same as in (17.53). Observe that for t ¼ t0 and γ ¼ 1, ΘII(0) ¼ 0. In
general, ΘII(α) is a nonlinear function. For choosing the stepsize αk at every iteration it is necessary
for αk to satisfy

ΘII ðαk Þ  0: ð17:66Þ

Proposition 17.6 Let {tk} be a sequence generated as solution of (17.38). Then,

 2
min f1, 0:5τ2 g Φðtk Þ  sTk zk þ wTk pk þ vTk qk  ð2n þ mÞΦðtk Þ: ð17:67Þ

Proof The second inequality follows from proposition 17.4. Hence, we prove only the first one.
Since Θi(αk)  0,for i ¼ 1, 2, from (17.64) with γk  1/2 we have
T 
sk zk þ wTk pk þ vTk qk  ð1=2Þτ2 kCOðtk Þk2 :

Therefore,
620 17 Interior-Point Methods

 
sTk zk þ wTk pk þ vTk qk
1

 kSZek2 þ kWPek2 þ kVQek2 þ 0:5τ2 kCOðtk Þk2

2
1
 min f1, 0:5τ2 gkFðtk Þk2 ,
2
which completes the proof. ♦

This proposition shows that the complementarity conditions are bounded and γ k must be selected
as a decreasing sequence with 1/2  γ k  γ k1.
Having in view all these developments, at each iteration, the stepsize αk is computed as a solution
of the following system of algebraic inequalities:
l  xk þ αk Δxk  u, ð17:68aÞ

sðαk Þ, zðαk Þ, wðαk Þ, pðαk Þ, vðαk Þ, qðαk Þ > 0, ð17:68bÞ

ΘI ðαÞ  0, α 2 ½0, αk , ð17:68cÞ

ΘII ðαk Þ  0, ð17:68dÞ

Φk ðαk Þ  Φk ð0Þ þ αk βΦ0k ð0Þ, ð17:68eÞ

∇Φk ðαk ÞT Δtk  δ∇Φk ð0ÞT Δtk , ð17:68fÞ

where 0 < β < δ < 1.

To determine αk satisfying (17.68), a strategy of reducing the interval can be used. The first two
relations (17.68a) and (17.68b) are easy to implement. As in linear programming, the corresponding
ratio test is performed, thus obtaining a value αm which maintains the positivity of the variables as
well as the simple bounds on variables. Then, a value αk 2 (0, αm] which satisfies the conditions
(17.68c), (17.68d), (17.68e), and (17.68f) is selected. However, the selection of αk from the interval
(0, αm] is not simple.

Primal-Dual Interior-Point Algorithm

The interior-point algorithm for solving (17.30) based on the above developments has three main
parts. They refer to the computations of the following: the search direction, the barrier parameter, and
the stepsize.

Algorithm 17.4 Primal-dual interior-point algorithm—PDIP

1. Initialization. Choose an initial point t0 ¼ [x0, y0, z0, p0, q0, s0, w0, v0], such that l  x0  u, (s0, z0) > 0,
(w0, p0) > 0, (v0, q0) > 0. Choose the parameters ε > 0, β 2 (0, 1/2], γ k1 ¼ 1 and ρ 2 (0, 1). Set k ¼ 0
2. Convergence test. Compute the value of the merit function Φ(tk) ¼ F(tk)TF(tk). If Φ(tk)  ε, stop;
otherwise, go to step 3
3. Evaluation of the barrier parameter. Choose σk 2 (0, 1). Using (17.59), compute the barrier parameter μk
4. Computing the search direction. Determine Δtk by solving the reduced Newton system (17.46) and by
using (17.45) and (17.41)
17.4 A Variant of the Line-Search Interior-Point Algorithm 621

5. Stepsize determination. Use the Armijo technique:

(a) Choose 1/2  γ k  γ k1.
(b) Compute αm as the longest value of αk which satisfies (17.68a) and (17.68b). 

(c) Determine αIk 2 ð0, αm  as the smallest positive root, such that ΘI(α)  0 for all α 2 0, αIk :

 
(d) Determine αIIk 2 0, αIk , such that Θ αIIk  0:
II

(e) Set αk ¼ min αIk , αIIk :
(f) Set αk ¼ ρj αk , such that αk satisfies (17.68e), where j ¼ 0, 1, . . . is the smallest integer with this
property
6. Updating the variables. Set tkþ1 ¼ tk þ αkΔtk, k ¼ k þ 1 and go to step 2 ♦

Some comments are as follows (details can be found in (Andrei, 1998c)).

1. (Search direction) In proposition 17.2, we showed that the step direction Δt given by (17.50) is
descent for the merit function provided that the value of the barrier parameter is selected as in
(17.51).

Proposition 17.7 Suppose that f, cE, and cI are twice continuously differentiable, the derivative
of CO(t) defined by (17.35) is Lipschitz continuous and the set of gradients
{∇cE1(xk), . . ., ∇cEme(xk)} [ {∇cIi(xk), i 2 A(x)}, is linearly independent for k sufficiently large,
where A(x) is the set of active inequality constraints at x. Then, the sequence {Δtk} generated by the
PDIP algorithm is bounded.

Proof By permuting the rows and columns, F0(tk) can be rearranged as

 
A B
F0 ð t k Þ ¼ ,
BT C

where
2 3
K ðxk , yk , zk Þ J E ðxk ÞT J I ðxk ÞT
6 7
A ¼ 4 J E ðxk Þ 0 0 5,
J I ðxk Þ 0 0

the matrices B and C being very easily identified from F0(t). By assumptions of the proposition, the
matrix A is invertible and kA1k is uniformly bounded. But
" #1 2 1  1  1 3
A B A þ A1 B C  BT A1 B BT A1 A1 B C  BT A1 B
¼4  1  1
5,
BT C  C  BT A1 B BT A1 C  BT A1 B

which is bounded, since every matrix involved is bounded. Hence, [F0(tk)]1 is uniformly bounded,
proving the proposition. ♦

2. (Stepsize) A crucial point of the algorithm is the stepsize computation. The following proposition
shows that the sequence fαk g generated in step 5e of the PDIP algorithm is bounded away
from zero.
622 17 Interior-Point Methods

Proposition 17.8 Suppose that the functions f(x), cE(x), and cI(x) are twice continuously differentia-
ble and the derivative of CO(t) is Lipschitz continuous with constant L. If the sequence {σk} is
bounded away from zero, then the sequence fαk g generated by the algorithm is also bounded away
from zero.
  
Proof Sine αk ¼ min αIk , αIIk , it suffices to show that the sequences αIk and αIIk are bounded
away from zero. Let us suppress the subscript k. As in (El-Bakry, Tapia, Tsuchiya, and Zhang, 1996),
let us define the vectors a(α) ¼ [s(α), w(α), v(α)]T 2 ℝmþ2n and b(α) ¼ [z(α), p(α), q(α)]T 2 ℝmþ2n.
Then, the function ΘI(α) from (17.53) can be written as
min i fai ðαÞbi ðαÞg
ΘI ðαÞ ¼  γτ1 :
aðαÞT bðαÞ
2n þ m

From the definition of αI (step 5c of the PDIP algorithm), we can see that α is the largest number in
[0, αm] such that

ai ðαÞbi ðαÞ  γτ1 aðαÞT bðαÞ=ð2n þ mÞ  0

for every i ¼ 1, . . ., 2n þ m and α 2 [0, αI]. Now, define

 
 ΔaT Δb

ηi ¼ Δai Δbi  γτ1 :
2n þ m

From proposition 17.7, it follows that Δt is bounded. Then, there is a positive constant M such that
ηi  M. Straightforward computation shows that for α 2 [0, αI], we have

aðαÞT bðαÞ
ai ðαÞbi ðαÞ γτ1
2n þ m
   
aT b aT Δb þ bT Δa
¼ ai bi  γτ1 þ α ai Δbi þ bi Δai  γτ1
2n þ m 2n þ m
   
Δa Δb
T 
2 ΔaT Δb
þα Δai Δbi  γτ1
2
 αð1  γτ1 Þμ  α Δai Δbi  γτ1
2n þ m 2n þ m
¼ αð1  γτ1 Þμ  ηi α2  αð1  γτ1 Þμ  Mα2  0:

Hence,

αI  ð1  γτ1 Þμ=M:
T
But, as we know, μ ¼ σ 2nþm
a b
and for σ bounded away from zero it follows that μ is bounded below.
Hence, α is bounded away from zero.
I

Let us now show that the sequence {αII} generated by step 5 of the PDIP algorithm is bounded
away from zero. According to the mean-value theorem for vector-valued functions (Dennis and
Schnabel, 1983), we have
17.4 A Variant of the Line-Search Interior-Point Algorithm 623

21 3
ð
COðt þ αΔtÞ ¼ COðtÞ þ α4 ∇COðt þ ξαΔtÞdξ5Δt
0 21 3
ð
¼ COðtÞ þ α∇COðtÞΔt þ α4 ð∇COðt þ ξαΔtÞ  ∇COðtÞÞdξ5Δt
21 0 3
ð
¼ ð1  αÞCOðtÞ þ α4 ð∇COðt þ ξαΔtÞ  ∇COðtÞÞdξ5Δt,
0

where the last equality is from (17.38). Having in view that the derivative of CO(t) is Lipschitz
continuous with constant L, we obtain

kCOðt þ αΔtÞk  kCOðtÞkj1  αj þ Lα2 kΔtk2 :

Using this inequality, we have

ΘII ðαÞ ¼ aðαÞT bðαÞ  γτ2 kCOðt þ αΔtÞk



¼ aT b þ α aT Δb þ bT Δa þ α2 ΔaT Δb  γτ2 kCOðt þ αΔtÞk

 
 j1  αjaT b þ ασaT b þ α2 ΔaT Δb  γτ2 kCOðtÞkj1  αj þ Lα2 kΔtk2
 
¼ j1  αjðaT b  γτ2 kCOðtÞkÞ þ ασaT b þ α2 ΔaT Δb  γτ2 LkΔtk2
h  i
 α σaT b  αΔaT Δb  γτ2 LkΔtk2  :

From proposition 17.7, there is a constant N > 0 such that

 T 
Δa Δb  γτ2 LkΔtk2   N:

Hence ΘII(α)  α[σaTb  αN]. From the condition (17.66), it follows that αII  σaTb/N. Since

the sequence {σk} is bounded away from zero, then the sequence αIIk is also bounded away
from zero. ♦

3. (Convergence) The convergence of the PDIP algorithm is proved as follows.

Theorem 17.3 Suppose that the functions f(x), cE(x), and cI(x) are twice continuously differentiable,
and the derivative of CO(t) is Lipschitz continuous. Let {tk} be the sequence generated by the PDIP
algorithm, where {σk} ⊂ (0, 1) is bounded away from zero and one. Then the sequence {F(tk)}
converges to zero and for any limit point t ¼ [x, y, z, p, q, s, w, v]T, x is a KKT point of the
problem (17.30).

Proof Firstly, note that the sequence {kF(tk)k} is monotonously decreasing, hence it is convergent.
By contradiction, suppose that the sequence {kF(tk)k} is not convergent to zero. Then, from
proposition 17.5, we have
Φk ðαk Þ=Φk ð0Þ  1  2αk βð1  σk Þ:
624 17 Interior-Point Methods

Therefore, from proposition 17.8 it follows that the sequence {Φk} linearly converges to
zero. This leads to a contradiction. On the other hand, from proposition 17.2 we have
∇Φðtk ÞΔtk ¼ 2Fðtk ÞT Fðtk Þ þ 2μk Fðtk ÞT eb: Since the sequence {αk} is bounded away from zero, it
follows that the backtracking line-search used in step 5 of the PDIP algorithm produces
h i
∇Φðtk ÞΔtk 2 Fðtk Þ Fðtk Þ  μk Fðtk Þ eb
T T

¼ ! 0:
kΔtk k kΔtk k

Therefore, from proposition 17.7, Δtk is bounded away from zero, i.e.,


Φðtk Þ  μk sTk zk þ wTk pk þ vTk qk ! 0:

However,


Φðtk Þ  μk sTk zk þ wTk pk þ vTk qk  ð1  σk ÞΦðtk Þ:

Therefore, it must hold that Φ(tk) ! 0 because the sequence {σk} is bounded away from one.
Again, this leads to a contradiction. Hence, the sequence {kF(tk)k} must be convergent to zero. Since
the KKT conditions for the problem (17.30) are satisfied by t, it follows that x is a KKT point for
(17.30). ♦

Example 17.1 Let us consider the following nonlinear optimization problem (Hock, & Schittkowski,
1981), (Andrei, 1998c)
min x1 x4 ðx1 þ x2 þ x3 Þ þ x3
subject to
x1 x2 x3 x4  25  0,
x21 þ x22 þ x23 þ x24  40 ¼ 0,
1  xi  5, i ¼ 1, 2, 3, 4:

The evolution of some elements corresponding to a simple variant of the PDIP algorithm is
presented in Table 17.1.

Table 17.1 Evolution of some elements of PDIP. Example 17.1

k f ( xk ) Φ(tk ) CO(tk ) CT (.)

2 2
0 16.0000 534.076 465.076 69.0000
1 16.9494 11.2666 8.97539 2.291261
2 17.1584 0.253076 8.97539 0.135262
3 16.9951 0.020296 1.41485 0.61477e-2
4 17.0168 0.45552e-3 0.18875e-3 0.26677e-3
5 17.0158 0.11076e-4 0.52654e-6 0.10549e-4
6 17.0144 0.42452e-6 0.56172e-9 0.42387e-6
7 17.0141 0.16245e-7 0.65172e-9 0.16244e-7
8 17.0140 0.24373-10 0.1999e-14 0.2437e-10
17.4 A Variant of the Line-Search Interior-Point Algorithm 625

Table 17.2 Evolution of parameters of PDIP. Example 17.1

k γk σk αmax αk μk
0 0.750000 0.1 1.090858 1.039720 0.233333
1 0.625000 0.2 0.992283 0.926926 0.90664e-1
2 0.562500 0.2 1.170237 1.0 0.22643e-1
3 0.531250 0.2 1.210422 1.0 0.48802e-2
4 0.515625 0.2 1.165723 1.0 0.10580e-2
5 0.507812 0.2 1.220644 1.0 0.21612e-3
6 0.503906 0.19530 1.238018 1.0 0.42383e-4
7 0.501953 0.03823 1.039720 1.0 0.16244e-5

In this variant of PDIP, the condition (17.68d) was not implemented. Although this condition was
not implemented, yet, in the last part of the optimization process we have kCO(tk)k2 < kCT(.)k2. This
behavior ensures the convergence of the algorithm. Table 17.2 shows the evolution of the algorithm
parameters.
The parameter γ k is updated as γ kþ1 ¼ 0.5 þ (γ k  0.5)/2. Some other updating formula may be
imagined so that γ k 2 [0.5, γ k  1]. The parameter σk is computed as
(  
η1 , if σk  η2 sTk zk þ wTk pk þ vTk qk ,
σk ¼    
η2 sTk zk þ wTk pk þ vTk qk , if σk > η2 sTk zk þ wTk pk þ vTk qk ,

where η1 ¼ 0.1 and η2 ¼ 100. Observe that the stepsize αk ! 1, exactly as in the “pure” Newton
method.
The solution of the problem is x ¼ [1, 5, 5, 1]. The objective value is 17.0140173. The
SPENBAR (Andrei, 1996a, b, c) gives the same solution involving 8 major iterations, 143 minor
iterations, and 591 evaluations of the functions defining the problem.

Example 17.2 Consider the problem (Andrei, 2015a, pp.777)

min 24:55x1 þ 26:75x2 þ 39x3 þ 40:5x4
subject to
2:3x1 þ 5:6x2 þ 11:1x3 þ 1:3x4  5  0,
12x1 þ 11:9x2 þ 41:8x3 þ 52:1x4  21
 1=2
1:645 0:28x21 þ 0:19x22 þ 20:5x23 þ 0:62x24  0,
x1 þ x2 þ x3 þ x4  1 ¼ 0,
0  xi  10, i ¼ 1, 2, 3, 4:

The PDIP algorithm gives the results from Tables 17.3 and 17.4. Again, observe that the algorithm
has the same behavior along the iterations. Even if the condition ΘII(αk)  0 was not implemented in
PDIP, we can see that after very few iterations, kCO(tk)k2 < kCT(.)k2. This ensures the convergence
of the algorithm. Again, note that the stepsize converges to 1, like in the pure Newton method.
The solution to this problem is x ¼ [0.6355, 0.286E  7, 0.3127, 0.05177]. MINOS gives the
same solution in 5 major iterations, 18 minor iterations, and 23 evaluations of the function defining
626 17 Interior-Point Methods

Table 17.3 Evolution of some elements of PDIP. Example 17.2

k f ( xk ) F(tk ) CO(tk ) CT (.)

2 2
0 130.8000 7422.758 7009.758 413.0000
1 38.03691 26.02877 17.60484 8.423930
2 30.49207 0.278046 0.40382e-3 0.2776424
3 29.96742 0.85973e-2 0.37688e-5 0.85935e-2
4 29.91532 0.51103e-3 0.86534e-4 0.42450e-3
5 29.89686 0.26626e-4 0.90809e-5 0.17545e-4
6 29.89512 0.59983e-6 0.49498e-8 0.59488e-6
7 29.89452 0.26198e-7 0.23005e-8 0.23898e-7
8 29.89439 0.18476e-9 0.12652e-9 0.58238e-10
9 29.89438 0.34199e-12 0.34150e-12 0.49357e-15

Table 17.4 Evolution of parameters of PDIP. Example 17.2

k γk σk αmax αk μk
0 0.750000 0.1 0.9581337 0.9498877 0.51
1 0.625000 0.2 1.098776 1.0 0.1158525
2 0.562500 0.2 1.032449 1.0 0.0282029
3 0.531250 0.2 1.058755 1.0 0.56835e-2
4 0.515625 0.2 1.006443 0.9938785 0.11798e-2
5 0.507812 0.2 1.106218 1.0 0.24329e-3
6 0.503963 0.2 1.244491 1.0 0.48777e-4
7 0.501953 0.048885 1.051240 1.0 0.23898e-5
8 0.500976 0.0024132 1.002405 1.0 0.58237e-8

Table 17.5 Performances of PDIP for solving 4 applications from the LACOP collection

n me mc #it #nf KKT CT cpu vfo

ELCH 10 3 0 18 19 0.185290e-12 0.147536e-11 0.01 -47.761090
ALKI 10 3 8 21 22 0.572627e-08 0.257441e-34 0.01 -1768.8069
MSP3 13 0 15 26 27 0.383793e-08 0.133552e-11 0.02 97.587510
POOL 34 20 0 19 20 0.896223e-10 0.896147e-10 0.1 2785.8000

In this table, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the number of
inequality constraints, #it ¼ the number of iterations to obtain a solution, #nf ¼ the number of evaluations of the
functions defining the problem (including the gradients and Hessians), KKT ¼ the norm of the KKT conditions,
CT ¼ the norm of the centrality conditions (see (17.36), cpu ¼ the CPU computing time for obtaining a solution
(seconds), vfo ¼ the value of the objective function at the optimal point

the problem. On the other hand, SPENBAR needs 10 major iterations, 455 minor iterations, and 2083
evaluations of the functions of the problem, giving the same solution. NLPQLP gives the same
solution, needing 14 iterations, and 17 evaluations of the functions.
Table 17.5 shows the performances of PDIP for solving some applications from the LACOP
collection.
17.5 Trust-Region Interior-Point Algorithm 627

17.5 Trust-Region Interior-Point Algorithm

There are two main differences between the trust-region interior-point and the line-search interior-
point algorithms. The first one is that the trust-region interior-point is not a real primal-dual method.
Unlike the line-search interior-point method in which both the primal and the dual variables are
simultaneously computed, the trust-region interior-point algorithm firstly computes a step in the
variables (x, s) and then the estimates of the multipliers (dual variables). The second difference is that
the trust-region interior-point algorithm uses a scaling of the variables, which discourages the
movement toward the boundary of the feasible region. This makes the trust-region interior-point
algorithms have better convergence properties than those of the line-search interior-point.
In the following, the trust-region algorithm for finding approximate solutions of a fixed barrier
problem is presented. Then a trust-region interior-point algorithm is described, where the barrier
parameter is driven to zero.

An Algorithm for the Barrier Problem

Let us consider the barrier problem (17.3). This is an equality constrained optimization problem that
can be solved by using a sequential quadratic programming method with trust-region. However, a
direct application of this method is not possible because it generates steps that tend to violate the
positivity of the slack variables and therefore they frequently cut short the trust-region constraint. To
overcome this difficulty, Nocedal and Write (2006) suggest designing another sequential quadratic
programming method.

At iterate (x, s), for a given value of the barrier parameter μ, the estimates of the Lagrange
multipliers (dual variables) (y, z) are firstly computed and then the step p ¼ ( px, ps) is determined,
which approximately solves the following subproblem:
1 1
min px ,ps ∇f ðxÞT px þ pTx ∇2xx Lpx  μeT S1 ps þ pTs Σps ð17:69aÞ
2 2
subject to
J E ðxÞpx þ cE ðxÞ ¼ r E , ð17:69bÞ

J I ðxÞpx  ps þ ðcI ðxÞ  sÞ ¼ r I , ð17:69cÞ

 
 px , S1 ps   Δ, ð17:69dÞ
2

ps  τs: ð17:69eÞ

In this problem, Σ ¼ S1Z and the scalar parameter τ 2 (0, 1) is chosen close to 1, (for
exampleτ ¼ 0.995). The inequality (17.69e) has the same role as the fraction to the boundary rule
given by (17.8). The constraints (17.69b) and (17.69c) are linear. Clearly, we would like to set
r ¼ (rE, rI) ¼ 0, but this can cause the constraints (17.69b) (17.69c), and (17.69d) to be incompatible
or to give a step p that makes little progress toward feasibility. Therefore, the parameter r is selected
by an auxiliary computation like in the sequential quadratic programming with trust-region.
The choice of the objective function (17.69a) is motivated by the fact that the first-order optimality
conditions of (17.69a), (17.69b), and (17.69c) are exactly those given by (17.2) in which (17.2b) is
scaled by S1.
628 17 Interior-Point Methods

The trust-region constraint (17.69d) guarantees that the problem (17.69) has a finite solution even
when ∇2xx Lðx, s, y, zÞ is not positive definite, and therefore this Hessian never needs to be modified.
Besides, the trust-region formulation ensures that adequate progress is made at every iteration.
The scaling S1 used in (17.69d) is motivated by the crucial fact that the shape of the trust-region
must take into account the requirement that the slacks should not approach zero prematurely. The
scaling S1 restricts those components i of ps for which si are close to their lower bound of zero.
In the following, we present the sequential quadratic programming algorithm with trust-region for
the barrier problem. This means that at every iteration the subproblem (17.69) is solved. The
condition of stopping the iterations is defined by the error function (17.9) which uses the perturbed
KKT system (17.2) as well as by the sufficient reduction of the merit function Φσ defined by (17.21).
The algorithm is as follows.

Algorithm 17.5 Trust-region algorithm for barrier problems

1. Choose the initial point x0 and s0 > 0. Compute the initial values of the multipliers y0 and z0 > 0. Select an
initial value of the barrier parameter μ > 0. Consider a tolerance εμ and set k ¼ 0
2. If E(xk, sk, yk, zk, μ)  εμ, then continue with step 3; otherwise, stop; the current point is the solution of
the problem
3. Compute p ¼ ( px, ps) as an approximate solution of the subproblem (17.69)
4. If p provides sufficient decrease in the merit function Φσ, then set xkþ1 ¼ xk þ px, skþ1 ¼ sk þ ps, compute
the new multiplier estimates ykþ1, zkþ1 > 0 and set Δkþ1  Δk. Continue with step 5. Otherwise, define
xkþ1 ¼ xk, skþ1 ¼ sk and set Δkþ1 < Δk. Continue with step 3
5. Set k ¼ k þ 1 and go to step 2 ♦

Now, let us discuss some aspects regarding the determination of an approximate solution of the
subproblem (17.69) as well as the estimates (ykþ1, zkþ1) of the Lagrange multipliers.

Solving the Subproblem (17.69)

This problem is approximately solved. At the very beginning, a change of variable is made, which
transforms the trust-region constraint (17.69d) into a ball. By defining
   
px px
pe ¼ ¼ 1 , ð17:70Þ
pes S ps

the subproblem (17.69) becomes

1 1
min p ,ep ∇f ðxÞT px þ pTx ∇2xx Lpx  μeT pes þ peTs SΣSe
ps ð17:71aÞ
x s 2 2
subject to
J E ðxÞpx þ cE ðxÞ ¼ r E , ð17:71bÞ

J I ðxÞpx  Se
ps þ ðcI ðxÞ  sÞ ¼ r I , ð17:71cÞ

kðpx , pes Þk2  Δ, ð17:71dÞ

pes  τe: ð17:71eÞ

17.5 Trust-Region Interior-Point Algorithm 629

To compute the vectors rE and rI consider the following subproblem in the variables v ¼ (vx, vs):

min v kJ E ðxÞvx þ cE ðxÞk22 þ kJ I ðxÞvx  Svs þ ðcI ðxÞ  sÞk22 ð17:72aÞ

subject to
kðvx , vs Þk2  0:8Δ, ð17:72bÞ
τ
vs   e: ð17:72cÞ
2
If the constraint (17.72c) is ignored, then the problem (17.72) has a standard form of a trust-region
problem and therefore an approximate solution of it can be obtained by the known techniques. If the
solution of this problem violates the bounds (17.72c), then a backtracking can be used to satisfy them.
Having a solution (vx, vs) for (17.72), the vectors rE and rI are computed as
r E ¼ J E ðxÞvx þ cE ðxÞ, ð17:73aÞ

r I ¼ J I ðxÞvx  Svs þ ðcI ðxÞ  sÞ: ð17:73bÞ

With these developments, we are ready to compute an approximate solution de for (17.71) as
follows. From (17.73), we can note that v is a particular solution of the linear constraints (17.71b) and
(17.71c). Therefore, the subproblem (17.71a), (17.71b), and (17.71c), which is a quadratic program-
ming problem with equality constraints, can be solved by using, for example, the projected conjugate
gradient algorithm. During the solving process, the satisfaction of the trust-region constraint (17.71d)
is monitored. If the boundary of this region is reached or if a negative curvature is detected or if an
approximate solution is obtained, then the algorithm is stopped. If the solution obtained by the
projected conjugate gradient algorithm does not satisfy the bounds (17.71e), then a backtracking is
used until all these are satisfied. After the step ðpx , pes Þ has been computed, the vector p is recovered
from (17.70).
It should be mentioned that at each iteration, the projected conjugate gradient algorithm requires
solving a linear system in order to perform the projection operation. For the quadratic programming
(17.71a), (17.71b), and (17.71c), this projection matrix is
" #  
I Ab
T
J E ðxÞ 0
b
, where A ¼ : ð17:74Þ
Ab 0 J I ðxÞ S

Although this trust-region approach requires the solution of an augmented linear system, the
matrix (17.74) is simpler than the primal-dual matrix from (17.11). Moreover, the advantage is that
the Hessian ∇2xx L needs never to be factored because the conjugate gradient approach requires only
products of this matrix with vectors. Besides, the matrix SΣS from (17.71a) has a much tighter
distribution of eigenvalues than Σ. Therefore, the conjugate gradient method will run much better in
the presence of ill-conditioning, being much more suitable for solving the quadratic problem
(17.71a), (17.71b), and (17.71c) (Nocedal, & Wright, 2006).

Lagrange Multipliers Estimates and Step Acceptance

As we have already seen in step 4 of Algorithm 17.5, the estimates of the Lagrange multipliers ykþ1
and zkþ1 > 0 are computed. This is done as follows. At the iterate (x, s), the (y, z) are chosen as the
least-square multipliers corresponding to the subproblem (17.71a), (17.71b), and (17.71c), i.e.,
630 17 Interior-Point Methods

     
y T 1 ∇f ðxÞ
b
¼ AA b b
A , ð17:75Þ
z μe

where Ab is the matrix from (17.74). It is quite possible that the estimates z obtained from (17.75) may
not always be positive. To enforce the positivity of this multiplier, the following computational
scheme can be used:

zi ¼ min 103 , μ=si , i ¼ 1, . . . , m: ð17:76Þ

Observe that if all the components of z are defined as in (17.76), then the matrix Σ reduces to μS2.
The quantities μ/si are called the i-th primal multiplier estimate of multipliers.

As in the standard trust-region method, step p is accepted if

aredðpÞ  ηpredðpÞ, ð17:77Þ

where

aredðpÞ ¼ Φσ ðx, sÞ  Φσ ðx þ px , s þ ps Þ, ð17:78Þ

predðpÞ ¼ qσ ð0Þ  qσ ðpÞ, ð17:79Þ

1 1
qσ ðpÞ ¼ ∇f ðxÞT px þ pTx ∇2xx Lpx  μeT S1 ps þ pTs Σps þ σmðpÞ, ð17:80Þ
2 2
 
 J E ðxÞpx þ cE ðxÞ 

m ð pÞ ¼   : ð17:81Þ
J I ðxÞpx  ps þ cI ðxÞ  s 2

In (17.77), η is a constant in (0, 1), for example η ¼ 108.

To determine a corresponding value for the penalty parameter σ, we impose that this is large
enough so that

predðpÞ  ρσ ðmð0Þ  mðpÞÞ, ð17:82Þ

where ρ is a parameter in (0, 1). Observe that this condition is exactly as (15.38). Therefore, the value
of σ can be computed by the procedure described in (15.39) from the sequential quadratic
programming.

Description of the Trust-Region Interior-Point Algorithm

For updating the barrier parameter, the Fiacco-McCormick strategy is used. The condition of stopping
the algorithm uses the error function E defined by (17.9). If the quasi-Newton approach is used, then
the Hessian ∇2xx L is replaced by a symmetric approximation (BFGS or limited memory BFGS).

Algorithm 17.6 Trust-region interior-point algorithm

1. Choose the value for the parameters η > 0, τ 2 (0, 1), σ 2 (0, 1), ρ 2 (0, 1), and ξ 2 (0, 1). Choose the
tolerances εμ and εTOL small enough. If the quasi-Newton approach is used, choose a symmetric n  n
matrix B0. Choose the initial values of the parameters μ > 0 and Δ0 as well as the initial points x0, s0 > 0.
Set k ¼ 0
2. If E(xk, sk, yk, zk, 0) > εTOL, then go to step 3; otherwise, stop
3. If E(xk, sk, yk, zk, μ) > εμ, then go to step 4; otherwise, go to step 13
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 631

4. Using (17.75) and (17.76), compute the Lagrange multipliers

 
5. Compute ∇2xx L xk , sk , yk , zk or update a quasi-Newton approximation Bk and define Σk ¼ S1k Zk
6. Compute the normal step vk ¼ (vx, vs)
7. Compute pek by applying the projected conjugate gradient algorithm to the problem (17.71)
8. Using (17.70), compute the total step pk
9. Update vk to satisfy ((17.82)
10. Compute pred( pk) and ared( pk) by (17.79) and (17.78), respectively
11. If ared( pk)  ηpred( pk), then set xkþ1 ¼ xk þ px, skþ1 ¼ sk þ ps, and choose Δkþ1  Δk. Otherwise, set
xkþ1 ¼ xk, skþ1 ¼ sk and choose Δkþ1 < Δk
12. Set k ¼ k þ 1 and go to step 3
13. Set μ ¼ σμ, update εμ and go to step 2 ♦

The tolerance on the barrier parameter can be defined as εμ ¼ μ, or an adaptive strategy can be used
for its updating. It is quite clear that the merit function can lead to the Maratos effect. In this case, the
second-order corrections or a nonmonotone strategy can be used. Algorithm 17.6 is implemented in
KNITRO/INTERIOR (Byrd, Hribar, and Nocedal, 1999), which uses the exact Hessian or a quasi-
Newton approximation of this matrix and follows to be presented in the next section.

17.6 Interior-Point Sequential Linear-Quadratic Programming

(KNITRO/INTERIOR)

In Chap. 15, the KNITRO/ACTIVE algorithm based on the active-set sequential linear-quadratic
programming method has been presented. This chapter describes the KNITRO/INTERIOR algo-
rithm, together with its numerical performances for solving large-scale general continuous nonlinear
optimization problems. KNITRO/INTERIOR provides two procedures for computing the steps
within the interior-point approach. In the version INTERIOR-CG, each step is computed using a
projected conjugate gradient iteration. It factors a projection matrix and uses the conjugate gradient
method to approximately minimize a quadratic model of the barrier problem. In the version
INTERIOR-DIRECT, the algorithm attempts to compute a new iterate by solving the primal-dual
KKT system using direct linear algebra. In case this step cannot be guaranteed to be of good quality or
if a negative curvature is detected, then the new iterate is computed by the INTERIOR-CG algorithm.
The description of the KNITRO/INTERIOR-CG algorithm is given in (Byrd, Hribar, & Nocedal,
1999) and its global convergence theory is presented in (Byrd, Gilbert, & Nocedal, 2000). The
method implemented in the KNITRO/INTERIOR-DIRECT algorithm is described in (Waltz,
Morales, Nocedal, and Orban, 2003).
Consider the problem

min x2ℝn f ðxÞ ð17:83aÞ

subject to
cE ðxÞ ¼ 0, ð17:83bÞ

cI ðxÞ  0, ð17:83cÞ

where f : ℝn ! ℝ, cE : ℝn ! ℝl, and cI : ℝn ! ℝm are twice continuously differentiable functions.

The interior-point or the barrier methods implemented in KNITRO associate the following barrier
problem to (17.83):
632 17 Interior-Point Methods

X
m
min x2ℝn ,s2ℝm f ðxÞ  μ log ðsi Þ ð17:84aÞ
i¼1

subject to
cE ðxÞ ¼ 0, ð17:84bÞ

cI ðxÞ  s ¼ 0, ð17:84cÞ

where s 2 ℝm is a vector of slack variables and μ > 0 is the barrier parameter. The interior-point
methods consist in finding the approximate solutions of the barrier problem (17.84) for a sequence of
positive barrier parameters {μk} that converges to zero.
The KKT optimality conditions for (17.84) are as follows:

∇f ðxÞ  J TE ðxÞy  J TI ðxÞz ¼ 0, ð17:85aÞ

μe þ Sz ¼ 0, ð17:85bÞ

cE ðxÞ ¼ 0, ð17:85cÞ

cI ðxÞ  s ¼ 0, ð17:85dÞ

where e ¼ [1, . . ., 1]T, S ¼ diag (s1, . . ., sm), JE(x), and JI(x) are the Jacobian matrices corresponding
to the equality and inequality constraints vectors, respectively. The vectors y and z are the Lagrange
multipliers (dual variables) associated to the equality and inequality constraints. We also have s, z > 0.
In the line-search approach, the Newton method is applied to (17.85), backtracking if necessary, so
that the variables s and z remain positive and a merit function is sufficiently reduced. In the trust-
region approach, a quadratic programming problem is associated to (17.84) and the step of the
algorithm is an approximate solution of this quadratic programming sub-problem. These two
approaches are implemented in the KNITRO/INTERIOR-DIRECT and KNITRO/INTERIOR-CG
algorithms, respectively.
A very important component in the interior-point methods is the procedure for choosing the
sequence of the barrier parameters {μk}. KNITRO contains several options. In the Fiacco-McCormic
monotone strategy, the barrier parameter μ is held fixed for a series of iterations until the KKT (17.85)
are satisfied to some accuracy. In the adaptive strategy, the barrier parameter is updated at every
iteration using different rules: the rule implemented in LOQO based on the deviation of the minimum
complementarity pair from the average (Vanderbei, & Shanno, 1999), a probing strategy that uses
Mehrotra’s predictor step to select a target value for μ, a quality-function approach and some other
rules described in (Nocedal, Wächter, & Waltz, 2005).
To control the quality of the steps, both interior-point algorithms implemented in KNITRO make
use of a nondifferentiable merit function
X
m
Φσ ðx, sÞ ¼ f ðxÞ  μ log ðsi Þ þ σkcE ðxÞk2 þ σkcI ðxÞ  sk2 , ð17:86Þ
i¼1

where σ > 0. A step is acceptable only if it provides a sufficient decrease of the merit function (17.86).
Let us now present the interior-point algorithms implemented in KNITRO.

KNITRO/INTERIOR-DIRECT Algorithm
In this algorithm, the search direction is determined by the direct solving of the Newton system
associated to the nonlinear system given by the KKT optimality conditions (17.85). To obtain global
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 633

convergence in the presence of the nonconvexity of the Hessian or of the Jacobian singularities, this
step may be replaced under certain circumstances by a safeguarding trust-region step. KNITRO/
INTERIOR-DIRECT is described in (Byrd, Nocedal, and Waltz, 2006). (See also (Waltz, Morales,
Nocedal, and Orban, 2003).)

By applying the Newton method to the system (17.85) in the variables x, s, y, z, we get
2 2 32 3 2 3
∇xx L 0 J TE ðxÞ J TI ðxÞ dx ∇f ðxÞ  J TE ðxÞy  J TI ðxÞz
6 0 S 7 6 7 6 Sz  μe 7
6 Z 0 76 d s 7 6 7
6 76 7 ¼ 6 7, ð17:87Þ
4 J E ðxÞ 0 0 0 54 d y 5 4 cE ðxÞ 5
J I ðxÞ I 0 0 dz cI ðxÞ  s

where, as usual, L represents the Lagrange function

Lðx, s, y, zÞ ¼ f ðxÞ  yT cE ðxÞ  zT ðcI ðxÞ  sÞ: ð17:88Þ

If the inertia of the matrix in (17.87) is

ðm þ n, l þ m, 0Þ, ð17:89Þ

then the step d determined as a solution of (17.87) can be guaranteed to be a descent direction for the
merit function (17.86). In this case, compute the scalars
αsmax ¼ max fα 2 ð0, 1 : s þ αds  ð1  τÞsg, ð17:90aÞ

αzmax ¼ max fα 2 ð0, 1 : z þ αdz  ð1  τÞzg, ð17:90bÞ


where τ ¼ 0.995. If min αsmax , αzmax is not too small, then perform a backtracking line-search that
computes the stepsizes



αs 2 0, αsmax , αz 2 0, αzmax , ð17:91Þ

which achieve a sufficient decrease of the merit function (17.86).

The new iterate is computed as

xþ ¼ x þ αs dx , sþ ¼ s þ αs d s , ð17:92aÞ

yþ ¼ y þ αz d y , zþ ¼ z þ αz dz : ð17:92bÞ

On the other hand, if the inertia of the matrix in (17.87) is not as in (17.89) or if the stepsize αs or αz
is less than a given threshold, then the step d solution of (17.87) is rejected. In this case, the search
direction d is not a descent one for the merit function (17.86). The algorithm reverts to the trust-region
method implemented in KNITRO/INTERIOR-CG, algorithm guaranteed to provide a successful step
even in the presence of a negative curvature or singularity and which is described in the next section.
It is worth mentioning that this strategy implemented in KNITRO/INTERIOR-DIRECT is differ-
ent from the line-search interior-point algorithms IPOPT (Wächter, & Biegler, 2005a, 2005b, 2006)
and LOQO (Vanderbei, & Shanno, 1999). Whenever the inertia condition is not satisfied, then IPOPT
and LOQO modify the Hessian ∇2xx L: Instead, KNITRO/INTERIOR-DIRECT implements the above
strategy because it permits to compute a step by using a null space approach without modifying the
Hessian ∇2xx L: Moreover, using the trust-region step guarantees progress in case the line-search
approach fails (Wächter, & Biegler, 2000), (Byrd, Marazzi, & Nocedal, 2004b).
634 17 Interior-Point Methods

It is known that the step lengths αs or αz in (17.92) converge to zero when the line-search iterations
converge to nonstationary points. In this case, the algorithm monitors these step lengths. If one of
them is smaller than a given threshold, then the iterate given by (17.87) and (17.92) is discarded and
replaced with the trust-region step.
Therefore, the algorithm monitors two criteria: the inertia of the matrix in (17.87) and the step
lengths (17.91). If these two criteria are satisfied, then we continue the iterations by using the Newton
method for solving (17.87) by the direct method.
For the initialization of the algorithm, it is necessary to compute the initial values of the multipliers
y0 and z0. The multipliers are computed as the least-squares solution of the system given by (17.85a)
and (17.85b). If the line-search is discarded, then one or more KNITRO/INTERIOR-CG iterations are
executed until one of them provides sufficient reduction in the merit function. The KNITRO/
INTERIOR-DIRECT algorithm presented below uses DΦσ(x, s, d ), which is the directional deriva-
tive of the merit function Φσ along a direction d.

Algorithm 17.7 KNITRO/INTERIOR-DIRECT—Byrd, Hribar, and Waltz

1. Choose the initial point x0, s0 > 0, and the parameters η > 0 and 0 < αmin < 1. Compute the initial values of
the multipliers y0 and z0 > 0. Choose a value of the trust-region radius Δ0 > 0 and a value for the barrier
parameter μ > 0. Set k ¼ 0
2. If a test for stopping the iterations is satisfied, then stop; otherwise, go to step 3
3. If the perturbed KKT optimality conditions (17.85) are approximately satisfied, then go to step 9;
otherwise, go to step 4
4. Factorize the matrix of the primal-dual system (17.87) and compute neig as the number of the negative
eigenvalues of this matrix
5. Set LineSearch ¼ False
6. If neig  l þ m, then:
Solve the system (17.87) to obtain the direction d ¼ (dx, ds,dy, dz)
Define w ¼ (xk, sk) and dw ¼ (dx, ds)
Compute αsmax and αzmax as in (17.90)

If min αsmax , αzmax > α min , then:
Update the penalty parameter σk(see the merit function below)
Compute a stepsize αs ¼ ααsmax , α 2 ð0, 1, such that
Φσ(w þ αsdw)  Φσ(w) þ ηαsDΦσ(w, dw)
If αs > αmin, then:
Set αz ¼ ααzmax
Compute (xkþ1, skþ1, ykþ1, zkþ1) as in (17.92)
Set LineSearch ¼ True
7. If LineSearch ¼ False, then compute (xkþ1, skþ1, ykþ1, zkþ1) using the algorithm KNITRO/INTERIOR-
CG described in the next section
8. Compute Δkþ1. Set k ¼ k þ 1 and go to step 3
9. Choose a new smaller value for the barrier parameter μ and go to step 2 ♦

Observe that at every iteration the algorithm computes and maintains a trust-region radius Δk in
case it needs to revert to the trust-region KNITRO/INTERIOR-CG algorithm in step 7.

KNITRO/INTERIOR-CG Algorithm
The second algorithm implemented in KNITRO computes the search directions by using a quadratic
model and trust-regions. This strategy permits great freedom in the choice of the Hessian and
provides a mechanism for coping with the Jacobian and Hessian singularities. However, the iterations
of this algorithm are more complex. The KNITRO/INTERIOR-CG algorithm is described in (Byrd,
Hribar, and Nocedal, 1999) and analyzed in (Byrd, Gilbert, and Nocedal, 2000).
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 635

To describe this algorithm, observe that the barrier problem (17.84) is an equality constrained
optimization problem which can be solved by using a sequential quadratic programming (SQP)
method with trust-regions. However, a straightforward application of the SQP method to the barrier
problem leads to inefficient steps that tend to violate the positivity of the slack variables and
frequently cuts short the trust-region constraint. To overcome this difficulty, Byrd, Nocedal, and
Waltz (2006) suggested the following SQP method associated to the barrier problem (17.84).
At the current iterate (xk, sk) and for a given value of the barrier parameter μ, firstly compute the
Lagrange multiplier estimates (yk, zk) and then the step d ¼ (dx, ds) as solution of the following
subproblem:
1
min dx ,ds ∇f ðxk ÞT dx þ d Tx ∇2xx Lðxk , sk , yk , zk Þd x
2 ð17:93aÞ
T 1 1
μe Sk ds þ dTs Σk d s
2
subject to
cE ðxk Þ þ J E ðxk Þdx ¼ r E , ð17:93bÞ

cI ðxk Þ þ J I ðxk Þd x  d s  sk ¼ r I , ð17:93cÞ

 
dx , S1 ds   Δk , ð17:93dÞ
k 2

ds  τs, ð17:93eÞ

where Σk ¼ S1 k Z k and τ ¼ 0.995. Ideally, the residues rE and rIshould be zero, i.e., r ¼ (rE, rI) ¼ 0,
but since this value can determine the constraints to be incompatible or produce a poor step, we would
rather choose r as the smallest vector such that the constraints (17.93b), (17.93c), and (17.93d) should
be consistent.
The choice of the objective function (17.93a) is motivated by the fact that the first-order optimality
conditions of (17.93a), (17.93b), and (17.93c) are given by (17.85) with the second block of equations
scaled by S1. Therefore, the steps computed by using (17.93) are related to those of the line-search
algorithm described in the previous section. The trust-region constraint (17.93d) guarantees that
(17.93) has a finite solution even if the Hessian of the Lagrangian ∇2xx Lðxk , sk , yk , zk Þ is not positive
definite. Hence, the Hessian matrix never needs to be modified in this algorithm. The scaling S1 k used
in the trust-region constraint is crucial in the economy of the algorithm.

Step Computation
The presence of the nonlinear constraints (17.93d) and of the bounds (17.93e) makes the highly
accurate solving of (17.93) difficult. However, useful inexact solutions can be computed at a
moderate cost. KNITRO uses the null space approach, in which step d is computed as a sum of a
normal step v that attempts to satisfy the linear constraints (17.93b) and (17.93c) with r ¼ 0 and also,
possibly, the trust-region, and of a tangential step that lies on the tangent space of the constraints and
that tries to achieve optimality.

The normal step v ¼ (vx, vs), is the solution of the following subproblem:

min v kJ E vx þ cE k22 þ kJ I vx  vs þ cI  sk22 ð17:94aÞ

subject to
 
 vx , S1 vs   0:8Δ, ð17:94bÞ
2
636 17 Interior-Point Methods

in which the arguments of the functions appearing in (17.94) have been omitted. An inexact solution
to the subproblem (17.94) is computed by using a dogleg approach, which minimizes (17.94a) along a
piecewise linear path composed of a steepest descent step in the norm used in (17.94b) and a
minimum-norm Newton step with respect to the same norm. The scaling S1vs in the norm from
(17.94b) tends to limit the extent to which the bounds on the slack variables are violated.
Once the normal step v has been computed, the vectors rE and rI from (17.93b) and (17.93c),
respectively, are computed as residuals, namely,
r E ¼ J E vx þ cE , r I ¼ J I vx  vs þ ðcI  sÞ:

Once the normal step v ¼ (vx, vs) is computed, the subproblem (17.93) can be written as
1 T 2 
min dx ,ds ∇f T dx  μeT S1 d s þ d ∇ Ld þ dTs Σds ð17:95aÞ
2 x xx x
subject to
J E d x ¼ J E vx , ð17:95bÞ

J I d x  d s ¼ J I vx  vs , ð17:95cÞ
 
 d x , S1 ds   Δ, ð17:95dÞ
2

which is called tangential subproblem. Now, to find an approximate solution d of the subproblem
(17.95), firstly introduce the scaling

des ¼ S1 ds , ð17:96Þ

which transforms (17.95d) into a sphere. Then the projected conjugate gradient (CG) method to the
transformed quadratic program is applied, where all the iterates are in the linear manifold defined by
(17.95b) and (17.95c). While solving by using CG, the strategy of Steihaug is used, with the
monitoring of the satisfaction of the trust-region constraint (17.95d). The iterations are stopped if
the boundary of this region is reached or if a negative curvature is detected. Finally, if necessary, step
d is truncated to satisfy (17.93e). The KNITRO/INTERIOR-CG algorithm is as follows.

Algorithm 17.8 KNITRO/INTERIOR-CG—Byrd, Hribar, and Waltz

1. Choose x0, s0 > 0 and Δ0 > 0. Choose a value for η > 0. Set k ¼ 0
2. If a test for stopping the algorithm is satisfied, stop; otherwise, go to step 3
3. If the perturbed KKK system (17.85) is approximately satisfied, go to step 11; otherwise, go to step 4
4. Compute the normal step vk ¼ (vx, vs)
5. Compute the Lagrange multipliers yk and zk > 0
6. Compute the total step dk by applying the projected conjugate gradient algorithm to the subproblem
(17.95a), (17.95b), and (17.95c)
7. Update the penalty parameter σk (see the merit function below)
8. Compute aredk(dk) and predk(dk) by using the relations below
9. If aredk(dk)  η predk(dk), then set xkþ1 ¼ xk þ dx, skþ1 ¼ sk þ ds and update Δkþ1; otherwise, set
xkþ1 ¼ xk, skþ1 ¼ sk and choose Δkþ1 < Δk
10. Set k ¼ k þ 1 and go to step 3
11. Choose a smaller value for the barrier parameter μ and go to step 2 ♦
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 637

In Algorithm 17.8,
ared ðdÞ ¼ Φσ ðx, sÞ  Φσ ðx þ dx , s þ ds Þ ð17:97Þ

is the actual reduction in the merit function. pred(d ) is immediately defined.

The Lagrange multiplier estimates (yk, zk) are computed by the least squares approximation to the
Eqs. (17.85a) and (17.85b) evaluated at the point xk and truncated to ensure the positivity of zk.
Observe that the interior-point Algorithm 17.8 is asymptotically equivalent to the standard line-
search interior-point algorithms, but it is significantly different in two respects. Firstly, it is not a fully
primal-dual method because the multipliers are computed as a function of the primal variables (x, s)
as opposed to the formulation (17.87) in which the primal and the dual variables are computed
simultaneously from their previous values. Secondly, the trust-region method uses a scaling of the
variables that avoids moving toward the boundary of the feasible region. Therefore, the algorithm
generates steps quite different from those produced by a line-search method.

Merit Function
The role of the merit function (17.86) is to determine whether a step can be accepted. Byrd, Hribar,
and Nocedal (1999) show that the efficiency of the algorithm depends on the choice of the penalty
parameter σ. In both interior-point algorithms implemented in KNITRO/INTERIOR, at every itera-
tion σ is chosen so that the decrease in the quadratic model of the merit function produced by a step
d is proportional with σ multiplied by the decrease in the linearized constraints.

More exactly, suppose that either the KNITRO/INTERIOR-DIRECT algorithm or KNITRO/

INTERIOR-CG has generated a step d. Then define the following linear/quadratic model of the
merit function Φσ:
ξ T 2 
Qσ ðd Þ ¼ ∇f T dx  μeT S1 ds þ d x ∇xx Ldx þ dTs Σds þ σðmð0Þ  mðdÞÞ, ð17:98Þ
2
where
 
 J E d x þ cE 
mðd Þ ¼ 

 ð17:99Þ
J I d x  d s þ cI  s  2

represents the first-order violation of the constraints and ξ is a parameter that follows to be discussed.
Define the predicted decrease in the merit function as
pred ðd Þ ¼ Qσ ð0Þ  Qσ ðdÞ: ð17:100Þ

In all the cases, the penalty parameter σ is chosen large enough such that
pred ðdÞ  ρσðmð0Þ  mðd ÞÞ, ð17:101Þ

where 0 < ρ < 1 (for example ρ ¼ 0.1).

If the value of σ from the previous iteration satisfies (17.101), then σ is left unchanged. Otherwise,
σ is increased so that it satisfies the inequality (17.101) with some accuracy.
For the trust-region method as implemented in KNITRO/INTERIOR-CG, set ξ ¼ 1 in (17.98). On
the other hand, in KNITRO/INTERIOR-DIRECT, parameter ξ is defined as
638 17 Interior-Point Methods

(
1, if dTx ∇2xx Ldx þ dTs Σds > 0,
ξ¼ ð17:102Þ
0, otherwise:

This choice of ξ guarantees that the directional derivative of Φσ in the direction d is negative.

Computational Aspects
KNITRO/INTERIOR contains many algorithmic options and features that are listed and discussed in
the documentation of the package (Waltz, 2004). In the following, we shall present some of these
options implemented in KNITRO.

Initial point strategy. It is well known that any interior-point algorithm is very sensitive to the
choice of the initial point. In KNITRO, several different strategies are implemented. One strategy is
 
as follows. At an initial point x0 specified by the user, an affine scaling step d A ¼ dAx , dAs , d Ay , dAz is
 
computed by solving the system (17.87) with μ ¼ 0. Then, s1 ¼ max 1, s0 þ dAs  and z1 ¼
 
max 1, z0 þ dAz  are defined, where the operators max and absolute are component-wise applied.
The primal variables x and the multipliers y associated to the equality constraints are not altered, i.e.,
(x1, y1) ¼ (x0, y0). Finally, the initial value of the barrier parameter is computed as μ1 ¼ sT1 z1 =m:
Hessian options. There are some options in KNITRO for using the second derivatives. One
possibility is that the user can supply the first and the second derivatives, which generally results in
the greatest level of robustness and efficiency for all the three algorithms implemented in KNITRO.
In some particular applications, the Hessian ∇2xx L cannot be computed or is too large to store. In these
cases, KNITRO/INTERIOR-CG and KNITRO/ACTIVE allow the user to provide products Hessian-
vectors at every iteration. Another variant implemented in all the three algorithms in KNITRO is to
approximate ∇2xx L by quasi-Newton updates: BFGS, memory-less BFGS, or SR1. For example,
BFGS is implemented as

qk qTk Bk sk sTk Bk
Bkþ1 ¼ Bk þ  T , ð17:103Þ
sTk qk s k Bk s k

where qk ¼ ∇xL(xþ, sþ, yþ, zþ)  ∇xL(x, sþ, yþ, zþ) and sk ¼ xkþ1  x.
Feasible iterations. In some applications, it is desirable for all the iterates generated by the
optimization algorithm to be feasible with respect to some or to all of the inequality constraints.
KNITRO uses the following procedure to generate feasible iterates. If the current iteration x satisfies
the constraints cI(x) > 0, then, after computing step d, set x+ ¼ x þ dx. The slack variables are
redefined as s+ ¼ cI(x+) and tested whether the point (x+, s+) is acceptable for the merit function Φσ. If
the case, this point is defined as the new iterate, otherwise this point is rejected and a new shorter one
is computed (in a line-search method by backtracking and in a trust-region method by reducing the
trust-region radius). The justification of this strategy is as follows. If at a trial point, we have ci(x+)  0,
for some inequality constraint, then the value of the merit function is +1, and this trial point is
rejected. This strategy also rejects the steps x þ dx that are too close to the boundary of the feasible
P
region because such steps increase the barrier term μ m i¼1 log ðsi Þ in the merit function.
Crossover technique. The situation is that the interior-point methods provide only an approximate
estimate of the solution and of the optimal active-set. However, in many practical applications, it is
useful to know accurately which constraints are active. Moreover, it is often important to know the
highly accurate estimates of the Lagrange multipliers. They are important in the sensitivity analysis of
the solution to some variations of certain parameters in the problem. Increasing the accuracy of the
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 639

solution or of the active-set can be done by switching from the interior-point method to an active-set
iteration, a process which is called crossover. This technique was introduced for the first time in linear
programming by Megiddo (1989). In linear programming, the crossover technique involves two
stages: identifying the active constraints and moving from a nonbasic optimal solution to a nearby
basic one. However, in nonlinear programming, this form of crossover cannot be used because we
cannot expect the set of active constraints to correspond to a basic solution. In nonlinear optimization,
the crossover was introduced for the first time by Byrd, Nocedal, and Waltz (2006). It seeks to
identify a set of active constraints (with linearly independent constraint gradients) and computes a
solution at which these constraints are approximately satisfied, solution which determines a stationary
point of the Lagrangian by using only these constraints.
The crossover procedure internally commutes to the KNITRO/ACTIVE algorithm after the
KNITRO/INTERIOR-DIRECT or the KNITRO/INTERIOR-CG algorithms have solved the problem
to the required tolerance. Firstly, solve the equality quadratic programming (15.110) to generate a
new solution estimate. If this step does not solve the problem immediately, then the full KNITRO/
ACTIVE algorithm is started with an initial LP trust-region radius ΔLP k computed on the basis of the
active set estimate. The idea is to choose ΔLP
k small enough to exclude all the inactive constraints, but
large enough to include the active ones. The crossover algorithm as described in (Byrd, Nocedal, and
Watz, 2006) is as follows.

Algorithm 17.9 KNITRO crossover algorithm

1. The interior-point algorithms DIRECT or CG terminate with a solution (xk, sk, yk, zk) which satisfies the
stopping tolerance εTOL
2. Estimate the set of the active constraints A
3. Using this active-set estimate, generate a step dQ by solving the equality quadratic programming problem
(15.110). Perform a line-search to determine the stepsize αQ. If xk þ αQdQ satisfies the stopping
tolerances, then the algorithm stops with this point and with the corresponding multipliers; otherwise, go
to step 4
4. Determine the initial LP trust-region radius ΔLP0 and the penalty parameter σ0 for the KNITRO/ACTIVE
(Algorithm 15.9) as n o
ci ðxk , sk Þ
0 ¼ min
ΔLP k∇ci ðxk , sk Þk : i=
2A ,
σ0 ¼ 10k(yk, zk)k1
5. Start the KNITRO/ACTIVE algorithm using the initial point (xk, sk, yk, zk), ΔLP
0 and σ0 ♦

In step 3 of Algorithm 17.9, the active set is estimated by using a tolerance test rather than by
solving the linear programming problem (15.105). This is because, in some difficult problems, the
cost of solving the linear programming subproblem can be nontrivial and, as motivated by Byrd,
Nocedal and Waltz (2006), the cost of the crossover procedure shall be a small part of the overall
solution time. Therefore, it is not necessary to solve (15.105) to identify the optimal active set. If strict
complementarity holds at the solution, the initial estimate of the active-set based on the simple
tolerance test will be correct and the crossover will succeed in one iteration without solving (15.105).
The formula for computing the initial trust-region radius ΔLP 0 from step 4 of Algorithm 17.9,
guarantees that if the active set estimate is correct, then this initial trust-region radius will be small
enough to exclude all the inactive constraints. As motivated by the theory of l1 exact penalty
functions, the penalty parameter is initialized to be a little larger than the Lagrange multiplier of
the largest magnitude at the interior-point solution.
640 17 Interior-Point Methods

Practical hints. KNITRO implements four algorithms. An algorithm is selected according to the
value of the parameter option. For option 0, KNITRO will automatically choose the best algorithm
based on the problem characteristics. For option 1, KNITRO will use the INTERIOR-DIRECT
algorithm. For option 2, KNITRO will use the INTERIOR-CG algorithm. For option 3, KNITRO
will use the ACTIVE algorithm (see Chap. 15).

Numerical Study—KNITRO: Solving Applications from the LACOP Collection

In the following, Tables 17.6, 17.7, and 17.8 present the performances of KNITRO with option
0 (KNITRO will automatically try to choose the best algorithm based on the problem characteristics),
option 1 (KNITRO will use the INTERIOR-DIRECT algorithm), and KNITRO with option
2 (KNITRO will use the INTERIOR-CG algorithm) for solving 12 nonlinear optimization
applications from the LACOP collection described in Appendix C.

Table 17.6 Performances of KNITRO for solving 12 applications from the LACOP collection. Option 0. Small-scale
nonlinear optimization applications

n me mc #it #itcg #nf #ng #nh cpu vfo

ELCH 10 3 0 6 0 7 7 6 0.143 -47.761090
ALKI 10 3 8 10 0 13 11 10 0.011 -1768.8069
PREC 8 0 6 10 9 11 11 10 0.081 3.95116321
PPSE 9 6 0 8 0 9 9 8 0.009 5055.01179
MSP3 13 0 15 26 7 36 27 26 0.044 97.5875095
MSP5 16 0 21 23 10 31 24 23 0.045 174.786942
POOL 34 20 0 9 0 10 10 9 0.026 2569.800
TRAFO 6 0 2 16 18 19 17 16 0.009 135.07592
LATHE 10 1 14 23 5 50 24 23 0.020 -4430.0879
DES 150 50 0 39 350 99 40 39 0.479 1055.1823
CSTC 303 200 0 4 0 5 5 4 0.035 3.4800745
DIFF 396 324 0 2 0 3 1 0 0.016 0

Table 17.7 Performances of KNITRO/INTERIOR-DIRECT for solving 12 applications from the LACOP collection.
Option 1. Small-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng #nh cpu vfo

ELCH 10 3 0 6 0 7 7 0 0.130 -47.761090
ALKI 10 3 8 10 0 13 11 10 0.009 -1768.8069
PREC 8 0 6 10 9 11 11 10 0.012 3.95116321
PPSE 9 6 0 8 0 9 9 8 0.008 5055.01179
MSP3 13 0 15 26 7 36 27 26 0.012 97.5875095
MSP5 16 0 21 23 10 31 24 23 0.014 174.786942
POOL 34 20 0 9 0 10 10 9 0.010 2569.800
TRAFO 6 0 2 16 18 19 17 16 0.009 135.07592
LATHE 10 1 14 23 5 50 24 23 0.017 -4430.0879
DES 150 50 0 39 350 99 40 39 0.470 1055.1823
CSTC 303 200 0 4 0 5 5 4 0.011 3.4800745
DIFF 396 324 0 2 0 3 1 0 0.020 0
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 641

Table 17.8 Performances of KNITRO/INTERIOR-CG for solving 12 applications from the LACOP collection.
Option 2. Small-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng #nh cpu vfo

ELCH 10 3 0 6 27 7 7 6 0.008 -47.761090
ALKI 10 3 8 20 79 21 21 20 0.011 -1768.8065
PREC 8 0 6 22 124 24 23 22 0.057 4.2867607
PPSE 9 6 0 5 13 6 6 5 0.008 5055.01300
MSP3 13 0 15 36 237 37 37 36 0.019 97.5875197
MSP5 16 0 21 84 325 146 85 84 0.035 174.795082
POOL 34 20 0 17 59 18 18 17 0.011 2569.800
TRAFO 6 0 2 37 103 52 38 37 0.017 135.076250
LATHE 10 1 14 27 118 33 28 27 0.016 -4430.0806
DES 150 50 0 28 262 79 29 28 0.237 1055.1823
CSTC 303 200 0 4 10 5 5 4 0.019 3.4800745
DIFF 396 324 0 1 0 2 1 0 0.024 0

In the tables above, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the number of
inequality constraints, #it ¼ the number of iterations to obtain a solution, #itcg ¼ the number of conjugate gradient
iterations, #nf ¼ the number of evaluations of the functions defining the problem, #ng ¼ the number of evaluations of
the gradients of the functions defining the problem, #nh ¼ the number of evaluations of the Hessian, cpu ¼ the CPU
computing time for obtaining a solution of the problem (seconds), and vfo ¼ the value of the objective function at the
optimal point

Table 17.9 Performances of the KNITRO algorithms. Small-scale nonlinear optimization applications

Option #itt #nft #ngt #nht cput

0 176 293 186 174 0.918
1 (DIRECT) 176 293 186 168 0.722
2 (CG) 287 430 298 286 0.462
3 (ACTIVE) 278 731 289 418 0.782

The performances of KNITRO/ACTIVE (option 3) for solving small-scale nonlinear optimization

applications are presented in Table 15.6. Table 17.9 presents the total number of iterations (#itt), the
total number of evaluations of the functions defining the problems (#nft), the total number of
evaluations of the gradients of the functions (#ngt), the total number of evaluations of the Hessian
(#nht), and the total CPU time (cput) in seconds for solving 12 small-scale nonlinear optimization
applications considered in this numerical study with option 0 (Table 17.6), option 1 (Table 17.7),
option 2 (Table 17.8), and option 3 (Table 15.6) respectively.
Tables 17.10, 17.11, and 17.12 show the numerical performances of KNITRO with option 0, option
1, and with option 2, respectively, for solving 6 large-scale nonlinear optimization applications of
different dimensions from the LACOP collection.
Performances of KNITRO/ACTIVE (option 3) for solving 15 large-scale nonlinear optimization
applications are presented in Table 15.7. Table 17.13 contains a comparison of these four variants of
KNITRO subject to the total number of iterations (#itt), the total number of conjugate gradient
iterations (#itcgt), the total number of evaluations of the function defining the problem (#nft), the total
number of evaluations of the gradients of the functions defining the problem (#ngt), the total CPU
computing time to obtain a solution (cput) for solving 15 large-scale nonlinear optimization
applications considered in this numerical study, with option 0 (Table 17.10), option
1 (Table 17.11), option 2 (Table 17.12), and option 3 (Table 15.7), respectively.
642 17 Interior-Point Methods

Table 17.10 Performances of KNITRO for solving 6 applications from the LACOP collection. Option 0. Large-scale
nonlinear optimization applications

n me mc #it #itcg #nf #ng cpu vfo

HANG 2002 1001 0 21 73 44 22 0.349 5.0685101
4002 2001 0 81 328 291 82 3.039 5.0684889
FLOW 1182 754 0 5 0 6 6 0.160 0.311e-11
FLOWO 1556 1005 0 10 19 11 11 1.031 0.107e-6
POL 4004 3000 0 27 35 29 28 1.203 14.009791
6004 4500 0 26 35 27 27 1.836 13.990035
8004 6000 0 28 36 29 29 2.568 14.009035
10004 7500 0 30 39 31 31 3.454 14.007359
CAT 3003 2000 0 8 0 9 9 0.301 -0.048052
6003 4000 0 8 0 9 9 0.674 -0.048048
9003 6000 0 7 0 8 8 0.993 -0.048045
CONT 2505 2000 0 9 0 10 10 0.109 1.0132439
5005 4000 0 10 0 11 11 0.403 1.0059324
7505 6000 0 11 0 12 12 0.743 1.0045764
10005 8000 0 11 0 12 12 0.947 1.0040918

Table 17.11 Performances of KNITRO/INTERIOR-DIRECT for solving 6 applications from the LACOP collection.
Option 1. Large-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng cpu vfo

HANG 2002 1001 0 21 73 44 22 0.425 5.068510
4002 2001 0 81 328 291 82 3.068 5.0684889
FLOW 1182 754 0 5 0 6 6 0.160 3.85e-11
FLOWO 1556 1005 0 10 19 11 11 1.088 1.071e-5
POL 4004 3000 0 27 35 29 28 1.291 14.009791
6004 4500 0 26 35 27 27 1.839 13.990035
8004 6000 0 28 36 29 29 2.586 14.009035
10004 7500 0 30 39 31 31 3.481 14.007359
CAT 3003 2000 0 8 0 9 9 0.353 -0.048052
6003 4000 0 8 0 9 9 0.650 -0.048048
9003 6000 0 7 0 8 8 1.055 -0.048045
CONT 2505 2000 0 9 0 10 10 0.190 1.0132439
5005 4000 0 10 0 11 11 0.408 1.0059324
7505 6000 0 11 0 12 12 0.734 1.0045764
10005 8000 0 11 0 12 12 0.968 1.0040918

Application Penici
Application L16 is described in the LACOP collection. Table 17.14 presents the performances of the
algorithms implemented in KNITRO for solving this application.

There are a number of packages implementing the interior-point methods for nonlinear optimiza-
tion, both in the line-search and in the trust-region framework. The line-search implementation
includes LOQO (Vanderbei, & Shanno, 1999), KNITRO/INTERIOR-DIRECT (Waltz, Morales,
Nocedal, & Orban, 2003), IPOPT (Wächter, & Biegler, 2006), BARNLP (Betts, Eldersveld, Frank,
& Lewis, 2000), and MOSEK (Andresen, & Andresen, 2000). The trust-region algorithm was
implemented in KNITRO/INTERIOR-CG (Byrd, Hribar, & Nocedal, 1999). All these interior-
point packages are strong competitors of the leading active-set and of the augmented Lagrangian
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 643

Table 17.12 Performances of KNITRO/INTERIOR-CG for solving 6 applications from the LACOP collection.
Option 2. Large-scale nonlinear optimization applications

n me mc #it #itcg #nf #ng cpu vfo

HANG 2002 1001 0 6 141 7 7 0.211 5.0685100
4002 2001 0 6 142 7 7 0.538 5.0684889
FLOW 1182 754 0 29 174 30 30 1.080 1.831e-11
FLOWO 1556 1005 0 48 731 49 49 4.104 1.8483e-5
POL 4004 3000 0 44 179 46 45 1.546 14.202608
6004 4500 0 43 353 45 44 2.866 14.243107
8004 6000 0 19 197 23 20 1.938 14.265885
10004 7500 0 19 153 23 20 2.208 14.260745
CAT 3003 2000 0 10 46 11 11 0.379 -0.048047
6003 4000 0 6 18 7 7 0.571 -0.047980
9003 6000 0 7 25 8 8 1.069 -0.047954
CONT 2505 2000 0 13 92 14 14 0.357 1.0134407
5005 4000 0 13 98 14 14 0.758 1.006334
7505 6000 0 14 95 15 15 1.211 1.005174
10005 8000 0 13 98 14 14 1.532 1.004447

Table 17.13 Performances of the KNITRO algorithms. Large-scale nonlinear optimization applications

Option #itt #itcgt #nft #ngt cput

0 292 565 539 307 17.81
1 (DIRECT) 292 565 539 307 18.296
2 (CG) 290 2542 313 305 20.367
3 (ACTIVE) 1123 2197 2711 1138 466.084

Table 17.14 Performances of KNITRO. Application PENICI (n ¼ 707, l ¼ 602, m ¼ 0)

Option #it #itcg #nf #ng cpu

0 397 2922 433 398 7.374
1 (DIRECT) 397 2922 433 398 7.357
2 (CG) 307 3994 559 308 4.786
3 (ACTIVE) Too many iterations

packages, like MINOS (Murtagh, & Saunders, 1987), SNOPT (Gill, Murray, & Saunders, 2002),
LANCELOT (Conn, Gould, & Toint, 1992b), and KNITRO/ACTIVE (Byrd, Gould, Nocedal, &
Waltz, 2004). For solving nonlinear optimization problems and applications, the interior-point and
the active-set methods are the most suitable, while the augmented Lagrangian methods seem to be
less efficient and robust. KNITRO implements a crossover technique from the interior-point to the
active-set modes (Byrd, Nocedal, & Waltz, 2006). KNITRO/INTERIOR-CG implements a trust-
region algorithm using the projected conjugate gradient iteration for computing the step, which
allows the method to work when only the Hessian-vector products are available, not the Hessian
itself.
In the interior-point methods, the primal-dual linear system to be solved at every iteration has the
same block structure which can be speculated to get efficient solutions. Both the direct solutions using
the sparse matrix technology and the conjugate gradient methods are available for solving the primal-
644 17 Interior-Point Methods

Table 17.15 Comparisons between KNITRO and CONOPT. Small-scale nonlinear optimization applications

CONOPT KNITRO 0 KNITRO 1 KNITRO 2 KNITRO 3

#itt 272 176 176 287 278
cput 0.80 0.918 0.722 0.462 0.782

Table 17.16 Comparisons between KNITRO and CONOPT. Large-scale nonlinear optimization applications

CONOPT KNITRO 0 KNITRO 1 KNITRO 2 KNITRO 3

#itt 520 292 292 290 1123
cput 9.124 17.81 18.296 20.368 466.084

dual system. However, the interior-point methods, unlike the active-set methods, consider all the
constraints at each iteration, even if they are irrelevant to the solution, thus enlarging the cost of the
primal-dual iteration. The main drawbacks of the interior-point methods is their sensitivity to the
choice of the initial point, the scaling of the problem and the update strategy for the barrier parameter
μ. The interior-point methods may have difficulties and their convergence can be slow if the iterates
approach the boundary of the feasible region prematurely. To get a robust code, some algebraic
techniques must be implemented, such as second-order corrections, iterative refinement, resetting the
parameters, procedure for computing the approximate solutions of the normal and tangential
subproblems, etc.
A comparison of codes KNITRO and CONOPT for solving 12 small-scale applications from the
LACOP collection is shown in Table 17.15. Table 17.16 presents a comparison of KNITRO versus
CONOPT for solving 15 large-scale applications from the LACOP collection.

Notes and References

Interior-point methods generate iterations that avoid the boundary of the feasibility region defined by
the inequality constraints. Along the optimization process, the iterates are allowed to get closer and
closer to the boundary of the feasibility region and converge to the optimal solution which might lie
on the boundary. From the practical point of view, interior-point methods typically require only tens
of iterations. Each iteration consists in solving a large linear system of equations which takes into
consideration all the variables and constraints so that each iteration is quite expensive. In other words,
interior-point methods approximately solve a sequence of perturbed KKT systems by driving a barrier
parameter to zero. These methods can be regarded as perturbed Newton methods applied to the KKT
system in which the primal/dual variables are kept positive. All the characteristics of the optimization
process given by optimality, feasibility, and complementarity are simultaneously reduced. The main
advantages of the interior-point methods are as follows: they can better exploit the second derivatives
and their structure; they are efficient for loosely constrained optimization problems and are easily
implemented in computing programs.

This chapter is based on the papers (Nocedal, & Wright, 2006) and (Andrei, 1998c). The interior-
point methods were first developed for linear programming by Ilya Dikin (1936–2008) (1967, 1974)
and then by Karmarkar (1984). The extension to the quadratic and nonlinear programming was quite
natural. There is a vast literature on nonlinear interior-point methods. For a comprehensive list of
references, we recommend the papers by Forsgren, Gill, and Wright (2002) and Gould, Orban, and
Toint (2005a). The book by Conn, Gould, and Toint (2000) also gives a thorough presentation of
several interior-point algorithms. The papers by Ulbrich, Ulbrich and Vicente (2004) and Wächter
17.6 Interior-Point Sequential Linear-Quadratic Programming (KNITRO/INTERIOR) 645

and Biegler (2005a, 2005b) present an analysis of interior-point algorithms that use filter globaliza-
tion. The primal barrier methods for nonlinear optimization were originally proposed by Frisch
(1955) and were further analyzed by Fiacco and McCormick (1968). In this chapter, the prototype
of interior-point algorithms was presented first. After that, a variant of the line-search interior-point
algorithm was deeply discussed, where a methodology of the interior-point algorithms development
and analysis was given. The conclusion is that for having an efficient and robust interior-point
algorithm, some methods based on sequential linearization combined with line-search or trust-
region globalization strategies have to be introduced.
A history of the barrier function methods was given by Nash (1998). Shanno (2012) presented
interesting comments on “who invented the interior-point algorithm.” His conclusion is that the
history of the interior-point method goes back to Fiacco and McCormick. It seems that the terms
interior-point and primal central path appear in the book by Fiacco and McCormick. The most
important packages implementing the interior-point methods are as follows.
KNITRO/INTERIOR implements two methods: trust-region and line-search interior-point/barrier
methods. The idea of the algorithm is to solve a sequence of barrier subproblems for a decreasing
sequence of barrier parameters. For accepting the step and for ensuring the global convergence it uses
a merit or a penalty function. The barrier subproblems are solved by a sequence of linearized primal-
dual equations. For solving the primal-dual system, KNITRO implements two options: the direct
factorization of the system and the preconditioned conjugate gradient method. The preconditioned
conjugate gradient method solves the indefinite primal-dual system by projecting constraints onto the
null space of the equality constraints. KNITRO uses the crossover technique to obtain an active set
from the solution of the interior-point method.
LOQO (Vanderbei, & Shanno, 1999) uses an infeasible primal-dual interior-point method for
solving general nonlinear optimization problems and applications. The inequality constraints are
added to the minimizing function by using a log-barrier function. To get a solution to the system of
nonlinear equations that is obtained by applying the first-order necessary optimality conditions to the
barrier function, Newton’s method is used. The solution of this system gives a search direction. A
filter or a merit function is used to accept the next iterate obtained by performing a line-search along
the search direction. The filter allows a point that improves either the feasibility or the barrier
objective as compared to the current iterate. The merit function is the barrier function and an
additional l2 norm of the violation of constraints. The exact Hessian of the Lagrangian is used in
Newton’s method. When the problem is nonconvex, then the Hessian is perturbed by adding the
matrix δI to it, where I is the identity matrix and δ > 0 is a scalar chosen in such a way that the
perturbed Hessian is positive definite.
Ipfilter (Ulbrich, Ulbrich, & Vicente, 2004) is an interior-point filter method. Firstly, a step is
generated by solving the KKT system. Then, a specially designed filter is used for selecting the
stepsize in the direction of the calculated step. The first component of the filter is the sum of the
constraint violation and centrality, while the second component is the norm of the gradient of the
Lagrangian.
IPOPT (Wächter, & Biegler, 2005a, 2005b) is a line-search filter interior-point method. The outer
loop approximately minimizes a sequence of nonlinear equality constrained barrier subproblems for a
decreasing sequence of barrier parameters. The inner loop uses a line-search filter sequential
quadratic programming method to approximately solve each barrier subproblem. IPOPT is described
in Chap. 19 of this monograph.
 Filter Methods
18

This chapter focuses on the filter methods developed by Fletcher and Leyffer (2002) as a new
technique for the globalization of nonlinear optimization algorithms. These methods are motivated
by the aim of avoiding the need to choose penalty parameters in penalty functions or in augmented
Lagrangian functions and their variants. Let us consider the nonlinear optimization problems with
inequality constraints
min f ðxÞ
subject to ð18:1Þ
ci ðxÞ  0, i ¼ 1, . . . , m,

where the objective function f : ℝn ! ℝ and the functions ci : ℝn ! ℝ, i ¼ 1, . . ., m, defining the

constraints are supposed to be twice continuously differentiable. The methods for solving this
problem are based on the Newton method. Given an estimate xk of the solution x of (18.1), a linear
or a quadratic approximation of (18.1) is solved, thus obtaining a new estimate xkþ1 which is expected
to be better than the previous one. Near a solution, this approach is guaranteed to be convergent.
However, far away from the solution the sequence {xk} generated by the above procedure may not
converge. In this situation, away from the solution, the idea is to use the Newton method again,
but considering the penalty or the merit functions. The penalty functions or the merit functions
are a combination of the objective function and a measure of the constraints violation, such as
h(x) ¼ kc(x)+k1, where c(x) ¼ [c1(x), . . ., cm(x)]T and cþi ¼ max f0, ci g: A well-known example is
the l1 exact penalty function p(x, σ) ¼ f(x) þ σh(x), where σ > 0 is the penalty parameter. If σ is
sufficiently large, then this penalty function can be minimized in an iterative procedure to ensure
progress to the solution.
This approach based on the penalty or the merit functions is interesting, but not without
difficulties. Unfortunately, a suitable value of the penalty parameter depends on the solution of the
problem (18.1), that is σ > kλkD, where λ is the vector of the Lagrange multipliers and k.kD is the
dual norm. Hence, the determination of a suitable penalty parameter is difficult. Moreover, if the
penalty parameter is too large, then any monotonic method would be forced to follow the nonlinear
constraint manifold very closely, resulting in much shortened Newton steps and therefore in slow
convergence. All these deficiencies are very well known and for their avoidance; some computational
schemes have been imagined like methods based on the augmented Lagrange function (modified or
not) or interior point methods. In this section, we present the methods for solving nonlinear

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 647
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_18
648 18 Filter Methods

optimization problems known as filter methods (Fletcher, & Leyffer, 1998, 1999, 2002), (Fletcher,
Leyffer, & Toint, 2002b, 2006).
The idea of a filter method is not of combining the objective and the constraints violation into a
single function. Instead, (18.1) is viewed as a biobjective optimization problem that minimizes f(x)
and h(x). However, as we can immediately see, the second objective is more important because we
must ensure that h(x) ¼ 0. Fletcher and Leyffer (2002) introduced the concept of domination taken
from the multi-objective optimization. A point xk dominates a point xl if and only if f(xk)  f(xl) and
h(xk)  h(xl). Therefore, a filter is defined as a list of pairs (h(xl), f(xl)) such that no pair dominates
another pair.
This concept can be implemented in different ways. The filter methods use the sequential linear
programming or the sequential quadratic programming in the context of the trust-region methods. At
iteration k ¼ 0, the filter is initialized as Fk ¼ {(u, 1)}, where u is an upper bound on the acceptable
constraint violation. Solving the linear programming problem or the quadratic programming problem,
a solution s is obtained. If this point is acceptable by the filter, then we set xkþ1 ¼ xk þ s and possibly
increase the trust-region radius and update the filter (adding the previous point and removing any
dominated entries). On the other hand, if the step is dominated by the current filter, then we reject it,
set xkþ1 ¼ xk,reduce the trust-region radius, and solve the linear or the quadratic programming
problem in a new context. For the process to be convergent, this general description of the filter
method needs some refinements:

1. Filter envelope. To avoid the convergence to an infeasible limit point where h(x) > 0, an envelope
around the current filter is added (Chin, 2002), (Chin, & Fletcher, 2003). A new iterate is
acceptable if for all l 2 Fk, hkþ1  βhl or fkþ1  fl  γhkþ1, where fk ¼ f(xk), hk ¼ h(xk) and
0 < β, γ < 1 are constants.
2. Sufficient reduction. The filter alone cannot ensure convergence to the stationary points. For
example, if the sequence {xk} satisfies the condition hkþ1  βhk, then the iterates could converge
to an arbitrary feasible point. Therefore, if the constraint violation becomes small, a sufficient
reduction condition can be imposed as follows: denote the predicted reduction by Δqk as the value
of the objective function of the linear or of the quadratic problem and introduce the following
switching condition: if Δqk > 0, then check fk  fkþ1  σΔqk, where σ 2 (0, 1) is a constant.
3. Feasibility restoration. By reducing the trust-region radius of the linear or the quadratic problems,
these may become inconsistent. That is, the current point is too far from the feasibility region to get
significant progress toward the solution. In this case, a sequential linear or quadratic programming
problem is invoked, that minimizes the constraint violation h(x). This restoration phase is left once
an acceptable point has been obtained.

The filter algorithm contains an inner and an outer iteration. During the inner iteration, the trust-
region radius is reduced until either an acceptable point is found or the restoration phase is initiated.
The aim of the restoration phase is to find an acceptable iterate xkþ1 such that the linear (or quadratic)
programming problem is compatible for some smaller values of the trust-region radius. The iterate
and the filter are updated in the outer iteration, which also initializes the trust-region radius to a given
lower bound. The filter is updated by adding (hk, fk) to Fk, which corresponds to an h-type iteration
after we move to xkþ1. Fletcher and Leyffer (2002) motivate the switching condition as follows.
When close to a feasible point, it is expected that the linear or the quadratic model of the problem
predicts a decrease in the objective function, that is Δqk > 0. However, far away from a feasible point
the predicted reduction is usually negative, that is Δqk < 0. The iterations that satisfy the switching
condition are called f-type iterations. All the other steps are called h-type iterations. Observe that if
hk ¼ 0 at a non-stationary point, then Δqk > 0, therefore implying that only f-typed steps can be
18.1 Sequential Linear Programming Filter Algorithm 649

accepted. In other words, no points for which hk ¼ 0 are added to the filter. This ensures that the
restoration phase generates only filter-acceptable points.

18.1 Sequential Linear Programming Filter Algorithm

Suppose that the constraints from (18.1) include some linear constraints, for example, simple bounds
on variables that define a non-empty bounded region X. In the current point x, for the problem (18.1)
we can associate the following linear programming sub-problem modified with the trust-region radius
ρ > 0, as

min d2ℝn gT d
subject to:
ð18:2Þ
ci þ aTi d  0, i ¼ 1, . . . , m,
kdk1  ρ,

where g ¼ ∇ f(x), ci ¼ ci(x) and ai ¼ ∇ ci(x). Observe that the l1 norm is used to define the trust-
region because it is very easy to be implemented by adding simple bounds to the linear programming
subproblem. If it exists, let d be the solution of (18.2). Denote

Δl ¼ gT d ð18:3Þ

as the predicted reduction in f(x) and with

Δf ¼ f ðxÞ  f ðx þ d Þ ð18:4Þ

as the actual reduction in f(x). The measure of the constraint infeasibility which is used in this text is
given by

hð c Þ ¼ kc þ k1 , ð18:5Þ

where cþ i ¼ max f0, ci g, i ¼ 1, . . ., m.

A filter is defined by pairs of values (h, f ) obtained by evaluating h(c(x)) and f(x) for different
values of x. A pair (hi, fi) dominates another pair (hj, fj) if and only if both hi  hj and fi  fj are
satisfied, thus indicating that the former point is at least as good as the latter in respect to both
measures. The filter is defined to be a list of pairs (hi, fi) such that no pair dominates any other. A point
x is acceptable for inclusion in the filter if its (h, f ) pair is not dominated by any entry in the filter. This
is the condition that
either h < hi or f < f i ð18:6Þ

for all i 2 F, where F denotes the current set of filter entries. We may include a point in the filter by
which we mean that its pair (h, f ) is added to the list of pairs in the filter and any pairs in the filter that
are dominated by the new pair are deleted from the filter.
Observe that this definition of the filter is not suitable for proving the convergence because it
allows points to accumulate in the neighborhood of a filter entry that has hi > 0. However, this may be
corrected by defining a small envelope around the current filter in which the points are accepted.
Therefore, the condition for a point to be acceptable to the filter is that its pair (h, f ) satisfies
650 18 Filter Methods

either h  βhi or f  f i  γhi ð18:7Þ

for all i 2 F, where β and γ are parameters such that 1 > β > γ > 0, with β close to one and γ close to
zero. Observe that the first inequality from (18.7) is exactly the reduction in h. The dependence on hi
of the second inequality is a small artifact to enable the convergence of the algorithm. Since γ is small,
the extra term has little practical impact.
It is also convenient to introduce an upper bound
hðcðxÞÞ  βu ð18:8Þ

(u > 0) on the constraint infeasibility, which is readily implemented by initializing the filter with the
entry (u, 1).
As we know, a common feature in a trust-region algorithm for the unconstrained minimization is
the use of a sufficient reduction criterion
Δf  σΔl, ð18:9Þ

where Δl is positive and σ 2 [0, 1) is a given parameter. However, in the trust-region algorithm, Δl
may be negative or even zero, case in which this test is no longer appropriate. Fletcher and Lyeffer
(2002) suggest using (18.9) with σ  γ only when Δl is sufficiently positive, which can be achieved by
testing the inequality

Δl  δh2 , ð18:10Þ

where h refers to h(c(x)) evaluated at the current point and δ > 0 is a given parameter close to zero.
Again, the dependence of the right-hand side of (18.10) on h is an artifact to enable the convergence to
be proved.
Let us denote the linear programming subproblem (18.2) by LP(x, ρ), then the sequential linear
programming filter algorithm is as follows.

Algorithm 18.1 Sequential linear programming filter—filterSD

1. Choose an initial point x 2 X and initialize the filter with (u, 1). Set k ¼ 1
2. Consider the restoration phase to find a point xk 2 X acceptable for inclusion in the filter such that
LPðxk , e
ρÞ is compatible for a eρ  ρ0 , and initialize ρ ¼ e ρ
3. Solve the subproblem LP(xk, ρ)
4. If LP(xk, ρ) is infeasible, then place (hk, fk) in the filter (h-type iteration). Set k ¼ k þ 1 and go to step 2
5. If LP(xk, ρ) has the solution d, then if d ¼ 0, stop; the current point is a KKT point for the problem
6. If d 6¼ 0, then evaluate f(xk þ d ) and c(xk þ d )
7. If xk þ d is not acceptable for inclusion in the filter, then set ρ ¼ ρ/2 and go to step 3
8. If xk þ d is acceptable for inclusion in the filter and Δf < σΔl and Δl  δ(hk)2, then set ρ ¼ ρ/2 and go to
step 3
9. If xk þ d is acceptable for inclusion in the filter and Δf < σΔl and Δl  δ(hk)2 are not satisfied, then set
ρk ¼ ρ, dk ¼ d, Δlk ¼ Δl and Δfk ¼ Δf
10. If Δlk < δ(hk)2, then the pair (hk, fk) is included in the filter (h-type iteration)
11. Set xkþ1 ¼ xk þ dk, k ¼ k þ 1, initialize ρ  ρ0 and go to step 3 ♦

Some important details as well as the convergence of the filter algorithm with sequential linear
programming are given in (Fletcher, Leyffer, & Toint, 1999). Mainly, the convergence theorem is
based on the necessary optimality condition by Fritz John and on the following standard hypothesis:
18.1 Sequential Linear Programming Filter Algorithm 651

the set X defined by the linear constraints of the problem (18.1) is non-empty and bounded and the
functions f(x) and c(x) of the problem are twice continuously differentiable on X. When the algorithm
is applied, one of four different possible outcomes can occur:

(i) The restoration phase iterates infinitely and fails to find a point x which is acceptable to the filter
and for which LP(x, ρ) is compatible for some ρ  ρ0.
(ii) A KKT point is found (d ¼ 0 solves LP(xk, ρ) for some k).
(iii) All the iterations are f-type iterations for k sufficiently large.
(iv) There exists an infinite subsequence of h-type iterations.

The following theorem proves the convergence of the sequential linear programming filter
algorithm, as specified in (Fletcher, Leyffer, & Toint, 1999).

Theorem 18.1 If the standard hypothesis holds, then for the sequential linear programming filter
algorithm either (i) or (ii) occurs or the sequence of the iterates generated by the algorithm has an
accumulation point that satisfies the Fritz John necessary optimality conditions. ♦

More discussions about the sequential linear programming filter algorithm (filterSD) can be found
in (Fletcher, & Leyffer, 1998, 2002). In the following, let us present some numerical results with this
algorithm.

Example 18.1 To illustrate the running of the filterSD algorithm, let us consider the nonlinear
optimization problem (Duran, & Grossmann, 1986):
min ð10x1  18 ln ðx2 þ 1Þ  7x3 þ 5x4 þ 6x5 þ 8x6  19:2 ln ðx1  x2 þ 1Þ þ 10Þ
subject to
0:8 ln ðx2 þ 1Þ þ 0:96 ln ðx1  x2 þ 1Þ  0:8x3  0,
ln ðx2 þ 1Þ þ 1:2 ln ðx1  x2 þ 1Þ  x3  2x6  2,
x2  x1  0,
x2  2x4  0,
x1  x2  2x5  0,
x4 þ x5  1,

where
0  x1  2, 0  x2  2, 0  x3  1, 0  x4  1, 0  x5  1, 0  x6  1:

Let x0 ¼ [0, 0, 0, 0, 0, 0]T be the initial point. The filterSD algorithm gives the following results.
The objective function value in the initial point is f(x0) ¼ 10. The value of the constraints at the initial
point are ci(x0) ¼ 0, i ¼ 1, . . ., 6. The optimal solution is x and the value of the constraints in the
optimal point c(x) are as follows:
652 18 Filter Methods

2 3 2 3
1:146514 0
6 0:546592 7 6 7
6 7 6 0 7
6 7 6 7
6 1 7 6 0:5999216 7
x ¼ 6 7
6 0:273296 7 cðx Þ ¼ 6
6
7:
7
6 7 6 0 7
6 7 6 7
4 0:299960 5 4 0 5
0 0:57325729

The value of the objective function in the optimal point is equal to f(x) ¼ 0.7592843922. This
solution was obtained in 5 iterations, 29 evaluations of the functions of the problem, and
23 evaluations of the gradients of the functions. Along the iterations, the value of the trust-region
radius was ρ ¼ 1000. ♦

Numerical Study—filterSD: Solving Applications from the LACOP Collection

Table 18.1 presents the performances of an implementation of the filterSD algorithm given by
Fletcher for solving 8 nonlinear optimization applications from the LACOP collection described in
Appendix C.

Table 18.2 presents a comparison of filterSD (Table 18.1) versus NLPQLP (Table 15.4) and
KNITRO/ACTIVE (option 3) (Table 15.6) for solving some nonlinear optimization applications from

Table 18.1 Performances of filterSD for solving 8 applications from the LACOP collection

n me mc #it #nf #ng vfo

ALKI 10 3 8 10 89 66 -1768.8069
ELCH 10 3 0 4 293 238 -47.761090
PREC 8 0 6 12 383 301 3.95116351
PPSE 9 6 0 5 27 24 5055.01180
MSP3 13 0 15 9 45 42 97.5482075
MSP5 16 0 21 21 195 165 174.786994
POOL 34 20 0 3 7 7 2785.80000
LATHE 10 1 14 12 43 43 -4434.0019

In this table we have: n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the number of
inequality constraints, #it ¼ the number of iterations to get a solution, #nf ¼ the number of the evaluations of the
functions defining the problem, #ng ¼ the number of the evaluations of the gradients of the functions, vfo ¼ the value of
the objective function in the optimal point

Table 18.2 Comparison of filterSD versus NLPQLP and KNITRO

filterSD NLPQLP KNITRO (3)

#it #nf #it #nf #it #nf
ELCH 4 293 21 30 18 19
ALKI 10 89 53 69 22 58
PREC 12 383 19 20 45 117
LATHE 12 43 19 19 83 242
PPSE 5 27 9 10 6 11
MSP3 9 45 146 264 7 9
POOL 3 7 22 22 15 33
TOTAL 55 887 289 434 196 489
18.2 Sequential Quadratic Programming Filter Algorithm 653

the LACOP collection subject to the number of iterations (#it) and to the number of evaluations of the
functions defining the problem (#nf).
In filterSD, for solving the linear programming subproblems with simple bounds on variables, a
recursive form of an active-set method is used at each iteration. To solve degeneracy, the Wolfe
method is implemented. A limited memory reduced gradient sweep method is used for minimization
in the null space, so the KKT point is usually a local minimizer. Observe that filterSD has very good
numerical performances for solving nonlinear optimization problems (see Table 18.2). The main
advantage of this algorithm is that it avoids using second derivatives and also avoids storing an
approximate reduced Hessian matrix by taking a limited memory spectral gradient approach based on
the Ritz values (Fletcher, 2011).

18.2 Sequential Quadratic Programming Filter Algorithm

In this section, we give some details on the filter algorithm with sequential quadratic programming for
solving the nonlinear optimization problem (18.1). The filter method views (18.1) as a biobjective
nonlinear optimization problem: the first is the minimization of the objective function f and the
second is the satisfaction of the constraints. These two conflicting aims can be formalized as
min f ðxÞ ð18:11Þ

and

min hðcðxÞÞ, ð18:12Þ

where
  X m
hðcðxÞÞ≜cðxÞþ 1 ≜ ci ðxÞþ ð18:13Þ
i¼1

is the l1 norm of the constraint violation and ci(x)+ ¼ max {0, ci(x)}, i ¼ 1, . . ., m. Observe that the
problem of satisfiability has been written as a minimization problem. Hence, (18.11) and (18.12)
represent a biobjective nonlinear optimization problem. Here, the l1 norm is used because it has some
convenient features that can be exploited in defining the algorithm.
The fundamental idea of the algorithm is to use the filter as a criterion for accepting or rejecting a
step in the sequential quadratic programming method. In other words, given the current point xk, the
solution of the quadratic programming subproblem
1 T
min d W k d þ d T gk
d 2
subject to ð18:14Þ
ATk d þ ck  0,
kdk1  ρ,

generates a step dk. Here, g(x) ¼ ∇ f(x), Wk is (an approximation to) the Hessian of the Lagrangian
L(x, λ) ¼ f(x) þ λTc(x), Ak is the Jacobian of the constraints and ck is the value of the constraints at the
point xk. Set xkþ1 ¼ xk þ dk. The new point xkþ1 is accepted by the filter if the corresponding pair
(hkþ1, fkþ1) is not dominated by any other point in the filter. Otherwise, the step is rejected and the
trust-region radius ρ is reduced. The use of the l1 norm in (18.14) ensures that (18.14) remains
tractable as quadratic programming.
654 18 Filter Methods

The above strategy based on solving a sequence of quadratic programming subproblems must be
completed in order to cope with the situations in which the subproblems are infeasible. In a sequential
quadratic programming trust-region method, reducing the trust-region radius will give rise to an
infeasible quadratic programming subproblem if the current point is infeasible in the nonlinear
optimization problem. Usually, a trust-region algorithm arrives at this situation after rejecting a
number of consecutive steps. Thus, it is not sufficient to simply increase the trust-region radius to
regain feasibility. An infeasible quadratic programming subproblem also occurs when the linearized
constraints are themselves inconsistent. To overcome this situation, Fletcher and Leyffer (2002)
chose the strategy to minimize h(c(x)).This is referred to as the restoration phase, which tries to get
close to the feasible region of the nonlinear optimization problem. A possible outcome is that the
restoration phase finds a nonzero minimum of h(c(x)) which is taken as an indication that the
nonlinear optimization problem is infeasible. Otherwise, we can assume that the algorithm finds a
point at which the quadratic programming subproblem is feasible and therefore the above method can
be continued from this point.
Having in view all these developments, the basic sequential quadratic programming filter algo-
rithm starts the iterations with an initial guess x0 and an initial estimate of the Lagrange multipliers λ0.
On the subsequent iterations the Lagrange multipliers are updated whenever the sequential quadratic
programming subproblem generates a new point that is accepted by the filter.

Algorithm 18.2 Basic sequential quadratic programming filter—filterSQP

1. Choose an initial point x0, a value for the parameter ρ and set k ¼ 1
2. Test a criterion for stopping the iterations
3. Solve the quadratic programming subproblem (18.14)
4. If (18.14) is infeasible, find a new point xkþ1 using the restoration phase; otherwise, determine the step dk
and provisionally set xkþ1 ¼ xk þ dk and continue with step 5
5. If the pair (hkþ1, fkþ1) is acceptable to the filter, then:
accept xkþ1 and add the pair (hkþ1, fkþ1) to the filter,
remove the points dominated by the pair (hkþ1, fkþ1),
possibly increase the trust-region radius,
otherwise,
set xkþ1 ¼ xk and reduce the trust-region radius
6. Set k ¼ k þ 1 and go to step 2 ♦

Algorithm 18.2 is the basic filter algorithm with sequential quadratic programming. It is very easy
to implement, but some algorithmic extensions are required to exclude the situations in which it might
fail or might converge slowly. Fletcher and Leyffer (1998, 2002) present some extensions and
techniques for its protection against certain difficulties in solving nonlinear optimization problems.
Some of these extensions are as follows.
Second-order correction step. One of the most important property of any sequential quadratic
programming algorithm is that it usually has second-order convergence near the solution. However,
the use of a non-differentiable penalty function can preclude the acceptance of the unit step arbitrarily
close to the solution, thus preventing second-order convergence. This is known as the Maratos effect.
This difficulty can be avoided by computing a correction to the step that eliminates second-order
contributions of the nonlinear constraints. This is the second-order correction step. In the filter
algorithm, it is quite possible for a sequential quadratic programming unit step to increase both the
objective and the constraint violation functions.
18.2 Sequential Quadratic Programming Filter Algorithm 655

The second-order correction step is computed whenever xkþ1 ¼ x(k þ 1, 0) is rejected by the filter in
Algorithm 18.2. In this case, the quadratic programming subproblem that is solved is defined as
1 T
min d W k d þ d T gk
d2
subject to ð18:15Þ
ATk d þ cðkþ1,lÞ  ATk d ðk,lÞ  0,
kd k1  ρ,

for l ¼ 0, 1, . . ., where c(k þ 1, 0) ¼ ckþ1 and d(k, 0) ¼ dk. Let dbk ≜dðk,lþ1Þ be the solution of (18.15). The
new point x(kþ1, l ) ¼ xk þ d(k, lþ1) is tested in the filter. If the pair (h(kþ1, l ), f(kþ1, l )) is acceptable to
the filter, then step d(k, lþ1) is accepted and the trust-region radius may be increased. Otherwise, a
sequence of second-order correction steps is performed, generating a sequence of trial points x(kþ1, l ),
l ¼ 0, 1, . . ., until one of the following holds:

(i) An acceptable trial point x(kþ1, j ) is found.

(ii) An infeasible quadratic programming problem is detected.
(iii) The rate of convergence of the second-order correction steps defined by

hðkþ1,lÞ
r≜
hðkþ1,l1Þ

is considered to be too slow, or

(iv) An almost feasible point with h(kþ1, l ) < ε is generated, where ε is a tolerance used in solving the
problem (18.1).

In case (i), the next iteration is xkþ1 ¼ x(kþ1, j ) and the filter with sequential quadratic programming
continues from step 2. In all the other cases, the steps are rejected and the trust-region radius is
reduced. The last case (iv) ensures the finiteness of this process. The best second-order correction step
is stored by using a penalty function estimate to rank the steps.
Feasibility restoration phase. This is the most complex part of the algorithm. One difficulty in
using the trust-region approach is that the reduction of the trust-region radius may cause the quadratic
programming subproblem to become infeasible. Besides, the linearizations of the nonlinear
constraints may themselves be inconsistent. The strategy of Fletcher and Leyffer (2002) for dealing
with this situation is to enter a restoration phase in which the purpose is to get closer to the feasible
region by minimizing h(c(x)).
If an infeasible quadratic programming problem is detected, then the solver exits with a solution of
the linear programming problem
X
min ∇cj ðxk ÞT d
d
j2J
subject to: ð18:16Þ
T ⊥
∇cj ðxk Þ d þ cj ðxk Þ  0, j 2 J ,
kdk1  ρ:
656 18 Filter Methods

In other words, the quadratic programming solver partitions the constraints into two index sets:
J ⊂ {1, 2, . . ., m} and its complement J⊥ ¼ {1, 2, . . ., m}\J. The set J contains infeasible
constraints at the point xk: cj(xk) þ ∇ cj(xk)Td > 0, j 2 J, whose l1 sum is minimized at the solution
to the linear programming problem (18.16) with respect to the constraints in J⊥ being satisfied.
The strategy in the restoration phase is to apply a sequential quadratic programming trust-region
method to the nonlinear problem
X
min cj ðxÞþ
x
j2J
subject to: ð18:17Þ
cj ðxÞ  0, j 2 J ⊥ ,

that is defined by this partitioning into the sets J and J⊥. However, the difficulty is that it is quite
possible for J and J⊥ to change from iteration k to iteration k þ 1, thus making it difficult to enforce
convergence. Therefore, the restoration phase consists of a sequence of sequential quadratic pro-
gramming iterations that continues while the subproblem (18.14) is infeasible. At each iteration, first
check the feasibility of the system

∇cj ðxk ÞT d þ cj ðxk Þ  0, j ¼ 1, . . . , m, ð18:18aÞ

kdk1  ρ: ð18:18bÞ

If the system (18.18) has a feasible solution, then the restoration phase ends. Otherwise, the sets
J and J⊥ which solve (18.16) are determined. Next, a quadratic programming subproblem is
constructed by adding the second-order term (1/2)dTWkd into the objective function of (18.16), where
X X
W ðx, λÞ ¼ ∇2 cj ðxÞ þ λj ∇2 cj ðxÞ: ð18:19Þ
j2J j2J ⊥

The multipliers λj, j 2 J⊥, are those obtained from the solution of (18.16). Note that for solving
(18.16) a warm start can be used, i.e., the solution of (18.16) can be used to initialize the solution of
the new quadratic programming problem. The iterations in the restoration phase are continued until
either a feasible quadratic programming problem is encountered (case in which the algorithm can
return to the basic sequential quadratic programming algorithm) or an infeasible KKT point of
(18.17) is found for some sets J and J⊥ (case in which the algorithm terminates with the indication
that (18.1) is locally infeasible).
For solving (18.17), Fletcher and Leyffer (2002) suggest using the filter method by introducing the
so-called restoration filter (or phase I filter) defined by
X
hJ ≜hðcJ ðxÞÞ≜ cj ðxÞþ
j2J

and similarly hJ ⊥ : A phase I filter is a list of pairs ðhJ , hJ ⊥ Þ such that no pair dominates any other.
Fletcher and Leyffer (2002) present a computational situation which leads to the introduction of
the blocking entry in the filter. Consider the situation in which (18.1) has a global solution x and a
worse local solution x, as well as that x0 is a feasible point fairly close to x, but the subsequent
iterates xk are convergent to x. Then, if f(x0)  f(x), then the filter entry (h0, f0) prevents the
sequential quadratic programming iteration from converging to x. In this case, we would like to
backtrack to x0, but our decision not to store x0 with the filter information precludes this. The pair
18.2 Sequential Quadratic Programming Filter Algorithm 657

(h0, f0) is called blocking entry in the filter. Some more details are found in (Fletcher, & Leyffer,
2002). The algorithm for the restoration phase is as follows.

Algorithm 18.3 Feasibility restoration algorithm

1. Consider the current point xk, the trust-region radius ρ and a constraint upper bound u which measures the
violation of the constraints
2. Test a criterion for stopping the iterations
3. Solve (18.16) and determine the sets J and J⊥, which remain fixed for this iteration
4. If (18.16) is feasible, then return to a normal sequential quadratic programming and clear the restoration
filter. Otherwise, add the quadratic term (18.19) to the objective function of (18.16) and solve phase I of
the quadratic programming algorithm. Let dk be the solution and set xkþ1 ¼ xk þ dk
 
5. If hkþ1 kþ1
J , hJ ⊥ is acceptable to the phase I filter, then:
 
accept xkþ1 and add the pair hkþ1 kþ1
J  , hJ ⊥ to the restoration filter,

remove the points dominated by hkþ1 J , hkþ1
J ⊥ from the filter,
possibly increase the trust-region radius ρ.
Otherwise, solve a sequence of quadratic programming second-order correction problems to get the step
dbk : Set xbkþ1 ¼ xk þ dbk and go to step 6
 kþ1 kþ1 
6. If hbJ , hbJ⊥ is acceptable to the phase I restoration filter, then:
 kþ1 kþ1 
accept xbkþ1 and add hbJ , hbJ⊥ to the restoration filter,
 kþ1 kþ1 
remove the points dominated by hbJ , hbJ⊥ from the restoration filter,
possibly increase the trust-region radius ρ.
Otherwise, if J has changed from the iteration k  1 and hbðk,LÞ < u, then:
accept the best second-order correction step, i.e., set xkþ1 ¼ xbkþ1 ,
remove all the blocking entries from then restorationofilter,
reduce the upper bound u to u ¼ max hbðk,LÞ , u=10 ,
otherwise:
reject the step, i.e., set xkþ1 ¼ xk,
reduce the trust-region radius ρ.
7. Set k ¼ k þ 1 and go to step 2 ♦

Observe that the trust-region radius is changed in the feasibility restoration algorithm. At its return
to the sequential quadratic programming, the current value of ρ reflects how well a first-order Taylor
series approximates the nonlinear constraints in xk. Therefore, this value of ρ is used in the next step of
the sequential quadratic programming.
Sufficient reduction. A new iterate xkþ1 is acceptable to the filter if
hkþ1  βhl ð18:20Þ

or if

f kþ1  f l  max fα1 Δql , α2 hl μl g ð18:21Þ

are satisfied for all the filter entries l. Here, β, α1 and α2 are positive constants which can be taken
to be: β ¼ 0.99, α1 ¼ 0.25, and α2 ¼ 0.0001, respectively. An estimation of the penalty parameter μl is
computed as the least power of ten but larger than kλlk1 and cuts this value off so as to lie in the
interval [106, 106]. The value of hlμlcan be used as a predicted reduction in f. Both Δql and μl are
stored along with fl and hl. Observe that μl measures the marginal effect of the changes in f due to
658 18 Filter Methods

changes in h. This is the reason for using the term α2hlμl in (18.21). A quadratic programming step
predicts a reduction of hl to zero, equivalent to a predicted reduction in f of hlμl.
Beyond the extreme points of the filter. Mainly, the filter is a reliable oracle for deciding whether or
not to accept a step as long as the new constraint violation lies within the range of the constraint
violations recordered in the filter. However, the current heuristics do not exclude the possibility of
generating a sequence of filter entries in which {fk} is monotonically increasing and {hk} is
monotonically decreasing without converging to a KKT point. But, as long as the sufficient reduction
criterion in h is satisfied by the sequence, the points will be accepted by the filter. To overcome this
possibility, a new additional heuristic is introduced, when hkþ1 provides sufficient reduction from h1,
where h1 now refers to the leftmost entry in the filter. An overestimate μ ¼ 1000μ1 of the penalty
parameter is considered. It requires xkþ1, which provides a reduction in the resulting exact penalty
function. The value of the exact penalty function corresponding to the leftmost entry f1, h1 is f1 þ μh1.
The new point is then accepted if, in addition to the sufficient reduction from h1, it satisfies
fkþ1 þ μhkþ1  f1 þ μh1. Fletcher and Leyffer (2002) called this the North-West corner rule. A
similar rule is applied in the South-East corner of the filter.
Now, the complete sequential quadratic programming algorithm based on the filter concept can be
presented as follows.

Algorithm 18.4 Filter sequential quadratic programming—filterSQP

1. Choose the initial point x0 and the initial trust-region radius ρ. Set k ¼ 0
2. Test a criterion for stopping the iterations
3. Solve (18.14) for a step dk. If (18.14) is infeasible, then continue with the restoration phase (see
Algorithm 18.3) and return to the normal sequential quadratic programming when xkþ1 is found, whose
corresponding quadratic programming is feasible. Otherwise, set xkþ1 ¼ xk þ dk and go to step 4
4. If (hkþ1, fkþ1) is acceptable to the filter, then:
accept xkþ1 and add (hkþ1, fkþ1) to the filter,
remove the points dominated by (hkþ1, fkþ1)from the filter,
possibly increase the trust-region radius ρ,
otherwise, go to step 5
5. If hkþ1 > 0, then solve a sequence of quadratic programming problems for the second-order correction
b b
 d k : Set xbkþ1
step  ¼ xk þ d k
If hbkþ1 , fbkþ1 is acceptable to the filter, then:
 
accept xbkþ1 and hbkþ1 , fbkþ1 to the filter,
 
remove the points dominated by hbkþ1 , fbkþ1 from the filter,
possibly increase the trust-region radius ρ,
otherwise, go to step 6
6. If there is no acceptable point in the filter, then:
If the algorithm is in the first iteration after the restoration phase, then:
accept the best second-order correction step, i.e., set xkþ1 ¼ xbkþ1 ,
remove all the blocking entries from n the filter, o
reduce the upper bound to: u ¼ max hbkþ1 , u=10 ,
otherwise, reject step dbk , i.e., set xkþ1 ¼ xk and reduce the trust-region radius ρ
7. Set k ¼ k þ 1 and go to step 2 ♦

Some more details regarding a specific implementation of this algorithm can be found in Fletcher
and Leyffer (2002).
18.2 Sequential Quadratic Programming Filter Algorithm 659

Example 18.2 Let us consider the nonlinear optimization problem from the example 18.1 solved by
means of the filter with sequential linear programming—filterSD. The filterSQP algorithm in the
implementation given by Fletcher and Leyffer (1999) with initial point x0 ¼ [0, 0, 0, 0, 0, 0]T gives
the following results.

The value of the objective function in the initial point is f(x0) ¼ 10. FilterSQP determines the
solution x ¼ [1.14651, 0.54659, 1, 0.27329, 0.29996, 0]Tfor which the value of the objective
is f(x) ¼ 0.75928439. This solution was obtained in 4 iterations, 5 evaluations of the functions
defining the problem, 5 evaluations of the gradients of the functions, and 5 evaluations to the Hessian
of the Lagrange function. The number of quadratic programming subproblems is 4. The value of the
norm of constraints at the solution is kc(x)k ¼ 1.3222E  10. The final trust-region radius is ρ ¼ 10.
The norm of the KKT optimality conditions is equal to 1.393E  8. ♦

The global convergence of the trust-region sequential quadratic programming filter algorithm,
filterSQP, under reasonable conditions (the iterates xk are in a compact set, the functions f(x) and c(x)
are twice continuously differentiable, the Hessian of the Lagrange function associated to the problem
is bounded), was proved by Fletcher, Gould, Leyffer, Toint, and Wächter (2002a). The idea of the
proof is to decompose the step into its normal and tangential components. This allows for an
approximate solution of the quadratic subproblem and incorporates the safeguarding test described
above. The normal step is responsible for the feasibility of the linearized constraints from (18.14).
The tangential step reduces the value of the objective function model while continuing to satisfy the
constraints from (18.14).

Notes and References

The filter method was suggested by Fletcher at a SIAM Conference in 1999. The filter method is
motivated by the aim of avoiding the need to choose penalty parameters, as it occurs with the use of l1
penalty functions or the augmented Lagrangian functions. The filter methods know two
implementations: with sequential linear programming (filterSD) and with sequential quadratic pro-
gramming (filterSQP). The package filterSD calls the linearly constrained problem solver glcpd or the
quadratic programming solver qlcpd. The package filterSQP (Fletcher, & Leyffer, 1998) implements
a sequential quadratic programming solver with a “filter” to promote global convergence. A quadratic
approximation to the original problem is solved within a trust-region, where trust the region-radius is
changed adaptively by the algorithm. The solver runs with a dense or a sparse linear algebra package
and a robust quadratic programming solver bqpd. bqpd is a null-space active set method that builds up
a factorization of the reduced Hessian, which is the projection of the Hessian onto the null space of
constraints that are currently regarded as being active.
The global convergence of the filter method with sequential linear programming was proved by
Fletcher, Leyffer, and Toint (1999). Later on, the global convergence of the filter method with
sequential quadratic programming was presented by Fletcher, Leyffer, and Toint (2002b), Ulbrich
(2004), and Fletcher, Leyffer, and Shen (2009). These filter methods were independently suggested
by Surry, Radcliffe, and Boyd (1995) and Lemaréchal, Nemirovskii, and Nesterov (1995). Ulbrich,
Ulbrich, and Vicente (2004) considered a trust-region filter method based on the acceptance of the
trial steps using the norm of the optimality conditions. On the other hand, Benson, Shanno, and
Vanderbei (2002a, 2002b) proposed several heuristics based on the idea of the filter method, for
which improved efficiency was reported versus the merit function approach. Even if the filter methods
can solve optimization problems, it is not known yet whether the filter strategy has advantages over
the merit functions.
 Interior-Point Filter Line-Search
19

In this chapter, a combination called IPOPT of the interior-point algorithm with the filter line-search
for solving large-scale nonlinear optimization problems, proposed by Wächter and Biegler (2005a, b),
is presented. As we know, to allow convergence from poor starting points and to enforce progress to
the solution, the interior point methods both in the trust-region and in the line-search frameworks with
exact penalty merit function were developed. For example, KNITRO uses the l1 exact penalty
function (Byrd, Gilbert, & Nocedal, 2000; Byrd, Hribar, & Nocedal, 1999). On the other hand,
Fletcher and Leyffer (2002) proposed the filter methods as an alternative to the merit functions, as a
tool for global convergence guarantee in the algorithms for nonlinear optimization. The idea of the
filter is that the trial points are accepted if they improve the objective function value or the constraint
violation instead of a combination of these two measures defined by the merit function. Even if the
filter methods include different heuristics, they have been adapted to the barrier methods in a number
of ways. For example, Ulbrich, Ulbrich, and Vicente (2004) considered a trust-region filter method
and accepted the trial step on the basis of the norm of the optimality conditions. Benson, Shanno, and
Vanderbei (2002a) proposed several heuristics using the filter methods, for which the improved
efficiency is reported compared to their previous merit function approach. The global convergence of
an interior point algorithm with filter line-search was analyzed by Wächter and Biegler (2001). Here,
the assumptions made for the analysis of the global convergence are less restrictive than those made
for the line-search interior-point methods for nonlinear optimization developed, for example, by
El-Bakry, Tapia, Tsuchiya, and Zhang (1996), Yamashita (1998) or Tits, Wächter, Bakhtiari, Urban,
and Lawrence (2003).
In this context, IPOPT is an interior-point filter line-search algorithm for solving large-scale
nonlinear optimization problems developed by Wächter and Biegler (2001, 2006). This algorithm
combines the primal-dual interior-point algorithms with the filter line-search for solving problems of
the form
min f ðxÞ ð19:1aÞ
subject to
cðxÞ ¼ 0, ð19:1bÞ

xL  x  xU , ð19:1cÞ

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 661
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_19
662 19 Interior-Point Filter Line-Search

where x 2 ℝn, xL 2 ℝn and xU 2 ℝn are lower and upper bounds on variables, f : ℝn ! ℝ and
c : ℝn ! ℝm, with m  n, are assumed to be twice continuously differentiable. Problems with general
nonlinear inequality constraints can be reformulated in the above form by introducing the slack
variables.
In the following, we shall present the primal-dual interior-point algorithm IPOPT, which involves
the step computation, the line-search with filter, the second-order corrections, and the restoration of
feasibility as well as some heuristics which improve the performances of the algorithm. Finally,
numerical results for solving some nonlinear optimization applications together with comparisons are
presented.

19.1 Basic Algorithm IPOPT

The Primal-Dual Barrier Approach

To simplify the presentation, we begin by presenting the method for solving the problem
min f ðxÞ ð19:2aÞ

subject to
ð19:2bÞ
cðxÞ ¼ 0,

x  0: ð19:2cÞ

Later on, we will see the method for solving general nonlinear optimization problems with simple
bounds on variables. As a barrier method, the proposed algorithm computes an approximate solution
of a sequence of barrier subproblems
X
n  
min x2ℝn φμ ðxÞ≜f ðxÞ  μ log xi ð19:3aÞ
i¼1

subject to
ð19:3bÞ
cðxÞ ¼ 0,

for a decreasing sequence of barrier parameters {μj} converging to zero, where xi is the i-th
component of vector x. This problem can be interpreted as applying a homotopy method to the
primal-dual equations
∇f ðxÞ þ ∇cðxÞλ  z ¼ 0, ð19:4aÞ

cðxÞ ¼ 0, ð19:4bÞ

XZe  μe ¼ 0, ð19:4cÞ

where the homotopy parameter μ is driven to zero (Byrd, Liu, & Nocedal, 1997). In (19.4), λ 2 ℝm
and z 2 ℝn are the Lagrange multipliers associated to (19.2b) and (19.2c), respectively. Observe that
the equations (19.4) with μ ¼ 0 and the constraints x, z  0 are exactly the KKT optimality conditions
for (19.2). Clearly, if the constraint qualifications (see Remark 11.2) are satisfied, then these are the
first-order optimality conditions.
19.1 Basic Algorithm IPOPT 663

The strategy for solving (19.2) is to compute an approximate solution to the barrier problem (19.3)
for a fixed value of μ, and then decrease the barrier parameter and continue the solution of the next
barrier problem from the approximate solution of the previous one, etc.
From the primal-dual equations (19.4), we can see that the optimality error for the barrier problem
can be defined as
 
k∇f ðxÞ þ ∇cðxÞλ  zk1 kXZe  μek1
Eμ ðx, λ, zÞ ¼ max , kcðxÞk1 , , ð19:5Þ
sd sc

where the scaling parameters sd, sc  1 are defined below. E0(x, λ, z) is the value of (19.5) for μ ¼ 0.
This measures the optimality error for the original problem (19.2). With this, we can consider that the
algorithm terminates with an approximate solution (x, λ, z) if

E0 ðx , λ , z Þ  εTOL , ð19:6Þ

where εTOL > 0 is a tolerance for stopping the algorithm.

Even if the problem is well scaled, the multipliers λ and z might become very large, for example
when the gradients of the active constraints are nearly linear dependent at the solution of (19.2). In
this case, the algorithm faces major numerical difficulties. In order to adapt the termination criteria to
handle such situations, in (19.5) the scaling factors are introduced as
   
kλk1 þ kzk1 kz k1
sd ¼ max 1, , sc ¼ max 1,
ðm þ nÞs max ns max

In this way, a component of the optimality error is scaled whenever the average value of the
multipliers becomes larger than a fixed number smax  1. (For example smax ¼ 100.)
The algorithm has two types of iterations: major iterations denoted by j, in which the value of the
barrier parameter μj is modified, and minor iterations denoted by k, in which the barrier problem is
solved for a fixed value μj of the barrier parameter. In order to achieve the superlinear convergence of
the algorithm, i.e., to get a local solution of (19.2) satisfying the second-order sufficient optimality
conditions (see Theorem 11.13), Wächter and Biegler (2006) follow the approach proposed by Byrd,
Liu, and Nocedal (1997). Therefore, at the major iteration j it is required that the approximate solution
 
xjþ1 , λjþ1 , zjþ1 of the barrier problem (19.3) for a given value of the barrier parameter μj should
satisfy the tolerance test
 
Eμj xjþ1 , λjþ1 , zjþ1  κ ε μj ð19:7Þ

before the algorithm continues with the solution of the next barrier problem, where κε > 0 is a given
constant. The barrier parameter is updated as
n n oo
ε θ
μjþ1 ¼ max TOL , min κ μ μj , μj μ , ð19:8Þ
10
with the constants κμ 2 (0, 1) and θμ 2 (1, 2). In this way, the barrier parameter is decreased at a
superlinear rate. On the other hand, given the desired tolerance εTOL, the updating given by (19.8)
does not allow μ to become smaller than necessary, thus avoiding numerical difficulties at the end of
the optimization procedure.
664 19 Interior-Point Filter Line-Search

Solving the Barrier Problem

For solving the barrier problem (19.3) in which the barrier parameter is fixed at the value μj, the
algorithm uses the Newton method applied to the primal-dual equations (19.4). As we have already
said, the minor iterations necessary for solving the barrier problem are denoted by k. Therefore, given
 
an iteration (xk, λk, zk) with xk, zk > 0, then the search direction dxk , d λk , d zk is obtained as solution of
the linearization of (19.4) at (xk, λk, zk), namely,
2 32 x 3 2 3
W k J k I dk ∇f ðxk Þ þ J k λk  zk
6 T 76 7 6 7
4 Jk 0 0 54 dλk 5 ¼ 4 c ðx k Þ 5 ð19:9Þ
Z k 0 Xk dkz
Xk Z k e  μj e

In (19.9), Jk ¼ ∇ c(xk) and Wk is the Hessian of the Lagrange function associated to the problem
(19.2)

Lðx, λ, zÞ ¼ f ðxÞ þ cðxÞT λ  z: ð19:10Þ

Instead of solving the nonsymmetric linear system (19.9), the proposed method computes the
solution equivalently by first solving the smaller symmetric linear system
  " #
W k þ Σk J k d xk ∇φμj ðxk Þ þ J k λk
¼ ð19:11Þ
J Tk 0 d λk cðxk Þ

derived from (19.9) by eliminating the last block row, where Σk ¼ X1 z
k Z k . The vector d k is then
computed as

dzk ¼ μj X1
k e  z k  Σk d k
x
ð19:12Þ

As we know, in most line-search methods, we have to ensure that the matrix Wk þ Σk from (19.11)
projected onto the null space of the constraint Jacobian J Tk , must be positive definite. This is
necessary to guarantee certain descent properties of the filter line-search used in the algorithm
described below. Also, if Jk does not have full rank, the iteration matrix in (19.11) is singular, i.e.,
the system (19.11) does not have any solutions. Therefore, as protection, it might be necessary to
modify the matrix of the system (19.11). Wächter and Biegler (2006) suggested the following
modification of the system (19.11):
  x " #
W k þ Σ k þ δw I Jk dk ∇φμj ðxk Þ þ J k λk
¼ , ð19:13Þ
J Tk δc I dλk c ðx k Þ

where δw, δc  0. The choice of these constants is to be discussed.

Having a solution to the systems (19.13) and using (19.12), we must determine the step sizes. Since
x and z are both positive at an optimal solution of the barrier problem (19.3), then this property has to
be maintained for all the iterations. Therefore, in order to achieve this property, we first compute

  
αkmax ¼ max α 2 ð0, 1 : xk þ αdxk  1  τj xk , ð19:14aÞ

  
αzk ¼ max α 2 ð0, 1 : zk þ αdzk  1  τj zk , ð19:14bÞ

where the parameter τj 2 (0, 1) is computed as

19.1 Basic Algorithm IPOPT 665


τj ¼ max τ min , 1  μj , ð19:15Þ

τmin 2 (0, 1) being its minimum value. In interior point methods, τj is known as the parameter which
represents the fraction-to-the-boundary of the feasible region. With this, the next iteration is
computed as

xkþ1 ¼ xk þ αk dxk , ð19:16aÞ

λkþ1 ¼ λk þ αk dλk , ð19:16bÞ

zkþ1 ¼ zk þ αzk dzk , ð19:16cÞ

where αzk is computed as in (19.14b) and, in order to ensure the global convergence, the step size
 
αk 2 0, αkmax for the remaining variables is determined by a backtracking line-search procedure
exploring a decreasing sequence of trial steps: αk,l ¼ 2l αkmax ðwith l ¼ 0, 1, . . .Þ of Armijo type.
Observe the difference between the computation of zkþ1 and the rest of the variables. We can see
that the parameter which represents the fraction-to-the-boundary is used only in computing the
variables xkþ1 and zkþ1.
For achieving the superlinear convergence of the algorithm, Wächter and Biegler (2001) prove that
the primal-dual barrier term Hessian Σk does not deviate too much from the primal Hessian μj X2 k :
i
This requirement is achieved by resetting the components zkþ1 of zkþ1 as
   
κ Σ μj μj
zkþ1 ¼ max min zkþ1 , i
i i
, , i ¼ 1, . . . , n, ð19:17Þ
xkþ1 κΣ xikþ1

where κΣ  1 is a fixed parameter. This strategy guarantees that each component σ ikþ1 of the diagonal
matrix Σkþ1 is in the interval
" #
1 μj μj
σ kþ1 2
i
, κ Σ  2 : ð19:18Þ
κ Σ xi 2 xi
k k

In IPOPT, κΣ ¼ 1010.

Line-Search Filter Method

This method interprets the solving of the barrier problem (19.3) for μj fixed as a biobjective
optimization problem with two goals: minimizing the objective function φμj ðxÞ and minimizing the
constraint violation defined by θ(x) ≜ kc(x)k. As seen in Chap. 18, for solving this biobjective
minimization problem, the emphasis is on minimizing the constraint violation.
In this context, in the line-search with backtracking a trial point xk ðαk,l Þ≜xk þ αk,l d xk is considered
as acceptable if it leads to sufficient progress toward either goal compared to the current iterate, i.e., if
θðxk ðαk,l ÞÞ  ð1  γ θ Þθðxk Þ, ð19:19aÞ

or φμj ðxk ðαk,l ÞÞ  φμj ðxk Þ  γ φ θðxk Þ, ð19:19bÞ

holds for the fixed constants γ θ, γ φ 2 (0, 1).

666 19 Interior-Point Filter Line-Search

However, as we can see, each of the above criteria is complicated enough and therefore is replaced
by requiring sufficient progress in the barrier objective function whenever for the current iterate, we
have θ(xk)  θmin for some constant θmin 2 (0, 1] and the following switching condition
h isφ
∇φμj ðxk ÞT dxk < 0, and αk,l ∇φμj ðxk ÞT dxk > δ½θðxk Þsθ , ð19:20Þ

with the constants δ > 0, sθ > 1, sφ  1, is satisfied. Observe that if θ(xk)  θmin and (19.20) is true for
the current stepsize αk, l, then the trial point has to satisfy the Armijo condition

φμj ðxk ðαk,l ÞÞ  φμj ðxk Þ þ ηφ αk,l ∇φμj ðxk ÞT dxk ð19:21Þ

instead of the condition (19.19) for the trial point to be acceptable. In (19.21), ηφ 2 (0, 1/2) is a
constant.
The algorithm maintains a filter, i.e., a set Fk ⊆ {(θ, φ) 2 ℝ2 : θ  0} computed for each iteration
k. The filter Fk contains the combinations of the constraint violation values θ and the objective
function values φ which are prohibited for a successful trial point at iteration k, i.e. during the line-
 
search, a trial point xk(αk, l) is rejected if θðxk ðαk,l ÞÞ, φμj ðxk ðαk,l ÞÞ 2 Fk : We say that the trial point
is not acceptable to the current filter. At the beginning of the optimization process, the filter is
initialized as


F0 ¼ ðθ, φÞ 2 ℝ2 : θ  θ max , ð19:22Þ

for some θmax, so that the algorithm will never allow trial points that have a constraint violation larger
than θmax to be accepted. Later, after every iteration, the filter is augmented using the update formula
n o
Fkþ1 ¼ Fk [ ðθ, φÞ 2 ℝ2 : θ  ð1  γ θ Þθðxk Þ and φ  φμj ðxk Þ  γ φ θðxk Þ , ð19:23Þ

where the accepted trial step size does not satisfy the switching condition (19.20) or in which the
Armijo condition (19.21) does not hold. This ensures that the iterates cannot return to the neighbor-
hood of xk. On the other hand, if both (19.20) and (19.21) hold for the accepted stepsize, the filter
remains unchanged. This procedure ensures that the algorithm cannot cycle indefinitely, for example
between two points that alternately decrease the constraint violation and the barrier objective
function.
Finally, there are cases in which it is not possible to find a trial step size αk, l that satisfies the above
criteria. In these cases, a minimum desired stepsize is approximated by using linear models of the
involved functions. For this, Wächter and Biegler (2006) define the function
8 8 9
>
> >
< >
=
>
> γ φ θ ðx k Þ s
δ½θðxk Þ θ
>
> min γ , , h isφ ,
>
> > θ
>
> : ∇φμj ðxk ÞT dxk ∇φμ ðxk ÞT dx > ;
>
> k
>
j
>
>
>
> if∇φμj ðxk ÞT d xk < 0 and θðxk Þ  θ min ,
>
< ( )
αkmin ≜γ α γ φ θðxk Þ ð19:24Þ
>
> min γ θ , ,
>
> ∇φμj ðxk Þ d kT x
>
>
>
>
>
> T
>
> if∇φμj ðxk Þ d k < 0 and θðxk Þ > θ min ,
x
>
>
>
> γθ ,
>
>
:
otherwise,
19.1 Basic Algorithm IPOPT 667

where γ α 2 (0, 1] is a “safety factor.” If the backtracking line-search encounters a trial step size with
αk,l  αkmin , then the algorithm reverts to a feasibility restoration phase. That is, the algorithm tries to
find a new iterate xkþ1 > 0 which is acceptable to the current filter and for which the condition (19.19)
holds by reducing the constraint violation using some kind of iterative method. Observe that the
restoration phase might not be able to produce a new iterate for the filter line-search method, for
example when the problem is infeasible.

Second-Order Corrections

Many methods for nonlinear optimization use second-order corrections to improve the proposed step
in case a trial point was rejected. A second-order correction (SOC) for some step dxk aims at reducing
the infeasibility by applying an additional Newton step for the constraints at the point xk þ d xk using
the Jacobian Jk evaluated at xk. The details are given in (Wächter & Biegler, 2006) and are as follows.
If the first trial, the stepsize αk,0 was rejected and if θ(xk(αk, 0))  θ(xk), then a second-order
correction d x,soc
k is computed, such that
 
J Tk dx,soc
k þ c xk þ αk,0 dxk ¼ 0: ð19:25Þ

The new corrected search direction is obtained as

d x,cor
k ¼ dx,soc
k þ αk,0 d xk : ð19:26Þ

The condition (19.25) does not uniquely define the second-order correction, and different choices
would be possible. In order to avoid additional matrix factorization, the proposed method uses the
same matrix as in (19.13) to compute the overall corrected step (19.26) as solution of the system
  x,cor 
W k þ Σ k þ δw I Jk dk ∇φμj ðxk Þ þ J k λk
¼ , ð19:27Þ
J Tk δc I dλk csoc
k

where
 
k ¼ αk,0 cðxk Þ þ c xk þ αk,0 d k
csoc ð19:28Þ
x

is obtained from (19.13), (19.25), and (19.26).

As soon as the corrected search direction dx,cor
k has been obtained, the fraction-to-the-boundary
rule is applied to get

  
αsoc
k ≜ max α 2 ð0, 1 : xk þ αd k
x,cor
 1  τ j xk ð19:29Þ

k ≜xk þ αk d k
soc x,cor
and we test if the resulting trial point xsoc is accepted in the filter. Note that the
x
original search direction dk is still used in (19.20) and on the right-hand side of (19.21). Also, xsock
replaces x(αk) in (19.21). If this trial point passes the tests, then it is accepted as the new iterate.
Otherwise, additional second-order corrections are applied until the correction step has not decreased
the constraint violation by a fraction κsoc 2 (0, 1) or until a maximum number pmax of second-order
corrections has been performed. In this case, the original search direction dxk is restored and the
regular backtracking line-search is restarted with a shorter step size αk,1 ¼ αk, 0/2.
668 19 Interior-Point Filter Line-Search

The Algorithm

The following filter line-search algorithm for solving the barrier subproblem (19.3) can be presented.

Algorithm 19.1 Line-search filter barrier algorithm—Wächter and Biegler

1. Choose an initial point (x0,λ0, z0) with x0, z0 > 0, an initial value for the barrier parameter μ0 > 0, and the
constants εTOL, smax  1, κ ε > 0, κ μ 2 (0, 1), θμ 2 (1, 2), τmin 2 (0, 1), κ Σ > 1, θmax 2 (θ(x0), 1], θmin > 0,
γ θ, γ φ 2 (0, 1), δ > 0, γ α 2 (0, 1], sθ > 1, sφ  1, ηφ 2 (0, 1/2), κ soc 2 (0, 1) and pmax 2 {0, 1, 2, . . .}
2. Initialization. Initialize j ¼ 0, k ¼ 0, as well as the filter F0 as in (19.22). Compute τ0 as in (19.15)
3. Test the convergence for the overall problem. If E0(xk, λk, zk)  εTOL, then stop
4. Test the convergence for the barrier problem. If Eμj ðxk , λk , zk Þ  κε μj , then go to step 5
5. Compute μjþ1 and τjþ1 as in (19.8) and (19.15), respectively
6. Re-initialize the filter Fk ¼ {(θ, φ) 2 ℝ2 : θ  θmax}
7. If k ¼ 0, then continue with step 4; otherwise, go to step 8
 
8. Compute the search direction. Compute d xk , dλk , d zk as solution of the system (19.13), where the
parameters δw and δc are computed from Algorithm IC for correction of inertia
9. Backtracking. Initialization of linear search. Set αk,0 ¼ αkmax , where αkmax is computed as in (19.14a)
and set l ¼ 0
10. Backtracking. Compute the new trial point. Set xk ðαk,l Þ ¼ xk þ αk,l dxk
 
11. Backtracking. Test on acceptability to the filter. If θðxk ðαk,l ÞÞ, φμj ðxk ðαk,l ÞÞ 2 Fk , then reject the
trial step and go to step 13
12. Backtracking. Test on sufficient decrease with respect to the current iterate
Case I: θ(xk)  θmin and (19.20) holds. If (19.21) holds, then accept the trial step xkþ1 ¼ xk(αk, l) and
continue with step 19; otherwise, go to step 13
Case II: θ(xk) > θmin or (19.20) is not satisfied. If (19.19) holds, then accept the trial step xkþ1 ¼ xk(αk, l)
and go to step 19; otherwise, continue with step 13
13. Backtracking. Initialize the second-order correction. If l > 0 or θ(xk, 0) < θ(xk), then skip the second-
order correction and go to step 18; otherwise, initialize the second-order correction by setting p ¼ 1 and
k as in (19.28). Initialize θold ¼ θðxk Þ
soc
csoc
14. Backtracking. Compute the second-order correction. Compute d x,cor k and dλk as solutions of the system
(19.27). Compute αsoc from (19.29) and then set x soc
¼ x k þ αsoc x,cor
d
k k
 k k   
15. Backtracking. Test on acceptability to the filter (in SOC). If θ xsoc , φμj xsoc 2 Fk , then reject the
k k
trial step size and go to step 18
16. Backtracking. Test on sufficient reduction with respect to the current iterate in SOC.
Case I: θ(xk)  θmin and (19.20) holds for αk, 0. If (19.21) holds with xk(αk, l) replaced by xsoc k , then the
trial step xkþ1 ¼ xsoc
k is accepted and go to step 19; otherwise, continue with step 17
Case II: θ(xk) > θmin or (19.20) is not satisfied for αk, 0. If (19.19) holds for xk(αk, l) replaced by xsoc
k , then
the trial step xkþ1 ¼ xsoc
k is accepted and go to step 19; otherwise, continue with step 17
 
17. Backtracking. The next second-order correction. If p ¼ pmax or θ xsoc > κ soc θsoc , then abort second-
k  old   soc 
order correction and go to step 18; otherwise, set p ¼ p þ 1, ck ¼ αk ck þ c xsoc
soc soc soc
k old ¼ θ xk
and θsoc :
Go back to step 14
18. Backtracking. Choose a new trial step size. Set αk,lþ1 ¼ αk, l/2 and l ¼ l þ 1. If the trial step size becomes
too small, i.e., αk,l < αkmin , where αkmin is defined as in (19.24), then continue with the feasibility
restoration phase, i.e., step 22; otherwise, go back to step 10
19. Accept the trial point. Set αk ¼ αk, l (or αk ¼ αsoc
k if the SOC point was accepted in step 16) and update
the Lagrange multiplier estimates λkþ1 and zkþ1 using (19.16b) and (19.16c) respectively, with αzk
computed as in (19.14b). If necessary, apply (19.17) to correct zkþ1
20. Augment the filter. If (19.20) or (19.21) do not hold for αk, then augment the filter as in (19.23);
otherwise, the filter is unchanged, i.e., Fkþ1 ¼ Fk
21. Continue with the next iteration. Set k ¼ k þ 1 and go back to step 3.
19.2 Implementation Details 669

22. Feasibility restoration phase. Augment the filter as in (19.23) and compute a new iterate xkþ1 > 0 by
decreasing the infeasibility
  measure θ(x), so that xkþ1 is acceptable to the augmented filter, i.e.,
θðxkþ1 Þ, φμj ðxkþ1 Þ 2
= Fkþ1 : Then continue with the regular iteration in step 21 ♦

At every iteration, at least one trial point will be tested before the algorithm may switch to the
feasibility restoration phase in step 22. Also, the condition in step 7 ensures that eventually, at least
one step is taken for each decreased value of the barrier parameter.
The algorithm is complex, with a multitude of parameters which specify the optimization
conditions. Wächter and Biegler (2006) give the values of the constants used in their implementation
as κ e ¼ 10, κμ ¼ 0.2, θμ ¼ 1.5, τmin ¼ 0.99, γ θ ¼ 105, γ φ ¼ 105, δ ¼ 1, γ α ¼ 0.05, sθ ¼ 1.1, sφ ¼ 2.3,
ηφ ¼ 104, κsoc ¼ 0.99, pmax ¼ 4, as well as μ0 ¼ 0.1, θmax ¼ 104 max {1, θ(x0)} and
θmin ¼ 104 max {1, θ(x0)}, where x0 is the starting point. At the same time, they present some
implementation details which lead to the IPOPT algorithm. In the following, we will only present
some of the most interesting details in order to complete Algorithm 19.1 for it to be as close as
possible to IPOPT described in (Wächter, 2002) or (Wächter & Biegler, 2006).

19.2 Implementation Details

General Lower and Upper Bounds

For simplicity, Algorithm 19.1 is dedicated to solving nonlinear optimization problems (19.2).
However, this algorithm can be immediately generalized for solving general problems (19.1). In
particular, for the problem (19.1) with simple bounds on variables xL  x  xU, the barrier problem
(19.3) becomes
X   X  
min x2ℝn φμj ðxÞ ¼ f ðxÞ  μj log xi  xiL  μj log xiU  xi ð19:30aÞ
i2I L i2IU
subject to
ð19:30bÞ
cðxÞ ¼ 0,



where I L ¼ i : xiL 6¼ 1 and I U ¼ i : xiU 6¼ þ1 At the same time, for the multipliers z the
simple bounds ziL and ziU are introduced for all the finite lower and upper bounds, and the primal-dual
 
Hessian of the barrier terms Σk is defined as the sum of the matrices ΣLk ¼ diag σ Lk,1 , . . . , σ Lk,n and
 U 
ΣUk ¼ diag σ k,1 , . . . , σ k,n , where
U

 i  i 
zL,k = xk  xiL , if i 2 I L ,
σ k,i ¼
L

 i  0,i  otherwise,
z = x  x i
k , if i 2 I U ,
σUk,i ¼
U,k U
0, otherwise:

Moreover, for i 2
= IL, define ziL,k ¼ 0 and for i 2
= IU, define ziU,k ¼ 0.
670 19 Interior-Point Filter Line-Search

Initialization

Since Algorithm 19.1 requires that the iterates should strictly satisfy the bounds constraints (19.1c), it
follows that it is often necessary to modify the user-provided initial point so that it is sufficiently away
from the boundary. For this purpose, each component i of the initial point which has only one (let us
say, a lower) bound is modified by



xi0 ¼ max xi0 , xiL þ κ 1 max 1, xiL ,

where κ 1 > 0 is a constant. Similar modifications can be made for the only upper-bounded variables.
The initial value of a variable xi bounded on two sides is projected into the interval
 i 
xL þ piL , xiU  piU , where the perturbations piL and piU are computed as


  
piL ≜ min
κ1 max
1, xiL , κ2 xiU  xiL ,
piU ≜ min κ1 max 1, xiU , κ2 xiU  xiL ,

where κ 2 2 (0, 1/2) (For example: κ1 ¼ κ2 ¼ 102.)

The dual variables corresponding to the bound constraints are initialized to one, component-wise.
Finally, using the possibly modified initial point x0 and the initial bound multipliers, the multipliers
λ0 for the equality constraints are obtained as least-square solutions for the dual infeasibility (19.4a),
i.e., by solving the linear system
  
I J ðx 0 Þ w ∇f ðx0 Þ  zL,0 þ zU,0
¼  , ð19:31Þ
J ðx0 ÞT 0 λ0 0

where w are temporary variables discarded after this computation. However, if λ0 obtained in this way
is too large, i.e., kλ0k1 > λmax, (with λmax ¼ 103 for example), then the least-square estimate λ0 is
discarded and set λ0 ¼ 0. (J(x0) ¼ ∇ c(x0)).

Handling Unbounded Solution Sets

In some cases, the optimal point set for (19.1) does not consist of isolated points, but contains an
unbounded connected component. Then the objective function of the corresponding barrier problem
(19.30) for a fixed value of the barrier parameter μj is unbounded below over the feasible set since a
log-barrier term converges to 1 as its argument goes to infinity. Therefore, the method for solving
the barrier problem might fail to converge, even though the original problem is well-posed. To
prevent this behavior, linear damping terms for all the variables with exactly one finite bound are
added to the barrier objective function (19.30a), which then becomes
P   P  
φμj ðxÞ ¼ f ðxÞ  μj log xi  xiL  μj log xiU  xi
i2I L  i2IU  
P P
þ κ d μj xi  xiL þ κ d μj xiU  xi ,
i2I L ∖IU i2IU ∖I L

where κd > 0 is a constant independent of μj (κd ¼ 104 for example). In this way, the divergence of
the variables that have only one bound is penalized. On the other hand, the effect of the damping term
is reduced when μj decreases.
19.2 Implementation Details 671

Inertia Correction

In order to be able to compute the search direction, we need to ensure that the matrix of the system
(19.11) is nonsingular. In addition, the filter line-search method requires the matrix Wk þ Σk projected
onto the null space of the constraint Jacobian J Tk being positive definite. But these conditions are
satisfied if the matrix of the system (19.11) has the inertia equal to (n, m, 0), i.e., if this matrix has
exactly n positive, m negative, and no zero eigenvalues. Therefore, if the inertia of this matrix is not
(n, m, 0), then the linear system (19.13) is re-solved with different trial values for the scalars
δw, δc  0 until the inertia is as desired. Observe that the desired inertia is obtained if δw is sufficiently
large and the constraint Jacobian J Tk has full rank. If J Tk is rank-deficient, the matrix of the system
(19.13) is singular as long as δc is zero. Therefore, a positive value for δc and a sufficiently large value
of δw ensures the correct eigenvalue signatures for the matrix from (19.13). However, in practice, the
matrix of the system (19.13) can become so ill-conditioned that the factorization cannot be success-
fully performed, even with very large values of δw and some positive values for δc In this case,
Algorithm 19.1 switches directly to the feasibility restoration phase in step 22. These observations
motivate the following heuristics for choosing the parameters δc and δw (Wächter & Biegler, 2005a).

Algorithm 19.2 Inertia correction algorithm

min 0 max þ
1. Choose the constants 0 < δw < δw < δw , δc > 0, κ c  0, 0 < κ  þ
w < 1 < κ w < κ w : Initialize
δlast
w ¼ 0
2. Attempt to factorize the unmodified matrix of the system (19.13) with δw ¼ δc ¼ 0. If the matrix is
nonsingular and its inertia is (n, m, 0), then use the resulting search direction in the line-search obtained
by solving the system (19.13); otherwise, continue with step 3
3. If the matrix from (19.13) has zero eigenvalues, then set δc ¼ δc μκc ; otherwise, set δc ¼ 0
n o
4. 0 min  last
If δlast
w ¼ 0, then set δw ¼ δw ; otherwise, set δw ¼ max δw , κ w δw

5. Attempt to factorize the matrix from (19.13) modified with these values of δc and δw. If inertia is equal to
w ¼ δw and use the resulting search direction in the line-search, otherwise, continue
(n, m, 0), then set δlast
with step 6
þ þ
6. If δlast
w ¼ 0, then set δw ¼ κ w δw ; otherwise, set δw ¼ κ w δw
max
7. If δw > δw , then abort the search direction computation, skip the backtracking line-search and switch
directly to the restoration phase in step 22 of Algorithm 19.1. Otherwise, continue with step 5 ♦

The above procedure was established by Wächter and Biegler (2006). In step 2 it first checks if the
unmodified matrix has the desired inertia so that the pure Newton search direction is used whenever
possible. If this is the case, then apply the Newton method for the search direction computation;
otherwise, increase δw. The first trial value is based on δlast
w , which stores the perturbation value from
the last modification of the matrix from (19.13). In this way, the smallest perturbation is found
necessary to avoid the factorization in step 5 of the algorithm for too small values of δw. The reason
for using a much larger factor κþ w in step 6 of the algorithm for the first necessary correction rather
than for the correction in later iterations is that we want to avoid a high number of trial factorizations
when the scale of the problem and the order of the magnitude for a successful correction is not known
yet. The selection of a nonzero value for δc in step 3 is based on the assumption that the singularity is
caused by the rank-deficient constraint Jacobian. Note that the nonzero value for δc in step 3 converges
to zero as μ ! 0 if κ c > 0, so that the perturbation δc is smaller when a solution of the problem is
approached.
672 19 Interior-Point Filter Line-Search

Automatic Scaling of the Problem

The Newton steps for the primal-dual system (19.11) for solving the nonlinear system (19.4) are
invariant to scaling the variables, the objective, and the constraints functions, i.e., to replacing x, f and
c by xe ¼ Dx x, feðxÞ ¼ df f ðxÞ and ceðxÞ ¼ Dc cðxÞ for some df > 0 and for the positive definite diagonal
   
matrices Dx ¼ diag d1x , . . . , d nx and Dc ¼ diag d1c , . . . , dm c . However, the optimization algorithm
with its initialization procedures, globalization strategy, and stopping criteria usually behaves very
differently for different scaling factors, particularly if the scaling factors are very large or very small.
The idea of scaling in IPOPT is that changing a variable by a given amount has a comparable effect
on any function which depends on this variable, or in other words, the nonzero elements of the
function gradients are of the same order of magnitude.
Another scaling procedure available in IPOPT as an option consists in applying an equilibration
algorithm to the first derivative matrix
" #
∇x c ð x 0 Þ T
M0 ¼
∇x f ð x 0 Þ T

to obtain the scaling matrices Dx and Dcf ¼ diag (Dc, df) so that the nonzero elements in Dcf M0 D1
x
are of order one. (The algorithm for the equilibration of the matrices is the one implemented in the
subroutines MC19 and MC29 from the Harwell library (AEA Technology, 2002)). Similarly, another
procedure for scaling computes the scaling factors so that the matrix
" #" #" #
D1
x 0 ∇2xx f ðx0 Þ ∇x cðx0 Þ D1 x 0
0 Dc ∇x c ð x 0 Þ T 0 0 Dc

has nonzero entries close to one. These procedures are available to users as options in IPOPT. Even if
these strategies seem to work well in certain instances, the performances of the algorithm are not
conclusive with scaling.
In IPOPT, Wächter and Biegler (2006) apply an automatic scaling procedure. Given a threshold
value gmax, for example gmax ¼ 100, then the scaling factors are chosen as


df ¼ min
1, g max =k∇x f ðx0 Þk1 ,
d jc ¼ min 1, g max =k∇x cj ðx0 Þk1 , j ¼ 1, . . . , m,

and Dx ¼ I. Observe that this procedure will never multiply a function by a number larger than one
and that all the gradient components in the scaled problem are at most of size gmax at the starting
point.
The scaling factors are computed only at the beginning of the optimization at x0.

Feasibility Restoration Phase

A very important component of the filter line-search algorithm is the feasibility restoration phase (see
step 22 of Algorithm 19.1). The task of the restoration phase is to compute a new iteration acceptable
to the augmented filter Fkþ1 by decreasing the infeasibility whenever the regular backtracking line-
search procedure cannot make sufficient progress and when the step size becomes too small (see step
18 of Algorithm 19.1). Besides, when the inertia is corrected as in Algorithm 19.2, the method
switches to the restoration phase whenever the linear system (19.13) is very ill-conditioned and
19.2 Implementation Details 673

cannot be successfully factorized, despite the modifications of the matrix from (19.13). The feasibility
restoration phase has another significant purpose, namely, to detect the local infeasibility. If the
problem is infeasible, then the algorithm is not able to generate sufficient progress in the regular
backtracking line-search procedure and reverts to the restoration phase. In Wächter and Biegler
(2001), the global convergence of the filter line-search method is proved under the hypothesis that in
the neighborhood of feasible points the gradients of the active constraints are linearly independent.
Consequently, the algorithm does not switch to the feasibility restoration phase at feasible points.
However, this assumption might be violated in practice and the restoration phase might be called at a
point with a very small value of θ. In IPOPT, two procedures for feasibility restorations are used.
(In order to avoid confusion, the overbars are used to denote quantities referring to the restoration
phase, and the subscript t is used for the restoration phase iteration counter).

Minimization of the constraint violation This is the first algorithm of the feasibility restoration
 
phase. The purpose of this method is to return a new iterate xkþ1 > 0 with θðxkþ1 Þ, φμj ðxkþ1 Þ 2= Fkþ1
for step 22 of Algorithm 19.1 or to converge to a nonzero minimizer (or at least to a stationary point)
of some norm of the constraint violation. The restoration phase algorithm applies the primal-dual
interior-point filter line-search algorithm to a smooth reformulation of the optimization problem
ξ
min x2ℝn kcðxÞk1 þ kDR ðx  xR Þk22 ð19:32aÞ
2
subject to
x  0: ð19:32bÞ

In (19.32a), a term is included in the objective function that penalizes the deviation from a
reference point xR , where ξ > 0 is a weighting parameter and the scaling matrix DR is defined by



DR ¼ diag min 1, 1= x1R , . . . , min 1, 1= xnR

The reference point xR is chosen to be the iterate xk at which the restoration phase is called in
step 22 of Algorithm 19.1. In this way, we seek to decrease the constraint violation and try to avoid a
large deviation from xR , which determines undesired significant increase in the barrier objective
function φμj :

A smooth reformulation of the problem (19.32) is obtained by introducing the nonnegative

variables p, n 2 ℝm that capture the positive and negative parts of the constraints
X
m   ξ
min x2ℝn ,p,n2ℝm pi þ ni þ kDR ðx  xR k22 ð19:33aÞ
i¼1
2
subject to
cðxÞ  p þ n ¼ 0, ð19:33bÞ

x, p, n  0: ð19:33cÞ

We can see that this optimization problem is of the form given in (19.2). Therefore, the filter line-
search interior-point algorithm can be applied to solve a sequence of barrier problems of the
following form:
674 19 Interior-Point Filter Line-Search

X
m   ξ
min x2ℝn ,p,n2ℝm ρ pi þ ni þ kDR ðx  xR k22
i¼1
2
X
n   X
m   X
m  
μ log xi  μ log pi  μ log ni ð19:34aÞ
i¼1 i¼1 i¼1

subject to
cðxÞ  p þ n ¼ 0: ð19:34bÞ

Observe that an additional scaling parameter ρ > 0 has been introduced. This is to allow a relative
scaling of the overall objective function (19.34a) with respect to the constraints (19.34b). Moreover,
if the regularization parameter ξ > 0 is chosen sufficiently small, the optimization problem (19.33) is
the exact penalty formulation of the problem: “find the feasible point that is closest to the reference
point xR in a weighted norm”:

min x2ℝn kDR ðx  xR Þk22

subject to
cðxÞ ¼ 0, x  0:

In addition to the original variables x, the barrier problem (19.34) contains the variables p and n:
The corresponding primal-dual equations (similar to (19.4)) include their dual variables, say zp and zn :
The search directions for the line-search are obtained by linearization of these equations and can be
written as
2 3" #
W þ ξD2R þ Σ ∇cðxÞ d
x
4 5 ¼
1 1 λ
∇cðxÞT Σp  Σn d
2 1
3 ð19:35Þ
ξD2R ðx  xR Þ þ ∇cðxÞλ  μX e
4 5,
1 1
cðxÞ  p þ n þ ρZ p ðμe  pÞ þ ρZ n ðμe  nÞ
P i 2 1 1 1 p n zp
where W ¼ m i¼1 λ ∇xx cðxÞ, Σ ¼ X Z, Σp ¼ P Z p and Σn ¼ N Z n : Subsequently, d , d , d
zn
and d are obtained from
   
p 1 λ zp 1 p
d ¼ Z p μe þ P λ þ d  ρp , d ¼ μP e  zp  Σp d ,
   
n 1 λ zn 1 n
d ¼ Z n μe þ N λ þ d  ρn , d ¼ μN e  zn  Σn d ,
z 1 x
d ¼ μX e  z  Σd :

Observe that the structure of the nonzero elements of the linear system (19.35) is identical to the
one in (19.13). Therefore, for solving (19.35) the same symbolic factorization may be used.
The filter line-search method applied to the problem (19.34) might itself revert to a restoration
phase. If this occurs, then the optimal solution of (19.34) is computed for a fixed value of x, namely,
the current iterate xt , and this solution is used as the “result” of the restoration phase. In this
case, since (19.34) becomes separable, this can be done by solving a quadratic equation for each
 i i
p , n , i.e.,
19.2 Implementation Details 675

s
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2 ffi
μ  ρci ðxÞ μ  ρci ðxÞ μci ðxÞ
ni ¼ þ þ :
2ρ 2ρ 2ρ
pi ¼ c i ð x Þ þ ni :

Reduction of the KKT error This method tries to achieve reduction in the norm of the primal-dual
equations by using the regular iteration steps. If in step 22 of Algorithm 19.1 the feasibility restoration
phase is invoked, then the algorithm does not immediately revert to the feasibility restoration phase.
Instead, the reduction in the norm of the primal-dual equations is tried. To describe this procedure, let
Fμ(x, λ, z) denote the left-hand side of the nonlinear system (19.4). Then, given a constant κ F 2 (0, 1),
for example κF ¼ 0.999, the following algorithm reduces the error in the KKT system.

Algorithm 19.3 KKT error reduction algorithm

1. Initialize
  feasibility restoration phase with the parameter t ¼ 0 and choose the initial point
the
x0 , λ0 , z0 ¼ ðxk , λk , zk Þ
 
2. x λ z
Compute a search direction dt , dt , dt using the system (19.11) and (19.12)
3. Apply the fraction-to-the-boundary rule

x   z   
βt ≜ max β 2 ð0, 1Þ : xt þ βdt  1  τj xt , zt þ βd t  1  τj zt
4. Test whether      
Fμ xtþ1 , λtþ1 , ztþ1   κ F Fμ xt , λt , zt  ,
     1 1
x λ z
where xtþ1 , λtþ1 , ztþ1 ¼ xt , λt , zt þ βt dt , dt , d t :
If this condition of reduction is not satisfied, then discard the trial point and switch to the procedure for
minimizing the constraint violation (described above) by using xt as a reference point. Otherwise,
continue with step 5
 
5. If θðxtþ1 Þ, φμj ðxtþ1 Þ 2= Fkþ1 , then continue the regular interior-point Algorithm 19.1 from the point
   
xtþ1 , fλgtþ1 , ztþ1 ¼ xtþ1 , λtþ1 , ztþ1 : Otherwise, set t ¼ t þ 1 and go to step 2 ♦

In a neighborhood of a strict local solution satisfying the second-order sufficient optimality

conditions (see Theorem 11.6) for the barrier problem, the projection of the Hessian Wt þ Σt onto
the null space of the constraint Jacobian ∇cðxt ÞT is positive definite. Therefore, no modification of the
matrix of the system (19.13) as described in the inertia correction procedure is applied. As a
consequence, the search directions computed by solving the system (19.11) and the relations
(19.12) are the Newton steps applied to the nonlinear system (19.4). Hence, the above algorithm
will accept those steps and will quickly converge toward this solution, provided it is started
sufficiently close. The performance and robustness of IPOPT on larger models heavily rely on the
solver for sparse symmetric indefinite linear systems.

Numerical Study—IPOPT: Solving Applications from the LACOP Collection

The performances of IPOPT for solving the applications from the LACOP collection described in
Appendix C are illustrated in Tables 19.1 and 19.2.
Appendix C includes the application L16 – Fed-batch fermenter for penicillin production.
(PENICI). Table 19.3 shows the performance of IPOPT for solving this application.
676 19 Interior-Point Filter Line-Search

Table 19.1 Performances of IPOPT for solving 12 applications from the LACOP collection. Small-scale nonlinear
optimization applications

n me mc #it #nf #ng #nh cpu vfo

ELCH 10 3 0 10 11 11 10 0.143 –47.761090
ALKI 10 3 8 13 14 14 13 0.011 –1768.8069
PREC 8 0 6 16 17 17 16 0.081 3.95116343
PPSE 9 6 0 11 12 12 11 0.009 5055.01180
MSP3 13 0 15 19 20 20 19 0.044 97.5875091
MSP5 16 0 21 28 29 29 28 0.045 174.786994
POOL 34 20 0 13 14 14 13 0.026 2569.7999
TRAFO 6 0 2 9 10 10 9 0.009 135.07596
LATHE 10 1 14 27 28 28 27 0.020 –4430.0875
DES 150 50 0 88 137 89 88 0.479 1055.18231
CSTC 303 200 0 5 6 6 5 0.035 3.4800741
DIFF 396 324 0 1 2 2 1 0.016 0

Table 19.2 Performances of IPOPT for solving 6 applications from the LACOP collection. Large-scale nonlinear
optimization applications

n me mc #it #nf #ng #nh cpu vfo

HANG 2002 1001 0 6 7 7 6 0.180 5.0685100
4002 2001 0 7 8 8 7 0.310 5.0684889
FLOW 1182 754 0 9 10 10 9 0.272 0.226e–10
FLOWO 1556 1005 0 19 20 20 19 0.597 0.689e–7
POL 4004 3000 0 66 67 67 66 3.845 14.216290
6004 4500 0 54 55 55 54 3.741 14.198908
8004 6000 0 67 68 68 67 6.337 14.190847
10004 7500 0 179 180 180 179 21.750 13.981795
CAT 3003 2000 0 15 16 16 15 0.252 –0.048055
6003 4000 0 15 16 16 15 0.639 –0.048055
9003 6000 0 15 16 16 15 0.909 –0.048055
CONT 2505 2000 0 14 15 15 14 0.412 1.0132389
5005 4000 0 13 14 14 13 0.706 1.0059224
7505 6000 0 15 16 16 15 0.962 1.0045614
10005 8000 0 15 16 16 15 1.264 1.0040718

Table 19.3 Performances of IPOPT for solving the PENICI application

n me mc #it #nf #ng #nh cpu vfo

PENICI 707 602 0 519 765 512 519 6.81 113.98986
1407 1202 0 423 623 424 423 9.93 113.98942

In the tables above, we have n ¼ the number of variables, me ¼ the number of equality constraints, mc ¼ the number of
inequality constraints, #it ¼ the number of iterations to obtain a solution, #nf ¼ the number of evaluations of the
functions defining the problem, #ng ¼ the number of evaluations of the gradients of the functions defining the problem,
#nh ¼ the number of evaluations of the Hessian, cpu ¼ the CPU computing time for obtaining a solution of the problem
(seconds), vfo ¼ the value of the objective function at the optimal point
19.2 Implementation Details 677

Table 19.4 Performances of the IPOPT algorithm. Small-scale nonlinear optimization applications

#itt #nft #ngt #nht cput

IPOPT 240 300 252 240 0.918

Table 19.5 Performances of the IPOPT algorithm. Large-scale nonlinear optimization applications

#itt #nft #ngt #nht cput

IPOPT 509 524 524 509 42.176

Table 19.4 presents the total number of iterations (#itt), the total number of evaluations of the
function defining the problem (#nft), the total number of evaluations of the gradients of the functions
defining the problem (#ngt), the total number of evaluations of the Hessian (#nht), and the total CPU
computing time to obtain a solution (cput) for solving 12 small-scale nonlinear optimization
applications (see Table 19.1) considered in this numerical study.
Table 19.5 shows the performances of IPOPT for solving 15 large-scale nonlinear optimization
applications (see Table 19.2) considered in this numerical study.
From Table 17.13, we can see that for solving 15 large-scale applications from the LACOP
collection subject to the CPU computing time, KNITRO with option 0 needs 17.81 seconds, being
top performer versus IPOPT which needs 42.176 seconds.
There is a great difference between KNITRO and IPOPT. In both variants KINTO/ACTIVE and
KNITRO/INTERIOR, the sequential linear or sequential quadratic programming in different compu-
tational structures are used. The KNITRO/ACTIVE uses a new active-set method based on the
sequential linear-quadratic programming (SLQP) and the projected conjugate gradient iteration. On
the other hand, in KNITRO/INTERIOR two procedures for computing the steps are used. In the
version INTERIOR-DIRECT, the algorithm attempts to compute a new iterate by solving the primal-
dual KKT system by using the direct linear algebra. In the version INTERIOR-CG, each step is
computed by using a projected conjugate gradient iteration. It factors a projection matrix and uses the
conjugate gradient method to approximately minimize a quadratic model of the barrier problem. The
stepsizes are determined by minimizing merit functions. All the algorithms included in KNITRO
have a very strong theoretical justification.
IPOPT, on the other hand, combines the primal-dual interior-point algorithms with the filter line-
search, where the search direction is computed as solution of the first-order optimality conditions, and
the stepsize is computed by the filter technique. The algorithm depends by a multitude of parameters
which have a great impact on its performance. Although both the merit functions and filters are
important globalization techniques of the interior-point methods, currently it is unclear whether filter
techniques are preferable to the merit functions.

Notes and References

This Chapter has described the interior-point method with filter line-search. The representative for
this method is IPOPT. The content of this chapter is based on the papers of Wächter and Biegler
(2005a, b, 2006). This is a line-search filter interior-point algorithm with two loops. The outer loop
approximately minimizes a sequence of nonlinearly equality constrained barrier subproblems for a
decreasing sequence of barrier parameters. The inner loop uses a line-search filter sequential
quadratic programming problem to approximately solve each barrier subproblem. The global
678 19 Interior-Point Filter Line-Search

convergence of each barrier subproblem is enforced through a line-search filter method (Fletcher &
Leyffer, 1999, 2002). The filter is reset after each barrier parameter update. The steps are computed
by solving a primal-dual system corresponding to the KKT conditions of the barrier problem, using
the subroutines from the HSL mathematical software library collection of Fortran codes for large
scientific computation (MA77, MA86, MA97). The algorithm controls the inertia of the primal-dual
system by adding δI to the Hessian of the Lagrangian, where δ is a positive parameter, thus ensuring
the descent properties. The inner iterations include the second-order correction steps and some
mechanisms for switching to a feasibility restoration if the stepsize becomes too small. To approxi-
mate the Hessian of the Lagrangian, IPOPT has an option for using the limited memory BFGS
updates
 Direct Methods for Constrained
Optimization 20

As we have already seen in Chap. 9, the direct methods for unconstrained optimization do not use
derivative information. From the multitude of these methods, only the NELMEAD by Nelder and
Mead (1965), NEWUOA by Powell (2004, 2006), and DEEPS by Andrei (2021a) have been
discussed. They are suitable for solving unconstrained optimization problems with a small number
of variables (let us say up to 100).
This chapter is dedicated to presenting the direct methods for solving constrained optimization
problems and applications. They use only the function values that define the problem along a
sequence of points {xk} that is expected to be convergent to a point in which the constraints are
satisfied, and the objective value is less than its value in the initial point. The direct methods are
suitable in situations in which the smoothness of the functions of the problem is not satisfied or when
the values of the functions are obtained by simulation of some complex mathematical structures.
The concept optimization based on simulation is applied in the analysis, design, and control of
complex physical systems by using methods of optimization. Thus, by computational simulation,
some quantities necessary for optimization are generated. Every simulation needs the execution of a
finite sequence of computing programs involving, for example, the generation of discretizations and
solving some systems of differential equations. The results of simulation must be processed in order
to obtain the final values of the minimizing function and of the constraints. Observe that these
computational complications are not suitable for using optimization methods based on derivative
information. Even if the automatic differentiation has been successfully applied for solving some
optimization problems, it is not appropriate for the optimization based on simulation. The concept of
nonsmooth optimization refers to the problems that involve, for example, functions of the type |x| or
more general nondifferentiable functions. On the other hand, there are some optimization problems
that involve some computational schemes of the type if-then-else which make the
corresponding problems nonsmooth.
These sorts of optimization problems are treated by the direct methods. The crucial aspect of the
direct methods is that they are based on the hypothesis that in the set of trying solutions there is an
order relation which allows us to specify that a trying solution is better than another one and at every
iteration, there is a finite number of new possible solutions determined by the searching method. This
aspect emphasizes the important difference between the direct methods of optimization and the
methods based on derivative information (gradient, Hessian) in which there is a continuum of trying
solutions.

# The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 679
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2_20
680 20 Direct Methods for Constrained Optimization

The purpose of this chapter is to detail two different direct methods for constrained optimization
together with their numerical performances for solving real applications of optimization from the
LACOP collection. The first method is COBYLA—Constrained Optimization BY Linear Approxi-
mation of Powell (1993). At every iteration of this method, the objective function and constraints are
linearly approximated by interpolation in vertices of a simplex structure. The corresponding linear
programming problem completed with a constraint of trust-region type is solved, thus obtaining a new
approximation to the solution of the original problem. The second method is an extension of the
successive quadratic penalty method which does not involve derivative information. This method
approximately minimizes a sequence of merit functions, in which the penalization of the violation of
the constraints is progressively enlarged (Liuzzi, Lucidi, & Sciandrone, 2010).

20.1 COBYLA Algorithm

The COBYLA algorithm is an extension of the Nelder-Mead direct method for solving constrained
optimization problems (see Chap. 9). For solving the unconstrained optimization problem minf(x),
where f : ℝn ! ℝ, the Nelder-Mead method computes at a given iteration, the minimizing function
values in n þ 1 points of a simplex {x j : j ¼ 0, 1, . . ., n} which satisfy the nondegeneracy condition.
Let xl be the vertex of the simplex for which f(xl) ¼ max {f(x j) : j ¼ 0, 1, . . ., n}. Since xl is the vertex
of the simplex in which the minimizing function has the highest value, the algorithm replaces xl with
the point

ð1 þ θ Þ X j
n
xlnew ¼ θxl þ x, ð20:1Þ
n
j¼0, j6¼l
 
where the reflection coefficient θ is a constant in the open interval (0, 1). Obviously, if f xlnew is the
smallest value of the minimizing function, then the reflection coefficient can be enlarged. Formula
(20.1) defines the point xlnew by extrapolation along a line connecting xl with the centroid of the rest of
 
the vertices of the simplex. Hence, if f is a linear function, then f xlnew is smaller than the average of
the values {f(x j) : j ¼ 0, 1, . . ., n, j 6¼ l}, so that the iteration is a success subject to the reduction of
the function’s values. On the other hand, if the iteration is not a success, then the algorithm shrinks the
current simplex defined by the vertices {x j : j ¼ 0, 1, . . ., n}. This idea of the Nelder-Mead algorithm
is adapted for solving the constrained optimization problems

min f ðxÞ
subject to ð20:2Þ
ci ðxÞ  0, i ¼ 1, . . . , m,

where it is supposed that the functions f and ci, i ¼ 1, . . ., m, can be evaluated in any point x. For
solving this problem, Powell (1993) suggests the following algorithm based on the evaluation of the
functions defining the problem in the vertices {x j : j ¼ 0, 1, . . ., n} of a nondegenerate simplex in ℝn.
In this case, there are the linear functions fb and cbi , i ¼ 1, . . . , m, that interpolate f and ci, i ¼ 1, . . .,
m, in the vertices of the current simplex with which (20.2) can be approximated by the following
linear programming problem:
20.1 COBYLA Algorithm 681

min fbðxÞ
subject to ð20:3Þ
cbi ðxÞ  0, i ¼ 1, . . . , m:

Observe that the functions fband cbi , i ¼ 1, . . ., m, are uniquely determined. The modification of the
variables is under a trust-region constraint which closes the feasible domain of the linear program
(20.3). The radius of the trust-region determines the stepsizes. For comparing two solutions, the
algorithm uses the following merit function:

ΦðxÞ ¼ f ðxÞ þ μ½ max fci ðxÞ : i ¼ 1, . . . , mgþ , ð20:4Þ

where μ is a parameter and vþ ¼ max {0, v}. Observe that if x 2 ℝn is feasible, then Φ(x) ¼ f(x). In
this context, the point x 2 ℝn is better than the point y 2 ℝn if and only if Φ(x) < Φ( y). Observe that
the merit function (20.4) includes the constraints of the problem penalized by μ. With this, the
COBYLA algorithm may be describes as follows.
Let us suppose that the vertices {x j : j ¼ 0, 1, . . ., n}, the radius ρ of the trust-region and a value of
the penalty parameter μ from the merit function (20.4) are known. The vertices of the current simplex
are ordered so that x0 is the best vertex, that is
   
Φ x0  Φ xj , j ¼ 1, . . . , n: ð20:5Þ

Let x be the vector of the variables of the linear programming problem (20.3). Then the trust-
region condition is
  
x  x0   ρ: ð20:6Þ
2

With this, let x be the minimum of the linear approximation fbðxÞ subject to the linear constraints
from (20.3) and the inequality (20.6). It is quite possible for the constraints from (20.3) to be
incompatible with (20.6). In this case, define x as solution of the minimization of the largest violation
of the constraints subject to (20.6). The computation of x is made by continuously increasing ρ from
the zero value to a current value. The sequence of points x generated for different values of ρ defines
a pice-wise linear continuous trajectory.
Let us describe the procedure for the modification of the penalty parameter μ which depends
on x. Initially, μ ¼ 0. Observe that the reduction of the merit function Φ(x) < Φ(x0) is not obtained
if the value of μ does not satisfy the condition Φ b ðx0 Þ, where Φ
b ðx Þ < Φ b is the linear approximation

b ðxÞ ¼ fbðxÞ þ μ½ max fb

Φ ci ðxÞ : i ¼ 1, . . . , mgþ : ð20:7Þ

Therefore, the following procedure for modifying the penalty parameter μ can be used. Let μ be the
smallest nonnegative value of μ for which Φ b ðx0 Þ: The existence of μ is a direct consequence
b ðx Þ < Φ

of the definition of x . With this, the modification of μ is the following. If μ  3μ=2, then μ is
unchanged, otherwise set μ ¼ 2μ: The selection of the factors 3/2 and 2 is obtained from numerical
experiments with COBYLA. By increasing μ, it is possible for (20.5) not to be satisfied. In this case,
the optimality of x0 is restored by changing two vertices of the simplex. The computation of x and the
modification of μ are repeated until x0 is the vertex with the smallest value of the minimizing function
and the value of μ is acceptable.
The strategy for updating the radius of the trust-region is based on the principle that the value of ρ
is maintained until the iterations cease to achieve a sufficient reduction of the merit function,
682 20 Direct Methods for Constrained Optimization

otherwise the value of ρ is reduced. However, a lot of simplexes may appear along the iterations.
Before updating ρ, the current simplex must be “acceptable.” An unacceptable simplex is the one for
which (20.3) is an inadequate approximation of the original problem. Therefore, define an acceptable
simplex as follows. For j ¼ 1, . . ., n, let σ j be the Euclidian distance from the vertex x j to the opposite
face of this vertex from the current simplex. Let η j be the length of the segment connecting the
vertices x j and x0. The simplex is called “acceptable” if and only if the inequalities

σ j  αρ, ð20:8aÞ

ηj  βρ, ð20:8bÞ

are satisfied for j ¼ 1, . . ., n, where α and β are two constants so that 0 < α < 1 < β.
(Powell recommends α ¼ 0.25 and β ¼ 2.1.) The radius of the trust-region is reduced if either
kx  x0k  ρ/2 or

Φðx0 Þ  Φðx Þ
< 0:1: ð20:9Þ
b ðx0 Þ  Φ
Φ b ðx  Þ

In COBYLA, two values of ρ are used: the initial value ρbeg and the final value ρend. (Powell
recommends ρbeg ¼ 1.5 and ρend ¼ 108.) If the conditions of reducing ρ are satisfied, then the
following procedure is considered. If ρ  ρend, then the algorithm is stopped, the solution of the
problem is given by x0, except for the case in which Φ(x) < Φ(x0), when x is preferable as solution.
On the other hand, if ρ > ρend, then the radius of the trust-region is updated as

ρ=2, if ρ > 3ρend ,
ρnew ¼ ð20:10Þ
ρend , if ρ  3ρend :

Numerical experiments show that whenever ρ is reduced, the parameter μ is also reduced. To
update the parameter μ the following procedure may be used. Let us say that the i-th constraint is
important in the merit function if i is in the set
 
I ¼ i : cimin < cimax =2 \ f1, . . . , mg, ð20:11Þ

where cimin and cimax are the smallest and the largest value of ci(x) in the vertices of the current
simplex, respectively. If I ¼ ∅ , then set μ ¼ 0. Otherwise, μ is updated as

max j¼0,1,...,n f ðxj Þ  min j¼0,1,...,n f ðxj Þ

μ¼ n o , ð20:12Þ
min cimax þ  cimin : i 2 I

if and only if μ is reduced. The initial simplex is obtained by using ρbeg and the initial point given by
the user. Let x0 be the initial point. Then, for j ¼ 1, . . ., n, set x j ¼ x0 þ ρbegej, where ej is the j-th
column of the identity matrix. For any value of j, if f(x j) < f(x0), then x0 is replaced by x j, so that x0
becomes the vertex in which the minimizing function is minimum.
The point x is not computed at every iteration because the aim is to satisfy the acceptability
conditions (20.8a, 20.8b) of the current simplex. Hence, a point xΔ which improves the acceptability
of the current simplex is computed. In COBYLA, at the current iteration, x is computed instead of xΔ
if and only if at least one of the following five conditions is satisfied: (C1) the algorithm is at the first
iteration, (C2) at the previous iteration the value of ρ was reduced, (C3) the previous iteration
computed xΔ, (C4) the previous iteration computed x and the value of the merit function was reduced
at least with one tenth from the predicted reduction (see (20.9)), (C5) the current simplex is
20.1 COBYLA Algorithm 683

acceptable. When none of the above five conditions is satisfied, then xΔ is defined in the following
way. If η j > βρ, for all j ¼ 1, . . ., n, then let l be the smallest integer from [1, n] which satisfies the
condition
 
ηl ¼ max ηj : j ¼ 1, . . . , n : ð20:13Þ

Otherwise, l is obtained from the condition

 
σ l ¼ min σ j : j ¼ 1, . . . , n : ð20:14Þ

Now the vertex xl is replaced with xΔ, that is xΔ is far away from the face of the simplex which is
opposite to the vertex xl. Therefore, let vl be a vector of unitary length, perpendicular to this face of
the simplex. Then,

xΔ ¼ x0  γρvl , ð20:15Þ

where  is selected to minimize the approximation Φ b ðxΔ Þ, while γ is a constant from the interval
(α, 1) (γ ¼ 1/2 in COBYLA). Therefore, the next iteration is given by the simplex with the vertices
{x j : j ¼ 0, 1, . . ., n, j 6¼ l} and xΔ. This is an iteration which computes xΔ.
When an iteration computes x, then we must choose one from the following three options: reduce
ρ, maintain the value of ρ for another iteration which will compute x or maintain ρ for an iteration
which will improve the acceptability of the simplex. Since (20.9) is used if and only if x satisfies the
condition
  
x  x0   ρ=2, ð20:16Þ
2

it follows that the values f(x) and ci(x), i ¼ 1, . . ., m, are computed only when the inequality (20.16)
holds. These values can be included in the linear approximation (20.3) by replacing a vertex from
{x j : j ¼ 1, . . ., n} of the current simplex with x as follows. Compute the scalars fσ j : j ¼ 1, . . . , ng,
where σ j is the distance from x to the face of the current simplex which is opposite to x j. With this,
the set J ¼ fj : σ j  σ j g [ fj : σ j  αρg is determined, where α is defined in (20.8). Then, the
optimum vertex from the next iteration is the point
 
x , Φðx Þ < Φðx0 Þ,
x ¼
0
ð20:17Þ
x0 , Φðx Þ  Φðx0 Þ:
 
If the set J is nonempty, then let l be the last element of J, so that xl  x0 2 ¼
n  o
max xj  x0 2 : j 2 J : If one of the following conditions Φ(x) < Φ(x0), σ l > σ l or both of
them are satisfied, then xl is replaced with x, where l is obtained from the relation σ l =σ l ¼
max fσ j =σ j : j ¼ 1, . . . :ng. Thus, the simplex is updated at the iterations which compute the values
of the minimizing function and of the constraints in x, the only exception being the iterations where,
 
besides the inequalities fσ j  σ j : j ¼ 1, . . . :ng and Φ(x)  Φ(x0), the distance xj  x0 2 is upper
bounded by δρ for any j 2 J, where δ is a constant from the interval (1, β]. (In COBYLA, δ ¼ 1.1).
With this, the description of COBYLA is complete.
To prove the convergence of the algorithm, the following points need to be established: (A1) the
penalty parameter μ from the merit function (20.4) remains finite, (A2) the number of reductions of
the radius of the trust-region ρ is finite, (A3) if and only if μ and ρ remain unchanged, then the
optimum vertex x0 cannot be retained for an infinite number of iterations, (A4) if μ and ρ remain
684 20 Direct Methods for Constrained Optimization

Table 20.1 Performances of COBYLA. rbeg ¼ 1.5d0, rend ¼ 1. d  8.

n m vfi nri #nf vfo nrf

ALKI 10 34 – 872.3874 0.448994 1552 –1550.38851 0.5086e –12
CAM 10 43 – 47.12388 0.912991 182 – 43.859947 0.2957e –14
DES 15 10 3.7046642 572.8970 1440 6.4746914 0.9222e –15
HANG 20 2 4.953514 21.98809 97484 5.0690569 0.7105e –13
MSP3 13 41 449.70 202.04113 1420391 97.587578 0.156e–13
MSP5 16 53 284.66957 85.358034 108956 175.619466 0.0
POOL 34 108 2743.3678 5886.4115 1025 2779.2255 0.9570e –10
PREC 8 22 3.6573657 0.4350181 3552 3.9511635 0.1665e –15
PPSE 9 30 4853.3335 1.184889 230 5055.0118 0.844e–15
TRAFO 6 2 137.06643 0.0 6986 135.075962 0.222e–15

In this table, we have n ¼ the number of variables, m ¼ the number of constraints including the simple bounds on
variables, vfi ¼ the value of the objective function in the initial point, nri ¼ the norm of the vector given by the violated
constraints in the initial point, #nf ¼ the number of the evaluations of the functions defining the problem, vfo ¼ the value
of the objective function in the final point, nrf ¼ the norm of the vector given by the violated constraints in the final
point

unchanged, then the number of replacements of the optimum vertex is finite. Powell (1993) proved
that any sequence of iterations generated by COBYLA which do not modify ρ or x0 is finite. Moreover,
consider an iteration which replaces the vertex xl of the current simplex with xΔ and take the distances
from the vertices (excluding x0) to the opposite faces in the old simplex and in the new simplex
n o  
σ jold : j ¼ 1, . . . , n and σ jnew : j ¼ 1, . . . , n , respectively. Then, besides σ lnew ¼ γρ, the follow-
ing inequalities σ jnew  σ jold , j ¼ 1, . . ., n, j 6¼ l, hold.

Numerical Study—COBYLA Algorithm

Table 20.1 shows the performances of COBYLA for solving 10 applications from the LACOP
collection described in Appendix C.

20.2 DFL Algorithm

The following algorithm is based on the penalization of the constraints in the objective function, in
the frame of the direct methods. The DFL method, developed by Liuzzi, Lucidi, and Sciandrone
(2010), consists in the approximate minimization of a sequence of merit functions, in which the
penalization of the violated constraints is increased along the iterations. The minimization of the
merit functions involves only the values of the functions defining the problem, without referring to the
derivative information about these functions. Consider the problem
20.2 DFL Algorithm 685

min f ðxÞ
subject to
ð20:18Þ
cðxÞ  0,
l  x  u,

where f : ℝn ! ℝ, c : ℝn ! ℝm are known functions for which their values can be computed in any
point from the domain X ¼ {x 2 ℝn : l  x  u} defined by the simple bound constraints. For solving
(20.18), the objective function is augmented with terms which penalize the violation of the
constraints, thus obtaining the following penalty function:
!
1 X þ 2
X X
n 
þ 2
n 
þ 2
m
Qðx, μÞ ¼ f ðxÞ þ c ðxÞ þ ðxi  ui Þ þ ðli  xi Þ , ð20:19Þ
μ j¼1 j i¼1 i¼1

where, as usual, v+ ¼ max {0, v}. As it is known, if the minimum xk of the penalty function Q(x, μ)
 
can be determined for a sequence {μk} so that μk ! 0, then the sequence xk is convergent to a
global minimum x of (20.18) (Fiacco, & McCormick, 1968). More exactly, Nocedal and Wright
(2006) prove the convergence to a stationary point of (20.18) in very mild conditions of regularity,
provided that, for every k, a stationary point of Q(x, μk),can be determined with increased accuracy.
In other words, if {xk} is a sequence of points satisfying the condition

k∇Qðxk , μk Þk  τk , ð20:20Þ

where {τk} is a sequence of scalars such that 0 < τkþ1 < τk, for any k, and τk ! 0, then if the sequence
{xk} (or, at least a subsequence of it) is convergent to the point xe in which the gradients of the active
constraints are linear independent, xe is a stationary point of (20.18).
These theoretical results, very well established in the case of differentiable functions, can be
extended in the case in which there is no access to the derivative information of the functions defining
the problem (20.18). Observe that the simple bound constraints can be explicitly treated because their
gradients are known. Therefore, the following penalty function can be introduced (Liuzzi, Lucidi, &
Sciandrone, 2010).

1 X q
m
Pðx, μÞ ¼ f ðxÞ þ cj ðxÞþ , ð20:21Þ
μ j¼1

where q > 1 (for example q ¼ 1.1) and where only the nonlinear constraints are introduced in (20.21).
(For the linear constraints, their gradients are known.) With this, for any fixed value of μ the following
problem is considered:
min Pðx, μÞ
subject to ð20:22Þ
l  x  u:

The direct methods are based only on the values of the functions defining the problem computed
along a set of directions able at limit to give sufficient information to recover the necessary optimality
condition of order one. In the context of the constrained optimization, in which the penalty parameter
is updated to zero, the procedure for its updating must be in accordance with the procedure of
selecting the points where the functions defining the problem are evaluated, the so-called probe
points. The idea is that the penalty parameter needs to be reduced to zero more slowly than the
686 20 Direct Methods for Constrained Optimization

maximum step used in the computation procedure of the probe points, where the values of the
problem’s functions are evaluated.
The following proposition, proved by Liuzzi, Lucidi, and Sciandrone (2010), shows a very general
result which can be used to show the convergence to the stationary points of a sequence of iterations
generated by a direct search algorithm (without involving the derivative information) based on the
approximate minimization of the penalty function P(x, μ) on X. The proposition gives sufficient
conditions of the procedure for the computation of the probe points, as well as of the procedure for
updating the penalty parameter which guarantees the convergence to a stationary point of (20.18).

Proposition 20.1 Let {μk} be a bounded sequence of penalty parameters. Let {xk} be a sequence of
points so that xk 2 X for any k and let x be a limit point of the subsequence {xk}K, where
K ⊆ {0, 1, . . .}. Suppose that x satisfies the Mangasarian-Fromovitz (MFCO) constraint qualifica-
tion (see Remark 11.2) and for any k large enough the following conditions hold:

(i) For any d i 2 D \ DðxÞ there exist the vectors yik and the scalars ξik > 0 such that

yik þ ξik di 2 X, ð20:23aÞ

     
P yik þ ξik di , μk  P yik , μk  o ξik , ð20:23bÞ
  
max ξik , xk  yik 
lim k!1,k2K ¼ 0, ð20:23cÞ
μk

where D(x) ¼ {d 2 ℝn : di  0 if xi ¼ li, di  0 if xi ¼ ui, i ¼ 1, . . ., n} is the cone of

admissible directions in x subject to the simple bound constraints and D ¼ {e1, . . ., en},
where ei is the i-th column of the unity matrix.

(ii) For any k 2 K

 
lim k!1,k2K μk cðxk Þþ  ¼ 0: ð20:23dÞ

Then x is a stationary point of (20.18). ♦

A measure of the stationarity of the current iteration xk for (20.22) is given by

  
max i:di 2D\DðxÞ ξik , xk  yik  , ð20:24Þ

(see (Kolda, Lewis, & Troczon, 2003)). Therefore, the limit from (20.23c) shows that the current
measure of the stationarity tends to zero faster than the penalty parameter μk does.
With these preparatives, let us present a variant of the direct search algorithm for solving the
problem (20.18). The algorithm uses a linear search which approximately minimizes a penalty
function along some search directions. At every iteration, the searching directions are parallel with
the axes of coordinates. The algorithm is based on the sensitivity of the objective function along the
searching directions. This is the reason why the algorithm computes different stepsizes along each
searching direction. In particular, at every iteration the following quantities are computed:
20.2 DFL Algorithm 687

• αi , i ¼ 1, . . ., n, as the maximum stepsize that can be taken along the searching directions without
leaving the domain X,
• αik , i ¼ 1, . . . , n, as the results of the approximate minimizations of the penalty function along the
directions dik ,
• e
αik , i ¼ 1, . . . , n, as the stepsizes from the previous iteration used as initial stepsizes at the current
iteration.

Algorithm 20.1 DFL Algorithm

1. Consider the initial point x0 2 X, as well as the parameters: μ0 > 0, γ > 0, θ 2 (0, 1), p > 1. Consider e
αi0 > 0
and set d0 ¼ ei , for i ¼ 1, . . ., n. Consider a sequence of positive numbers ηk ! 0
i

2. Minimization on the cone D. Set i ¼ 1 and yik ¼ xk

 
3. Compute αi so that yik þ αi d ik 2 X: Set b αik ¼ min αi , e
αik
     i 2
αik > 0 and P yik þ b
If b αik dik , μk  P yik , μk  γ b αk , then compute αik using the procedure
 i i i i 
PE α , bαk , yk , dk , γ, αk : Set e
i
αkþ1 ¼ αk , dkþ1 ¼ dik and go to step 6
i i i
 
4. Compute α so that yk  α d k 2 X: Set b
i i i i
αik ¼ min αi , e
αik
     i 2
αik > 0 and P yik  b
If b αik dik , μk  P yik , μk  γ b αk , then compute αik using the procedure
 i i i 
PE α , bαk , yk , dk , γ, αk : Set e
i i
αkþ1 ¼ αk , dkþ1 ¼ dik and go to step 6
i i i

5. Set αik ¼ 0 and e

αikþ1 ¼ θe
αik
6. Set ykiþ1 ¼ yik þ αik dik : If i < n, then set i ¼ i þ 1 and go to step 3
 i i
7. αk , αk  μpk and kc(xk)+k > ηk, then set μkþ1 ¼ θμk; otherwise, set μkþ1 ¼ μk
If max i¼1,...,n e
 
8. Determine xkþ1 2 X such that Pðxkþ1 , μk Þ  P ykiþ1 , μk : Set k ¼ k þ 1 and go to step 2 ♦

The procedure PE(.) used in the frame of Algorithm 20.1 (see steps 3 and 4) is a step for computing
the stepsize αik which approximately minimizes the penalty function along the directions dik :

Algorithm 20.2 Procedure PEðα, b

α, y, p, γ, αÞ

1. Select δ 2 (0, 1)
2. Set α ¼ bα
3. Determine eα ¼ min fα, ðα=δÞg
4. If α ¼ α or Pðy þ e αÞ2 , return
αp, μk Þ > Pðy, μk Þ  γ ðe
5. Set α ¼ e
α and go to step 3 ♦

Liuzzi, Lucidi, and Sciandrone (2010) prove that the DFL algorithm is well defined. Moreover, if
the sequence of positive numbers {μk} monotone decreasing is such that lim k!1 μk ¼ μ > 0, then
lim k!1 αik ¼ 0, for i ¼ 1, . . ., n, and lim k!1 e
αik ¼ 0, for i ¼ 1, . . ., n. On the other hand, if
the sequence of positive numbers {μk} monotone decreasing is such that limk ! 1μk ¼ 0,
then lim k!1,k2K αik ¼ 0, for i ¼ 1, . . ., n, and lim k!1,k2K e αik ¼ 0, for i ¼ 1, . . ., n, where
K ¼ {k : μkþ1 < μk}.
688 20 Direct Methods for Constrained Optimization

Observe that the DFL algorithm considers the following search directions at every iteration k:
 
Dk ¼ d 1k , d1k , . . . , dnk , d nk ¼ fe1 , . . . , en g ¼ D,

where ei is the i-th column of the unity matrix. At every iteration, the algorithm gets information on
the behavior of the penalty function both in the direction dik and in the direction dik : In particular,
along all the directions dik , i ¼ 1, . . ., n, the algorithm identifies the following circumstances:

(a) If the initial stepsize e αik does not produce a reduction of the penalty function, then either yik þ
     i 2
e = X, or P yik þ e
αik d ik2 αik d ik , μk > P yik , μk  γ e
αk :
(b) If the initial stepsize e αik produces a reduction of the penalty function, then both conditions
     i 2
yik þ eαik d ik 2 X and P yik þ e
αik dik , μk  P yik , μk  γ eαk are satisfied and a stepsize αik is

  
αi αi
generated by the linear search, so that either yik þ δk d ik2 = X, or P yik þ δk d ik , μk > P yik , μk 

i 2
α
γ δk :

A similar analysis may be obtained along the direction dik (Liuzzi, Lucidi, & Sciandrone, 2010).
 
This analysis determines that the initial stepsize e αik should satisfy either yik þ e αik d ik 2= X, or
 i  i   i   i 2
P yk þ e αk dk , μk > P yk , μk  γ e
i
αk : Otherwise, compute a stepsize αk such that either yik þ
i


i 2
αik   αik    i  αk
δ d i
k 2= X or P yi
k þ δ d i
k , μ k > P y k , μ k  γ δ : In other words, the algorithm considers
the search along the directions d ik , extracting information from the behavior of the penalty function
P(.) along these directions. The following theorem shows the convergence of the DFL algorithm.

Theorem 20.1 Let {xk} be the sequence generated by the DFL algorithm. Suppose that any limit
point of the sequence {xk} satisfies the Mangasarian-Fromovitz (MFCO) constraint qualification (see
Remark 11.2). Then there exists a limit point x of the sequence {xk}, which is a stationary point of the
problem (20.18). ♦

Let {xk} and {μk} be the sequences generated by the DFL algorithm and assume that any limit
point of {xk} satisfies the Mangasarian-Fromovitz constraint qualification. If lim k!1 μk ¼ μ > 0,
then any limit point of {xk} is a stationary point of (20.18).
In step 1 of DFL the parameters are initialized as γ ¼ 106, θ ¼ 0.5, p ¼ 2, and e αi0 ¼
 3  
max 10 , min f1, ðx0 Þi  , i ¼ 1, . . ., n.

Numerical Study—DFL Algorithm

Table 20.2 shows the performances of DFL for solving 9 applications from the LACOP collection
presented in Appendix C.
A comparison between COBYLA and DFL (see Tables 20.1 and 20.2, respectively) shows that for
minimizing the applications considered in this numerical study, COBYLA generates better approxi-
mate solutions than DFL. An explanation of this behavior is that at every iteration, COBYLA
considers a linear approximation of the minimizing function and of the constraints and solves a linear
programming problem, which is more appropriate than the minimization of a simple penalty function.
20.2 DFL Algorithm 689

Table 20.2 Performances of DFL

n m vfi nri #iter #nf vfo nrf

ALKI 10 14 – 872.387 0.448994 672 1336 – 931.120 0.15e–10
ELCH 10 6 – 20.9602 1.449137 271 499 – 40.89478 0.218e– 6
CAM 10 23 – 47.1238 1.71630 306622 809653 – 45.54602 0.368799
LATHE 10 16 – 2931.46 0.0 714 961 – 4429.1529 0.0
MSP3 13 15 449.70 202.0401 1068 1774 50.0 0.09901
MSP5 16 21 284.669 85.35803 1778 2049 209.96775 0.741584
PPSE 9 12 4853.33 1.696721 6460942 >2e+7 8063.5088 0.003715
PREC 8 6 3.65736 0.435018 489 979 4.1431963 0.0
TRAFO 6 2 137.066 0.020303 20865 63112 136.27182 0.000126

In this table, we have n ¼ the number of variables, m ¼ the number of constraints, vfi ¼ the value of the objective
function in the initial point, nri ¼ the norm of the vector given by the violated constraints in the initial point, #iter ¼ the
number of the iterations, #nf ¼ the number of evaluations of the functions defining the problem, vfo ¼ the value of the
objective function in the final point, nrf ¼ the norm of the vector given by the violated constraints in the final point

In other words, the use of successive linear programming is more advantageous than the penalty
function. The weakness of the DFL algorithm is that at every iteration the penalty function is
approximately minimized along the axes of coordinates.

Notes and References

We emphasized that for derivative-free methods like the ones presented in this book, the only thing
we can obtain is a point where the minimizing function value is smaller than or equal to the value in
its initial point. Nothing can be said about its optimality, but having in view the derivative scarcity of
information on the minimizing function, the result obtained may be of use for practical
considerations.
Excellent reviews and perspectives highlighting the recent developments of the derivative-free
optimization methods for both unconstrained and constrained optimization with deterministic, sto-
chastic, or structured objectives were given by Rios and Shainidis (2013) and by Larson, Menickelly,
and Wild (2019). A review of the derivative-free algorithms followed by a systematic comparison of
22 related implementations using a test set of 502 problems was given by Rios and Shainidis (2013).
Their conclusion is not definitive. They emphasize that the ability of all these solvers to obtain good
solutions diminishes with the increasing size of the problem. Besides, attaining the best solutions
even for small problems is a challenge for most current derivative-free solvers and there is no single
solver whose performance dominates that of all the others. The dimension (the number of variables)
of the problems and the nonsmoothness rapidly increase the complexity of the search and decrease the
performances of all solvers.
 Appendix A: Mathematical Review

A1. Elements of Applied Numerical Linear Algebra

Vectors

Define a column n-vector to be an array of n numbers denoted as

2 3
x1
6x 7
6 27
x ¼ 6 7:
4⋮5
xn

The number xi, i ¼ 1, . . ., n, is called the i-th component of the vector x. Define the set of real
numbers by ℝ. The space of the real vectors of length n is denoted by ℝn. The vectors are always
column vectors. The transpose of x is denoted by xT. Therefore, xT is a row vector. Given the vectors
x, y 2 ℝn, the scalar product is defined by
Xn
xT y ¼ xy:
i¼1 i i

The vectors x, y 2 ℝn are orthogonal (perpendicular) if xTy ¼ 0. This is denoted by writing x ⊥ y.

If x and y are orthogonal and xTx ¼ 1 and yTy ¼ 1, then we say that x and y are orthonormal.
A set of vectors v1, . . ., vk is said to be linearly dependent if there are the scalars λ1, . . ., λk, not all
P
zero, so that ki¼1 λi vi ¼ 0: If no such set of scalars exists, then the vectors are said to be linearly
P
independent. A linear combination of the vectors v1, . . ., vk is a vector of the form ki¼1 λi vi , where all
λi are scalars.
Let {x1, . . ., xn} be a set of vectors. The span of this set of vectors denoted span{x1, . . ., xn} is the
set of all the vectors that can be expressed as a linear combination of {x1, . . ., xn}. That is,
n Xn o
spanfx1 , . . . , xn g ¼ v : v ¼ i¼1
αi x i , αi 2 ℝ :

If {x1, . . ., xn} is a set of n linearly independent vectors where each xi 2 ℝn, then span{x1, . . .,
xn} ¼ ℝn. In other words, any vector v 2 ℝn can be written as a linear combination of x1, . . ., xn. A
linearly independent set of vectors that span ℝn is said to be a basis for ℝn.

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 691
Switzerland AG
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
692 Appendix A: Mathematical Review

Norms of Vectors

For a vector x 2 ℝn, the following norms can be defined:

X
n  1=2
kx k1 ¼ jxi j, kxk2 ¼ xT x , kxk1 ¼ max jxi j:
i¼1, ..., n
i¼1

The norm k.k2 is often called the Euclidean norm or the l2 norm. On the other hand, k.k1 is referred
to as the l1 norm and k.k1 as the l1 norm. All these norms measure the length of the vector in some
sense, and they are equivalent, i.e., each one is bounded above and below by a multiple of the other.
More exactly, for all x 2 ℝn it follows that
pffiffiffi
kxk1  kxk2  nkxk1 and kxk1  kxk1  nkxk1 :

In general, a norm is any mapping k.k from ℝn to the nonnegative real numbers that satisfy the
following properties:

1. For all x, y 2 ℝn, kx þ yk  kxk þ kyk, with equality if and only if one of the vectors x and y is a
nonnegative scalar multiple of the other one.
2. kxk ¼ 0 ) x ¼ 0,
3. kαxk ¼ |α|kxk, for all α 2 ℝ and x 2 ℝn.

The magnitude of a vector x is kxk2 ¼ (xTx)1/2. The angle between two nonzero vectors
x, y 2 ℝn is defined to be the number θ 2 [0, π] so that cosθ ¼ xTy/kxkkyk.
For the Euclidian norm, the Cauchy-Schwarz inequality holds |xTy|  kxkkyk, with equality if and
only if one of these vectors is a nonnegative multiple of the other one. In particular,
 T  X  X

X 
x y ¼  x i y   j x i j j y j  ð max i j x i j Þ j y j ¼ kx k1 ky k1 :
i i i i i i

The H€ older inequality, a generalization of the Cauchy-Schwarz inequality, states that for all ai > 0,
bi > 0, i ¼ 1, . . ., n, p, q > 0 so that 1/p þ 1/q ¼ 1
!1=p !1=q
X
n X
n X
n
ai bi  api bqi :
i¼1 i¼1 i¼1

Matrices

A matrix is a rectangular array of numbers with m rows and n columns specified by its elements
aij, i ¼ 1, . . ., m, j ¼ 1, . . ., n. The space of the real m  n matrices is denoted by ℝmn. A submatrix of
a given matrix A is an array obtained by deleting any combination of rows and columns from A. The
leading j  j principal submatrix of A is denoted by A(1 : j, 1 : j). The transpose of A 2 ℝmn denoted
by AT is the n  m matrix with elements aji. In other words, the (i, j)-th entry of AT is the ( j, i)-th
entry of A. Therefore, if A 2 ℝmn, then AT 2 ℝn  m. The matrix A is squared if m ¼ n. For a square
matrix A ¼ (aij) 2 ℝnn the elements a11, a22, . . ., ann define the main diagonal of the matrix. A
squared matrix is symmetric if A ¼ AT. A matrix A 2 ℝnn is diagonal if aij ¼ 0 for all i 6¼ j.
The identity matrix denoted by I is the square diagonal matrix whose diagonal elements are all 1.
Appendix A: Mathematical Review 693

A square matrix A ¼ (aij) is said to be lower triangular if aij ¼ 0 for i < j. A unit lower triangular
matrix is a lower triangular matrix with all the diagonal elements equal to 1. The matrix A is said to be
upper triangular if aij ¼ 0 for i > j. A matrix A 2 ℝnn is tridiagonal if aij ¼ 0 for |i  j| > 1. A matrix
A 2 ℝnn is pentadiagonal if aij ¼ 0 for |i  j| > 2. A matrix A is normal if ATA ¼ AAT.

Matrix Norms

The most widely used matrix norms are defined in terms of vector norms. If A is a matrix and kxk is a
vector norm, then the induced matrix norm kAk is defined by

kAk ¼ max kAxk:

kxk¼1

Every induced matrix norm satisfies kAxk  kAkkxk for all vectors x. Also, kABk  kAkkBk for
the matrices A and B. The matrix norms corresponding to the above vector norms are
X
n   qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
  X
n  
kAk1 ¼ max Aij , kAk ¼ λ max AT A , kAk ¼ max Aij ,
2 1
1jn 1in
i¼1 j¼1

where λmax(ATA) is the largest eigenvalue of ATA. If A is a square symmetric matrix, then
kAk2 ¼ |λmax(A)|, where λmax(A) is the eigenvalue of the largest magnitude of A.
The Frobenius norm of A 2 ℝmn is defined as
X 2  
kAk2F ¼ aij  ¼ tr AT A ,
i, j

where for the matrix Ann ¼ (aij), tr(A) ¼ a11 þ ⋯ þ ann is the trace of A. The ellipsoid norm is
defined as kxkA ¼ (xTAx)1/2, where A is a symmetric and positive definite matrix.

Subspaces

For a function f : ℝn ! ℝm, let R( f ) denote the range of f. That is R( f ) ¼ {f(x) : x 2 ℝn} ⊆ ℝm is the
set of all the “images” when x varies over ℝn. The range of a matrix A 2 ℝmn denoted R(A) is the
span of the columns of A. That is,
RðAÞ ¼ fv 2 ℝm : v ¼ Ax, x 2 ℝn g:

Therefore, R(A) is the space spanned by the columns of A (column space). The range of AT is the
span of the columns of AT. But, the columns of AT are just the rows of A. Therefore,
  
R AT ¼ w 2 ℝn : w ¼ AT y, y 2 ℝm

is the space spanned by the rows of A (row space).

The dimension of R(A) is the rank of A, denoted rank(A). The rank of a matrix A is equal to the
maximum number of linearly independent columns in A. This number is also equal to the maximum
number of linearly independent rows in A. The rank of A 2 ℝmn can never be greater than the
minimum of m and n. The m  n matrix A is said to be of full rank if the rank of A is equal to the
minimum of m and n.
694 Appendix A: Mathematical Review

The null space of a matrix A 2 ℝmn is the set

N ðAÞ ¼ fx : Ax ¼ 0g ⊆ ℝn :

In other words, N(A) is the set of all the solutions to the homogeneous system Ax ¼ 0. For
A 2 ℝmn, the set N(AT) ¼ {y 2 ℝm : ATy ¼ 0} ⊆ ℝm is called the left-hand null space of A because
N(AT) is the set of all the solutions to the left-hand homogeneous system yTA ¼ 0T. Observe that the
vectors in R(A) are of size m, while the vectors in N(A) are of size n. Therefore, the vectors in R(AT)
and N(A) are both in ℝn. The following equations are true:

1. {w : w ¼ u þ v, u 2 R(AT), v 2 N(A)} ¼ ℝn.

2. R(AT) \ N(A) ¼ {0}.

In other words, R(AT) and N(A) are disjoint subsets that together span the entire space of ℝn. The
fundamental theorem of linear algebra states that
 
N ðAÞ  R AT ¼ ℝn ,

where n is the number of columns of A and  denotes the direct sum of two sets. (If S1 and S2 are two
sets, then S1  S2 ¼ {u þ v : u 2 S1, v 2 S2}.) Often, the sets of this type are called orthogonal
complements and we write R(AT) ¼ N(A)⊥.
If A 2 ℝmn then:

1. N(A) ¼ {0} if and only if rank(A) ¼ n.

2. N(AT) ¼ {0} if and only if rank(A) ¼ m.

For A 2 ℝmn the following statements are true:

1. R(ATA) ¼ R(AT) and R(AAT) ¼ R(A).

2. N(ATA) ¼ N(A) and N(AAT) ¼ N(AT).

For all the matrices A 2 ℝmn, dimR(A) þ dim N(A) ¼ n. Traditionally, dimN(A) is known as the
nullity of A.

Inverse of a Matrix

A squared n  n matrix A is nonsingular if for any vector b 2 ℝn there exists x 2 ℝn so that Ax ¼ b.

For nonsingular matrices A, there exists a unique n  n matrix B so that AB ¼ BA ¼ I. The matrix B is
denoted by A1 and is called the inverse of A. For nonsingular matrices A and B, the following
properties hold:

1. (A1)1 ¼ A,
2. If the product AB exists and it is nonsingular, then (AB)1 ¼ B1A1,
3. (AT)1 ¼ (A1)T.
4. (cA)1 ¼ c1A1, for any nonzero scalar c.
5. If A is nonsingular and symmetric, then A1 is symmetric.
6. If A 2 ℝnn is nonsingular, then rank(A) ¼ n.
7. det(A) 6¼ 0, where det(A) is the determinant of A.
Appendix A: Mathematical Review 695

If A, B 2 ℝnn, then the matrix B is the approximate inverse of A if kI  BAk < 1.

Banach lemma: If A, B 2 ℝnn and B is an approximate inverse of A, then A and B are both
nonsingular and

1
k Bk

kA k

A
 ,
B1
 ,
1  kI  BAk 1  kI  BAk

and

1

A  B
 kBkkI  BAk ,
A  B1
 kAkkI  BAk :
1  kI  BAk 1  kI  BAk

Von Neumann lemma: If kAk < 1, then I  A is nonsingular.

Sherman-Morrison formula. Let a, b 2 ℝn be two vectors so that 1 þ bTa 6¼ 0. It is straightforward
to verify by direct multiplication that
 1 abT
I þ abT ¼I :
1 þ bT a
Let A 2 ℝnn be a nonsingular matrix and a, b 2 ℝn two vectors, so that 1 þ bTA1a 6¼ 0. Then, the
inverse of the matrix B ¼ A þ abT is
 1   1  1
B1 ¼ A þ abT ¼ A I þ A1 abT ¼ I þ A1 abT A1
 
A1 abT A1 abT A1
¼ I T 1 A1 ¼ A1  :
1þb A a 1 þ bT A1 a

If 1 þ bTA1a ¼ 0, then B is a singular matrix. This is often called the Sherman-Morrison rank-one
update formula because, when a 6¼ b 6¼ 0, then rank(abT) ¼ 1.
A generalization of the Sherman-Morrison formula is as follows. If C, D 2 ℝn  p so that
(I þ DTA1C)1 exists, then
 1  1
A þ CDT ¼ A1  A1 C I þ DT A1 C DT A1 ,

known as Sherman-Morrison-Woodbury formula.

Some results for the unconstrained optimization quasi-Newton BFGS methods.

(1) Let

Bk sk sTk Bk yk yTk
Bkþ1 ¼ Bk  þ T
sTk Bk sk yk sk

be the BFGS updating formula, where Bk 2 ℝnn is invertible and sk, yk 2 ℝn so that yTk sk > 0: If
H k ¼ B1
k , then the inverse of Bkþ1, denoted by Hkþ1, is computed by twice applying the Sherman-
Morrison update formula as
 
Hk yk sTk þ sk yTk H k yTk Hk yk sk sTk
Hkþ1 ¼ Hk  þ 1þ T :
yTk sk y k sk yTk sk
696 Appendix A: Mathematical Review

(2) Let

Bk sk sTk Bk y yT
Bkþ1 ¼ δ k Bk  T þ γ k kT k
sk B k sk y k sk

be the scaled BFGS updating formula, where Bk 2 ℝnn is invertible, sk, yk 2 ℝn so that yTk sk > 0 and
δk, γ k 2 ℝ are two known nonzero scalar parameters. If H k ¼ B1 k , then the inverse of Bkþ1, denoted
by Hkþ1, is computed by twice applying the Sherman-Morrison update formula as
  
1 H k yk sTk þ sk yTk H k δk yTk H k yk sk sTk
Hkþ1 ¼ Hk  þ þ :
δk yTk sk γk yTk sk yTk sk

(3) Consider
" #
sk sTk yk yTk
Bkþ1 ¼ δk I  þ ,
ks k k 2 yTk sk

where sk, yk 2 ℝn so that yTk sk > 0, sk 6¼ 0 and δk 2 ℝ is a known nonzero scalar parameter. Then the
inverse of Bkþ1, denoted by Hkþ1, is computed by twice applying the Sherman-Morrison update
formula as
2
1 1 sk yTk þ yk sTk 1 kyk k sk sTk
H kþ1 ¼ I  þ 1 þ :
δk δk yTk sk δk yTk sk yTk sk

(4) Consider
" #
sk sTk yk yTk
Bkþ1 ¼ δk I  þ γk ,
ks k k 2 yTk sk

where sk, yk 2 ℝn so that yTk sk > 0, sk 6¼ 0 and δk, γ k 2 ℝ are two known nonzero scalar parameters.
Then the inverse of Bkþ1, denoted by Hkþ1, is computed by twice applying the Sherman-Morrison
update formula as
2
1 1 sk yTk þ yk sTk 1 1 kyk k sk sTk
H kþ1 ¼ I  þ þ :
δk δk yTk sk γ k δk yTk sk yTk sk

Orthogonality

A square matrix Q 2 ℝnn is orthogonal if it has the property QQT ¼ QTQ ¼ I, where I is the n  n
identity matrix. Therefore, the inverse of an orthogonal matrix is its transpose.
Suppose that kuk ¼ 1 and let u⊥ denote the space consisting of all the vectors that are perpendicu-
lar on u. u⊥ is called the orthogonal complement of u. The matrix P ¼ I  uuT is the orthogonal
projector onto u⊥ in the sense that P maps each x to its orthogonal projection in u⊥. For a subspace
S ⊂ ℝn, the orthogonal complement S⊥ of S is defined as the set of all the vectors in ℝn that are
orthogonal to every vector in S. In this case, dimS⊥ ¼ n  dim S.
Appendix A: Mathematical Review 697

Eigenvalues

A scalar value λ is an eigenvalue of the n  n matrix A if there exists a nonzero vector u 2 ℝn so that
Au ¼ λu. The vector u is called an eigenvector of A. The spectrum of a matrix is the set of all its
eigenvalues. Let λ1, . . ., λn be the eigenvalues of the matrix A, real or complex. Then, its spectral
radius ρ(A) is defined as ρ(A) ¼ max {|λ1|, . . ., |λn|}. Observe that ρ(A)  kAk for every matrix norm.
The condition number of A can be expressed as κ(A) ¼ ρ(A)ρ(A1). A matrix A is nonsingular if all its
eigenvalues are different from zero. The eigenvalues of the symmetric matrices are all real numbers.
The nonsymmetric matrices may have imaginary eigenvalues.
Two matrices A, B 2 ℝnn are similar if there exists a nonsingular matrix P 2 ℝnn so that
B ¼ P1AP. Similar matrices represent the same linear operator in different bases, with P being the
change of the basis matrix. Two similar matrices have the same eigenvalues, even though they will
usually have different eigenvectors.

Positive Definite Matrices

A square matrix A is positive definite if and only if xTAx > 0 for every nonzero x 2 ℝn. For the real
symmetric matrices A, the following statements are equivalent:

1. All the eigenvalues of A are positive.

2. A ¼ BTB for some nonsingular B. While B is not unique, there is one and only one upper-triangular
matrix R with positive diagonals so that A ¼ RTR. This is the Cholesky factorization of A.
3. A has an LU (or LDU) factorization with all the pivots being positive. The LDU factorization is of
the form A ¼ LDLT ¼ RTR, where R ¼ D1/2LT is the Cholesky factor of A.

Any of the statements above can serve as the definition of a positive definite matrix.
A matrix A is positive semidefinite if for all x 2 ℝn, xTAx  0. The following statements are
equivalent and can serve as the definition of a positive semidefinite matrix:

1. All the eigenvalues of A are nonnegative.

2. A ¼ BTB for some B with rank(B) ¼ r.

If a matrix is symmetric and positive definite, then its eigenvalues are all positive real numbers. A
symmetric matrix can be tested if it is positive definite by computing its eigenvalues and by verifying
if they are all positive or by performing a Cholesky factorization.

Gaussian Elimination (LU Factorization)

For solving the system Ax ¼ b, where A is nonsingular, the Gaussian elimination consists of the
following four steps:

1. Factorize the matrix A as A ¼ PLU, where:

P is a permutation matrix
L is a unit lower triangular matrix
U is a nonsingular upper triangular matrix
698 Appendix A: Mathematical Review

2. Solve the system PLUx ¼ b subject to LUx by permuting the entries of b, i.e., LUx ¼ P1b ¼ PTb.
3. Solve the system LUx ¼ P1b subject to Ux by forward substitution, i.e., Ux ¼ L1(P1b).
4. Solve the system Ux ¼ L1(P1b) subject to x by backward substitution, i.e., x ¼ U1(L1(P1b)).

The following result is central in the Gaussian elimination.

The following two statements are equivalent:

1. There exist a unique unit lower triangular matrix L and a nonsingular upper triangular matrix U
such that A ¼ LU. This is called the LU factorization of A.
2. All the leading principal submatrices of A are nonsingular.

The LU factorization without pivoting can fail on nonsingular matrices, and therefore we need to
introduce permutations into the Gaussian elimination.
If A is a nonsingular matrix, then there exist the permutation matrices P1 and P2, a unit lower
triangular matrix L, and a nonsingular upper triangular matrix U such that P1AP2 ¼ LU. Observe
that P1A reorders the rows of A. AP2 reorders the columns of A. P1AP2 reorders both the rows and the
columns of A.
The next two results state simple ways to choose the permutation matrices P1 and P2 to guarantee
that the Gaussian elimination will run on nonsingular matrices.

Gaussian Elimination with Partial Pivoting

The permutation matrices P02 ¼ I and P01 can be chosen in such a way that a11 is the largest entry in
absolute value in its column. More generally, at step i of the Gaussian elimination, where the i-th
column of L is computed, the rows i through n are permuted so that the largest entry in the column is
on the diagonal. This is called “Gaussian elimination with partial pivoting,” or GEPP for short.
GEPP guarantees that all the entries of L are bounded by one in absolute value.

Gaussian Elimination with Complete Pivoting

The permutation matrices P02 and P01 are chosen in such a way that a11 is the largest entry in absolute
value in the whole matrix. More generally, at step i of the Gaussian elimination, where the i-th
column of L is computed, the rows and the columns i through n are permuted so that the largest entry
in this submatrix is on the diagonal. This is called “Gaussian elimination with complete pivoting,” or
GECP for short.

Cholesky Factorization

The Cholesky factorization of a symmetric and positive definite matrix A, is defined as A ¼ LLT,
where L is a lower triangular matrix as follows:
Appendix A: Mathematical Review 699

The Cholesky factorization method for solving a symmetric positive definite system Ax ¼ b by
using the factorization A ¼ LDLT computes the elements of the diagonal matrix D and the lower
triangular matrix L as follows.

When A is symmetric and positive definite, then the Cholesky factorization requires about n3/6
multiplications per iteration. If A is indefinite, then the Cholesky factorization may not exist. Even if it
does exist, numerically it is unstable when it is applied to such matrices, in the sense that the elements
of L can become arbitrarily large. In this case, the modified Cholesky factorization may be used, as
described in Gill, Murray, and Wright (1981) or in Moré and Sorensen (1984).

Modified Cholesky Factorization

In this factorization the matrix A is modified during the course of the factorization in such a way that
all the elements of D are sufficiently positive and the elements of D and L are not too large. To control
the quality of the modification, two positive parameters δ and β are selected. They require that during
the computation of the j-th columns of L and D in the Cholesky factorization, the following bounds be
pffiffiffiffi
satisfied: dj  δ and |mij|  β, i ¼ j þ 1, . . ., n, where mij ¼ lij dj : To satisfy these bounds, we only
need to change one step in the Cholesky factorization algorithm. Indeed, the formula for computing
the diagonal element dj is replaced by
   
  θj  

dj ¼ max cjj ,  , δ , with θj ¼ max cij  :
β j<in

To verify that the above bounds hold, observe that in the Cholesky factorization, cij ¼ lijdj and
therefore
700 Appendix A: Mathematical Review

   
   pffiffiffiffi cij  cij β
mij  ¼ lij d j  ¼ pffiffiffiffi   β for all i > j:
dj θj

Details on the modified Cholesky factorization can be found in (Gill, Murray, & Wright, 1981).

QR Decomposition

A QR decomposition of a matrix A 2 ℝn  m is given by A ¼ QR, where Q 2 ℝnn is an orthogonal

matrix and R 2 ℝn  m is an upper-triangular matrix. If n  m, then R has the form
" #
Rb
R¼ ,
0

where Rb 2 ℝmm is an upper-triangular square matrix. Therefore, A can be expressed as A ¼ Q bR,

b
b b b
where Q is the matrix formed by the first m columns of Q. Now, if Q ¼ ½q1 q2 ⋯ qm , then A ¼ QRb
can be written as
X
m
bRx
Ax ¼ Q bx ¼
b ¼ Qb xbi qi :
i¼1

In other words, if A has full column rank m, then the first m columns of Q form an orthogonal basis
for the range of A.

Singular Value Decomposition

Suppose A 2 ℝmn with rank(A) ¼ r. Then A can be factored as A ¼ UΣVT, where U 2 ℝm  r satisfies
UTU ¼ I, V 2 ℝn  r satisfies VTV ¼ I and Σ ¼ diag (σ 1, . . ., σ r) with σ 1  σ 2  ⋯  σ r > 0. The
columns of U are called left singular vectors of A, the columns of V are called right singular vectors of
A, and the numbers σ i are the singular value. The matrix decomposition A ¼ UΣVT is known as the
singular value decomposition (SVD) of A. Observe that
" # " #
Σ2 0 T Σ2 0 T
AA ¼ U
T
U and A A ¼ V
T
V :
0 0 0 0

Therefore, the singular values of A are the positive square roots of the nonzero eigenvalues of AAT
(or ATA), the i-th left singular vector ui is the i-th eigenvector of AAT, and the i-th right singular vector
vi is the i-th eigenvector of ATA. It has many applications in optimization and in other fields. For
example:

1. The l2 norm and the Frobenius norm of matrix A 2 ℝmn of rank r are given by
sffiffiffiffiffiffiffiffiffiffiffiffiffi
X r
kAk2 ¼ σ 1 , and kAkF ¼ σ 2i :
i¼1
Appendix A: Mathematical Review 701

2. The condition number of a nonsingular matrix A 2 ℝnn is defined as

σ
cond ðAÞ ¼ κ ðAÞ ¼ kAk2
A1
2 ¼ 1 :
σn

3. The range and the null space of a matrix A 2 ℝmn of rank r have the form
RðAÞ ¼ spanfu1 , . . . , ur g and N ðAÞ ¼ spanðvrþ1 , . . . , vn g:

4. The properties and the computation of the Moore-Penrose pseudo-inverse of a matrix A 2 ℝmn is
defined as the matrix A+ 2 ℝn  m that satisfies the following four conditions:
(i) AAþ A ¼ A,
(ii) Aþ AAþ ¼ Aþ ,
(iii) ðAAþ Þ ¼ AAþ ,
T

(iv) ðAþ AÞ ¼ Aþ A:
T

Using the SVD of A, the Moore-Penrose pseudo-inverse of A can be obtained as

 
Aþ ¼ VΣþ U T , where Σþ ¼ diag σ 1 1
1 , . . . , σr :

5. For an underdetermined system of linear equations Ax ¼ b, where A 2 ℝmn, b 2 ℝm with m < n

and b 2 R(A), all its solutions are characterized by

x ¼ Aþ b þ V r z,

where A+ is the Moore-Penrose pseudo-inverse of A, Vr ¼ [vrþ1 vr þ 2 ⋯ vn] 2 ℝn  (n  r) composed

of the last n  r columns of the matrix V which is obtained by computing the SVD of A, and z 2 ℝn  r
is an arbitrary vector. Observe that A+b is a solution of the system Ax ¼ b, while Vrz belongs to the
null space of A. In other words, the vector z parameterizes all the solutions of an underdetermined
system of linear equations.

Spectral Decomposition (Symmetric Eigenvalue Decomposition)

Suppose A 2 ℝnn is a real symmetric matrix. Then A can be factored as A ¼ QΛQT, where
Q ¼ [q1, . . ., qn] 2 ℝnn is an orthogonal matrix with columns qi, i ¼ 1, . . ., n, as eigenvectors of
A and Λ ¼ diag (λ1, . . ., λn), where λi are the eigenvalues of A. When A is positive definite as well as
symmetric, this spectral decomposition is identical to the singular value decomposition. In this case,
the singular values σ i and the eigenvalues λi coincide.

Elementary Matrices

Matrices of the form I  uvT, where u, v 2 ℝn such that vTu 6¼ 1, are called elementary matrices. All
such matrices are nonsingular and
 1 uvT
I  uvT ¼I :
vT u1
702 Appendix A: Mathematical Review

The inverses of the elementary matrices are elementary matrices. The elementary matrices are
used for (i) interchanging the rows i and j, (ii) multiplying the row (column) i by α 6¼ 0, (iii) adding a
multiple of the row (column) i to the row (column) j.
An elementary lower-triangular matrix is defined as an n  n triangular matrix of the form

T k ¼ I  ck eTk ,

where ck is a column with zero in the first k positions. In particular, if

2 3
2 3 1 0 ⋯ 0 0 ⋯ 0
0 6 7
6 0 7 6 0 1 ⋯ 0 0 ⋯ 0 7
6 7 6 7
6 7 6⋮ ⋮ ⋱ ⋮ ⋮ ⋯ ⋮7
6 ⋮ 7 6 7
6 7 6 7
ck ¼ 6 7, then T k ¼ 6 0 0 ⋯ 1 0 ⋯ 0 7:
6 μkþ1 7 6 7
6 0 7
6
4 ⋮ 5
7 6 0 0 ⋯ μkþ1 1 ⋯ 7
6 7
4⋮ ⋮ ⋮ ⋮ ⋱ ⋮5
μn
0 0 ⋯ μn 0 ⋯ 1

Observe that
2 3
1 0 ⋯ 0 0 ⋯ 0
6 0 ⋯ ⋯ 0 7
6 1 0 0 7
6 7
6⋮ ⋮ ⋱ ⋮ ⋮ ⋯ ⋮7
6 7
6 7
T 1 ¼ I þ ck ek ¼ 6 0
T
0 ⋯ 1 0 ⋯ 0 7,
k
6 7
6 0 0 ⋯ μkþ1 1 ⋯ 0 7
6 7
6 7
4⋮ ⋮ ⋮ ⋮ ⋱ ⋮5
0 0 ⋯ μn 0 ⋯ 1

which is also an elementary lower-triangular matrix.

The importance of the elementary lower-triangular matrices lies in the fact that all the row
operations needed to annihilate the entries below the k-th pivot can be accomplished with one
multiplication by Tk. If
2   
3
⋯ α1 ⋯ 
6   7
6 0 ⋯ α2 ⋯  7
6 7
6⋮ ⋮ ⋱ ⋮ ⋮ ⋯ ⋮7
6 7
6  7
Ak1 ¼ 6 0 0 ⋯ αk ⋯  7
6 7
6 0 0 ⋯ αkþ1  ⋯  7
6 7
6 7
4⋮ ⋮ ⋮ ⋮ ⋱ ⋮5

0 0 ⋯ αn ⋯ 

is the partially triangularized result after k  1 elimination steps, then

Appendix A: Mathematical Review 703

 
T k Ak1 ¼ I  ck eTk Ak1 ¼ Ak1  ck eTk Ak1
2  
3 2 3
⋯ α1  ⋯  0
6 7 6 7
6 0 
⋯ α2  ⋯  7 6 0 7
6 7 6 7
6 7 6 7
6⋮ ⋮ ⋱ ⋮ ⋮ ⋯ ⋮7 7 6 7
6 6 ⋮ 7
6 7 6 7
6 7 6 7
¼ 6 0 0 ⋯ αk  ⋯  7, where ck ¼ 6 0 7
6 7 6 7
6   7 6 7
6 0 0 ⋯ 0 ⋯ 7 6 αkþ1 =αk 7
6 7 6 7
6 7 6 7
6⋮ ⋮ ⋮ ⋮ ⋱ ⋮5 7 6 7
4 4 ⋮ 5

0 0 ⋯ 0 ⋯  αn =αk

contains the multipliers used to annihilate the entries below αk. Therefore, if no row interchanges are
required, then reducing the nonsingular matrix A to an upper triangular matrix U by the Gaussian
elimination is equivalent to executing a sequence of n  1 left-hand multiplications with elementary
lower-triangular matrices. That is, Tn  1⋯T2T1A ¼ U, and hence A ¼ T 1 1 1
1 T 2 ⋯T n1 U: Having in
view that ej ck ¼ 0 whenever j  k, it follows that
T

    
T 1 1 1
1 T 2 ⋯T n1 ¼ I þ c1 e1 I þ c2 e2 ⋯ I þ cn1 en1
T T T

¼ I þ c1 eT1 þ c2 eT2 þ ⋯ þ cn1 eTn1 :

But
2 3
0 0 ⋯ 0 0 ⋯ 0
6 7
6 0 0 ⋯ 0 0 ⋯ 0 7
6 7
6⋮ ⋮ ⋱ ⋮ ⋮ ⋯ ⋮7
6 7
6 7
ck eTk ¼ 6 0 0 ⋯ 0 0 ⋯ 0 7,
6 7
6 0 0 ⋯ lkþ1,k 0 ⋯ 0 7
6 7
6 7
4⋮ ⋮ ⋮ ⋮ ⋱ ⋮5
0 0 ⋯ ln,k 0 ⋯ 0

where the li, k’s are the multipliers used at the k-th stage to annihilate the entries below the k-th pivot.
Therefore, from all these developments, it follows that
A ¼ LU,

where
2 3
1 0 0 ⋯ 0
6 7
6 l2,1 1 0 ⋯ 0 7
6 7
L ¼ I þ c1 eT1 þ c2 eT2 þ ⋯ þ cn1 eTn1 ¼6
6 l3,1 l3,2 1 ⋯ 0 7
7,
6 7
4⋮ ⋮ ⋮ ⋱ ⋮5
ln,1 ln,2 ln,3 ⋯ 1

is the lower-triangular matrix with 1’s on the main diagonal, and where li, j is the multiplier used to
annihilate the (i, j)-position during the Gaussian elimination. Therefore, the factorization A ¼ LU is
the matrix formulation of the Gaussian elimination with no row interchanges.
704 Appendix A: Mathematical Review

Conditioning and Stability

These are two terms used in numerical computations when a problem is solved with an algorithm.
Conditioning is a property of the problem, irrespective of its being a linear algebra problem, an
optimization or a differential equation. A problem is well-conditioned if its solution is not greatly
affected by small perturbations to the data that define the problem. Otherwise, it is ill-conditioned. On
the other hand, stability of an algorithm is a property of the algorithm. An algorithm is stable if it is
guaranteed to generate accurate answers to well-conditioned problems.
The condition number of a nonsingular matrix A 2 ℝnn denoted as cond(A) or κ(A) is defined
as cond(A) ¼ kAkkA1k. If the 2-norm is used, then κ(A) ¼ σ max(A)/σ min(A), where σ max(A) and
σ min(A) are the largest and the smallest singular values of A, respectively. For normal matrices,
κ(A) ¼ |λmax(A)|/|λmin(A)|, where λmax(A) and λmin(A) are the largest and the smallest eigenvalues of A,
respectively. The matrix A is well-conditioned if κ(A) is small (close to 1). The matrix A is
ill-conditioned if κ(A) is large.
For general linear systems Ax ¼ b, where A 2 ℝnn, the condition number of the matrix can be
used to see the conditioning of the system. If the matrix A is perturbed to A and b to b and consider x as
the solution of the perturbed system Ax ¼ b, it can be shown that (Golub & Van Loan, 1996)
"



#
kx  x k
A  A

b  b

κ ðAÞ þ :
kx k k Ak kbk

Therefore, a large condition number κ(A) indicates that the problem Ax ¼ b is ill-conditioned,
while a small value shows well-conditioning of the problem.
To see the significance of the stability of an algorithm, let us consider the linear system Ax ¼ b
solved by means of the Gaussian elimination with partial pivoting and triangular substitution. It is
shown that this algorithm gives a solution x whose relative error is approximately

kx  x k gr ðAÞ
κ ð AÞ u,
kx k k Ak

where gr(A) is the size of the largest element that arises in A during the execution of the Gaussian
elimination with partial pivoting and u is the unit roundoff. (In double-precision IEEE arithmetic, u is
about 1.1  1016.) In the worst case, it can be shown that gr(A)/kAk may be around 2n  1, which
indicates that the Gaussian elimination with partial pivoting is an unstable algorithm (Demmel,
1997). However, in practice, after decades of numerical experience with the Gaussian elimination
with partial pivoting, it was noticed that gr(A) is growing slowly as a function of n. In practice, gr(A)
is almost always n or less. The average behavior seems to be n2/3 or perhaps even n1/2 (Trefethen &
Schreiber, 1990). Therefore, the Gaussian elimination with partial pivoting is stable for all practical
purposes. However, the Gaussian elimination without pivoting is definitely unstable. For the system
Ax ¼ b where A is a symmetric and positive definite matrix, the Cholesky factorization method with
triangular substitution is a stable algorithm.

Determinant of a Matrix

The determinant is a scalar defined only for square matrices. A permutation p ¼ ( p1, p2, . . ., pn) of
the numbers (1, 2, . . ., n) is simply any rearrangement of these numbers. The sign of a permutation
p is defined to be the number
Appendix A: Mathematical Review 705


þ1, if p can be restored to the natural order by an even number of interchanges,
σ ðpÞ ¼
1, if p can be restored to the natural order by an odd number of interchanges:

Let A ¼ (aij) 2 ℝnn be an arbitrary matrix, where all its elements aij are real numbers. The
determinant of A is defined to be the scalar
X
detðAÞ ¼ σ ðpÞa1p1 a2p2 ⋯anpn ,
p

where the sum is taken over the n! permutations p ¼ ( p1, p2, . . ., pn) of (1, 2, . . ., n).
(n ! ¼ 1  2  . . .  n. For example 3 ! ¼ 1  2  3 ¼ 6.) Each term a1p1 a2p2 ⋯anpn contains
exactly one entry from each row and from each column of A.
Some properties of the determinants are as follows:

1. The determinant of a diagonal matrix det[diag(x1, x2, . . ., xn)] ¼ x1x2. . .xn.

2. Let In be the identity matrix of order n. Then det(In) ¼ 1.
3. The determinant of a triangular matrix is the product of its diagonal entries.
4. For any matrix A 2 ℝnn and constant c, det(cA) ¼ cn det (A).
5. Suppose that B is obtained from A by swapping two of the rows (columns) of A. Then
det(B) ¼  det (A).
6. If there is a row (column) of A all zero, then det(A) ¼ 0.
7. If two rows (columns) of A are equal, then det(A) ¼ 0.
 
8. det AT ¼ detðAÞ:
 
9. det A1 ¼ 1=detðAÞ:
10. detðABÞ ¼ detðAÞdetðBÞ:
11. If λ1, λ2, . . ., λn are the eigenvalues of A 2 ℝnn, then det(A) ¼ λ1λ2⋯λn.

For a matrix A 2 ℝnn, the polynomial p(λ) ¼ det (A  λI) is called the characteristic polynomial
of A. The set of all the eigenvalues of A is the set of all the roots of its characteristic polynomial. The
Cayley-Hamilton theorem says that p(A) ¼ 0.
Let In be the identity matrix of order n and u1, u2 2 ℝn arbitrary vectors. Then,
 
det I n þ u1 uT2 ¼ 1 þ uT1 u2 :

Let In be the identity matrix of order n and u1, u2, u3, u4 2 ℝn arbitrary vectors. Then,
       
det I n þ u1 uT2 þ u3 uT4 ¼ 1 þ uT1 u2 1 þ uT3 u4  uT1 u4 uT2 u3 :

Indeed,
 h  1 T i
I n þ u1 uT2 þ u3 uT4 ¼ I n þ u1 uT2 I n þ I n þ u1 uT2 u3 u4 :

Therefore,
706 Appendix A: Mathematical Review

    h  1 T i
det I n þ u1 uT2 þ u3 uT4 ¼ det I n þ u1 uT2 det I n þ I n þ u1 uT2 u3 u4
  h   i
1
¼ 1 þ uT1 u2 1 þ uT4 I n þ u1 uT2 u3
  
  T
u1 u2
¼ 1 þ uT1 u2 1 þ uT4 I n  u3
    1 þ uT1u2 

¼ 1 þ uT1 u2 1 þ uT3 u4  uT1 u4 uT2 u3 :

Determinant of the quasi-Newton BFGS update.

(1) Let

Bk sk sTk Bk yk yTk
Bkþ1 ¼ Bk  þ T ,
sTk Bk sk yk sk

be the BFGS updating of the matrix Bk, where Bk 2 ℝnn and sk, yk 2 ℝn so that yTk sk > 0. Then,
 
sk sTk Bk B1 k yk yk
T
detðBkþ1 Þ ¼ det Bk I  T þ T
s Bs y k sk
 k k k 
ðBk sk ÞT 1 yTk y T sk
¼ detðBk Þdet I  sk T þ Bk yk T ¼ detðBk Þ T k :
sk B k sk yk sk s k Bk s k

(2) Consider

Bk sk sTk Bk y yT
Bkþ1 ¼ δ k Bk  T þ γ k kT k ,
sk B k sk yk sk

where sk, yk 2 ℝn so that yTk sk > 0 and δk, γ k 2 ℝ are two known nonzero scalar parameters. Then,

yTk sk n1
detðBkþ1 Þ ¼ detðBk Þ δ γ :
sTk Bk sk k k

Trace of a matrix

The trace of a square matrix A ¼ (aij) 2 ℝnn is

X
n
traceðAÞ ¼ tr ðAÞ ¼ aii :
i¼1

The trace satisfies:

1. tr(AT) ¼ tr(A).
2. tr(AB) ¼ tr(BA).
3. tr(αA þ βB) ¼ αtr(A) þ βtr(B), α, β 2 ℝ.
Appendix A: Mathematical Review 707

If λ1, λ2, . . ., λn are the eigenvalues of A 2 ℝnn, then trace(A) ¼ λ1 þ λ2 þ ⋯ þ λn. If

    Xm X
n
A ¼ aij 2 ℝmn , then tr AT A ¼ a2ij :
i¼1 j¼1

Bk sk sTk Bk y yT
Let Bkþ1 ¼ Bk  sTk Bk sk
þ ykT skk be the BFGS updating of the matrix Bk, where Bk 2 ℝnn and
k

sk, yk 2 ℝn so that yTk sk > 0: Then,

k Bk s k k 2 k y k k 2
tr ðBkþ1 Þ ¼ tr ðBk Þ  þ T :
sTk Bk sk y k sk

A2. Elements of Analysis

Let {xk} be a sequence of points from ℝn. A sequence {xk} converges to a point x, written as
limk ! 1xk ¼ x, if for any ε > 0 there exists an index K so that kxk  xk  ε for all k  K. Given an
index set K ⊂ f1, 2, . . .g, a subsequence of {xk} corresponding to K can be defined and denoted by
fxk gk2K : Consider a convergent sequence {xk} with the limit x. Then, any subsequence of {xk} also
converges to x. A convergent sequence has only one limit. A sequence {xk} in ℝn is bounded if there
exists a number B  0 such that kxkk  B for all k ¼ 1, 2, . . . . Every convergent sequence is bounded.
A sequence {xk} in ℝn is uniformly bounded away from zero if there exists ε > 0 such that |xk|  ε for
any k  1.

Theorem A2.1 (Bolzano-Weierstrass Theorem). Each bounded sequence in ℝn has a convergent

subsequence. ♦

The point x 2 ℝn is an accumulation point or a limit point or a cluster point for the sequence {xk}
if there is an infinite set of indices k1, k2, k3, . . . so that the subsequence fxki gi¼1,2,3,...converges to x,
i.e., lim i!1 xki ¼ x : A sequence is a Cauchy sequence if for any ε > 0 there exists an integer K > 0 so
that kxk  xmk  ε for all the indices k  K and m  K. A sequence converges if and only if it is a
Cauchy sequence.
A function f : ℝn ! ℝm is continuous at x 2 ℝn if for all ε > 0 there exists a δ(ε, x) > 0 so that for
any y 2 ℝn, ky  xk2  δ(ε, x) )kf( y)  f(x)k2  ε. The continuity can be described in terms of
limits: whenever the sequence {xk} in ℝn converges to a point x 2 ℝn, the sequence {f(xk)} in ℝm
converges to f(x), i.e., limk ! 1f(xk) ¼ f(limk ! 1xk). A function f is continuous if it is continuous at
every point in ℝn.
A function f : ℝn ! ℝm is uniformly continuous at x 2 ℝn if for all ε > 0 there exists a δ(ε) > 0 so
that for any y 2 ℝn, ky  xk2  δ(ε) )kf( y)  f(x)k2  ε. It is obvious that a uniformly continuous
function is continuous.
If {xk} is a Cauchy sequence and f is uniformly continuous on a convex domain, then {f(xk)} is also
a Cauchy sequence.
A function f : ℝn ! ℝm is bounded if there exists a constant C  0 so that kf(x)k  C for all x 2 ℝn.
A continuous function f : ℝn ! ℝ is coercive if limkxk ! 1 ¼ þ 1 . This means that for any
constant M there must be a positive number RM such that f(x)  M whenever kxk  RM. In particular,
the values of f(x) cannot remain bounded on a set in ℝn which is not bounded. For f(x) to be coercive,
it is not sufficient that f(x) ! 1 as each coordinate tends to 1. Rather, f(x) must become infinite
708 Appendix A: Mathematical Review

along any path for which kxk becomes infinite. If f(x) is coercive, then f(x) has at least one global
minimizer, and these minimizers can be found among the critical points of f(x).
Let f : ℝ ! ℝ be a real-valued function of a real variable. The first derivative is defined by
f ð x þ εÞ  f ð x Þ
f 0 ðxÞ ¼ lim ε!0 :
ε
The second derivative is defined by

f 0 ð x þ εÞ  f 0 ð x Þ
f 00 ðxÞ ¼ lim ε!0 :
ε
The directional derivative of a function f : ℝn ! ℝ in the direction p 2 ℝn is given by

f ðx þ εpÞ  f ðxÞ
Dðf ðxÞ; pÞ ¼ lim ε!0 :
ε
Let f : ℝn ! ℝ be a continuously differentiable function. The conditions which characterize a
minimum can be expressed in terms of the gradient ∇f(x) with the first partial derivatives defined as
T
∂f ∂f
∇f ðxÞ ¼ , ⋯,
∂x1 ∂xn

and of n  n Hessian matrix ∇2f(x) with the second partial derivatives whose (i, j)-th element is
 2  2
∇ f ðxÞ ij ¼ ∂ f ðxÞ=∂xi ∂xj , i, j ¼ 1, . . . , n:

When f is twice continuously differentiable, the Hessian matrix is always symmetric. As a simple
example, let us consider the quadratic function f : ℝn ! ℝ, f(x) ¼ (1/2)xTAx þ bTx þ a, where
A 2 ℝnn is a symmetric matrix. Then, ∇f(x) ¼ Ax þ b. The Hessian of f is given by ∇2f(x) ¼ A, i.e.,
the second order approximation of a quadratic function is itself.
If f is continuously differentiable in a neighborhood of x, then

Dðf ðxÞ; pÞ ¼ ∇f ðxÞT p:

Theorem A2.2 (Mean Value Theorem) Given a continuously differentiable function f : ℝ ! ℝ and
two real numbers x1 and x2 that satisfy x2 > x1, then

f ðx2 Þ ¼ f ðx1 Þ þ f 0 ðξÞðx2  x1 Þ

for some ξ 2 (x1, x2).

For a multivariate function f : ℝn ! ℝ the mean value theorem says that for any vector d 2 ℝn,

f ðx þ dÞ ¼ f ðxÞ þ ∇f ðx þ αd ÞT d

for some α 2 (0, 1). ♦

Theorem A2.3 (Taylor’s Theorem). If f is continuously differentiable in a domain containing the line
segment [x1, x2], then there is a θ, 0  θ  1, so that
Appendix A: Mathematical Review 709

f ðx2 Þ ¼ f ðx1 Þ þ ∇f ðθx1 þ ð1  θÞx2 ÞT ðx2  x1 Þ:

Moreover, if f is twice continuously differentiable in a domain containing the line segment [x1, x2],
then there is a θ, 0  θ  1, so that
1
f ðx2 Þ ¼ f ðx1 Þ þ ∇f ðx1 ÞT ðx2  x1 Þ þ ðx2  x1 ÞT ∇2 f ðθx1 þ ð1  θÞx2 Þðx2  x1 Þ: ♦
2

For twice continuously differentiable functions f : ℝn ! ℝm and for any vector d 2 ℝn, one form of
the Taylor theorem is
1
f ðx þ dÞ ¼ f ðxÞ þ ∇f ðxÞT d þ dT ∇2 f ðx þ αd Þd,
2
for some α 2 (0, 1).
The level set of a function f : ℝn ! ℝ at level c is the set of points
S ¼ fx : f ðxÞ ¼ cg:

Theorem A2.4 Suppose that f is continuously differentiable. Then the vector ∇f(x0) is orthogonal to
the tangent vector to an arbitrary smooth curve passing through x0 on the level set determined by
f(x) ¼ f(x0). ♦

Consider the minimization problem min x2ℝn f ðxÞ with the minimum point x. Let e ¼ x  x be the
error and B(δ) the ball of the radius δ about x.

Theorem A2.5 Suppose that f is twice continuously differentiable with k∇2f(x)  ∇2f( y)k  γkx  yk,
∇f(x) ¼ 0 and ∇2f(x) is positive definite. Then there is δ > 0 such that for all x 2 B(δ),

2

∇ f ðxÞ
 2
∇2 f ðx Þ
,

 1

 1

∇ f ðxÞ
 2
∇2 f ðx Þ
,

1
 2  1
1

∇ f ðx Þ
kek  k∇f ðxÞk  2
∇2 f ðx Þ
kek: ♦
2

In the point x0 the gradient ∇f(x0) is the direction of the maximum rate of increase of f at x0. Since
∇f(x0) is orthogonal to the level set through x0 determined by f(x) ¼ f(x0), it follows that the direction
of the maximum rate of increase of a real-valued differentiable function at a point is orthogonal to the
level set of the function through that point.

Theorem A2.6 (Implicit Function Theorem). Let h : ℝn  ℝm ! ℝn be a function such that:

1. h(z, 0) ¼ 0 for some z 2 ℝn.

2. The function h(., .) is continuously differentiable in some neighborhood of (z, 0).
3. ∇zh(z, t) is nonsingular at the point (z, t) ¼ (z, 0).
710 Appendix A: Mathematical Review

Then, there are the open sets Nz ⊂ ℝn and Nt ⊂ ℝm containing z and 0, respectively, and a
continuous function z : Nt ! Nz such that z ¼ z(0) and h(z(t), t) ¼ 0 for all t 2 Nt. z(t) is uniquely
defined. If h is q times continuously differentiable with respect to both its arguments for some q > 0,
then z(t) is also q times continuously differentiable with respect to t and

∇zðtÞ ¼ ∇t hðzðtÞ, tÞ½∇z hðzðtÞ, tÞ1 ,

for all t 2 Nt. ♦

The implicit function theorem is applied to parameterized systems of linear equations in which z is
obtained as the solution of M(t)z ¼ g(t), where M(.) 2 ℝnn has M(0) nonsingular and g(t) 2 ℝn (see
the algebraic characterization of a tangent space). To apply the theorem, define h(z, t) ¼ M(t)z  g(t).
If M(.) and g(.) are continuously differentiable in some neighborhood of 0, the theorem implies that
z(t) ¼ M(t)1g(t) is a continuous function of t in some neighborhood of 0.

Rates of Convergence

Let {xk} be a sequence from ℝn that converges to x 2 ℝn. This sequence converges Qlinear if there
is a constant r 2 (0, 1) so that
kxkþ1  x k
r
kxk  x k

for all k sufficiently large. The convergence is Qsuperlinear if

kxkþ1  x k
lim ¼ 0:
k!1 kxk  x k

The convergence is Qquadratic if

kxkþ1  x k
M
kx k  x  k2

for all k sufficiently large, where M is a positive constant, not necessarily smaller than 1.
Typically, under appropriate assumptions, the quasi-Newton methods for unconstrained optimiza-
tion converge Qsuperlinearly, whereas Newton’s method converges Qquadratically. The steepest
descent algorithms converge only at a Qlinear rate, and when the problem is ill-conditioned, the
convergence constant r is close to 1.

Finite-Difference Derivative Estimates

Let f : ℝ ! ℝ. Finite-differencing refers to the estimation of f 0(x) using the values of f(x). Finite-
difference estimates can be obtained from Taylor’s series. In one dimension we have
1
f ðx þ hÞ ¼ f ðxÞ þ hf 0 ðxÞ þ h2 f 00 ðξÞ,
2
Appendix A: Mathematical Review 711

where ξ is on the line segment connecting x and x þ h. From the above equality, we get
f ðx þ hÞ  f ðxÞ 1 00
f 0 ðxÞ ¼  hf ðξÞ,
h 2
thus obtaining the approximation
f ð x þ hÞ  f ð x Þ
f 0 ðxÞ :
h
This is the most used finite-difference formula. It is called the forward finite-difference formula.
Now, let us estimate the error in finite differencing, and determine an acceptable value for h. For
this, observe that part of the error is due to the inaccuracies in the formula itself, called the truncation
error
1
truncated error ¼ hjf 00 ðξÞj:
2
Besides, there are rounding errors from the evaluation of ( f(x þ h)  f(x))/h which depends on
εmach, the precision of the computer calculation. Therefore, the rounding error from the evaluation of
the function f in the numerator is
ðrounding errorÞ1 jf ðxÞjεmach :

Hence, the rounding error is obtained as

jf ðxÞjεmach
ðrounding errorÞ2 þ jf 0 ðxÞjεmach ,
h
where the first rounding error is magnified by 1/h, and then there is an additional rounding error from
the division that is proportional to the result f 0(x). When h is small and f 0(x) is not large, then the first
term will dominate, thus leading to the estimate

jf ðxÞjεmach
rounding error :
h
Therefore, the total error is a combination of the above truncation error and the rounding error, i.e.,

1 jf ðxÞjεmach
error hjf 00 ðξÞj þ :
2 h
The best value of h is obtained by minimizing the error as a function of h. Differentiating the error
with respect to h and setting the derivative to zero, we get

1 00 jf ðxÞjεmach
j f ðξÞj  ¼ 0:
2 h2
Therefore, an estimate for h is obtained as
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2jf ðxÞjεmach
h¼ :
jf 00 ðξÞj
712 Appendix A: Mathematical Review

00
When f(x) and f (ξ) are neither large nor small, then a simpler estimation
pffiffiffiffiffiffiffiffiffiffi
h εmach

can be used. Observe that if h < |x|εmach, then the computed value of x þ h will be equal to x and the
finite-difference estimate will be zero. Thus, in the general case the choice of h will depend on εmach,
00
|x| and on the values of |f |. Some software packages use this estimation of h or some simple
modification of it which takes into account |x| or |f(x)|. see Dennis and Schnabel (1983) and Gill,
Murray, and Wright (1981).
A more accurate finite-difference formula is given by the central finite-difference formula
f ðx þ hÞ  f ðx  hÞ 1 2  000 000

f 0 ðxÞ ¼  h f ðξ 1 Þ þ f ðξ 2 Þ :
2h 12
It can be derived using the Taylor series for f(x þ h) and f(x  h) about point x. Higher derivatives
can also be obtained by finite differencing. For example,
f ðx þ hÞ  2f ðxÞ þ f ðx  hÞ 1 2  ð4Þ 
f 00 ðxÞ ¼  h f ð ξ 1 Þ þ f ð4Þ
ðξ 2 Þ
h2 24

can be obtained from the Taylor series for f(x þ h) and f(x  h) about point x.
For multidimensional functions F(x) : ℝn ! ℝm, the derivatives can be estimated by applying the
finite-difference formulas to each component of the gradient or the Hessian matrix
 
f x þ hej  f ðxÞ
½∇f ðxÞj ,
h
where ej is the j-th column of the unity matrix.
If the gradient is known, then the Hessian can be approximated as

 2  ∂ f ðxÞ ½∇f ðx þ hek Þ  ∇f ðxÞj

2
∇ f ðxÞ j,k ¼ :
∂xj ∂xk h

Special finite-differencing techniques to approximate sparse Hessian matrices were developed by

Curtis, Powell, and Reid (1974) and by Powell and Toint (1979).
In this presentation it was assumed that the rounding errors were proportional to the machine
precision εmach. More generally, the rounding errors will be proportional to the accuracy εf with which
the function f can be computed, which may be larger than the machine precision. In this case, the
formula for the estimation of h will include εf as well as εmach.

Automatic Differentiation

The automatic differentiation is a computational technique implemented in software tools which

apply rules for differentiating arithmetic operations, functions of functions, etc. These tools can
compute derivatives of the first and higher order. The automatic differentiation is different from
software for the symbolic differentiation which operates on the mathematical formulae and produces
the corresponding formulae for derivatives with respect to the chosen variables. The automatic
differentiation takes a user-supplied program for evaluating the function f(x) : ℝn ! ℝ and, using
Appendix A: Mathematical Review 713

the rules of calculus, determines the corresponding derivative values. In the following, a short
description of a method for the automatic differentiation which generates the function values, its
gradient, and the Hessian in a certain point x 2 ℝn is given.
 
Every variable xi, i ¼ 1, . . ., n, is represented by a triplet X ¼ xi , x0i , x00i , where xi 2 ℝ, x0i 2 ℝn
and x00i 2 ℝnn : Denote the set of all these triplets by Tn. On Tn, for any U and V from Tn, the following
arithmetic operations can be defined:

U þ V ¼ ðu þ v, u0 þ v0 , u00 þ v00 Þ,
U  V ¼ ðu  v, u0  v0 , u00  v00 Þ,
 
U V ¼ uv, uv0 þ vu0 , uv00 þ u0 v0 þ v0 u0 þ vu00 ,
T T

     
U=V ¼ u=v, ðvu0  uv0 Þ=v2 , v2 u00  v v0 u0 þ u0 v0 þ 2uv0 v0  uvv00 =v3 ,
T T T

for v 6¼ 0.
An independent variable in Tn is represented as Xi ¼ (xi, ei, 0), where ei is the i-th column of the
unitary matrix and 0 is an n  n matrix. All the constants c are represented by C ¼ (c, 0, 0). Let
00
C ¼ (c, 0, 0) be a constant and the triplet U ¼ (u, u0, u ) 2 Tn. Then,

C þ U ¼ U þ C ¼ ðc þ u, u0 , u00 Þ,
C  U ¼ ðc  u, u0 , u00 Þ,
U  C ¼ ðu  c, u0 , u00 Þ,
C U ¼ U  C ¼ ðc u, cu0 , cu00 Þ,
   
C=U ¼ c=u, cu0 =u2 , 2cu0 u0  cuu00 =u3 ,
T

U=C ¼ ðu=c, u0 =c, u00 =cÞ,

for c 6¼ 0 and u 6¼ 0.
Consider the following example:
 2
f ðxÞ ¼ 100 x2  x21 þ ð1  x1 Þ2

for which let us compute f(x0), ∇f(x0) and ∇2f(x0), where x0 ¼ [1.2, 1]T. For this, the point x0 is
represented in T2 as
" # " #!
1 0 0
X1 ¼ 1:2, , ,
0 0 0
" # " #!
0 0 0
X1 ¼ 1:0, , :
1 0 0

Taking into consideration the above arithmetic operations, the following steps determine f(x0),
∇f(x0) and ∇2f(x0), as in the table below
714 Appendix A: Mathematical Review

If g : ℝ ! ℝ is a twice continuous differentiable function, then its extension on Tn is the triplet

  
gðU Þ ¼ gðu, u0 , u00 Þ ¼ gðuÞ, g0 ðuÞu0 , g0 ðuÞu00 þ g00 ðuÞu0 u0 :
T

For example,
 
sin ðU Þ ¼ sin ðu, u0 , u00 Þ ¼ sin ðuÞ, u0 cos ðuÞ, u00 cos ðuÞ  u0 u0 sin ðuÞ :
T

Order Notation

Order notation is a concept used to see how the members of a sequence behave when we get far
enough along in the sequence. Let us consider two nonnegative sequences of scalars {ηk} and {θk}.
ηk ¼ o(θk) if the sequence of ratios {ηk/θk} approaches zero, i.e., limk ! 1ηk/θk ¼ 0. ηk ¼ O(θk) if there
is a positive constant c so that |ηk|  c|θk| for all k sufficiently large. If η : ℝ ! ℝ is a function, then
η(t) ¼ o(t) to specify that the ratio η(t)/t approaches zero either as t ! 0 or t ! 1 . Similarly,
η(t) ¼ O(t) if there is a constant c so that |η(t)|  c|t| for all t 2 ℝ. A slight variant of the above
definitions is as follows. ηk ¼ o(1) to specify that limk ! 1ηk ¼ 0. Similarly, ηk ¼ O(1) to indicate that
there is a constant c so that |ηk|  c for all k. Sometimes, in the above definitions, there are vectors or
matrices as arguments. In these cases, the definitions apply to the norms of these quantities. For
instance, if f : ℝn ! ℝn, then f(x) ¼ O(kxk) if there is a positive constant c so that kf(x)k  ckxk for all
x in the domain of f.
Appendix A: Mathematical Review 715

A3. Elements of Topology in the Euclidian Space ℝn

The open ball of radius ε centered at x is defined as the set B(x, ε) ¼ {x 2 ℝn : kx  xk < ε} in
any norm.
A subset D ⊂ ℝn is open if for every x 2 D there exists a positive number ε > 0 so that the ball of
radius ε centered at x is contained in D, i.e., {y 2 ℝn : ky  xk  ε} ⊂ D. The intersection of a finite
number of open sets is open. Any union of open sets is open.
A point x 2 ℝn is an interior point of the set D if there is an open ball B(x, ε) so that B(x, ε) ⊂ D.
The interior of a set D, denoted by intD, is the set of the interior points of D.The interior of a set is the
largest open set contained in D.
A point x 2 ℝn is an exterior point of D if it is an interior point of ℝn\D. Notice that the set D is
open if every point of D is an interior point of D. Obviously, if D is open, then intD ¼ D.
A point xe is said to be a limit point of the set D if every open ball Bðxe, εÞ contains a point x 6¼ xe so
that x 2 D. Note that xe does not necessarily have to be an element of D for being a limit point of D.
The set D is closed if for all the possible sequences of points {xk} in D, all the limit points of {xk}
are elements of D. The union of a finite number of closed sets is closed. Any intersection of closed sets
is closed.
The set D is bounded if there is some real number M > 0 so that kxk  M for all x 2 D.
The set D is compact if every sequence {xk} of points in D has at least one limit point and all such
limit points are in D. A central result in topology is that in ℝn the set D is compact if it is both closed
and bounded.

Theorem A3.1 (Weierstrass Extreme Value Theorem.) Every continuous function on a compact set
attains its extreme values on that set. ♦

The closure of the set D is the set cl(D) ¼ D [ L, where L denotes the set of all the limit points of D.
For a given point x 2 ℝn,a neighborhood of x is an open set containing x. A useful neighborhood is
the open ball of radius ε centered at x.
A point x 2 ℝn is a boundary point of the set D if every neighborhood of x contains points both
inside and outside of D. The set of the boundary points of D is denoted by ∂D.
Let f : D ⊂ ℝn ! ℝm. Then f is Lipschitz continuous on an open set N ⊂ D if there is a constant
0 < L < 1 so that

kf ðxÞ  f ðyÞk  Lkx  yk

for all x, y 2 N. L is called the Lipschitz constant. If g, h : D ⊂ ℝn ! ℝm are two Lipschitz

continuous functions on a set N ⊂ D, then their sum g þ h is also Lipschitz continuous, with the
Lipschitz constant equal to the sum of the Lipschitz constants for f and g, respectively. If
g, h : D ⊂ ℝn ! ℝm are two Lipschitz continuous functions and bounded on a set N ⊂ D, i.e.,
there is a constant M > 0 such that |g(x)|  M and |h(x)|  M for all x 2 N, then the product gh is
Lipschitz continuous on N.
If f is Lipschitz continuous on a set D ⊂ ℝn, then f is uniformly continuous on D. The reverse is
not true.
716 Appendix A: Mathematical Review

A4. Elements of Convexity:Convex Sets and Convex Functions

Convex Sets

A set C ⊂ ℝn is a convex set if for every point x, y 2 C the point z ¼ λx þ (1  λ)y is also in the set
C for any λ 2 [0, 1]. The intersection of any family of convex sets is a convex set. An affine set in ℝn
is the set of all the vectors {x}  S, where x 2 ℝn and S is a subspace of ℝn. A cone is a set V with the
property that for all x 2 V it follows that αx 2 V for all α > 0. A cone generated by {x1, x2, . . ., xm} is
the set of all the vectors of the form
X
m
x¼ αi xi , where αi  0 for all i ¼ 1, . . . , m:
i¼1

Observe that all the cones of this form are convex sets. A convex combination of a finite set of
vectors {x1, x2, . . ., xm} in ℝn is any vector x of the form
X
m X
m
x¼ αi xi , where αi ¼ 1, αi  0 for all i ¼ 1, . . . , m:
i¼1 i¼1

The following two results on the separation of the convex sets are used in specifying the optimality
conditions for nonlinear programming problems. We present them without proof (see Bazaraa,
Sherali, & Shetty, 1993).

Proposition A4.1 (Separation of a Convex Set and a Point). Let C ⊂ ℝn be a nonempty and convex
set. Consider a point y 2
= C. Then, there exist a nonzero vector a 2 ℝn and a scalar c 2 ℝ such that
a y > c and a x  c for any x 2 C.
T T
♦

Proposition A4.2 (Separation of Two Convex Sets). Let C1 and C2 be two nonempty and convex sets
in ℝn. Suppose that C1 \ C2 ¼ ∅ . Then, there exists a hyperplane that separates C1 and C2,
i.e., there is a nonzero vector p 2 ℝn such that pTx1  pTx2 for any x1 2 cl(C1) and for any
x2 2 cl(C2). ♦

Convex Functions

A function f : C ! ℝ defined on a convex set C ⊂ ℝn is a convex function if

f(λx þ (1  λ)y)  λf(x) þ (1  λ)f( y) for every x, y 2 C and every λ 2 (0, 1). Moreover, f is said to be
strictly convex if for every x, y 2 C and every λ 2 (0, 1), f(λx þ (1  λ)y) < λf(x) þ (1  λ)f( y).
In other words, this means that if we take any two points x and y, then f evaluated at any convex
combination of these two points should be no larger than the same convex combination of f(x) and
f( y). A function that is not convex is said to be nonconvex. A function f is concave if f is convex.
Any linear function of n variables is both convex and concave on ℝn. The following result shows why
the convex functions are of interest in optimization problems.

Theorem A4.1 Any local minimum of a convex function f : C ! ℝ defined on a convex set C ⊂ ℝn
is also a global minimum on C. Any local minimum of a strictly convex function f : C ! ℝ defined on a
convex set C ⊂ ℝn is the unique strict global minimum of f on C. ♦
Appendix A: Mathematical Review 717

Strong Convexity

A differentiable function f is called strongly convex on S with the parameter μ > 0 if for all the points
x, y 2 S,
μ
f ðyÞ  f ðxÞ þ ∇f ðxÞT ðy  xÞ þ ky  xk2 :
2
Intuitively, strong convexity means that there exists a quadratic lower bound on the growth of the
function. Observe that a strongly convex function is strictly convex since the quadratic lower bound
growth is strictly greater than the linear growth. An equivalent condition for the strong convexity of
the function f on S is

ð∇f ðxÞ  ∇f ðyÞÞT ðx  yÞ  μkx  yk2

for some μ > 0 and for all x, y 2 S.

For differentiable strongly convex functions, it is easy to prove that:

1. k∇f(x)k2  2μ( f(x)  f(x)) for all x 2 S, where x is a local minimum of the function f.
2. k∇f(x)  ∇ f( y)k  μkx  yk for all x 2 S.
3. f ðyÞ  f ðxÞ þ ∇f ðxÞT ðy  xÞ þ 2μ
1
k∇f ðyÞ  ∇f ðxÞk2 , for all x 2 S.

If the function f is twice continuously differentiable, then it is strongly convex with the parameter
μ > 0 on S if and only if ∇2f(x)  μI for all x 2 S, where I is the identity matrix and the inequality 
means that ∇2f(x)  μI is positive semidefinite.

Proposition A4.3 (Convexity of Level Set). Let C be a convex set in ℝn and let f : C ! ℝ be a convex
function. Then, the level set Cα ¼ {x 2 C : f(x)  α}, where α is a real number, is a convex set.

Proof Let x1, x2 2 C. Of course, x1, x2 2 Cα. f(x1)  α and f(x2)  α. Now, let λ 2 (0, 1) and consider
x ¼ λx1 þ (1  λ)x2. By the convexity of C, it follows that x 2 C. On the other hand, by the convexity
of f on C,

f ðxÞ  λf ðx1 Þ þ ð1  λÞf ðx2 Þ  λα þ ð1  λÞα ¼ α,

i.e., x 2 Cα. ♦

Proposition A4.4 (Convexity of a domain defined by a set of convex functions). Let C be a convex set
in ℝn and let ci : C ! ℝ, i ¼ 1, . . ., m, be convex functions on C. Then, the set defined by
X ¼ {x 2 C : ci(x)  0, i ¼ 1, . . ., m} is convex.

Proof The result follows from Proposition A4.3 and from the property of the intersection of the
convex sets. ♦

The following two propositions give differential criteria of checking the convexity of a function.

Proposition A4.5 (First-Order Condition for Convexity). Let C be a convex set in ℝn with nonempty
interior. Consider the function f : C ! ℝ which is continuous on C and differentiable on int(C). Then f
is convex on int(C) if and only if f( y)  f(x) þ ∇ f(x)T(y  x) for any points x, y 2 C. ♦
718 Appendix A: Mathematical Review

Proposition A4.6 (Second-Order Condition for Convexity). Let C be a convex set in ℝn with
nonempty interior. Consider the function f : C ! ℝ which is continuous on C and twice differentiable
on int(C). Then, f is convex on int(C) if and only if the Hessian ∇2f(x) is positive semidefinite at each
x 2 int (C). ♦

The convexity of the objective function and of the constraints is crucial in nonlinear optimization.
The convex programs have very nice theoretical properties which can be used to design efficient
optimization algorithms. Therefore, it is important to know how to detect the convexity and the
operations that preserve the convexity of functions.

Theorem A4.2 (Farkas’ Theorem). Let A be an m  n matrix and c an n vector. Then, just one of the
following two statements holds:

System 1. There exists x 2 ℝn such that Ax  0 and cTx > 0.

System 2. There exists y 2 ℝm such that ATy ¼ cand y  0.

Proof Suppose that System 2 has a solution, i.e., there exists y  0 such that ATy ¼ c. Let x be such
that Ax  0. Then, cTx ¼ yTAx  0. Therefore, System 1 has no solution.
Now, suppose that System 2 has no solution. Consider the set X ¼ {x : x ¼ ATy, y  0}. Observe
that X is a closed and convex set. By Proposition A4.1, there exist a vector p 2 ℝn and a scalar α such
that pTc > α and pTx  α for all x 2 X. Since 0 2 X, then α  0 and so pTc > 0. Also, α  pTATy ¼ yTAp
for all y  0. Since y  0 can be made arbitrarily large, it follows that the last inequality implies that
Ap  0. Therefore, we have constructed a vector p 2 ℝn such that Ap  0 and cTp > 0, i.e., System
1 has a solution. ♦

Theorem A4.3 (Gordan’s Theorem). Let A be a m  n matrix. Then just one of the following two
statements holds:

System 1. There exists x 2 ℝn such that Ax < 0.

System 2. There exists y 2 ℝm, y 6¼ 0 such that ATy ¼ 0 and y  0.

Proof System 1 can be equivalently written as Ax þ es  0 for some x 2 ℝn and s > 0, s 2 ℝ, where
e is a vector of m ones. Now, rewriting this system in the form of System 1 of Theorem A4.2, we
    
x x x
obtain ½A e  0 and ð0, . . . , 0, 1Þ > 0 for some 2 ℝnþ1 : By Theorem A4.2, the
s s s
" #
AT
associated System 2 states that y ¼ ð0, . . . , 0, 1ÞT and y  0 for some y 2 ℝm, that is, ATy ¼ 0,
eT
eTy ¼ 1 and y  0 for some y 2 ℝm. But this is equivalent to System 2. ♦

Theorem A4.4 Let P and Q be two symmetric matrices such that Q  0 and P > 0 on the null space
of Q (i.e., yTPy > 0 for any y 6¼ 0 with Qy ¼ 0). Then, there exists c > 0 such that P þ cQ > 0 for any
c > c:
Appendix A: Mathematical Review 719

Proof Assume the contrary. Then, for any k > 0 there exists xk, kxkk ¼ 1 such that
(xk)TP(xk) þ k(xk)TQ(xk)  0. Consider a subsequence {xk}K convergent to some x with kxk ¼ 1:
Dividing the above inequality by k and taking the limit as k 2 K ! 1, we get xT Qx  0: On the
other hand, Q being semipositive definite we must have xT Qx  0, hence xT Qx ¼ 0: Therefore,
using the hypothesis it follows that xT Px  0: But this contradicts the fact that
 k T  k 
x Px þ lim sup k x Q x  0:
T
♦
k!1, k2K

Proposition A4.7 (Linear combination with nonnegative coefficients). Let C be a convex set in ℝn. If
f : C ! ℝ and g : C ! ℝ are convex functions on C, then their linear combination λf þ ηg, where the
coefficients λ and η are nonnegative, is also convex on C. ♦

Proposition A4.8 (Composition with affine mapping). Let C and D be convex sets in ℝm and ℝn,
respectively. If g : C ! ℝ is a convex function on C and h : D ! ℝm is an affine mapping, i.e.,
h(x) ¼ Ax þ b with range(h) ⊂ C, then the composite function f : D ! ℝ defined as f(x) ¼ g(h(x)) is
convex on D. ♦

Notes and References

The material in this appendix is covered in (Dennis & Schnabel, 1983; Peressini, Sullivan, & Uhl,
1988; Trefethen & Schreiber, 1990; Bazaraa, Sherali, & Shetty, 1993; Golub & Van Loan, 1996;
Demmel, 1997; Trefethen & Bau III, 1997; Meyer, 2000; Nocedal & Wright, 2006; Bartholomew-
Biggs, 2008; Griva, Nash, & Sofer, 2009), etc.
 Appendix B: The SMUNO Collection

Small-Scale Continuous Unconstrained Optimization Applications

Application S1 Weber (1) Locate a central facility (Kelley, 1999, pp. 118–119; Andrei, 2003, p. 58)
[WEBER-1]
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
f ðxÞ ¼ 2 ðx1  2Þ þ ðx2  42Þ þ 4 ðx1  90Þ þ ðx2  11Þ þ 5 ðx1  43Þ2 þ ðx2  88Þ2 :
2 2 2 2

Initial point: x0 ¼ [10, 10]. f(x0) ¼  0.374731E þ 02.

Application S2 Weber (2) Locate a central facility (Kelley, 1999, pp. 118–119) [WEBER-2]
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
f ðxÞ ¼ 2 ðx1 þ 10Þ2 þ ðx2 þ 10Þ2  4 ðx1 Þ2 þ ðx2 Þ2
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
þ2 ðx1  5Þ2 þ ðx2  8Þ2 þ ðx1  25Þ2 þ ðx2  30Þ2 :

Initial point: x0 ¼ [1.2, 1]. f(x0) ¼ 0.785943E þ 02.

Application S3 Weber (3) Locate a central facility (Kelley, 1999, pp. 118–119) [WEBER-3]
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
f ðxÞ ¼ 2 ðx1 þ 10Þ þ ðx2 þ 10Þ  4 ðx1 Þ2 þ ðx2 Þ2
2 2
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
þ2 ðx1  5Þ2 þ ðx2  8Þ2 þ ðx1  25Þ2 þ ðx2  30Þ2
  
þ sin 0:0035 x21 þ x22 :

Initial point: x0 ¼ [1.2, 1]. f(x0) ¼ 0.786028E þ 02.

Application S4 Analysis of enzymes reaction (Andrei, 2003, p. 62) [ENZIMES]

11 
  2
X x1 u2i þ ui x2
f ðxÞ ¼ yi  2 ,
i¼1
ui þ ui x3 þ x4

where yi and ui have the following values:

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 721
Switzerland AG
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
722 Appendix B: The SMUNO Collection

i yi ui i yi ui
1 0.1957 4.000 7 0.0456 0.125
2 0.1947 2.000 8 0.0342 0.100
3 0.1735 1.000 9 0.0323 0.0833
4 0.1600 0.500 10 0.0235 0.0714
5 0.0844 0.250 11 0.0246 0.0625
6 0.0627 0.167

Initial point: x0 ¼ [0.25, 0.39, 0.415, 0.39]. f(x0) ¼ 0.531317E  02.

Application S5 Stationary solution of a chemical reactor (Shacham, 1986, pp. 1455–1481)

[REACTOR]

f ðx Þ ¼ ð 1  x 1  k 1 x 1 x 6 þ r 1 x 4 Þ2
þ ð 1  x 2  k 2 x 2 x 6 þ r 2 x 5 Þ2
þðx3 þ 2k3 x4 x5 Þ2
þð k1 x1 x6  r 1 x4  k3 x4 x5 Þ2
þð 1:5ðk2 x2 x6  r 2 x5 Þ  k3 x4 x5 Þ2
þð 1  x4  x5  x6 Þ2

where: k1 ¼ 31.24 k2 ¼ 0.272 k3 ¼ 303.03 r1 ¼ 2.062 r2 ¼ 0.02.

Initial point: x0 ¼ [1.09, 1.05, 3.05, 0.99, 6.05, 1.09]. f(x0) ¼ 0.196173E þ 08.

Application S6 Robot kinematics problem (Kearfott & Novoa, 1990, pp. 152–157; Andrei, 2013e,
pp. 101–103; Floudas, Pardalos, Adjiman, Esposito, Gümüs, Harding, Klepeis, Meyer, & Schweiger,
1999, pp. 329–331) [ROBOT]
 2
f ðxÞ¼ 4:731
103 x1 x3  0:3578x2 x3  0:1238x1 þ x7  1:637
103 x2  0:9338x4  0:3571
þð0:2238x1 x3 þ 0:7623x2 x3 þ 0:2638x1  x7  0:07745x2  0:6734x4  0:6022Þ2
 2
þ x6 x8 þ 0:3578x1 þ 4:731
103 x2
þð0:7623x1 þ 0:2238x2 þ 0:3461Þ2
 2
þ x21 þ x22  1
 2
þ x23 þ x24  1
 2
þ x25 þ x26  1
 2 2
þ x7 þ x28  1 :

Initial point: x0 ¼ [0.164, 0.98, 0.94, 0.32, 0.99, 0.05, 0.41, 0.91].
f(x0) ¼ 0.533425E þ 01.

Application S7 Solar spectroscopy (Andrei, 2003, p. 68) [SPECTR]

13 
X   2
ði þ x 3 Þ2
f ðx Þ ¼ x1 þ x2 exp   yi ,
i¼1
x4
Appendix B: The SMUNO Collection 723

where yi, i ¼ 1, . . ., 13 are as in the table below:

i yi i yi
1 0.5 8 2.5
2 0.8 9 1.6
3 1 10 1.3
4 1.4 11 0.7
5 2 12 0.4
6 2.4 13 0.3
7 2.7

Initial point: x0 ¼ [1, 1, 1, 1]. f(x0) ¼ 0.995870E þ 01.

Application S8 Estimation of parameters (Himmelblau, 1972, p. 430) [ESTIMP]

!2
X7
x21 þ ai x22 þ a2i x23
f ðx Þ ¼   1 ,
i¼1 1 þ ai x24 bi

where the parameters ai, bi, i ¼ 1, . . ., 7 have the following values:

i ai bi
1 0.0 7.391
2 0.000428 11.18
3 0.0010 16.44
4 0.00161 16.20
5 0.00209 22.20
6 0.00348 24.02
7 0.00525 31.32

Initial point: x0 ¼ [2.7, 90, 1500, 10]. f(x0) ¼ 0.290530E þ 01.

Application S9 Propan combustion in air – reduced variant (Meintjes & Morgan, 1990, pp. 143–
151; Averick, Carter, Moré, & Xue, 1992, pp. 18–19; Andrei, 2013e, pp. 54–56; Floudas, Pardalos,
Adjiman, Esposito, Gümüs, Harding, Klepeis, Meyer, & Schweiger, 1999, p. 327) [PROPAN]

f ðxÞ ¼ ðx1 x2 þ x1  3x5 Þ2

 2
þ 2x1 x2 þ x1 þ 2R10 x22 þ x2 x23 þ R7 x2 x3 þ R9 x2 x4 þ R8 x2  Rx5
 2
þ 2x2 x23 þ R7 x2 x3 þ 2R5 x23 þ R6 x3  8x5
 2
þ R9 x2 x4 þ 2x24  4Rx5
 2
þ x1 x2 þ x1 þ R10 x22 þ x2 x23 þ R7 x2 x3 þ R9 x2 x4 þ R8 x2 þ R5 x23 þ R6 x3 þ x24  1

where:
724 Appendix B: The SMUNO Collection

R5 ¼ 0:193 R6 ¼ 0:4106217541E  3 R7 ¼ 0:5451766686E  3

R8 ¼ 0:44975E  6 R9 ¼ 0:3407354178E  4 R10 ¼ 0:9615E  6
R ¼ 10

Initial point: x0 ¼ [10, 10, 0.05, 50.5, 0.05]. f(x0) ¼ 0.331226E þ 08.

Application S10 Gear train of minimum inertia (Sandgren & Ragsdell, 1980; Schittkowski, 1987,
Problem 328, p. 149) [GEAR-1]
     
f ðxÞ ¼ 0:1 12 þ x21 þ 1 þ x22 =x21 þ x21 x22 þ 100 =x41 x42 :

Initial point: x0 ¼ [0.5, 0.5]. f(x0) ¼ 0.256332E þ 04.

Application S11 Human heart dipole (Andrei, 2003, p. 65; Averick, Carter, Moré, & Xue, 1992,
p. 17; Andrei, 2013e, pp. 51–54; Nelson & Hodgkin, 1981, pp. 817–823)) [HHD]

f ðxÞ ¼ ðx1 þ x2  smx Þ2

 2
þ x3 þ x4  smy
þðx1 x5 þ x2 x6  x3 x7  x4 x8  sA Þ2
þðx1 x7 þ x2 x8 þ x3 x5 þ x4 x6  sB Þ2
     2
þ x1 x25  x27  2x3 x5 x7 þ x2 x26  x28  2x4 x6 x8  sC
     2
þ x3 x25  x27 þ 2x1 x5 x7 þ x4 x26  x28 þ 2x2 x6 x8  sD
         2
þ x1 x5 x25  3x27 þ x3 x7 x27  3x25 þ x2 x6 x26  3x28 þ x4 x8 x28  3x26  sE
  2        2
þ x3 x5 x5  3x27  x1 x7 x27  3x25 þ x4 x6 x26  3x28  x2 x8 x28  3x26  sF

where:

smx ¼ 0:485 sA ¼ 0:0581 sC ¼ 0:105 sE ¼ 0:167

:
smy ¼ 0:0019 sB ¼ 0:015 sD ¼ 0:0406 sF ¼ 0:399

Initial point:
x0 ¼ ½ 0:299 0:186 0:0273 0:0254 0:474 0:474 0:0892 0:0892 T :
f ðx0 Þ ¼ 0:190569E þ 00:

Application S12 Neurophysiology (Andrei, 2013e, pp. 57–61; Verschelde, Verlinden, & Cools,
1994, pp. 915–930) [NEURO]
 2  2
f ðxÞ ¼ x21 þ x23  1 þ x22 þ x24  1
 3  2  2
þ x5 x3 þ x6 x34  1 þ x5 x31 þ x6 x32  2
 2  2
þ x5 x1 x23 þ x6 x2 x24  1 þ x5 x3 x21 þ x6 x4 x22  4 :

Initial point: x0 ¼ [0.01, . . ., 0.001]. f(x0) ¼ 0.239991E þ 02.

Appendix B: The SMUNO Collection 725

Application S13 Combustion application (Morgan, 1987; Andrei, 2013e, pp. 61–63) [COMBUST]
 2
f ðxÞ ¼ x2 þ 2x6 þ x9 þ 2x10  105
 2
þ x3 þ x8  3
105
 2
þ x1 þ x3 þ 2x5 þ 2x8 þ x9 þ x10  5
105
 2
þ x4 þ 2x7  105
 2
þ 0:5140437
107 x5  x21
 2
þ 0:1006932
106 x6  2x22
 2
þ 0:7816278
1015 x7  x24
 2
þ 0:1496236
106 x8  x1 x3
 2
þ 0:6194411
107 x9  x1 x2
 2
þ 0:2089296
1014 x10  x1 x22 :

Initial point: x0 ¼ [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]. f(x0) ¼ 0.017433088.

Application S14 Circuit design (Ratschek & Rokne, 1993, p. 501; Andrei, 2009e, pp. 243–244],
[CIRCUIT]

X
4  
f ðxÞ ¼ ðx1 x3  x2 x4 Þ2 þ a2k þ b2k ,
k¼1

where
   
ak ¼ ð1  x1 x2 Þx3 exp x5 g1k  g3k x7
103  g5k x8
103  1 þ g4k x2  g5k , k ¼ 1, . . . , 4,
   
bk ¼ ð1  x1 x2 Þx4 exp x6 g1k  g2k  g3k x7
103  g4k x9
103  1 þ g4k  g5k x1 ,
2 3
k ¼ 1, . . . , 4, 0:4850 0:7520 0:8690 0:9820
6 7
6 0:3690 1:2540 0:7030 1:4550 7
6 7
g¼66 5:2095 10:0677 22:9274 20:2153 7 7,
6 7
4 23:3037 101:7790 111:4610 191:2670 5
28:5132 111:8467 134:3884 211:4823

Initial point: x0 ¼ [0.7, 0.5, 0.9, 1.9, 8.1, 8.1, 5.9, 1, 1.9]. f(x0) ¼ 0.296457E þ 04.

Application S15 Thermistor (Andrei, 2009e, pp. 722–723]) [THERM]

X16   2
x2
f ðx Þ ¼ yi  x1 exp
i¼1
45 þ 5i þ x3

where
726 Appendix B: The SMUNO Collection

i yi i yi
1 34780 9 8261
2 28610 10 7030
3 23650 11 6005
4 19630 12 5147
5 16370 13 4427
6 13720 14 3820
7 11540 15 3307
8 9744 16 2872

Initial point: x0 ¼ [0.01, 6100, 340]. f(x0) ¼ 0.233591E þ 10.

Application S16 Optimal design of a gear train (Sandgren, 1988, pp. 95–105; Andrei, 2013e, p. 79)
[GEAR-2]
 2
1 x1 x2
f ðx Þ ¼  :
6:931 x3 x4

Initial point: x0 ¼ [15, 14, 35, 35]. f(x0) ¼ 0.737081E  03.

Notes and References

This appendix includes 16 unconstrained optimization applications taken from literature.
 Appendix C: The LACOP Collection

Large-Scale Continuous Nonlinear Optimization Applications

Application L1. Chemical equilibrium (ELCH)

This application is described in (Hock & Schittkowski, 1981, pp. 121; Andrei, 1999a, pp. 804;
Andrei, 2015a, pp. 972).

X
10  
xj
min xj cj þ ln
j¼1
x1 þ ⋯ þ x10
subject to ð1Þ
x1 þ 2x2 þ 2x3 þ x6 þ x10  2 ¼ 0,
x4 þ 2x5 þ x6 þ x7  1 ¼ 0,
x3 þ x7 þ x8 þ 2x9 þ x10  1 ¼ 0:

Solution of the application:

Nr. cj Initial point Lower bound Solution

1 –6.089 0.1 1.e–6 0.0406685
2 –17.164 0.1 1.e–6 0.1477301
3 –34.054 0.1 1.e–6 0.7831533
4 –5.914 0.1 1.e–6 0.001414229
5 –24.721 0.1 1.e–6 0.4852466
6 –14.986 0.1 1.e–6 0.0006931799
7 –24.100 0.1 1.e–6 0.02739941
8 –10.708 0.1 1.e–6 0.01794741
9 –26.662 0.1 1.e–6 0.03731418
10 –22.179 0.1 1.e–6 0.09687152
f ðx0 Þ ¼ 20:96029 f ðx Þ ¼ 47:761090859

Application L2. Optimization of an alkilation process (ALKI)

This application is taken from (Hock & Schittkowski, 1981, pp. 123; Andrei, 1999a, pp. 803; Andrei,
2015a, pp. 974).

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 727
Switzerland AG
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
728 Appendix C: The LACOP Collection

min ð5:04x1 þ 0:035x2 þ 10x3 þ 3:36x5  0:063x4 x7 Þ

subject to ð2Þ
35:82  0:222x10  bx9  0,
133 þ 3x7  ax10  0,
35:82 þ 0:222x10 þ bx9 þ ð1=b  bÞx9  0,
133  3x7 þ ax10 þ ð1=a  aÞx10  0,
1:12x1 þ 0:13167x1 x8  0:00667x1 x28  ax4  0,
57:425 þ 1:098x8  0:038x28 þ 0:325x6  ax7  0,
1:12x1  0:13167x1 x8 þ 0:00667x1 x28 þ ax4 þ ð1=a  aÞx4  0,
57:425  1:098x8 þ 0:038x28  0:325x6 þ ax7 þ ð1=a  aÞx7  0,
1:22x4  x1  x5 ¼ 0,
98000x3 =ðx4 x9 þ 1000x3 Þ  x6 ¼ 0,
ðx2 þ x5 Þ=x1  x8 ¼ 0,

where a ¼ 0.99 and b ¼ 0.9.

Solution of the application:

Lower Upper
Nr. Initial point Solution
bound bound
1 1745 0.00001 1698.094 2000
2 12000 0.00001 15818.51 16000
3 110 0.00001 54.10228 120
4 3048 0.00001 3031.226 5000
5 1974 0.00001 2000 2000
6 89.2 85 90.11548 93
7 92.8 90 95 95
8 8 3 10.49324 12
9 3.6 1.2 1.561636 4
10 145 145 153.5354 162
f ðx0 Þ ¼ 872:3874 f ðx Þ ¼ 1768:80696

Application L3. Optimal design of a reactor as a geometric programming problem (PREC)

This application is presented in (Dembo, 1976; Rijckaert, 1973; Andrei, 1999a, pp. 801; Hock &
Schittkowski 1981, pp. 113).
Appendix C: The LACOP Collection 729

 0:67 0:67

min 0:4x0:67
1 x7 þ 0:4x0:67
2 x8  x1  x2 þ 10
subject to ð3Þ
1  0:0588x5 x7  0:1x1  0,
1  0:0588x6 x8  0:1x1  0:1x2  0,
1  4x3 x1 0:71 1
5  2x3 x5  0:0588x1:3
3 x7  0,
1  4x4 x1 0:71 1
6  2x4 x6  0:0588x1:3
4 x8  0,
0:67 0:67
0:4x0:67
1 x7 þ 0:4x0:67
2 x8  x1  x2 þ 10  1,
0:67 0:67
0:4x0:67
1 x7 þ 0:4x0:67
2 x8  x1  x2 þ 10  4:2:

Solution of the application:

Initial Lower Upper

Nr. Solution
point bound bound
1 6 0.1 6.465115 10
2 3 0.1 2.232708 10
3 0.4 0.1 0.6673975 10
4 0.2 0.1 0.5957564 10
5 6 0.1 5.932676 10
6 6 0.1 5.527235 10
7 1 0.1 1.013322 10
8 0.5 0.1 0.4006682 10
f ðx0 Þ ¼ 3:657366 f ðx Þ ¼ 3:951163508

Application L4. Cost minimization of a transformer design (TRAFO)

The objective function of this application represents the worth of the transformer, including the
operating cost. The constraints refer to the rating of the transformer and to the allowable transmission
loss. The variables x1, x2, x3 and x4 are physical dimensions of winding and core. The variables x5 and
x6 are the magnetic flux density and the current density, respectively (Price, 1983; Ballard, Jelinek, &
Schinzinger, 1974). The mathematical model is described in (Hock & Schittkowski, 1981) and
(Andrei, 2003, Application A1, pp. 344).

min 0:0204x1 x4 ðx1 þ x2 þ x3 Þ þ 0:0187x2 x3 ðx1 þ 1:57x2 þ x4 Þþ

0:0607x1 x4 x25 ðx1 þ x2 þ x3 Þ þ 0:0437x2 x3 x26 ðx1 þ 1:57x2 þ x4 Þ
subject to ð4Þ
0:001x1 x2 x3 x4 x5 x6  2:07  0,
1  0:00062x1 x4 x25 ðx1 þ x2 þ x3 Þ  0:00058x2 x3 x26 ðx1 þ 1:57x2 þ x4 Þ  0,

where the variables are bounded as: xi  0, i ¼ 1, . . ., 6.

The solution of this application is
x1 ¼ 5:3326663, x2 ¼ 4:6567441, x3 ¼ 10:4329919,
x4 ¼ 12:0823063, x5 ¼ 0:7526074, x6 ¼ 0:8786509:
730 Appendix C: The LACOP Collection

Application L5. Optimization of a multi-spindle automatic lathe (LATHE)

The optimization of a multi-spindle automatic lathe is to minimize a nonlinear objective subject to
15 nonlinear constraints with 10 variables. It is taken from (Schittkowski, 1987, pp. 195) and has the
following mathematical expression.
 
0:15x1 þ 14x2  0:06
min 20000
0:002 þ x1 þ 60x2
subject to ð5Þ
0:75
x1   0,
x3 x4
x
x1  9  0,
x4 x5
x 10
x1  10   0,
x4 x6 x4
0:19 10
x1    0,
x4 x7 x4
0:125
x1   0,
x4 x8
10000x2  0:00131x9 x0:666
5 4  0,
x1:5
10000x2  0:001038x10 x1:6
6 x4  0,
3

10000x2  0:000223x0:666
7 4  0,
x1:5
10000x2  0:000076x3:55
8 x4
5:66
 0,
10000x2  0:000698x1:2
3 x4  0,
2

10000x2  0:00005x1:6
3 x4  0,
3

10000x2  0:00000654x2:42
3 x4
4:17
 0,
10000x2  0:000257x0:666
3 4  0,
x1:5
30  2:003x4 x5  1:885x4 x6  0:184x4 x8  2x4 x0:803
3  0,
x9 þ x10  0:255 ¼ 0,

where the variables are bounded as

0  x1  10, 0  x2  0:1, 0:5e  4  x3  0:0081, 10  x4  1000,

0:5e  4  x5  0:0017, 0:5e  4  x6  0:0013, 0:5e  4  x7  0:0027,
0:5e  4  x8  0:002, 0:5e  4  x9  1, 0:5e  4  x10  1:

The solution of the problem is

x1 ¼ 0:1472722, x2 ¼ 0:1, x3 ¼ 0:0081, x4 ¼ 628:7173075, x5 ¼ 0:0017,
x6 ¼ 0:0011816, x7 ¼ 0:0027, x8 ¼ 0:00135, x9 ¼ 0:1574074,
x10 ¼ 0:0975926:
Appendix C: The LACOP Collection 731

Application L6. Static power scheduling (PPSE)

This application involves two electrical generators connected into a net with three nodes. The
variables x1 and x2 are the real power outputs from the generators, x3 and x4 represent the reactive
power outputs, and x5, x6, and x7 are the voltage magnitudes at the nodes of the electrical network.
Finally, x8 and x9 are the voltage phase angles at two of the nodes. The constraints of the model, other
than the simple bounds on variables, are the real and the reactive power balance equations, the
constraint stating that the power flowing into a node must balance the power flowing out. The
mathematical model, described in (Bartholomew-Biggs, 1976; Hock & Schittkowski, 1981,
pp. 116; Andrei, 2003, pp. 347; Andrei, 2013e), is as follows:
 
min 3000x1 þ 1000x31 þ 2000x2 þ 666:667x32
subject to ð6Þ
0:4  x1 þ 2Cx25 þ x5 x6 ðD sin ðx8 Þ  C cos ðx8 ÞÞþ
x5 x7 ðD sin ðx9 Þ  C cos ðx9 ÞÞ ¼ 0,
0:4  x2 þ 2Cx26 þ x5 x6 ðD sin ðx8 Þ  C cos ðx8 ÞÞþ
x6 x7 ðD sin ðx8  x9 Þ  C cos ðx8  x9 ÞÞ ¼ 0,
0:8 þ 2Cx27 þ x5 x7 ðD sin ðx9 Þ  C cos ðx9 ÞÞþ
x6 x7 ðD sin ðx9  x8 Þ  C cos ðx9  x8 ÞÞ ¼ 0,
0:2  x3 þ 2Dx25 þ x5 x6 ðC sin ðx8 Þ þ D cos ðx8 ÞÞ
x5 x7 ðC sin ðx9 Þ þ D cos ðx9 ÞÞ ¼ 0,
0:2  x4 þ 2Dx26  x5 x6 ðC sin ðx8 Þ þ D cos ðx8 ÞÞ
x6 x7 ðC sin ðx8  x9 Þ þ D cos ðx8  x9 ÞÞ ¼ 0,
0:337 þ 2Dx27  x5 x7 ðC sin ðx9 Þ þ D cos ðx9 ÞÞ
x6 x7 ðC sin ðx9  x8 Þ þ D cos ðx9  x8 ÞÞ ¼ 0,

where C ¼ sin (0.25)48.4/50.176 and D ¼ cos (0.25)48.4/50.176. The simple bounds on variables
are xi  0, i ¼ 1, 2 and 0.90909  xi  1, 0909, for i ¼ 5, 6, 7.
The solution is

x1 ¼ 0:6670128, x2 ¼ 1:0223847, x3 ¼ 0:2282871, x4 ¼ 0:1848218,

x5 ¼ 1:0909, x6 ¼ 1:0909, x7 ¼ 1:069036, x8 ¼ 0:1066106,
x9 ¼ 0:3387876:

Application L7. Optimization of a separation process in a membrane with three stages (MSP3)
This application is taken from (Dembo, 1976; Hock & Schittkowski, 1981, pp. 124; Andrei, 1999a,
pp. 802).
732 Appendix C: The LACOP Collection

min ðx11 þ x12 þ x13 Þ

subject to ð7Þ
x3  x2  0,
x2  x1  0,
1  0:002x7 þ 0:002x8  0,
x13  1:262626x10 þ 1:231059x3 x10  0,
x5  0:03475x2  0:975x2 x5 þ 0:00975x22  0,
x6  0:03475x3  0:975x3 x6 þ 0:00975x23  0,
x5 x7  x1 x8  x4 x7 þ x4 x8  0,
1  0:002ðx2 x9 þ x5 x8  x1 x8  x6 x9 Þ  x5  x6  0,
x2 x9  x3 x10  x6 x9  500x2 þ 500x6 þ x2 x10  0,
x2  0:9  0:002ðx2 x10  x3 x10 Þ  0,
x4  0:03475x1  0:975x1 x4 þ 0:00975x21  0,
x11  1:262626x8 þ 1:231059x1 x8  0,
x12  1:262626x9 þ 1:231059x2 x9  0,
x11 þ x12 þ x13  50,
x11 þ x12 þ x13  250:

The solution of this application is as follows:

Initial Lower Upper

Nr. Solution
point bound bound
1 0.5 0.1 0.8037730 1
2 0.8 0.1 0.8999858 1
3 0.9 0.1 0.9709695 1
4 0.099 0.0001 0.09999994 0.1
5 0.899 0.1 0.1908132 0.9
6 0.5 0.1 0.4605417 0.9
7 489 0.1 574.0773 1000
8 80 0.1 74.0776 1000
9 650 500 500.0162 1000
10 450 0.1 0.1 500
11 149.9 1 20.23311 150
12 149.9 0.0001 77.34769 150
13 149.9 0.0001 0.006730541 150
f ðx0 Þ ¼ 449:700 f ðx Þ ¼ 97:587532422

Application L8. Optimization of a separation process in a membrane with five stages (MSP5)
This application is taken from (Dembo, 1976; Andrei, 2003, pp. 367).
Appendix C: The LACOP Collection 733

min f1:262626ðx12 þ x13 þ x14 þ x15 þ x16 Þ

 
1:231060 x1 x12 þ x2 x13 þ x3 x14 þ x4 x15 þ x5 x16 g
subject to ð8Þ
x6  0:03475x1  0:975x1 x6 þ 0:00975x21  0,
x7  0:03475x2  0:975x2 x7 þ 0:00975x22  0,
x8  0:03475x3  0:975x3 x8 þ 0:00975x23  0,
x9  0:03475x4  0:975x4 x9 þ 0:00975x24  0,
x10  0:03475x5  0:975x5 x10 þ 0:00975x25  0,
x7 x11  x6 x11  x1 x12 þ x6 x12  0,
x8  x7  0:002ðx7 x12 þ x2 x13  x8 x13  x1 x12 Þ  0,
1  x8  x9  0:002ðx8 x13 þ x3 x14  x2 x13  x9 x14 Þ  0,
x3 x14  x9 x14  x4 x15  500x10 þ 500x9 þ x8 x15  0,
x4 x15  x5 x16  x10 x15  500x4 þ 500x10 þ x4 x16  0,
x4 x15  x5 x16  x10 x15  500x4 þ 500x10 þ x4 x16  0,
x4  0:002x4 x16 þ 0:002x5 x16  0:9  0,
1  0:002x11 þ 0:002x12  0,
x11  x12  0,
x5  x4  0,
x4  x3  0,
x3  x2  0,
x2  x1  0,
x10  x9  0,
x9  x8  0,
1:262626ðx12 þ x13 þ x14 þ x15 þ x16 Þ
1:231060ðx1 x12 þ x2 x13 þ x3 x14 þ x4 x15 þ x5 x16 Þ  50  0,
1:262626ðx12 þ x13 þ x14 þ x15 þ x16 Þ
þ1:231060ðx1 x12 þ x2 x13 þ x3 x14 þ x4 x15 þ x5 x16 Þ þ 250  0,

Solution of the application is as follows:

Initial Lower Upper
Nr. Solution
point bound bound
1 0.8 0.1 0.8037651 0.9
2 0.83 0.1 0.8161088 0.9
3 0.85 0.1 0.9 0.9
4 0.87 0.1 0.9 0.9
5 0.91 0.9 0.9 1
6 0.09 0.0001 0.0999996 0.1
7 0.12 0.1 0.1070319 0.9
8 0.19 0.1 0.1908370 0.9
9 0.25 0.1 0.1908370 0.9
10 0.29 0.1 0.1908370 0.9
11 512 1 505.0219 1000
12 13.1 0.000001 5.046301 500
13 71.8 1 72.63798 500
14 640 500 500.0 1000
15 650 500 500.0 1000
16 5.7 0.000001 0.1042061e–5 500
734 Appendix C: The LACOP Collection

f ðx0 Þ ¼ 284:6696 f ðx Þ ¼ 174:787136606

Application L9. Blending/pooling with five feeds and two products (POOL) (Andrei, 1999a,
pp. 808; Andrei, 2003, pp. 382)
min ð1:1x1 þ 1:1x2 þ 1:1x3 þ 0:878x4 þ 0:878x5 þ 0:878x6 þ 0:878x7
þ1:6x8 þ 1:6x9 þ 0:5x10 þ 0:5x11 þ 0:98x12 þ 0:998x13 Þ
subject to ð9Þ
90x1 þ 89x2 þ 91x3  ðx1 þ x2 þ x3 Þx14 ¼ 0,
87x4 þ 90:5x5 þ 89:5x6 þ 94x7  ðx4 þ x5 þ x6 þ x7 Þx15 ¼ 0,
89x8 þ 89:1x9  ðx8 þ x9 Þx16 ¼ 0,
89:2x10 þ 89:3x11 þ x14 x29 þ x15 x30  ðx10 þ x11 þ x29 þ x30 Þx17 ¼ 0,
89:4x12 þ 89:5x13 þ x15 x31 þ x16 x32  ðx12 þ x13 þ x31 þ x32 Þx18 ¼ 0,
86x1 þ 85:5x2 þ 86:5x3  ðx1 þ x2 þ x3 Þx19 ¼ 0,
83:2x4 þ 86:9x5 þ 85x6 þ 89:8x7  ðx4 þ x5 þ x6 þ x7 Þx20 ¼ 0,
85:1x8 þ 85:3x9  ðx8 þ x9 Þx21 ¼ 0,
85:4x10 þ 85:2x11 þ x19 x29 þ x20 x30  ðx10 þ x11 þ x29 þ x30 Þx22 ¼ 0,
85:5x12 þ 85:6x13 þ x20 x31 þ x21 x32  ðx12 þ x13 þ x31 þ x32 Þx23 ¼ 0,
0:78x1 þ 0:8x2 þ 0:81x3  ðx1 þ x2 þ x3 Þx24 ¼ 0,
0:77x4 þ 0:775x5 þ 0:78x6 þ 0:82x7  ðx4 þ x5 þ x6 þ x7 Þx25 ¼ 0,
0:785x8 þ 0:79x9  ðx8 þ x9 Þx26 ¼ 0,
0:787x10 þ 0:776x11 þ x24 x29 þ x25 x30  ðx10 þ x11 þ x29 þ x30 Þx27 ¼ 0,
0:783x12 þ 0:779x13 þ x25 x31 þ x26 x32  ðx12 þ x13 þ x31 þ x32 Þx28 ¼ 0,
x1 þ x2 þ x3  x29 ¼ 0,
x4 þ x5 þ x6 þ x7  x30  x31 ¼ 0,
x8 þ x9  x32 ¼ 0,
x10 þ x11 þ x29 þ x30  x33 ¼ 0,
x12 þ x13 þ x31 þ x32  x34 ¼ 0:
Appendix C: The LACOP Collection 735

Solution of the application:

Nr. Initial point Lower bound Solution Upper bound

1 182 0 131.9361 11000
2 140 0 132.5016 11000
3 70 0 135.5624 11000
4 1070 0 264.3126 11000
5 47 0 273.9741 11000
6 11 0 270.9078 11000
7 12 0 290.8055 11000
8 700 0 398.3295 11000
9 60 0 401.6705 11000
10 190 0 100.0567 11000
11 0.1 0 99.94331 11000
12 0.1 0 0 11000
13 0.1 0 0 11000
14 88 0 90.00765 11000
15 88 0 90.33802 11000
16 88 0 89.05021 11000
17 88 0 89.98826 11000
18 88 0 89.65119 11000
19 88 0 86.00383 11000
20 88 0 86.30969 11000
21 88 0 85.20042 11000
22 88 0 85.98542 11000
23 88 0 85.71808 11000
24 0.77 0 0.7967923 11000
25 0.77 0 0.7869266 11000
26 0.77 0 0.7875104 11000
27 0.77 0 0.7897882 11000
28 0.77 0 0.7872380 11000
29 401 400 400 11000
30 402 400 400 11000
31 702 700 700 11000
32 810 800 800 11000
33 1001 1000 1000 11000
34 1510 1500 1500 11000
f ðx0 Þ ¼ 2743:368 f ðx Þ ¼ 2785:80

Application L10. Distribution of electrons on a sphere (DES)

The problem is described in (Dolan, Moré, & Munson, 2004, pp. 5; see also Andrei, 2001; Andrei,
2003, pp. 348). Given np electrons, find the equilibrium state distribution (of minimal Coulomb
potential) of the electrons positioned on a conducting sphere. The problem, known as the Thomson
problem (raised in 1904), consists in finding the lowest energy configuration of np point charges on a
conducting sphere. This is an important problem in physics and chemistry, which determines a
minimal structure with respect to atomic positions.
736 Appendix C: The LACOP Collection

If (xi, yi, zi) are the positions of the np points (electrons), then the potential energy is
p 1
nX np 
X  2   1
 2  2 2
xi  xj þ y i  y j þ zi  zj , ð10aÞ
i¼1 j¼iþ1

which must be minimized subject to the np constraints

x2i þ y2i þ z2i ¼ 1, i ¼ 1, . . . , np : ð10bÞ

The problem has a multitude of local minimizers at which the objective value is relatively close to
the objective value at the global minimum. Also, the number of the local minimizers grows
exponentially with np. Therefore, determining the global minimum is a difficult task, and solvers
are usually expected to find only a local minimum.
For np ¼ 50 the value of the potential energy is 1055.1823147.

Application L11. Hanging chain (HANG)

Find the chain of uniform density of length L suspended between two points with minimal potential
energy. This is a classical problem, known as the dog problem (Cesari, 1983). It was suggested by
Mittelmann, and it is described in (Dolan, Moré, & Munson, 2004, pp. 9). The problem is to
determine a function x(t), the shape of the chain that minimizes the potential energy

ð1  1=2
xðtÞ 1 þ x0 ðtÞ2 dt,
0

subject to the constraints on the length of the chain

ð1  1=2
1 þ x0 ðtÞ2 dt ¼ L,
0

as well as the end conditions x(0) ¼ a and x(1) ¼ b, where a and b are given constants.
Another formulation of this problem is possible by introducing a control variable uðtÞ ¼ x01 ðtÞ and
the potential energy function
ðt 1=2
x2 ðtÞ ¼ x1 ðsÞ 1 þ uðsÞ2 ds:
0

This formulation leads to minimizing the total potential energy x2(1) subject to the differential
equations

x01 ðtÞ ¼ u,
 1=2
x02 ðtÞ ¼ x1 ðtÞ 1 þ uðtÞ2 ,
 1=2
x03 ðtÞ ¼ 1 þ uðtÞ2 :

By introducing n discretization points by means of a uniform time step of length h ¼ 1/(n þ 1),
then the discrete variant of the problem is
Appendix C: The LACOP Collection 737

Fig. L1 Hanging chain of minimal potential energy of length L ¼ 4 or L ¼ 6

nþ1 
X rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
 
xi þ xi1 x  xi1 2
min h 1þ i ð11aÞ
i¼1
2 h

subject to
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
 
X
nþ1
x  xi1 2 L ð11bÞ
1þ i ¼ ,
i¼1
h h

where x0 ¼ a and xnþ1 ¼ b (Bondarenko, Bortz, & Moré, 1999).

Figure L1a shows the form of the chain for a ¼ 1, b ¼ 3, L ¼ 4, and nh ¼ 400 while Fig. L1b shows
the form of the chain for a ¼ 1, b ¼ 3, L ¼ 6, and nh ¼ 400

Application L12. Determine the optimal mixing policy of two catalysts along the length of a
tubular plug flow reactor involving several reactions (CAT)
This application is described in (von Stryk, 1999) and (Dolan, Moré, & Munson, 2004, pp. 33) and
has the following form:

x01 ðtÞ ¼ uðtÞð10x2 ðtÞ  x1 ðtÞÞ, ð12aÞ

x02 ðtÞ ¼ uðtÞðx1 ðtÞ  10x2 ðtÞÞ  ð1  uðtÞÞx2 ðtÞ: ð12bÞ

The initial conditions are x1(0) ¼ 1 and x2(t) ¼ 0. The control variable u(t) represents the mixing
ratio of the catalysts and satisfies the bounds
0  uðtÞ  1: ð13Þ

The problem is to minimize

   
1 þ x1 tf þ x2 tf , tf ¼ 1: ð14Þ

A uniform partition of the interval [0, 1] with nh subintervals is considered and the equations
discretized in a natural way. As initial points we take u ¼ 0, x1 ¼ 1, and x2 ¼ 0, evaluated in the grid
points.
738 Appendix C: The LACOP Collection

Fig. L2 Evolution of u(t) given by CONOPT and MINOS

Fig. L3 Evolution of x1(t) and x2(t) given by CONOPT

Figure L2 presents the time evolution of the control variable u(t) given by CONOPT and MINOS,
respectively. We see the bang-bang character of this evolution, which is imposed by the presence of
bounds on the control variable.
Figure L3 presents the evolution of the variables x1(t) and x2(t) given by CONOPT.

Application L13. Optimal control of a continuous stirred-tank chemical reactor (CSTC)

Let us consider a continuous stirred-tank chemical reactor where the flow of a coolant through a coil
inserted in the reactor is to control the first-order, irreversible exothermic reaction taking place in the
reactor (Lapidus & Luus, 1967). The state variables are x1(t) and x2(t), which represent the deviation
from the steady-state temperature and the deviation from the steady-state concentration. u(t) is the
normalized control variable representing the effect of the coolant flow on the chemical reaction. The
state equations are
 
0 25x1 ðtÞ
x1 ðtÞ ¼ 2ðx1 ðtÞ þ 0:25Þ þ ðx2 ðtÞ þ 0:5Þ exp  ðx1 ðtÞ þ 0:25ÞuðtÞ, ð15aÞ
x2 ðt Þ þ 2
 
0 25x1 ðtÞ
x2 ðtÞ ¼ ð0:5  x2 ðtÞÞ  ðx2 ðtÞ þ 0:5Þ exp : ð15bÞ
x2 ðtÞ þ 2
Appendix C: The LACOP Collection 739

The initial conditions are x1(0) ¼ 0.05 and x2(0) ¼ 0. The performance measure to be minimized is

ð
0:78
 
J¼ x21 ðtÞ þ x22 ðtÞ þ Ru2 ðtÞ dt, ð16Þ
0

showing that the desired objective is to maintain the temperature and the concentration close to their
steady-state values, without expending large amounts of control effort. Here R is a weighting factor
that we arbitrarily shall select as 0.2.
The above optimal control problem can be represented as a nonlinear programming problem, thus
avoiding the backward in time solution of the Riccati equation. To solve the problem, we approximate
the state differential equations by differences, while the integral term in the performance measure, by
summation.
Taking a discretization of the time interval [0, tf] in 100 subintervals, Figs. L4 and L5 show the
time evolution of the state variables and of the control, respectively.
If for t 2 [0, 0.78] the control variable is bounded as 0.2  u(t)  1.2, then the time evolution of the
variables and of the control is as in Figs. L6 and L7, respectively.

Fig. L4 Evolution of
x1(t) and x2(t)

Fig. L5 Evolution of u(t)

740 Appendix C: The LACOP Collection

Fig. L6 Evolution of
x1(t) and x2(t)

Fig. L7 Evolution of u(t)

Application L14. Optimal temperature field in a rectangular area (DIFF)

This application considers the problem of determining the stationary temperature field in a rectangu-
lar area with heterogeneous thermal conductivity and with some source points of heat inside the area,
as well as the fixed temperatures on the boundaries of the solution domain (McKinney & Savitsky,
2003). Therefore, consider an area with the thermal conductivity V[ Wm1K1] heterogeneously and
symmetrically distributed in the area of interest, i.e., there is a symmetric zone at the center of the
solution domain, where V becomes larger than in the rest of the area. Then the heat transportation
equation is

∂ðcρT Þ
¼ ∇:ðV∇ðcρT ÞÞ þ I, ð17Þ
∂t
where c is the specific heat capacity of the substance (cal/kg degree), ρ is the density of the substance
(kg/m3), T is the temperature of the substance (degree), V is the thermal conductivity, I is the point
source of heat (cal/m3s), and t is the time in seconds. On the boundary of the considered area, the
temperature is fixed as
Appendix C: The LACOP Collection 741

T ¼ T 0 ðx, yÞ for ðx, yÞ on the boundary:

In the steady-state case, no characteristics of the problem change in time. Therefore, the transpor-
tation equation becomes
∇:ðV∇ðcρT ÞÞ þ I ¼ 0: ð18Þ

Now, if c and ρ do not vary spatially, the above equation can be written as follows:
I
∇:ðV∇T Þ þ ¼ 0: ð19Þ
cρ

To solve this equation, the differential operators must be approximated by algebraic analogues,
and a conservative scheme of solving the resulting system of algebraic equations must be used. For
this, the rectangular area is subdivided into a number of small rectangles with sides parallel to the
x and y axes. Thus, a number of intersection points known as nodes are obtained. Using this system of
nodes, the algebraic analogues for the differential operators ∇. () and ∇() can be constructed.
If a and B are differential functions, then
∇:ða∇BÞ ¼ a∇:ð∇BÞ þ ∇a:∇B:

Using this formula, we get the following equation for the heat transportation:
I
V∇:ð∇T Þ þ ∇T:∇V þ ¼ 0: ð20Þ
cρ

Using the finite difference in two dimensions, this equation can be discretized as
   
T iþ1,j  2T i,j þ T i1,j T i,jþ1  2T i,j þ T i,j1
V i,j þ V i,j
ðΔxÞ2 ðΔyÞ2
     
V iþ1,j  V i1,j T iþ1,j  T i1,j V i,jþ1  V i,j1 T i,jþ1  T i,j1 I
þ þ þ ¼ 0,
2ðΔxÞ 2ðΔxÞ 2ðΔyÞ 2ðΔyÞ cρ

for i ¼ 1,. . ., nx and j ¼ 1,. . ., ny where nx and ny are the numbers of the discretization points along the
sides of the area, respectively.
Consider the case in which the temperature is zero on the boundary of the area, except for the cells
(I1, J10), (I1, J11), (I10, J1), and (I11, J1), where the temperature is about 100  C. If nx ¼ 20 and
ny ¼ 20, then Fig. L8 presents the solution to the heat transportation problem with two fixed boundary
conditions (Andrei, 2013e).
Let us now consider the situation in which an additional source of heat is placed in the center of the
cell (I5, J5) of value 10,000 (cal/m3s). The distribution of the temperature and the constant level
curves are as in Fig. L9.
Observe that the symmetry of the problem and the boundary conditions determine the symmetry of
the solution.
In the following, let us consider another situation in which, in the center of the cell (I17, J17), an
additional source of value 15,000 is introduced. Figure L10 shows the temperature distribution in the
rectangular area.
In the following, we consider a time-dependent temperature field in a rectangular area with
heterogeneous thermal conductivity and a point source of heat and heat flows through the border of
the solution domain. Let a point source of heat begin to heat the area in the point (I10, J18) of value
10,000 (cal/m3s). The heat from this source is distributed over the solution domain according to the
742 Appendix C: The LACOP Collection

20

18
100 16

80 14

60 12

40 10

20 8

0 6
20
15 20 4
10 15
10 2
5 5
0 2 4 6 8 10 12 14 16 18 20
0
Fig. L8 Solution to the heat transportation problem with two fixed boundary conditions

20

18
120 16
100 14
80
12
60
10
40
8
20
6
0
20 4
15 20
10 15
10 2
5 5 2 4 6 8 10 12 14 16 18 20
0 0
Fig. L9 Solution to the heat transportation problem with two fixed boundary conditions and one heat source on the cell
(I5, J5)

20

18
200 16

150 14

12
100
10
50
8

0 6
20
15 20 4
10 15
10 2
5 5
0 0 2 4 6 8 10 12 14 16 18 20

Fig. L10 Solution to the heat transportation problem with two fixed boundary conditions and two heat sources
Appendix C: The LACOP Collection 743

heat transportation equation (19). The solution of the time-dependent temperature field consists in
calculating the changes in the temperature field inside the solution domain at one time step on the
basis of the previous temperature field. After the current period temperature field has been computed
from the previous time period, it is saved in order to be used in the next time step.
Figure L11 presents the solution of the transient heat transportation problem at a number of some
periods: t1, t5, t10, t15, t20, and t25. A stationary solution to the transient equation is found by using a

Fig. L11 Solution of the transient heat transportation problem at 6 time periods
744 Appendix C: The LACOP Collection

large number of iterations under stationary boundary conditions. After a number of iterations, the
influence of the initial conditions becomes very small and ∂T/∂t ! 0. In Fig. L11 observe that the
field temperature converges to a stationary solution after about 15–20 iterations.

Application L15. Stationary flow of an incompressible fluid in a rectangular area (FLOW/

FLOWO)
The equations that describe the flow of a fluid are the conservation of the linear momentum and the
mass conservation (McKinney & Savitsky, 2003). The conservation of the linear momentum of a
viscous, incompressible fluid in horizontal plane area in the direction x is as follows:

∂V x ∂V x ∂V y 1 ∂P
þ Vx þ Vy ¼ þ μ∇2 V x , ð21Þ
∂t ∂x ∂y ρ ∂x

and in the direction y

∂V y ∂V y ∂V y 1 ∂P
þ Vy þ Vy ¼ þ μ∇2 V y : ð22Þ
∂t ∂x ∂y ρ ∂y

The equation of the mass conservation (continuity equation) for the fluid is

∂V x ∂V y
þ ¼ 0, ð23Þ
∂x ∂y

where Vx is the x component of velocity [m/s], Vy is the y component of velocity [m/s], P is the
pressure [pa], μ is the kinematic viscosity [m2/s], and ρ is the density of the fluid [Kg/m3].
Observe that these equations are valid in the case when the viscosity is constant and the fluid
weight acts perpendicularly to the x  y plane. Therefore, the gravitational force is not present in
these equations.
With these preparatives, let us consider a stationary flow of an incompressible fluid in a rectangu-
lar area with given inflow of fluid on the borders of the solution domain (McKinney & Savitsky,
2003). In this application, the zones of inflow and outflow of water are on two opposite sides. On the
zone of inflow, the boundary conditions are Vx ¼ 0.5 [m/s] and Vy ¼ 0 [m/s]. On the zone of outflow,
the boundary conditions are ∂V ∂x
x
¼ 0 and Vy ¼ 0. On the other sides of the rectangle, the so-called
“non-slip” conditions are taken as Vx ¼ 0 [m/s] and Vy ¼ 0 [m/s].
To compute the flow of water, a finite difference analog of the differential equations will be
considered. The solution domain is divided into small rectangles by means of a grid of parallel lines
and to each intersection point the indices (i, j) are associated. At each point, the fluid velocity and the
pressure are computed. The fluid velocity is not computed in points, but on the lines connecting the
points. On the lines parallel to the x axis, Vx is computed. On the other hand, on the lines parallel to the
y axis, we compute Vy. Therefore, the continuity equation has the following finite difference form:
V x,i,j  V x,i1,j V y,i,j  V y,i,j1
þ ¼ 0: ð24Þ
Δx Δy

The pressure is computed at the node points (i, j) of each line, where the pressure gradient is
computed from the velocity:
Appendix C: The LACOP Collection 745


∂P Piþ1,j  Pi,j
¼ , ð25Þ
∂x i,j Δx

∂P Pi,jþ1  Pi,j
 ¼ : ð26Þ
∂y i,j Δy

The diffusive terms from (21) and (22) are computed as

!
V x,iþ1,j  2V x,i,j þ V x,i1,j V x,i,jþ1  2V x,i,j þ V x,i,j1
μ∇ V x μ
2
þ , ð27Þ
ðΔxÞ2 ðΔyÞ2
!
V y,iþ1,j  2V y,i,j þ V y,i1,j V y,i,jþ1  2V y,i,j þ V y,i,j1
μ∇ V y μ
2
þ : ð28Þ
ðΔxÞ2 ðΔyÞ2

Since there are errors introduced by the algebraic approximations, it is not always possible to
approximate the differentials by finite differences. Analyzing the system of Eqs. (21) and (22), we see
that these equations contain the velocity components connected by the pressure gradient ∇P. If P is
not connected with boundary conditions, then all the approximation errors will leave the solution
domain through the borders. In the Eq. (23) the velocity vector is in strong contact with the boundary
conditions. Therefore, in the calculations of the continuity equation, we expect big errors to appear at
each point. To overcome this difficulty, the continuity equation is written in a slightly different
(relaxed) form as

∂V x ∂V y
þ ¼ δ, ð29Þ
∂x ∂y

where δ is the error in the approximation of the differentials, which must be minimized. The final
system of equations which have to be solved has the following form:

1 ∂P
 þ μ∇2 V x ¼ 0, ð30Þ
ρ ∂x

1 ∂P
 þ μ∇2 V y ¼ 0, ð31Þ
ρ ∂y

∂V x ∂V y
þ ¼ δ: ð32Þ
∂x ∂y

The finite difference form of (30), (31), and (32) is as follows:

!
1 Piþ1,j  Pi,j V x,iþ1,j  2V x,i,j þ V x,i1,j V x,i,jþ1  2V x,i,j þ V x,i,j1
¼μ þ ,
ρ Δx ðΔxÞ2 ðΔyÞ2
!
1 Pi,jþ1  Pi,j V y,iþ1,j  2V y,i,j þ V y,i1,j V y,i,jþ1  2V y,i,j þ V y,i,j1
¼μ þ ,
ρ Δy ðΔxÞ2 ðΔyÞ2
V x,i,j  V x,i1,j V y,i,j  V y,i,j1
þ ¼ δi,j :
Δx Δy

In the following, consider a discretization of the rectangular area in which the x axis is discretized
in 15 intervals and the y axis in 20, where Δx ¼ Δy ¼ 1, Fig. L12 (McKinney, & Savitsky, 2003).
746 Appendix C: The LACOP Collection

Fig. L12 Flow domain and its discretization

Fig. L13 Velocity in the

direction x
0.8

0.6

0.4.

0.2

0
15
10 20
15
5 10
5
0 0

Let the fluid density be equal to 1000 [Kg/m3]. The kinematic viscosity of the fluid is 0.005 [m2/s].
The flow of the fluid is parallel to the x axis. Now, consider that in the interval [u5, u15] on the y axis,
the following boundary conditions are active: Vx ¼ 0.5 [m/s] and Vy ¼ 0 [m/s]. On the opposite side,
the boundary conditions are ∂V ∂x
x
¼ 0 and Vy ¼ 0. On the rest of the intervals of discretization, the
boundary conditions are Vx ¼ 0 [m/s] and Vy ¼ 0 [m/s].
The error in the approximation through finite differences is zero. Besides, notice that the optimal
value of the objective function is also zero. Figure L13 presents the velocity evolution in the
direction x.
Figure L14 shows three cross-sections of velocity Vx in the direction x.
Appendix C: The LACOP Collection 747

Fig. L14 Three cross-

sections of velocity in the
direction x

Fig. L15 Flow domain with an obstacle

In the following, we compute the stationary flow of water in a rectangular area in the presence of
an obstacle (Fig. L15). As above, let us consider a rectangular area. On the top edge of the area an
inflow of water with velocity of 0.5 m/s occurs.
The same flow leaves the area through the bottom edge of the domain. The boundary conditions
are the same as in the previous application, but in the middle of the domain there is a symmetrically
748 Appendix C: The LACOP Collection

Fig. L16 Velocity in the

direction x
1

0.5

−0.5
20
15
20
10 15
5 10
5
0 0

Fig. L17 Three cross-

sections of velocity in the
direction x

located obstacle around which the fluid is compelled to flow as represented in Fig. L15 (McKinney &
Savitsky, 2003). Figure L16 presents the evolution of the velocity Vx in the direction x.
Figure L17 shows three cross-sections of velocity Vx in the direction x.
Observe the asymmetry of the approximations. This happens because the velocities are not
determined at the cell centers, but on the cell faces.
If the inertial terms of the equations are considered, then they can be computed by using the
“marker and cell” method, as described in (Peyret & Taylor, 1985)

∂V x ∂V x V x,iþ1,j  V x,i,j
Vx þ Vy ffi V x,i,j þ
∂x ∂y 2Δx
V y,iþ1,j þ V y,i,j þ V y,i,j1 þ V y,iþ1,j1 V x,i,jþ1  V x,i,j1
,
4 2Δy

∂V y ∂V y V y,i,jþ1  V y,i,j1
Vx þ Vy ffi V y,i,j þ
∂x ∂y 2Δy
V x,i,j þ V x,i,jþ1 þ V x,i1,jþ1 þ V x,i1,j V y,iþ1,j  V y,i1,j
:
4 2Δx
Figure L18 presents three cross-sections of velocity in the direction x, with inertial terms.
It is worth seeing the solution of the flow in a rectangular area with two obstacles (u10, u5) and
(u10, u10). Figure L19 shows the velocity in the direction x.
Figure L20 shows three cross-sections of velocity Vx in the direction x.
Appendix C: The LACOP Collection 749

Fig. L18 Three cross- 0.9

sections of velocity in the vx10
0.8
direction x with inertial
effects 0.7
vx4
0.6 vx1

0.5

0.4

0.3

0.2

0.1

0
0 2 4 6 8 10 12 14 16 18 20

Fig. L19 Velocity in the 1

direction x for two
obstacles 0.8

0.6

0.4

0.2

20

10

15 20
5 10
0

Fig. L20 Three cross-

sections of velocity in the
direction x

Application L16. Fed-batch fermenter for penicillin production (PENICI)

This problem deals with the dynamic optimization of a fed-batch fermenter for the production of
penicillin through anaerobic glucose fermentation. The dynamic optimization of this process with
fixed final time was shown in (Banga, Alonso, & Singh, 1997; Cuthrell & Biegler, 1989; Luus, 1993;
750 Appendix C: The LACOP Collection

Larrosa, 2008). The problem consists in maximizing the total amount of penicillin produced by using
the feed rate of substrate as control variable. The mathematical model of this application is as follows:
    
max y2 tf y4 tf
subject to ð33Þ
dy1 uy1
¼ h1 y 1  ,
dt 500y4
dy2 uy2
¼ h2 y1  0:01y2  ,
dt 500y4
dy3 hy h y 0:029y1 y3  y 
u
¼ 1 1 2 1 þ 1 3 ,
dt 0:47 1:2 0:0001 þ y3 y4 500
dy4 u
¼ ,
dt 500

where
0:11y3 0:0055y3
h1 ¼ , h2 ¼ :
0:006y1 þ y3 0:0001 þ y3 ð1 þ 10y3 Þ

In this model, y1 represents the concentration of the biomass, y2 is the penicillin concentration, y3
is the concentration of the substrate, and y4 is the fermenter volume (in L). The initial conditions are
y1(0) ¼ 1.5, y2(0) ¼ 0.1, y3(0) ¼ 0.1 and y4(0) ¼ 7. The final product is destined to human
consumption. Therefore, the concentrations of the present species are subject to the following path
constraints:

0  y1  40, 0  y3  25, 0  y4  10:

The control variable is bounded as 0  u  50 and the total process time is fixed as tf ¼ 100 h.
For solving this application and for showing some other optimal control examples, let us discuss
some aspects of the numerical solutions of the ordinary differential equations

y0 ¼ f ðt, yÞ, t0  t  tf , yðt0 Þ ¼ y0 : ð34Þ

One of the simplest methods for solving (34) is Euler’s method. For this, the interval [t0, tf] is
divided by the mesh-points tk ¼ t0 þ kh, k ¼ 0, . . ., N, where h ¼ (tf  t0)/N is the step size and N is a
positive integer. Now, consider that for each k we seek a numerical approximation yk to y(tk), the
value of the analytical solution at the mesh point tk. Euler’s method proceeds by integrating the
differential equation (34) between two consecutive points tk and tkþ1:
ð
tkþ1

yðtkþ1 Þ ¼ yðtk Þ þ f ðt, yðtÞÞdt, k ¼ 0, . . . , N  1, ð35Þ

tk

and then, by applying the numerical integration rule,

ð
tkþ1

gðtÞdt hgðtk Þ, ð36Þ

tk

called the rectangle rule, with g(t) ¼ f(t, y(t)),to get

Appendix C: The LACOP Collection 751

yðtkþ1 Þ yðtk Þ þ hf ðtk , yðtk ÞÞ, k ¼ 0, . . . , N  1, yðt0 Þ ¼ y0 :

This simple derivation motivates the definition of Euler’s method as

ykþ1 ¼ yk þ hf ðtk , yk Þ, k ¼ 0, . . . , N  1: ð37Þ

This idea can be very easily generalized by replacing the rectangle rule in the derivation of Euler’s
method with one-parameter family of the integration rule of the form
ð
tkþ1

gðtÞdt h½ð1  θÞgðtk Þ þ θgðtkþ1 Þ,

tk

where θ 2 [0, 1] is a parameter. Now, applying it in (35) with g(t) ¼ f(t, y(t)) we get

yðtkþ1 Þ yðtk Þ þ h½ð1  θÞf ðtk , yðtk ÞÞ þ θf ðtkþ1 , yðtkþ1 ÞÞ, k ¼ 0, . . . , N  1, yðt0 Þ ¼ y0 :

Therefore, this motivates the introduction of the following one-parameter family of methods:
given that y0 is specified by (34), define
  
ykþ1 ¼ yk þ h ð1  θÞf ðtk , yk Þ þ θf tkþ1 , ykþ1 , k ¼ 0, . . . , N  1, ð38Þ

known as the θmethod. Observe that for θ ¼ 0 we recover Euler’s method. Now, for θ ¼ 1 and y0
given by (34), we get
 
ykþ1 ¼ yk þ hf tkþ1 , ykþ1 , k ¼ 0, . . . , N  1, ð39Þ

known as the implicit or the backward Euler method, since, unlike Euler’s method defined by (37)
and (39) requires the solution of an implicit equation in order to determine ykþ1, given yk. In this
context, (37) is called the explicit or the forward Euler method. For θ ¼ 1/2 in (38), we get another
interesting computational scheme: given that y0 is specified by (34), ykþ1are computed as
1   
ykþ1 ¼ yk þ h f ðtk , yk Þ þ f tkþ1 , ykþ1 , k ¼ 0, . . . , N  1, ð40Þ
2
known as the trapezoidal rule method. Like the backward Euler method, the trapezoidal rule is
implicit: in order to compute ykþ1, a nonlinear system of algebraic equations must be solved.
Each of these methods is consistent with the ordinary differential equation (34). That is, if we plug
solutions to the exact equation into the numerical method, we get a small local error. For example, the
forward Euler has consistency of order 1, and the trapezoidal rule has second-order consistency.
While the trapezoidal rule method leads to more accurate approximations, the forward Euler
method is less convenient from the computational viewpoint since it requires the solution of the
implicit equations at each mesh point xkþ1 to get ykþ1. To overcome this difficulty, an attractive
computational scheme lies in using the forward Euler method to compute an initial crude approxi-
mation for y(tkþ1) and then to use this value within the trapezoidal rule for obtaining a more accurate
approximation of y(tkþ1). The resulting numerical method is
1
ykþ1 ¼ yk þ h½f ðtk , yk Þ þ f ðtkþ1 , yk þ hf ðtk , yk ÞÞ, k ¼ 0, . . . , N  1, ð41Þ
2
known as the improved Euler method. This is an explicit one-step computational scheme.
Using the trapezoidal rule (40), Fig. L21 presents the evolution of the control u(t). Observe the
bang-bang character of this evolution.
752 Appendix C: The LACOP Collection

Fig. L21 Evolution of the 30

control u(t) u(t)
25

20

15

10

0
0 20 40 60 80 100 120

Fig. L22 State variables y1(t) and y2(t)

Fig. L23 State variables y3(t) and y4(t)

Figure L22 presents the evolution of the state variables y1(t) (biomass) and y2(t) (penicillin).
Figure L23 presents the time evolution of the state variables y3(t) (substrate concentration) and
y4(t) (volume).
Appendix C: The LACOP Collection 753

12
40

10

39.5
y1(t) 8 y2(t)
biomass penicillin
6

39 4

2 Backward Euler method

Backward Euler method

38.5 0
0 20 40 60 80 100 120 0 20 40 60 80 100 120

1.5 10

9.5 y4(t)
y3(t) fermenter volume
1
substrate concentration 9

8.5

0.5 8

7.5
Backward Euler method Backward Euler method
7
0 0 20 40 60 80 100 120
0 20 40 60 80 100 120

15

10
u(t)

Backward Euler method

0
0 20 40 60 80 100 120

Fig. L24 Time evolution of variables (backward Euler method)

It is interesting to see the solution of this application when the backward Euler method is used.
Figure L24 presents the time evolution of the variables associated to this application.
The trapezoidal rule method is more accurate, better spotting the characteristics of the optimal
control u(t) as well as of the state variables yi(t), i ¼ 1, . . ., 4.
754 Appendix C: The LACOP Collection

Application L17. A standard linear lumped parameter system (CONT)

For a linear dynamic system, it is desired to find the optimal control which will drive the state vector
from its initial state to the origin in minimum time. The mathematical model of this application is as
follows:

X
4
min x2i ðtÞ
i¼1
subject to ð42Þ
dx1 dx2
¼ 0:5x1 þ 5x2 , ¼ 5x1  0:5x2 þ u,
dt dt
dx3 dx4
¼ 0:6x3 þ 10x4 , ¼ 10x3  0:6x4 þ u,
dt dt
1  uðtÞ  1,

with the initial conditions: x(t0) ¼ [10 10 10 10] and the final time tf ¼ 4.2 (Nishida, Liu,
Lapidus, & Hiratsuka, 1976; Irizarry, 2005).
Figures L25 and L26 present the time evolution of the state variables and the control of this
application, respectively.

15 10

x2(t)
x1(t)
10 5

5 0

0 −5

−5 −10

−10 −15
0 100 200 300 400 500 600 0 100 200 300 400 500 600

15 15

x4(t)
10 x3(t) 10

5 5

0 0

−5 −5

−10 −10

−15 −15
0 100 200 300 400 500 600 0 100 200 300 400 500 600

Fig. L25 Time evolution of the state variables

Appendix C: The LACOP Collection 755

Fig. L26 Time evolution 1

of the control u(t) 0.8

0.6

0.4

0.2

0 u(t)

−0.2

−0.4

−0.6

−0.8

−1
0 100 200 300 400 500 600

Fig. L27 Evolution of

y1(t) and y2(t)

Application L18. Van der Pol oscillator (POL)

One of the classical equations of nonlinear dynamics was formulated by van der Pol (1927). It was a
model for an electrical circuit with a triode. A brief description of this circuit is given in Strogatz
(1994). The van de Pol oscillator has the following mathematical expression:
 
max y3 tf
uðtÞ
subject to ð43Þ
dy1  
¼ 1  y22  y2 þ u,
dt
dy2
¼ y1 ,
dt
dy3
¼ y21 þ y22 þ u2 ,
dt
0:3  uðtÞ  1:0,

with the initial conditions y(0) ¼ [0, 1, 0] and the final time tf ¼ 5.
Figures L27, L28, and L29 show the evolution of the state variables and of the control for this
application.
756 Appendix C: The LACOP Collection

Fig. L28 Evolution of

y3(t)

Fig. L29 Control u(t)

More optimization applications are found in (Andrei, 2003, 2011c, 2013e). Observe that the
applications described in this chapter are classified in two classes. The first one is the static nonlinear
optimization, and the second one is the dynamic nonlinear optimization, often called optimal control.
The strategy for solving optimal control problems is to “kill” the dynamics, i.e., to transform the
dynamic nonlinear optimization problem into a static one. In this way, we get a large-scale optimiza-
tion problem for which the Jacobian of the constraints has a block diagonal or a block triangular
structure. Some optimizers (e.g., CONOPT) can take into consideration this structure to solve the
problem in an efficient and robust manner.

Notes and References

The description of all these 18 nonlinear optimization applications (both in algebraic form and in the
GAMS technology) can be seen in (Andrei, 2013e, 2015a, and 2017c).
 Appendix D: The MINPACK-2 Collection

Large-Scale Unconstrained Optimization Applications

This Appendix includes a number of six applications from the MINPACK-2 collection (Averick,
Carter, & Moré, 1991; Averick, Carter, Moré, & Xue, 1992). The mathematical models of these
applications are expressed as unconstrained optimization problems.

Application A1. Elastic-Plastic Torsion

The description and the physical interpretation of the torsion problem are discussed in Glowinski
(1984, pp. 41–55). We follow the presentation of this problem from (Averick, Carter, Moré, & Xue,
1992). The elastic plastic torsion problem arises from the determination of the stress field on an
infinitely long cylindrical bar. The infinite-dimensional version of this problem is of the
following form:

min fqðvÞ : v 2 K g,

where q : K ! ℝ is a quadratic function

ð ð
1 2
qð v Þ ¼ k ∇v ð x Þ k dx  c vðxÞdx
2
D D

for some constant c, and D is a bounded domain with smooth boundary. The convex set K is defined as

K ¼ v 2 H 10 ðDÞ : jvðxÞj  distðx, ∂DÞ, x 2 D ,

where dist(., ∂D) is the distance function to the boundary of D and H 10 ðDÞ is the Hilbert space of all
the functions with compact support in D, such that v and k∇vk2 belong to L2(D).
A finite element approximation to the torsion problem is obtained by triangulating D and then by
replacing the minimization of q over H 10 ðDÞ with the minimization of q over the set of piecewise
linear functions that satisfy the constraints specified by K. The finite element approximation thus
gives rise to a finite-dimensional minimization problem whose variables are the values of the
piecewise linear function at the vertices of the triangulation.
In (Averick, Carter, Moré, & Xue, 1992) a finite element approximation to a minimization
problem with a quadratic q of the general form

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 757
Switzerland AG
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
758 Appendix D: The MINPACK-2 Collection

ð ð
1
qðvÞ ¼ wq ðxÞk∇vðxÞk2 dx  wl ðxÞvðxÞdx ð1Þ
2
D D

is described, where wq : D ! ℝ and wl : D ! ℝ are functions defined on the rectangle D. In the torsion
problem, wq ¼ 1 and wl ¼ c.
Let D ¼ (ξ1, l, ξ1, u)  (ξ2, l, ξ2, u) be a rectangle in ℝ2. The vertices zi, j 2 ℝ2 for a triangulation of
D are obtained by choosing the grid spacing hx and hy and by defining the grid points
 
zi,j ¼ ξ1,l þ ihx , ξ2,l þ jhy , 0  i  nx þ 1, 0  j  ny þ 1,
 
such that znx þ1,ny þ1 ¼ ξ1,u , ξ2,u : The triangulation consists of the triangular elements TL with the
vertices at zi, j, zi þ 1, j, and zi, jþ1 and the triangular elements TU with the vertices zi, j, zi  1, j, and
zi, j  1.
A finite element approximation to the torsion problem is obtained by minimizing q over the space
of piecewise linear functions v with the values vi, jat zi, j. The approximation to the integral
ð
wq ðxÞk∇vðxÞk2 dx
D

over the element TL is the quadratic qLi,j ðvÞ, where

 2  2 
viþ1,j vi,j vi,jþ1 vi,j
qLi,j ðvÞ ¼ μi,j hx þ hy ,
hx hy       
μi,j ¼ wq zi,j þ wq ziþ1,j þ wq zi,jþ1 :
6
Similarly, the approximation over the element TU is the quadratic qU i,j ðvÞ, where
( 2  2 )
v i1,j  v i,j v i,j1  v i,j
i,j ðvÞ ¼ λi,j
qU
hx
þ
hy
,

hx hy       
λi,j ¼ wq zi,j þ wq zi1,j þ wq zi,j1 :
6
Therefore, the finite element approximation to the quadratic (1) leads to a quadratic programming
problem of the following form:
min fqðvÞ : v 2 Ωg, ð2Þ

where qis the quadratic function

 
1X L X  
qðvÞ ¼ qi,j ðvÞ þ qU
i,j ðv Þ  hx hy wl zi,j vi,j : ð3Þ
2

Observe that in this formulation, the quadratic qLi,j is defined only when 0  i  nx and 0  j  ny,
i,j is defined when 1  i  nx þ 1 and 1  j  ny þ 1. Besides, note that for the torsion problem,
while qU
  
wq ¼ 1 and wl ¼ c, and the feasible set Ω, is Ω ¼ v 2 Rnx ny : vi,j   d i,j , where di, j is the value of
dist(., ∂D) at the vertices zi, j.
Appendix D: The MINPACK-2 Collection 759

Fig. A1 Solution of
Application A1. nx ¼ 200,
ny ¼ 200 0.4

0.3

0.2

0.1

0
200
150 200
100 150
100
50 50
0 0

Considering D ¼ (0, 1)  (0, 1), c ¼ 5 and nx ¼ 200, ny ¼ 200, then a minimization problem with
40,000 variables is obtained. The solution of this application without simple bounds is illustrated in
Fig. A1.

Application A2. Pressure Distribution in a Journal Bearing

This problem consists in determining the pressure distribution in a thin film of lubricant between two
circular cylinders. The infinite-dimensional version of this problem is of the following form:

min fqðvÞ : v 2 K g, ð4Þ

ð ð
1
qðvÞ ¼ wq ðxÞk∇vðxÞk2 dx  wl ðxÞvðxÞdx
2
D D

with

wq ðz1 , z2 Þ ¼ ð1 þ ε cos z1 Þ3 , wl ðz1 , z2 Þ ¼ ε sin z1 ,

for some constant ε 2 (0, 1) and D ¼ (0, 2π)  (0, 2b), where b > 0 is again an arbitrary constant. The
convex set K is defined as

K ¼ v 2 H 10 ðDÞ : v 2 D, v  0 :

A finite element approximation to the journal bearing problem is obtained as in the torsion
problem. Thus, a quadratic programming problem of the form (2) where q is the quadratic defined
by (3) is obtained. In the case of the journal bearing problem, wq(ξ1, ξ2) ¼ (1 þ ε cos ξ1)3 and

wl(ξ1, ξ2) ¼ ε sin ξ1.The feasible set Ω is given by Ω ¼ v 2 ℝnx ny : vi,j  0 : Considering b ¼ 10
and ε ¼ 0.1, as well as a discretization nx  ny of the domain D ¼ (0, 2π)  (0, 2b), where nx ¼ 200
and ny ¼ 200, then the solution of this application without simple bounds is represented in Fig. A2.
Numerical results for this problem are given, for example, by Lin and Cryer (1985), Cimatti and
Menchi (1978), and Moré and Toraldo (1991).

Application A3. Optimal Design with Composite Materials

This application requires determining the placement of two elastic materials in the cross-section of a
rod with maximal torsional rigidity. The formulation of this problem is given in (Averick, Carter,
Moré, & Xue, 1992) and follows the presentation from (Goodman, Kohn, & Reyna, 1986).
760 Appendix D: The MINPACK-2 Collection

Fig. A2 Solution of
Application A2. nx ¼ 200,
ny ¼ 200 0.2

0.1

−0.1

−0.2
200
150 200
100 150
100
50 50
0 0

Let D ⊂ ℝ2 be a bounded domain and let w < |D|, where |D| is the area of D. The solution of the
optimal design problem is a subset Ω of D that solves the problem

min Fðv, ΩÞ : v 2 H 10 ðDÞ, jΩj ¼ w , ð5Þ

where
ðn o
1
Fðv, ΩÞ ¼ μðxÞk∇vðxÞk2 þ vðxÞ dx,
2
D

and μ(x) ¼ μ1 for x 2 Ω, and μ(x) ¼ μ2 for x 2 = Ω. The reciprocals of the constants μ1 and μ2 are the
shear moduli of the elastic materials in the rod. It is assumed that μ1 < μ2.
Goodman, Kohn, and Reyna (1986) formulate the optimal design problem in terms of a family of
problems of the form

min f λ ðvÞ : v 2 H10 ðDÞ ,

where f λ : H10 ðDÞ ! ℝ is the functional

ð
f λ ðvÞ ¼ fψ λ ðk∇vðxÞkÞ þ vðxÞg dx
D

and ψ λ : ℝ ! ℝ is a piecewise quadratic function. In this formulation, λ is a Lagrange multiplier

associated with the optimal design problem. The piecewise quadratic ψ λ : ℝ ! ℝ is of the form
8
>
>
1 2
μ2 t , 0  t  t1 ,
>
>
>
<2  
1
ψ λ ðtÞ ¼ μ2 t1 t  t1 , t1  t  t2 ,
>
> 2
>
>  
>1  2
:
 1
μ1 t  t22 þ μ2 t1 t2  t1 , t2  t,
2 2
with the breakpoints t1 and t2 defined by
Appendix D: The MINPACK-2 Collection 761

 1=2  1=2
μ μ
t1 ¼ 2λ 1 and t2 ¼ 2λ 2 :
μ2 μ1

The definition of these breakpoints implies that μ1t2 ¼ μ2t1 and thus ψ is continuously differentia-
ble. The solution of the optimum design problem considered by Averick, Carter, Moré, and Xue
(1992) is minimizing fλ for a fixed value of λ, where μ1 ¼ 1 and μ2 ¼ 2, such that t21 ¼ λ and t22 ¼ 2λ:
A finite element approximation to this problem is obtained by minimizing fλ over the space of the
piecewise linear functions v with the values vij at zij, where zij 2 ℝ2 are the vertices of a triangulation
of D with the grid spacings hx and hy. The values vij are obtained by solving the minimization problem
nX  o
min f Li,j ðvÞ þ f U
i,j ð v Þ þ h h v
x y i,j : v 2 ℝ n
,

where the functions f Li,j and f U

i,j are defined by

hx hy  þ  U hx hy   
f Li,j ðvÞ ¼ ψ λ di,j ðvÞ , f i,j ðvÞ ¼ ψ d ðvÞ ,
2 2 λ i,j
with
( 2  2 )1=2
vi 1,j  vi,j vi,j 1  vi,j
d
i,j ðvÞ ¼ þ :
hx hy

Observe that in this formulation f Li,j is defined only for 0  i  nx and 0  j  ny, while f U
i,j is defined
for 1  i  nx þ 1 and 1  j  ny þ 1.
Figure A3 presents the solution of this application for μ1 ¼ 1 and μ2 ¼ 2, with nx ¼ 200 and
ny ¼ 200.

Application A4. Steady-State Combustion

This application is taken from (Averick, Carter, Moré, & Xue, 1992). The infinite-dimensional
optimization problem is as follows:

min f λ ðvÞ : v 2 H10 ðDÞ , ð6Þ

where f λ : H10 ðDÞ ! ℝ is the functional

Fig. A3 Solution of
Application A3. nx ¼ 200,
ny ¼ 200 0
−0.01
−0.02
−0.03
−0.04
−0.05
−0.06
200
150 200
100 150
100
50 50
0 0
762 Appendix D: The MINPACK-2 Collection

ðn o
1
f λ ðvÞ ¼ k∇vðxÞk2  λ exp ½vðxÞ dx,
2
D

and λ  0 is a known parameter. This problem is the variational formulation of the boundary value
problem
ΔvðxÞ ¼ λ exp ½vðxÞ, x 2 D,
vðxÞ ¼ 0, x 2 ∂D,

where Δ is the Laplacian operator. Aris (1975) and Bebernes and Eberly (1989) discuss this
application in the context of combustion problems.
An interesting property of the variational Bratu problem is that fλ is unbounded below for any λ > 0.
This can be seen by observing that if v is any positive constant function, then fλ(αv) !  1 as
α ! 1 . Another interesting property of the variational Bratu problem is that if λFK > 0 is the Frank-
Kamenetskii parameter, then fλ has a unique minimizer for λ 2 [0, λFK], but no minimizers for λ > λFK.
If D is the unit square, then λFK 6.81, known as the Frank-Kamenetskii parameter.
A finite element approximation to this problem is obtained by minimizing f over the space of the
piecewise linear functions v with the values vij at zij, where zij 2 ℝ2 are the vertices of a triangulation
of D with the grid spacings hx and hy. The values of vij are computed by solving the following
minimization problem:
nX  o
min f Li,j ðvÞ þ f U
i,j ðvÞ : v 2 ℝ
n
,

where
(    )
hx hy viþ1,j  vi,j 2 vi,jþ1  vi,j 2
f Li,j ðvÞ
¼ þ  λμi,j ,
L
4 hx hy
2      
μLi,j ¼ exp vi,j þ exp viþ1,j þ exp vi,jþ1 ,
3
and
(    )
hx hy vi1,j  vi,j 2 vi,j1  vi,j 2
i,j ðvÞ
fU ¼ þ  λμi,j ,
U
4 hx hy
2      
i,j ¼ 3 exp vi,j þ exp vi1,j þ exp vi,j1
μU :

In this formulation f Li,j is defined only for 0  i  nx and 0  j  ny, while f U

i,j is defined when
1  i  nx þ 1 and 1  j  ny þ 1.
Considering λ ¼ 5, Fig. A4 shows the solution of this application, where nx ¼ 200 and ny ¼ 200.

Application A5. Minimal Surfaces with Enneper Boundary Conditions

Determination of the surface with minimal area and with given boundary values in a convex domain
D is an infinite-dimensional optimization problem of the form

min ff ðvÞ : v 2 K g, ð7Þ

where f : K ! ℝ is the functional

Appendix D: The MINPACK-2 Collection 763

Fig. A4 Solution of
Application A4. nx ¼ 200,
ny ¼ 200 0.8

0.6

0.4

0.2

0
200
150 200
100 150
100
50 50
0 0

ð  1=2
f ðv Þ ¼ 1 þ k∇vðxÞk2 dx,
D

and the set K is defined by


K ¼ v 2 H1 ðDÞ : vðxÞ ¼ vD ðxÞ for x 2 ∂D

for some boundary data function vD : ∂D ! ℝ. The boundary function vD uniquely defines the
solution to the minimal surface problem. An interesting minimal surface, given by Enneper, is
obtained by defining vD on D ¼ (1/2, 1/2)  (1/2, 1/2) by

vD ðξ1 , ξ2 Þ ¼ u2  v2 ,

where u and v are the unique solutions to the nonlinear equations

1 1
ξ1 ¼ u þ uv2  u3 , ξ2 ¼ v  u2 v þ v3 :
3 3
A finite element approximation to the minimal surface problem is obtained by minimizing f over
the space of the piecewise linear functions v with the values vi, j at zi, j, where zi, j 2 ℝ2 are the vertices
of a triangulation of D with the grid spacings hx and hy. The values vi, j are obtained by solving the
minimization problem
nX  o
min f Li,j ðvÞ þ f U
i,j ð vÞ : v 2 ℝ n
,

where the functions f Li,j and f U

i,j are defined by

(  2  2 )1=2
hx hy viþ1,j  vi,j vi,jþ1  vi,j
f Li,j ðvÞ ¼ 1þ þ ,
2 hx hy
(  2  2 )1=2
hx hy vi1,j  vi,j vi,j1  vi,j
i,j ðvÞ
fU ¼
2
1þ
hx
þ
hy
:

Note that in this formulation f Li,j is defined only when 0  i  nx and 0  j  ny, while f U
i,j is defined
when 1  i  nx þ 1 and 1  j  ny þ 1. Figure A5 shows the solution of this application, where
nx ¼ 200 and ny ¼ 200.
764 Appendix D: The MINPACK-2 Collection

Fig. A5 Solution of
0.4
Application A5. nx ¼ 200,
ny ¼ 200 0.3

0.2

0.1

−0.1

−0.2

−0.3

−0.4
200
100 150 200
0 0 50 100

Numerical results for this problem are given by Nitsche (1989), Averick, Carter, Moré, and Xue
(1992), Andrei (2009e).

Application A6. Inhomogeneous Superconductors: 1-D Ginzburg-Landau

This problem arises in the solution of the Ginzburg-Landau equations for inhomogeneous
superconductors in the absence of a magnetic field. The formulation of this application is based on
the work of Garner and Benedek (1990) and is presented in (Averick, Carter, Moré, & Xue, 1992).
The optimization problem is to minimize the Gibbs free energy as a function of the temperature.
The infinite-dimensional version of this problem is of the form

min f ðvÞ : vðd Þ ¼ vðdÞ, v 2 C1 ½d, d , ð8Þ

where 2d is the width of the material and f is the Gibbs free energy function. In this problem,

ðd  
1 1 ħ2 0 2
f ðv Þ ¼ αðξÞjvðξÞj2 þ βðξÞjvðξÞj4 þ jv ðξÞj dξ,
2d 2 4m
d

the functions α and β are piecewise constants for a fixed value of the temperature, ħ is Planck’s
constant (1.05459e27 erg-s), and m is the mass of the electron (9.11e28 grams).
The functions α and β are constant in the intervals that correspond to lead and tin. Since in this
problem the lead in the material corresponds to the interval [dS, dS] and the tin in the remaining part
of the interval [d, d], the functions α and β are defined by
8 8
< αN , d  ξ  dS ,
> < βN , d  ξ  dS ,
>
αðξÞ ¼ αS , d S < ξ  d S , β ðξÞ ¼ βS , dS < ξ  dS ,
>
: >
:
αN , dS < ξ  d, βN , dS < ξ  d:

The constants αS and αN are negative, but βS and βN are positive.

A finite element approximation to the superconductivity problem is obtained by minimizing f over
the space of the piecewise linear functions v with the values vi at ti, where d ¼ t1 < t2 < ⋯ < tn < tnþ1 ¼ d.
It is assumed that there are the indices n1 and n2 such that tn1 ¼ dS and tn2 ¼ dS , where
1 < n1 < n2 < n. This guarantees that the ti does not straddle a point of discontinuity of the functions
α and β. The values vi are obtained by solving the following minimization problem:
Appendix D: The MINPACK-2 Collection 765

Fig. A6 Solution of 77.05

Application A6. n ¼ 1000
77

76.95

76.9

76.85

76.8
0 100 200 300 400 500 600 700 800 900 1000

( )
1 X
n
min f ðvÞ : v 2 ℝn ,
2d i¼1 i

where
(  2 )
αi v3iþ1  v3i βi v5iþ1  v5i ħ2 viþ1  vi
f i ð v Þ ¼ hi þ þ ,
3 viþ1  vi 10 viþ1  vi 4m hi

with hi ¼ tiþ1  ti the length of the i-th interval and the constants αi and βi as the values of the
functions α and β in the interval [ti, tiþ1]. The constraint that v(d ) ¼ v(d ) is taken into account by
requiring that vnþ1 ¼ v1.
Considering d ¼ 3.2Å and the temperature T ¼ 5, for n ¼ 1000 the solution of this application is
presented in Fig. A6.

Notes and References

The applications included in this Appendix are taken from the MINPACK-2 collection (Averick,
Carter, & Moré, 1991; Averick, Carter, Moré, & Xue, 1992). They are also described in (Andrei,
2017c, Chapter 4).
 References

Abadie, J.: Un nouvel algorithme pour la programmation non lineaire. RAIRO/Rech. Oper. 12(2), 233–238 (1978)
Abadie, J.: Une modification de la methode GRG. RAIRO/Rech. Oper. 13(3), 323–326 (1979)
Abadie, J., Carpentier, J.: Generalization of the Wolfe reduced gradient method to the case of nonlinear constraints. In:
Fletcher, R. (ed.) Optimization, pp. 37–47. Academic Press, London (1969)
Abadie, J., Guerrero, G.: Methode du GRG, Methode de Newton globale et application a la programmation
mathematique. RAIRO/Rech. Oper. 18(4), 319–351 (1984)
Abadie, J., Guigou, J.: Numerical experiments with GRG method. In: Abadie, J. (ed.) Integer and Nonlinear Program-
ming, pp. 529–536. North-Holland, Amsterdam (1970)
Abadie, J., Haggag, A.: Performance du gradient reduit generalise avec une methode quasi Newtonienne pour la
programmation non lineaire. RAIRO/Rech. Oper. 13(2), 209–216 (1979)
Al-Baali, M.: Descent property and global convergence of the Fletcher-Reeves method with inexact linesearch. IMA
J. Numer. Anal. 5, 121–124 (1985)
Al-Baali, M.: Numerical experience with a class of self-scaling quasi-Newton algorithms. J. Optim. Theory Appl. 96,
533–553 (1998)
Al-Baali, M., Grandinetti, L.: On practical modifications of the quasi-Newton BFGS method. AMO-Adv. Model.
Optim. 11(1), 63–76 (2009)
Al-Baali, M., Narushima, Y., Yabe, H.: A family of three-term conjugate gradient methods with sufficient descent
property for unconstrained optimization. Comput. Optim. Appl. 60, 89–110 (2015)
Andersen, E.D., Andersen, K.D.: The MOSEK interior point optimizer for linear programming: an implementation of
the homogeneous algorithm. In: Frenk, T.T.H., Roos, K., Zhang, S. (eds.) High Performance Optimization,
pp. 197–232. Kluwer Academic Publishers, New York (2000)
An, H.-B., Mo, Z.-Y., Liu, X.-P.: A choice of forcing terms in inexact Newton method. J. Comput. Appl. Math. 200(1),
47–80 (2007)
Andreani, R., Birgin, E., Martı́nez, J., Schuverdt, M.: On augmented Lagrangian methods with general lower-level
constraints. SIAM J. Optim. 18, 1286–1309 (2007)
Andreani, R., Birgin, E., Martı́nez, J., Schuverdt, M.: Augmented Lagrangian methods under the constant positive
linear dependence constraint qualification. Math. Program. 111, 5–32 (2008)
Andrei, N.: A simple algorithm for computing a zero of a nonlinear function of a variable in a given interval [a,b].
(Technical Report, Research Institute for Informatics, Bucharest, Romania, March 28, 1–9) (1975a)
Andrei, N.: A simple algorithm for computing all zeros of a nonlinear function of a variable in a given interval [a,b].
(Technical Report, Research Institute for Informatics, Bucharest, Romania, April 16, 1–21) (1975b)
Andrei, N.: RP: a package for efficient calculation of sparse Jacobian matrix for nonlinear systems of equations using
finite differences. (Technical Report, Research Institute for Informatics, Bucharest, Romania, April 15, 1–31)
(1983)
Andrei, N.: Application of sparse matrix techniques to the GRG algorithm for large-scale non-linear programming. Rev.
Roumaine Sci. Techn. Electrotechn. et Energy. 30(2), 175–186 (1985)
Andrei, N.: Application of sparse matrix techniques in GRG algorithm for very large-scale non-linear programming.
Rev. Roumaine Sci. Techn. Electrotechn. et Energy. 32(4), 457–464 (1987)
Andrei, N.: Computational experience with conjugate gradient algorithms for large-scale unconstrained optimization.
(Technical Report, Research Institute for Informatics-ICI, Bucharest, July 21) (1995)
Andrei, N.: Computational experience with a modified penalty-barrier method for large-scale nonlinear constrained
optimization. (Working Paper No. AMOL-96-1, Research Institute for Informatics-ICI, Bucharest, February 6)
(1996a)

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 767
Switzerland AG
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
768 References

Andrei, N.. Computational experience with a modified penalty-barrier method for large-scale nonlinear, equality and
inequality constrained optimization. (Technical Paper No. AMOL-96-2, Research Institute for Informatics-ICI,
Bucharest, February 12) (1996b)
Andrei, N.: Computational experience with “SPENBAR” a sparse variant of a modified penalty-barrier method for
large-scale nonlinear, equality and inequality, constrained optimization. (Working Paper No. AMOL-96-3,
Research Institute for Informatics-ICI, Bucharest, March 10) (1996c)
Andrei, N.: Penalty-barrier algorithms for nonlinear optimization. Preliminary computational results. Stud. Inform.
Control. 7(1), 15–36 (1998a)
Andrei, N.: Predictor-Corrector interior point methods for linear constrained optimization. Stud. Inform. Control. 7(2),
155–177 (1998b)
Andrei, N.: An interior point algorithm for nonlinear programming. Stud. Inform. Control. 7(4), 365–395 (1998c)
Andrei, N.: Programarea matematică avansată. Teorie, Metode computaţionale, Aplicaţii. [Advanced Mathematical
Programming. Theory, Computational Methods, Applications]. Editura Tehnică, Bucureşti (1999a)
Andrei, N.: Programarea matematică. Metode de punct interior [Mathematical Programming. Interior Point Methods].
Editura Tehnică, Bucureşti (1999b)
Andrei, N.: Optimizare fără Restricţii – Metode de direcţii conjugate [Unconstrained Optimization – Conjugate
Direction Methods]. MATRIXROM Publishing House, Bucharest (2000)
Andrei, N.: Numerical examples solved with SPENBAR – modified penalty barrier method for large-scale nonlinear
programming problems. (Technical Report No. 1/2001, Research Institute for Informatics ICI – Bucharest,
February) (2001)
Andrei, N.: Modele, Probleme de Test şi Aplicaţii de Programare Matematică. [Models, Test Problems and
Applications for Mathematical Programming]. Editura Tehnică, Bucureşti (2003)
Andrei, N.: Convergenţa algoritmilor de optimizare. [Convergence of the Optimization Algorithms] Editura Tehnică,
Bucureşti (2004a)
Andrei, N.: Teorie versus empirism ı̂n analiza algoritmilor de optimizare. [Theory Versus Empiricism in Analysis of the
Optimization Algorithms] Editura Tehnică, Bucureşti (2004b)
Andrei, N. Relaxed gradient descent method with backtracking for unconstrained optimization. În: Works 2005,
Manuscript. Biblioteca Academiei Române, pp. 1–10, March 2, 2005a
Andrei, N.: An acceleration of gradient descent algorithm with backtracking for unconstrained optimization. În: Works
2005, Manuscript. Biblioteca Academiei Române, pp. 1–13, April 20, 2005b
Andrei, N.: An acceleration of gradient descent algorithm with backtracking for unconstrained optimization. Numer.
Algorithms. 42, 63–73 (2006a)
Andrei, N.: Performance of conjugate gradient algorithms on some MINPACK-2 unconstrained optimization
applications. Stud. Inform. Control. 15(2), 145–168 (2006b)
Andrei, N.: Scaled conjugate gradient algorithms for unconstrained optimization. Comput. Optim. Appl. 38(3),
401–416 (2007a)
Andrei, N.: A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization. Appl. Math.
Lett. 20, 645–650 (2007b)
Andrei, N.: Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization.
Optim. Methods Softw. 22(4), 561–571 (2007c)
Andrei, N.: Numerical comparison of conjugate gradient algorithms for unconstrained optimization. Stud. Inform.
Control. 16(4), 333–352 (2007d)
Andrei, N.: A scaled nonlinear conjugate gradient algorithm for unconstrained optimization. Optimization. 57(4),
549–570 (2008a)
Andrei, N.: Another hybrid conjugate gradient algorithm for unconstrained optimization. Numer. Algorithms. 47,
143–156 (2008b)
Andrei, N.: A Dai-Yuan conjugate gradient algorithm with sufficient descent and conjugacy condition for uncon-
strained optimization. Appl. Math. Lett. 21(2), 165–171 (2008c)
Andrei, N.: New hybrid conjugate gradient algorithms for unconstrained optimization. In: Floudas, C.A., Pardalos,
P. (eds.) Encyclopedia of Optimization, 2nd edn, pp. 2560–2571. Springer Science + Business Media, New York
(2008d)
Andrei, N.: Performance profiles of conjugate gradient algorithms for unconstrained optimization. In: Floudas, C.A.,
Pardalos, P. (eds.) Encyclopedia of Optimization, 2nd edn, pp. 2938–2953. Springer Science + Business Media,
New York (2008e)
Andrei, N.: A hybrid conjugate gradient algorithm for unconstrained optimization as a convex combination of
Hestenes-Stiefel and Dai-Yuan. Stud. Inform. Control. 17(1), 55–70 (2008f)
Andrei, N.: 40 conjugate gradient algorithms for unconstrained optimization – a survey on their definition. (Technical
Report, Research Institute for Informatics-ICI, Bucharest, Romania, March 14, 1–13) (2008g)
References 769

Andrei, N.: Noether theorem and fundamentals of mathematical modeling. Revista Română de Informatică şi
Automatică. 18(4), 11–22 (2008h)
Andrei, N.: Hybrid conjugate gradient algorithm for unconstrained optimization. J. Optim. Theory Appl. 141(2),
249–264 (2009a)
Andrei, N.: Another nonlinear conjugate gradient algorithm for unconstrained optimization. Optim. Methods Softw.
24(1), 89–104 (2009b)
Andrei, N.: Acceleration of conjugate gradient algorithms for unconstrained optimization. Appl. Math. Comput. 213(2),
361–369 (2009c)
Andrei, N.: Accelerated conjugate gradient algorithm with finite difference Hessian/vector product approximation for
unconstrained optimization. J. Comput. Appl. Math. 230, 570–582 (2009d)
Andrei, N.: Critica Raţiunii Algoritmilor de Optimizare fără Restricţii. [Criticism of the Unconstrained Optimization
Algorithms Reasoning]. Editura Academiei Române, Bucureşti (2009e)
Andrei, N.: Metode Avansate de Gradient Conjugat pentru Optimizare fără Restricţii. [Advanced Conjugate Gradient
Methods for Unconstrained Optimization]. Editura Academiei Oamenilor de Ştiinţă din România, Bucureşti (2009f)
Andrei, N.: Accelerated conjugate gradient algorithm with modified secant condition for unconstrained optimization.
Stud. Inform. Control. 18(3), 211–232 (2009g)
Andrei, N.: Accelerated hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimi-
zation. Numer. Algorithms. 54, 23–46 (2010a)
Andrei, N.: Accelerated scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained
optimization. Eur. J. Oper. Res. 204, 410–420 (2010b)
Andrei, N.: New accelerated conjugate gradient algorithms as a modification of Dai-Yuan’s computational scheme for
unconstrained optimization. J. Comput. Appl. Math. 234, 3397–3410 (2010c)
Andrei, N.: A modified Polak-Ribiere-Polyak conjugate gradient algorithm for unconstrained optimization. Optimiza-
tion. 60(12), 1457–1471 (2011a)
Andrei, N.: Open problems in conjugate gradient algorithms for unconstrained optimization. Bull. Malaysian Math. Sci.
Soc. 34(2), 319–330 (2011b)
Andrei, N.: CAON: A collection of nonlinear optimization applications in GAMS language. (Technical Report, No.1/
2011, Research Institute for Informatics-ICI, Bucharest, January 31, (105 pages and CD)) (2011c)
Andrei, N.: Critica Raţiunii Algoritmilor de Programare Liniară. [Criticism of the Linear Programming Algorithms
Reasoning]. Editura Academiei Române, Bucureşti (2011d)
Andrei, N.: A accelerated conjugate gradient algorithm with guaranteed descent and conjugacy conditions for
unconstrained optimization. Optim. Methods Softw. 27(4–5), 583–604 (2012)
Andrei, N.: A simple three-term conjugate gradient algorithm for unconstrained optimization. J. Comput. Appl. Math.
241, 19–29 (2013a)
Andrei, N.: On three-term conjugate gradient algorithms for unconstrained optimization. Appl. Math. Comput. 219,
6316–6327 (2013b)
Andrei, N.: Another conjugate gradient algorithm with guaranteed descent and conjugacy conditions for large-scale
unconstrained optimization. J. Optim. Theory Appl. 159, 159–182 (2013c)
Andrei, N.: A numerical study on efficiency and robustness of some conjugate gradient algorithms for large-scale
unconstrained optimization. Stud. Inform. Control. 22(4), 259–284 (2013d)
Andrei, N.: Nonlinear Optimization Applications using the GAMS Technology. Springer Optimization and its
Applications Series, vol. 81. Springer Science + Business Media, New York (2013e)
Andrei, N.: An accelerated subspace minimization three-term conjugate gradient algorithm for unconstrained optimi-
zation. Numer. Algorithms. 65(4), 859–874 (2014)
Andrei, N.: Critica Raţiunii Algoritmilor de Optimizare cu Restricţii. [Criticism of the Constrained Optimization
Algorithms Reasoning]. Editura Academiei Române, Bucureşti (2015a)
Andrei, N.: A new three-term conjugate gradient algorithm for unconstrained optimization. Numer. Algorithms. 68(2),
305–321 (2015b)
Andrei, N.: An adaptive conjugate gradient algorithm for large-scale unconstrained optimization. J. Comput. Appl.
Math. 292, 83–91 (2016)
Andrei, N.: Eigenvalues versus singular values study in conjugate gradient algorithms for large-scale unconstrained
optimization. Optim. Methods Softw. 32(3), 534–551 (2017a)
Andrei, N.: Accelerated adaptive Perry conjugate gradient algorithms based on the self-scaling memoryless BFGS
update. J. Comput. Appl. Math. 325, 149–164 (2017b)
Andrei, N.: Continuous Nonlinear Optimization for Engineering Applications in GAMS Technology. Springer Optimi-
zation and Its Applications Series, vol. 121. Springer Science + Business Media, New York (2017c)
Andrei, N.: An adaptive scaled BFGS method for unconstrained optimization. Numer. Algorithms. 77(2), 413–432
(2018a)
770 References

Andrei, N.: A Dai-Liao conjugate gradient algorithm with clustering the eigenvalues. Numer. Algorithms. 77(4),
1273–1282 (2018b)
Andrei, N.: A double parameter scaled BFGS method for unconstrained optimization. J. Comput. Appl. Math. 332,
26–44 (2018c)
Andrei, N.: A double parameter scaling Broyden-Fletcher-Goldfarb-Shanno based on minimizing the measure function
of Byrd and Nocedal for unconstrained optimization. J. Optim. Theory Appl. 178, 191–218 (2018d)
Andrei, N.: A diagonal quasi-Newton method based on minimizing the measure function of Byrd and Nocedal for
unconstatined optimization. Optimization. 67(9), 1553–1568 (2018e)
Andrei, N.: A double parameter scaled modified Broyden-Fletcher-Goldfarb-Shanno method for unconstrained optimi-
zation. Stud. Inform. Control. 27(2), 135–146 (2018f)
Andrei, N.: A diagonal quasi-Newton updating method for unconstrained optimization. Numer. Algorithms. 81(2),
575–590 (2019a)
Andrei, N.: A new diagonal quasi-Newton updating method with scaled forward finite differences directional derivative
for unconstrained optimization. Numer. Funct. Anal. Optim. 40(13), 1467–1488 (2019b)
Andrei, N.: Conjugate gradient algorithms closest to self-scaling memoryless BFGS method based on clustering the
eigenvalues of the self-scaling memoryless BFGS iteration matrix or on minimizing the Byrd-Nocedal measure
function with different Wolfe line searches for unconstrained optimization. (Technical Report No.2/2019. Academy
of Romanian Scientists, Bucharest, Romania, April 18, 1–119) (2019c)
Andrei, N.: Acceleration by modifying the stepsize versus preconditioning with diagonalized quasi-Newton of self-
scaling memoryless BFGS conjugate gradient methods of Hager-Zhang and of Dai-Kou. (Technical Report No.3/
2019. Academy of Romanian Scientists, Bucharest, Romania, October 14, 1–20) (2019d)
Andrei, N.: Nonlinear Conjugate Gradient Methods for Unconstrained Optimization. Springer Optimization and Its
Applications Series, vol. 158. Springer Science + Business Media, New York (2020a)
Andrei, N.: Diagonal approximation of the Hessian by finite difference for unconstrained optimization. J. Optim.
Theory Appl. 185, 859–879 (2020b)
Andrei, N.: A new accelerated diagonal quasi-Newton updating method with scaled forward finite differences
directional derivative for unconstrained optimization. Optimization. 70(2), 345–360 (2020c)
Andrei, N.: New conjugate gradient algorithms based on self-scaling memoryless Broyden-Fletcher-Goldfarb-Shanno
method. Calcolo. 57-17, 1–17 (2020d)
Andrei, N.: A Derivative-Free Two Level Random Search Method for Unconstrained Optimization. Springer Briefs in
Optimization. Springer, New York (2021a)
Andrei, N.: More on the efficiency and robustness of conjugate gradient methods subject to procedures for stepsize
computation. (Technical Report 4/2021, March 3) (2021b)
Andrei, N.: Accelerated memory-less SR1 with generalized secant equation method for unconstrained optimization.
(Technical Report 8/2021, May 26) (2021c)
Andrei, N.: A note on memory-less SR1 and memory-less BFGS methods for large-scale unconstrained optimization.
Numer. Algorithms. 90(1), 223–240 (2021d)
Andrei, N.: Accelerated memory-less SR1 method with generalized secant equation for unconstrained optimization.
Calcolo. 59-16, 1–20 (2022)
Andrei, N., Bărbulescu, M.: Balance constrained reduction of large-scale linear programming problems. Ann. Oper.
Res. 43, 149–170 (1993)
Antoniou, A., Lu, W.-S.: Practical Optimization. Algorithms and Engineering Applications. Springer Science +
Business Media, LLC, New York (2007)
Aris, R.: The Mathematical Theory of Diffusion and Reaction in Permeable Catalysts. Clarendon Press, Oxford (1975)
ARKI Consulting and Development A/S: CONOPT version 3 (2004)
Armijo, L.: Minimization of functions having Lipschitz continuous first partial derivatives. Pac. J. Math. 16(1), 1–3
(1966)
Arrow, K.J., Sollow, R.M.: Gradient methods for constrained maxima, with weakened assumptions. In: Arrow, K.J.,
Hurwicz, L., Uzawa, H. (eds.) Studies in Linear and Nonlinear Programming, pp. 166–176. Stanford University
Press, Stanford (1958)
Arzam, M.R., Babaie-Kafaki, S., Ghanbari, R.: An extended Dai-Liao conjugate gradient method with global conver-
gence for nonconvex functions. Glasnik Matematicki. 52(72), 361–375 (2017)
Averick, B.M., Carter, R.G., Moré, J.J.: The MINPACK-2 test problem collection (Preliminary Version). (Technical
Memorandum No.150, Mathematics and Computer Science Division, Argonne National Laboratory, 9700 South
Cass Avenue, Argonne, Illinois, Technical Memorandum No. 150, May) (1991)
Averick, B.M., Carter, R.G., Moré, J.J., Xue, G.L.: The MINPACK-2 test problem collection. (Mathematics and
Computer Science Division, Argonne National Laboratory, 9700 South Cass Avenue, Argonne, Illinois, Preprint
MCS-P153-6092, June) (1992)
References 771

Axelsson, O., Lindskog, G.: On the rate of convergence of the preconditioned conjugate gradient method. Numer. Math.
48, 499–523 (1986)
Babaie-Kafaki, S.: A modified BFGS algorithm based on a hybrid secant equation. Sci. China Math. 54(9), 2019–2036
(2011)
Babaie-Kafaki, S.: A note on the global convergence theorem of the scaled conjugate gradient algorithms proposed by
Andrei. Comput. Optim. Appl. 52(2), 409–414 (2012)
Babaie-Kafaki, S.: A modified scaled memoryless BFGS preconditioned conjugate gradient method for unconstrained
optimization. 4OR. 11(4), 361–374 (2013)
Babaie-Kafaki, S.: Two modified scaled nonlinear conjugate gradient methods. J. Comput. Appl. Math. 261(5),
172–182 (2014)
Babaie-Kafaki, S., Ghanbari, R.: A modified scaled conjugate gradient method with global convergence for nonconvex
functions. Bull. Belg. Math. Soc. Simon Stevin. 21(3), 465–477 (2014)
Babaie-Kafaki, S., Ghanbari, R.: A hybridization of the Hestenes–Stiefel and Dai–Yuan conjugate gradient methods
based on a least-squares approach. Optim. Methods Softw. 30(4), 673–681 (2015)
Babaie-Kafaki, S., Rezaee, S.: Two accelerated nonmonotone adaptive trust region line search methods. Numer.
Algorithms. 78, 911–928 (2018)
Ballard, D.H., Jelinek, C.A., Schinzinger, R.: An algorithm for the solution of constrained polynomial programming
problems. Comput. J. 17, 261–266 (1974)
Banga, J.R., Alonso, A.A., Singh, R.P.: Stochastic dynamic optimization of batch and semicontinuous bioprocesses.
Biotech. Progress. 13, 326–335 (1997)
Bartels, R.H.: A penalty linear programming method using reduced-gradient basis-exchange techniques. Linear
Algebra Appl. 29, 17–32 (1980)
Bartlett, R., Biegler, L.T.: rSQO++: An object oriented framework for successive quadratic programming. In: Biegler,
L.T., Ghattas, O., Heinkenschloss, M., van Bloemen Waanders, B. (eds.) Large-Scale PDE-Constrained Optimiza-
tion Lecture Notes in Computational Science and Engineering, vol. 30, pp. 316–330. Springer, New York (2003)
Bartholomew-Biggs, M.C.: A numerical comparison between two approachs to nonlinear programming problems.
(Technical Report, No. 77, Numerical Optimization Centre, Hatfield, England) (1976)
Bartholomew-Biggs, M.: Nonlinear Optimization with Engineering Applications. Springer Science + Business Media,
New York (2008)
Barzilai, J., Borwein, J.M.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)
Bazaraa, M.S., Sherali, H.D., Shetty, C.M.: Nonlinear Programming Theory and Algorithms, 2nd edn. Wiley,
New York (1993)
Beale, E.M.L.: On quadratic programming. Naval Res. Logist. Q. 6, 227–244 (1959)
Beale, E.M.L.: A derivation of conjugate gradients. In: Lotsma, F.A. (ed.) Numerical Methods for Nonlinear Optimi-
zation, pp. 39–43. Academic Press, New York (1972)
Bebernes, J., Eberly, D.: Mathematical Problems from Combustion Theory. Applied Mathematical Sciences, vol. 83.
Springer, New York (1989)
Ben-Tal, A., El Ghaoui, L., Nemirovski, A.: Robust Optimization. Princeton Series in Applied Mathematics. Princeton
University Press, Princeton (2009)
Ben-Tal, A., Yuzefovich, I., Zibulevsky, M.: Penalty/barrier multiplier methods for minimax and constrained smooth
convex problems. (Research Report 9/92, Optimization Laboratory, Faculty of Industrial Engineering and Manage-
ment, Technion, Haifa, Israel) (1992)
Ben-Tal, A., Zibulevsky, M.: Penalty-barrier multipier methods for large-scale convex programming problems.
(Research Report 6/93. Optimization Laboratory, Faculty of Industrial Engineering and Management, Technion,
Haifa, Israel) (1993)
Benson, H.Y., Shanno, D.F.: Cubic regularization in symmetric rank-1 quasi-Newton methods. Math. Program.
Comput. 10, 457–486 (2018)
Benson, H.Y., Shanno, D.F., Vanderbei, R.J.: Interior-point methods for nonconvex nonlinear programming: Filter
methods and merit functions. Comput. Optim. Appl. 23, 257–272 (2002a)
Benson, H.Y., Shanno, D.F., Vanderbei, R.J.: A comparative study of large-scale nonlinear optimization algorithms.
(Technical Report ORFE-01-04, Operations Research and Financial Engineering, Princeton University, July 17)
(2002b)
Bergman, L.: Energy policy modeling: a survey of general equilibrium approaches. J. Policy Model. 10(3), 377–399
(1988)
Bertsekas, D.P.: On the Goldstein-Levitin-Polyak gradient projection method. IEEE Trans. Automat. Control. 21,
174–184 (1976)
Bertsekas, D.P.: Projected Newton mothods for optimization problems with simple constraints. SIAM J. Control Optim.
20, 221–246 (1982a)
Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York (1982b)
772 References

Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999)
Betts, J., Eldersveld, S.K., Frank, P.D., Lewis, J.G.: An interior-point nonlinear programming algorithm for large scale
optimization. (Technical Report MCT TECH-003, Mathematics and Computing Technology, The Boeing Com-
pany, P.O. Box 3707, Seattle, WA 98124-2207, USA) (2000)
Bicanic, N., Johnson, K.: Who was “Raphson”? Int. J. Numer. Methods Eng. 14, 148–152 (1979)
Biegler, L.T., Nocedal, J., Schmid, C.: A reduced Hessian method for large-scale constrained optimization. SIAM
J. Optim. 5, 314–347 (1995)
Biggs, M.C.: Minimization algorithms making use of non-quadratic properties of the objective function. J. Inst. Math.
Appl. 8, 315–327 (1971)
Biggs, M.C.: Constrained minimization using recursive equality quadratic programming. In: Lootsma, F.A. (ed.)
Numerical Methods for Nonlinear Optimization, pp. 411–428. Academic Press, London (1972)
Biggs, M.C.: A note on minimization algorithms making use of non-quadratic properties of the objective
function. J. Inst. Math. Appl. 12, 337–338 (1973)
Birge, J.R., Louveaux, F.: Introduction to Stochastic Programming. Springer, New York (1997)
Birgin, E.G., Martı́nez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. (Techni-
cal Paper, June 7) (1999)
Birgin, E.G., Martı́nez, J.M.: A spectral conjugate gradient method for unconstrained optimization. Appl. Math. Optim.
43, 117–128 (2001)
Birgin, E.G., Martı́nez, J.M.: Practical Augmented Lagrangian Methods for Constrained Optimization. SIAM,
Philadelphia (2014)
Birgin, E.G., Martı́nez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM
J. Optim. 10, 1196–1211 (2000)
Birgin, E.G., Martı́nez, J.M., Raydan, M.: Algorithm 813: SPG – software for convex-constrained optimization. ACM
Trans. Math. Softw. 27, 340–349 (2001)
Bisschop, J., Meeraus, A.: On the development of the general algebraic modeling systems in a strategic planning
environment. Math. Program. Study. 20, 1–29 (1982)
Boggs, P.T., Tolle, J.W.: A strategy for global convergence in a sequential quadratic programming algorithm. SIAM
J. Numer. Anal. 21, 600–623 (1989)
Boggs, P.T., Tolle, J.W.: Convergence properties of a class of rank-two updates. SIAM J. Optim. 4, 262–287 (1994)
Boggs, P.T., Tolle, J.W.: Sequential quadratic programming. Acta Numer. 4, 1–51 (1995)
Bondarenko, A.S., Bortz, D.M., Moré, J.J.: COPS: Large-scale nonlinearly constrained optimization problems. (Tech-
nical Report ANL/MCS-TM-237) (1999)
Bongartz, I., Conn, A.R., Gould, N.I.M., Toint, P.L.: CUTE: Constrained and unconstrained testing environment. ACM
Trans. Math. Softw. 21(1), 123–160 (1995)
Bonnans, J., Gilbert, J., Lemaréchal, C., Sagastizábal, C.: Numerical Optimization: Theory and Practical Aspects, 2nd
edn. Springer, Berlin (2006)
Bonnans, J.F., Panier, E.R., Tits, A.L., Zhou, J.L.: Avoiding the Maratos effect by means of a nonmonotone line search.
II. Inequality constrained problems – feasible iterates. SIAM J. Numer. Anal. 29, 1187–1202 (1992)
Boyd, S., El Ghaoui, L., Feron, E., Balakrishnan, V.: Linear Matrix Inequalities in System and Control Theory. Society
for Industrial and Applied Mathematics, Philadelphia (1994)
Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
Breitfeld, M.G., Shanno, D.F.: Preliminary computational experience with modified log-barrier functions for large-
scale nonlinear programming. In: Hager, W.W., Hearn, D.W., Pardalos, P.M. (eds.) Large Scale Optimization, State
of the Art, vol. 1994, pp. 45–67. Kluwer Academic Publishers, Dordrecht-Boston-London (1994a)
Breitfeld, M.G., Shanno, D.F.: Computational experience with penalty-barrier methods for nonlinear programming.
(RUTCOR Research Report, RRR 17-93, August 1993, Revised March 1994. Rutgers Center for Operations
Research, Rutgers University, New Brunswick, New Jersey 08903, March) (1994b)
Breitfeld, M.G., Shanno, D.F.: A globally convergent penalty-barrier algorithm for nonlinear programming and its
computational performance. (RUTCOR Research Report, RRR 12-94, April 1994, Rutgers Center for Operations
Research, Rutgers University, New Brunswick, New Jersey 08903, March) (1994c)
Brent, R.P.: Algorithms for Minimization Without Derivatives. Prentice Hall, Englewood Cliffs (1973)
Brooke, A., Kendrick, D., Meeraus, A., Raman, R., Rosenthal, R.E.: GAMS: A User’s Guide. GAMS Development
Corporation (1998)
Brooke, A., Kendrick, D., Meeraus, A., Raman, R.: GAMS: A User Guide. GAMS Development Corporation (2005)
Brown, A.A., Bartholomew-Biggs, M.C.: ODE vs SQP methods for constrained optimisation. (Technical Report
No. 179, Numerical Optimisation Centre, The Hatfield Polytechnic, Hatfield, June) (1987)
Brown, A.A., Bartholomew-Biggs, M.C.: Some effective methods for unconstrained optimization based on the solution
of systems of ordinary differential equations. J. Optim. Theory Appl. 62, 211–224 (1989)
References 773

Broyden, C.G.: The convergence of a class of double-rank minimization algorithms. I. General considerations. J. Inst.
Math. Appl. 6, 76–90 (1970)
Broyden, C.G., Dennis, J.E., Moré, J.J.: On the local and superlinear convergence of quasi-Newton methods. J. Inst.
Math. Appl. 12, 223–246 (1973)
Buckley, A.G., Lenir, A.: Algorithm 630-BBVSCG: A variable storage algorithm for function minimization. ACM
Trans. Math. Softw. 11, 103–119 (1985)
Bulirsch, R., Stoer, J.: Introduction to Numerical Analysis. Springer, New York (1980)
Bunch, J.R., Kaufman, L.: Some stable methods for calculating inertia and solving symmetric linear systems. Math.
Comput. 31, 163–179 (1977)
Bunch, J.R., Parlett, B.N.: Direct methods for solving symmetric indefinite systems of linear equations. SIAM J. Numer.
Anal. 8, 639–655 (1971)
Byrd, R.H., Curtis, F.E., Nocedal, J.: An inexact Newton method for nonconvex equality constrained optimization.
Math. Program. Serie A. 122, 273–299 (2010)
Byrd, R.H., Gilbert, J.-C., Nocedal, J.: A trust region method based on interior point techniques for nonlinear
programming. Math. Program. 89, 149–185 (2000)
Byrd, R.H., Gould, N.I.M., Nocedal, J., Waltz, R.A.: On the convergence of successive linear-quadratic programming
algorithms. (Technical Report OTC 2002/5, Optimization Technology Center, Northwestern University, Evanston,
IL) (2002)
Byrd, R.H., Gould, N.I.M., Nocedal, J., Waltz, R.A.: An algorithm for nonlinear optimization using linear programming
and equality constrained subproblems. Math. Program. Ser. B. 100, 27–48 (2004a)
Byrd, R.H., Hribar, M.E., Nocedal, J.: An interior point method for large scale nonlinear programming. SIAM J. Optim.
9, 877–900 (1999)
Byrd, R.H., Liu, G., Nocedal, J.: On the local behavior of an interior point method for nonlinear programming. In:
Griffths, D.F., Higham, D.J. (eds.) Numerical Analysis, pp. 37–56. Addison-Wesley Longman, Reading (1997)
Byrd, R.H., Liu, D.C., Nocedal, J.: On the behavior of Broyden’s class of quasi-Newton methods. SIAM J. Optim. 2,
533–557 (1992)
Byrd, R.H., Lu, P., Nocedal, J.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Stat.
Comput. 16(5), 1190–1208 (1995)
Byrd, R.H., Marazzi, M., Nocedal, J.: On the convergence of Newton iterations to non-stationary points. Math.
Program. Ser. A. 99, 127–148 (2004b)
Byrd, R.H., Nocedal, J.: A tool for the analysis of quasi-Newton methods with application to unconstrained minimiza-
tion. SIAM J. Numer. Anal. 26, 727–739 (1989)
Byrd, R.H., Nocedal, J., Waltz, R.A.: Feasible interior methods using slacks for nonlinear optimization. Comput.
Optim. Appl. 26(1), 35–61 (2003)
Byrd, R.H., Nocedal, J., Waltz, R.A.: KNITRO: An integrated package for nonlinear optimization. In: Di Pillo, G.,
Roma, M. (eds.) Large-Scale Nonlinear Optimization, pp. 35–59 (2006)
Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. (Technical
Report NAM-08, [Revised May 1994], Department of Electrical Engineering and Computer Science, Northwestern
University, Evanston, Illinois 60208) (1994)
Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm, for bound constrained optimization. SIAM J. Sci.
Comput. 16, 1190–1208 (1995)
Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representation of quasi-Newton matrices and their use in limited memory
methods. Math. Program. 63(2), 129–156 (1994)
Byrd, R.H., Nocedal, J., Yuan, Y.: Global convergence of a class of quasi-Newton methods on convex problems. SIAM
J. Numer. Anal. 24, 1171–1190 (1987a)
Byrd, R.H., Schnabel, R.B., Shultz, G.A.: A trust region algorithm for nonlinearly constrained optimization. SIAM
J. Numer. Anal. 24, 1152–1170 (1987b)
Byrd, R.H., Schnabel, R.B., Shultz, G.A.: Approximate solution of the trust-region problem by minimization over
two-dimensional subspaces. Math. Program. 40, 247–263 (1988)
Byrd, R.H., Tapia, R.A., Zhang, Y.: An SQP augmented Lagrangean BFGS algorithm for constrained optimization.
(Technical Report, University of Colorado at Boulder) (1990)
Carolan, W.J., Hill, J.E., Kennington, J.L., Niemi, S., Wochmann, S.J.: An empirical evaluation of the KORBX
algorithms for military airlift applications. Oper. Res. 38, 240–248 (1990)
Cauchy, A.: Méthodes générales pour la resolution des systèmes déquations simultanées. Comptes Rendus de
l’Académie des Sciences Paris. 25(1), 536–538 (1847)
Castillo, E., Conejo, A.J., Pedregal, P., Garcı́a, R., Alguacil, N.: Building and Solving Mathematical Programming
Models in Engineering and Science. Wiley, New York (2001)
Cesari, L.: Optimization – Theory and Applications. Springer, Bonn (1983)
774 References

Chachuat, B.C.: Nonlinear and Dynamic Optimization – from Theory to Practice, IC-31: Winter Semester 2006/2007.
École Politechnique Fédérale de Lausanne (2007)
Chen, H., Lam, W.H., Chan, S.C.: On the convergence analysis of cubic regularized symmetric rank-1 quasi-Newton
method and the incremental version in the application of large-scale problems. IEEE Access. 7, 114042–114059
(2019)
Cheng, S.H.: Symmetric indefinite matrices: Linear system solvers and modified inertia problems. Ph.D. Thesis,
University of Manchester. Faculty of Science and Engineering, January 1998
Cheng, S.H., Higham, N.J.: A modified Cholesky algorithm based on a symmetric indefinite factorization. SIAM
J. Matrix Anal. Appl. 19, 1097–1110 (1998)
Cheng, W.Y., Li, D.H.: Spectral scaling BFGS method. J. Optim. Theory Appl. 146, 305–319 (2010)
Chin, C.M.: A global convergence theory of a filter line search method for nonlinear programming. (Technical Report,
Department of Electrical Engineering, University of Malaya, Kuala Lumpur, Malaysia, August) (2002)
Chin, C.M., Fletcher, R.: On the local convergence of an SLP-filter algorithm that takes EQP steps. Math. Program.
Ser. A. 96, 161–177 (2003)
Cimatti, G., Menchi, O.: On the numerical solution of a variational inequality connected with the hydrodynamic
lubrication of a complete journal bearing. Calcolo. 15, 249–258 (1978)
Choi, T.D., Kelley, C.T.: Superlinear convergence and implicit filtering. SIAM J. Optim. 10, 1149–1162 (2000)
Cohen, A.: Rate of convergence of several conjugate gradient algorithms. SIAM J. Numer. Anal. 9, 248–259 (1972)
Coleman, T.F., Conn, A.R.: Nonlinear programming via an exact penalty function: Asymptotic analysis. Math.
Program. 24, 123–136 (1982a)
Coleman, T.F., Conn, A.R.: Nonlinear programming via an exact penalty function: Global analysis. Math. Program. 24,
137–161 (1982b)
Coleman, T.F., Hulbert, L.A.: A direct active set algorithm for large sparse quadratic programs with simple bounds.
Math. Program. Ser. A. 45, 373–406 (1989)
Coleman, T.F., Li, Y.: On the convergence of interior-reflective Newton methods for nonlinear minimization subject to
bounds. Math. Program. 67, 189–224 (1994)
Coleman, T.F., Li, Y.: An interior trust region approach for nonlinear minimization subject to bounds. SIAM J. Optim.
6, 418–445 (1996)
Coleman T.F., Li, Y.: A trust region and affine scaling interior point method for nonconvex minimization with linear
inequality constraints. (Technical Report, Cornell University, Ithaca, NY) (1997)
Concus, P., Golub, G., O’Leary, D.P.: A generalized conjugate gradient method for the numerical solution of elliptic
partial differential equations. In: Bunch, J., Rose, D. (eds.) Sparse Matrix Computation, pp. 309–332. Academic
Press, New York (1976)
Conejo, A.J., Castillo, E., Minguez, R., Garcia-Bertrand, R.: Decomposition Techniques in Mathematical Program-
ming: Engineering and Science Applications. Springer, Heidelberg (2006)
Conn, A.R.: Linear programming via a nondifferentiable penalty function. SIAM J. Numer. Anal. 13, 145–154 (1976)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Global convergence of a class of trust region algorithms for optimization with
simple bounds. SIAM J. Numer. Anal. 25(2), 433–460 (1988a)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Testing a class of methods for solving minimization problems with simple
bounds on the variables. Math. Comput. 50, 399–430 (1988b)
Conn, A.R., Gould, N.I.M., Toint, P.L.: A globally convergent augmented Lagrangean algorithm for optimization with
general constraints and simple bounds. SIAM J. Numer. Anal. 28, 545–572 (1991a)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Convergence of quasi-newton matrices generated by the symmetric rank one
update. Math. Program. 50(1–3), 177–195 (1991b)
Conn, A.R., Gould, N.I.M., Toint, Ph.L.: A globally convergent Lagrangian barrier algorithm for optimization with
general inequality constraints and simple bounds. (Technical Report 92/07, Department of Mathematics, Faculté
Universitaires de Namur, Namur, Belgium) (1992a)
Conn, A.R., Gould, N.I.M., Toint, P.L.: LANCELOT – A Fortran Package for Large-Scale Nonlinear Optimization
(Release A). Springer Series in Computational Mathematics, vol. 17. Springer, Berlin (1992b)
Conn, A.R., Gould, N.I.M., Toint, P.L.: A globally convergent Lagrangian barrier algorithm for optimization with
general inequality constraints and simple bounds. Math. Comput. 66, 261–288 (1997a)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Numerical experiments with the LANCELOT package (Release A) for large-
scale nonlinear optimization. Math. Program. Ser. A. 73(1), 73–110 (1996a)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. MPS-SIAM Series on Optimization. SIAM,
Philadelphia (2000)
Conn, A.R., Gould, N.I.M., Sartenaer, A., Toint, P.L.: Convergence properties of an augmented Lagrangean algorithm
for optimization with a combination of general equality and linear constraints. SIAM J. Optim. 6(3), 674–703
(1996b)
References 775

Conn, A.R., Scheinberg, K., Toint, P.L.: Recent progress in unconstrained nonlinear optimization without derivatives.
Math. Program. Ser. B. 79, 397–414 (1997b)
Contreras, M., Tapia, R.A.: Sizing the BFGS and DFP updates: A numerical study. J. Optim. Theory Appl. 78, 93–108
(1993)
Cottle, R.W.: William Karush and the KKT theorem. In: Gr€ otschel, M. (ed.) Optimization stories. Documenta
Mathematica, Journal der Deutschen Methematiker-Vereinigung, Extra volume, 21st International Symposium on
Mathematical Programming, pp. 255–269, Berlin, August 19–24, 2012
Crowder, H.P., Wolfe, P.: Linear convergence of the conjugate gradient method. IBM J. Res. Dev., 431–433 (1969)
Courant, R.: Variational methods for the solution of problems of equilibrium and vibrations. Bull. Am. Math. Soc. 49,
1–23 (1943)
Curtis, A., Powell, M.J.D., Reid, J.K.: On the estimation of sparse Jacobian matrices. J. Inst. Math. Appl. 13, 117–119
(1974)
Cuthrell, J.E., Biegler, L.T.: Simultaneous optimization and solution methods for batch reactor control profiles.
Comput. Chem. Eng. 13, 49–62 (1989)
Dai, Y.H.: Analyses of conjugate gradient methods. Ph.D. Thesis, Institute of Computational Mathematics and
Scientific/Engineering Computing, Chinese Academy of Sciences (1997)
Dai, Y.H.: A nonmonotone conjugate gradient algorithm for unconstrained optimization. J. Syst. Sci. Complex. 15(2),
139–145 (2002a)
Dai, Y.H.: On the nonmonotone line search. J. Optim. Theory Appl. 112, 315–330 (2002b)
Dai, Y.H.: Convergence properties of the BFGS algorithm. SIAM J. Optim. 13, 693–701 (2003)
Dai, Y.H.: Chapter 8: Convergence analysis of nonlinear conjugate gradient methods. In: Wang, Y., Yagola, A.G.,
Yang, C. (eds.) Optimization and Regularization for Computational Inverse Problems and Applications,
pp. 157–181. Higher Education Press/Springer, Beijing/Berlin, Heidelberg (2010)
Dai, Y.H.: Nonlinear conjugate gradient methods. Wiley Encyclopedia of Operations Research and Management
Science. https://doi.org/10.1002/9780470400531.eorms0183. Published on line, February 15, 2011
Dai, Y.H., Hager, W.W., Schittkowski, K., Zhang, H.: The cyclic Barzilai-Borwein method for unconstrained optimi-
zation. IMA J. Numer. Anal. 26, 604–627 (2006)
Dai, Y.H., Han, J.Y., Liu, G.H., Sun, D.F., Yin, H.X., Yuan, Y.X.: Convergence properties of nonlinear conjugate
gradient methods. SIAM J. Optim. 10(2), 345–358 (1999)
Dai, Y.H., Kou, C.X.: A nonlinear conjugate gradient algorithm with an optimal property and an improved Wolfe line
search. SIAM J. Optim. 23(1), 296–320 (2013)
Dai, Y.H., Liao, L.Z.: New conjugate conditions and related nonlinear conjugate gradient methods. Appl. Math. Optim.
43, 87–101 (2001)
Dai, Y.H., Liao, L.Z.: R-linear convergence of the Barzilai and Borwein gradient method. IMA J. Numer. Anal. 22(1),
1–10 (2002)
Dai, Y.H., Liao, L.Z., Li, D.: On restart procedures for the conjugate gradient method. Numer. Algorithms. 35, 249–260
(2004)
Dai, Y.H., Ni, Q.: Testing different conjugate gradient methods for large-scale unconstrained optimization. J. Comput.
Math. 22(3), 311–320 (2003)
Dai, Z., Wen, F.: Another improved Wei-Yao-Liu nonlinear conjugate gradient method with sufficient descent
property. Appl. Math. Comput. 218, 7421–7430 (2012)
Dai, Y.H., Yuan, Y.X.: Convergence properties of the conjugate descent method. Adv. Math. (China). 26, 552–562
(1996)
Dai, Y.H., Yuan, Y.: A nonlinear conjugate gradient method with strong global convergence property. SIAM J. Optim.
10, 177–182 (1999)
Dai, Y.H., Yuan, Y.: An efficient hybrid conjugate gradient method for unconstrained optimization. Ann. Oper. Res.
103, 33–47 (2001)
Dai, Y.H., Yuan, Y.: A class of globally convergent conjugate gradient methods. Sci. China Math. Ser. A. 46(2),
251–261 (2003)
Daniel, J.W.: The conjugate gradient method for linear and nonlinear operator equations. SIAM J. Numer. Anal. 4,
10–26 (1967)
Daniel, J.W., Graggs, W.B., Kaufman, L., Stewart, G.W.: Reorthogonalization and stable algorithms for updating the
Gram-Schmidt QR factorizations. Math. Comput. 30, 772–795 (1976)
Dantzig, G.B.: Linear Programming and Extensions. Princeton University Press, Princeton (1963)
Davidon, W.C.: Variable metric method for minimization. (Research and Development Report ANL-5990. Argonne
National Laboratories) (1959)
Davidon, W.C.: Conic approximation and collinear scalings for optimizers. SIAM J. Numer. Anal. 17(2), 268–281
(1980)
Debreu, G.: Definite and semidefinite quadratic forms. Econometrica. 20, 295–300 (1952)
776 References

Dehmiry, A.H.: The global convergence of the BFGS method under a modified Yuan-Wei-Lu line search technique.
Numer. Algorithms. (2019). https://doi.org/10.1007/s11075-019-00779-7
Dembo, R.S.: A set of geometric programming test problems and their solutions. Math. Program. 10, 192–213 (1976)
Dembo, R.S., Eisenstat, S.C., Steihaug, T.: Inexact Newton methods. SIAM J. Numer. Anal. 19, 400–408 (1982)
Dembo, R.S., Steihaug, T.: Truncated Newton algorithms for large-scale unconstrained optimization. Math. Program.
26, 190–212 (1983)
Dembo, R.S., Tulowitzki, U.: On the minimization of quadratic functions subject to box constraints. (Technical Report,
School of Organization and Management, Yale University, New Haven, CT) (1983)
Demmel, J.W.: Applied Numerical Linear Algebra. SIAM, Philadelphia (1997)
Dener, A., Denchfield, A., Munson, T.: Preconditioning nonlinear conjugate gradient with diagonalized quasi-Newton.
(Mathematics and Computer Science Division, Preprint ANL/MCS-P9152-0119, January 2019, Argonne National
Laboratory, 9700 South Cass Avenue, Argonne, Illinois 60430) (2019)
Deng, N.Y., Xiao, Y., Zhou, F.J.: Nonmonotonic trust-region algorithm. J. Optim. Theory Appl. 26, 259–285 (1993)
Dennis, J.E., Heinkenschlos, M., Vicente, L.N.: Trust-region interior-point algorithms for a class of nonlinear
programming problems. SIAM J. Control. Optim. 36, 1750–1794 (1998)
Dennis, J.E., Moré, J.J.: Quasi-Newton methods, motivation and theory. SIAM Rev. 19(1), 46–89 (1977)
Dennis, J.E., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Reprinted
as Classics in Applied Mathematics, vol. 16. Prentice-Hall/SIAM, Englewood Cliffs/Philadelphia (1983)
Dennis, J.E., Schnabel, R.B.: A view of unconstrained optimization. In: Optimization, Vol. 1 of Handbooks in
Operations Research and Management, pp. 1–72, Elsevier Science Publishers, Amsterdam (1989)
Dennis, J.E., Wolkowicz, H.: Sizing and least-change secant methods. SIAM J. Numer. Anal. 30(5), 1291–1314 (1993)
Deuflhard, P.: Global inexact Newton methods for very large scale nonlinear problems. In: Proceedings of the Cooper
Mountain Conference on Iterative Methods. Cooper Mountain, Colorado, April 1–5 (1990)
Dikin, I.I.: Iterative solution of problems of linear and quadratic programming. Soviet Math. Doklady. 8, 674–675
(1967)
Dikin, I.I.: On the convergence of an iterative process. Upravlyaemye Sistemi. 12, 54–60 (1974)
Dixon, L.C.W., Price, R.C.: Numerical experience with truncated Newton method for unconstrained
optimization. J. Optim. Theory Appl. 56, 245–255 (1988)
Dolan, E.D., Moré, J.J.: Benchmarching optimization software with performance profiles. Math. Program. 91, 201–213
(2002)
Dolan, E.D., Moré, J.J., Munson, T.S.: Benchmarking Optimization Software with COPS 3.0. Preprint ANL/MCS-TM-
273, Argonne National Laboratory, Argonne (2004)
Drud, A.: Application of sparse matrix techniques in large scale nonlinear programming. In: Prekopa, A. (ed.) Survey of
Mathematical Programming, vol. 1, part 3, Nonlinear Programming. (Proceedings of the 9th International Mathe-
matical Programming Symposium, Budapest, August 23–27, 1976, pp. 429–445) (1976)
Drud, A.: CONOPT: A GRG code for large sparse dynamic nonlinear optimization problems. (Report No. DRD59,
Development Research Department, Economics and Research Staff, The World Bank, 1818 H Street, Washington,
D.C. 20433, August) (1983)
Drud, S.A.: CONOPT: A GRG code for large sparse dynamic nonlinear optimization problems. Technical Note No.21,
Development Research Center, The World Bank, 1818 H Street, Washington, D.C. 20433, March (1984)
Drud, S.A.: CONOPT: A GRG code for large sparse dynamic nonlinear optimization problems. Math. Program. 31,
153–191 (1985)
Drud, S.A.: CONOPT – A large-scale GRG code. ORSA J. Comput. 6, 207–216 (1994)
Drud, S.A.: CONOPT – A system for large-scale nonlinear optimization. (Tutorial for CONOPT subroutine library,
16p. ARKI Consulting and Development A/S, Bagsvaerd, Denmark) (1995)
Drud, S.A.: CONOPT: A system for large-scale nonlinear optimization. (Reference Manual for CONOPT subroutine
library, 69p. ARKI Consulting and Development A/S, Bagsvaerd, Denmark) (1996)
Drud, S.A.: CONOPT. In: GAMS the Solver Manuals, pp. 39–82. GAMS Development Corporation, Washington, DC
(2005)
Drud, S.A. CONOPT – A system for large scale nonlinear optimization. Tutorial for CONOPT dynamic link library.
Version 3.15 (Fortran 90 examples). (ARKI Consulting and Development A/S, Bagsvaerd, Denmark, August)
(2011)
Duran, M., Grossmann, I.E.: An outer-approximation algorithm for a class of mixed-integer nonlinear programs. Math.
Program. 36, 307–339 (1986)
Ecker, J.G., Kupferschmid, M.: An ellipsoid algorithm for nonlinear programming. Math. Program. 27, 83–106 (1983)
Ecker, J.G., Kupferschmid, M.: A computational comparison of the ellipsoid algorithm with several nonlinear
programming algorithms. SIAM J. Control. Optim. 23(5), 657–674 (1985)
Eisenstat, S.C., Walker, H.F.: Choosing the forcing terms in an inexact Newton method. SIAM J. Sci. Comput. 17(1),
16–32 (1996)
References 777

El-Bakry, A.S., Tapia, R.A., Tsuchiya, T., Zhang, Y.: On the formulation and theory of the Newton interior-point
method for nonlinear programming. J. Optim. Theory Appl. 89(3), 507–541 (1996)
Eldersveld, S.K.: Large-scale sequential quadratic programming algorithms. Ph.D. Thesis, Department of Operations
Research, Stanford University, Stanford, CA (1991)
Facchinei, F., Júdice, J., Soares, J.: An active set Newton’s algorithm for large-scale nonlinear programs with box
constraints. SIAM J. Optim. 8, 158–186 (1998)
Facchinei, F., Lucidi, S.: A class of penalty functions for optimization problems with bounds constraints. Optimization.
26, 239–259 (1992a)
Facchinei, F., Lucidi, S.: A class of methods for optimization problems with simple bounds. Part 2: Algorithms and
numerical results. (Technical Report R.336, IASI-CNR, Roma, Italy) (1992b)
Facchinei, F., Lucidi, S.: Convergence to second order stationary points in inequality constrained optimization. Math.
Oper. Res. 23, 746–766 (1998)
Facchinei, F., Lucidi, S., Palagi, L.: A truncated Newton algorithm for large-scale box constrained optimization. SIAM
J. Optim. 4, 1100–1125 (2002)
Fiacco, A.V., McCormick, G.P.: The sequential unconstrained minimization technique for nonlinear programming. A
primal-dual method. Manag. Sci. 10, 360–366 (1964)
Fiacco, A.V., McCormick, G.P.: Extensions of SUMT for nonlinear programming: equality constraints and extrapola-
tion. Manag. Sci. 12, 816–828 (1966)
Fiacco, A.V., McCormick, G.P.: Nonlinear Programming: Sequential Unconstrained Minimization Technique. Wiley,
New York (1968). [Republished in 1990 by SIAM, Philadelphia]
Finkel, D.E.: DIRECT Optimization Algorithm User Guide. Center for Research in Scientific Computation, North
Carolina State University, Raleigh (2003)
Fletcher, R.: A new approach to variable metric algorithms. Comput. J. 13, 317–322 (1970)
Fletcher, R.: A general quadratic programming algorithm. J. Inst. Math. Appl., 76–91 (1971)
Fletcher, R.: Second order corrections for nondifferentiable optimization. In: Griffith, D. (ed.) Numerical Analysis,
pp. 85–114. Springer (1982). Proceedings Dundee 1981
Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, New York (1987)
Fletcher, R.: A new variational result for quasi-Newton formulae. SIAM J. Optim. 1, 18–21 (1991)
Fletcher, R.: An optimal positive definite update for sparse Hessian matrices. SIAM J. Optim. 5, 192–218 (1995)
Fletcher, R.: Stable reduced Hessian updates for indefinite quadratic programming. Math. Program. 87, 251–264 (2000)
Fletcher, R.: A package of subroutines for NLP and LCP. Open Source Initiative OSI – Eclipse Public License 1.0
(ELP-1.0), Release 1.0, 2011 (2011)
Fletcher, R., Sainz de la Maza, E.: Nonlinear programming and nonsmooth optimization by successive linear
programming. Math. Program. 43, 235–256 (1989)
Fletcher, R., Freeman, T.L.: A modified Newton method for minimization. J. Optim. Theory Appl. 23, 357–372 (1977)
Fletcher, R., Gould, N.I.M., Leyffer, S., Toint, P.T., Wächter, A.: Global convergence of a trust-region SQP filter
algorithm for general nonlinear programming. SIAM J. Optim. 13, 635–659 (2002a)
Fletcher, R., Grothey, A., Leyffer, S.: Computing sparse Hessian and Jacobian approximations with optimal hereditary
properties. (Technical Report, Department of Mathematics, University of Dundee) (1996)
Fletcher, R., Leyffer, S.: User Manual for FilterSQP. (Technical Report NA/181, Department of Mathematics,
University of Dundee, Scotland, April. [Updated, March 1999]) (1998)
Fletcher, R., Leyffer, S.: A bundle filter method for nonsmooth nonlinear optimization. (Numerical Analysis Report
NA/195, Dundee University, April) (1999)
Fletcher, R., Leyffer, S.: Nonlinear programming without a penalty function. Math. Program. Ser. A. 91, 239–269
(2002)
Fletcher, R., Leyffer, S.: Filter-type algorithms for solving systems of algebraic equations and inequalities. In: di Pillo,
G., Murli, A. (eds.) High Performance Algorithms and Software for Nonlinear Optimization, pp. 259–278. Kluwer
Academic Publishers, Central and South America (2003)
Fletcher, R., Leyffer, S., Toint, Ph.: On the global convergence of an SLP-filter algorithm. (Numerical Analysis Report
NA/183, August 1998, revised October 1999) (1999)
Fletcher, R., Leyffer, S., Toint, P.: On the global convergence of a filter-SQP algorithm. SIAM J. Optim. 13, 44–59
(2002b)
Fletcher, R., Leyffer, S., Toint, Ph.: A brief history of filter methods. (Argonne National Laboratory, Mathematics and
Computer Science Division, Preprint ANL/MCS-P1372-0906, September 26, 2006, revised October 9, 2006) (2006)
Fletcher, R., Leyffer, S., Shen, C.: Nonmonotone filter method for nonlinear optimization. (Argonne National Labora-
tory, Mathematics and Computer Science Division, Preprint ANL/MCS-P1679-0909, October 14) (2009)
Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J., 163–168 (1963)
Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradient. Comput. J. 7, 149–154 (1964)
Floudas, C.A., Pardalos, P.M.: Recent Advances in Global Optimization. Princeton University Press, Princeton (1992)
778 References

Floudas, C.A., Pardalos, M.P., Adjiman, C.S., Esposito, W.R., Gümüs, Z.H., Harding, S.T., Klepeis, J.L., Meyer, C.A.,
Schweiger, C.A.: Handbook of Test Problems in Local and Global Optimization. Kluwer Academic Publishers,
Dordrecht (1999)
Ford, J.A., Moghrabi, I.A.: Multi-step quasi-Newton methods for optimization. J. Comput. Appl. Math. 50(1–3),
305–323 (1994)
Ford, J.A., Moghrabi, I.A.: Minimum curvature multi-step quasi-Newton methods. Comput. Math. Appl. 31(4–5),
179–186 (1996a)
Ford, J.A., Moghrabi, I.A.: Using function-values in multi-step quasi-Newton methods. J. Comput. Appl. Math.
66(1–2), 201–211 (1996b)
Ford, J.A., Narushima, Y., Yabe, H.: Multi-step nonlinear conjugate gradient methods for unconstrained minimization.
Comput. Optim. Appl. 40(2), 191–216 (2008)
Forsgren, A., Gill, P.E., Murray, W.: Computing modified Newton directions using a partial Cholesky factorization.
SIAM J. Sci. Comput. 16(1), 139–150 (1995)
Forsgren, A., Gill, P.E., Wright, M.H.: Interior points for nonlinear optimization. SIAM Rev. 44, 525–597 (2002)
Forsythe, G.E., Hestenes, M.R., Rosser, J.B.: Iterative methods for solving linear equations. Bull. Am. Math. Soc. 57,
480 (1951)
Fourer, R.: Modeling languages versus matrix generators for linear programming. ACM Trans. Math. Softw. 9,
143–183 (1983)
Fourer, R., Gay, M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming, 2nd edn.
Duxbury Press, Scituate (2002)
Frank, M., Wolfe, P.: An algorithm for quadratic programing. Naval Res. Logist. Q. 3, 95–110 (1956)
Friedlander, A., Martı́nez, J.M., Santos, S.A.: A new trust region algorithm for bound constrained minimization. Appl.
Math. Optim. 30, 235–266 (1994)
Friedlander, M.P., Saunders, M.: A globally convergent linearly constrained Lagrangian method for nonlinear optimi-
zation. SIAM J. Optim. 15, 863–897 (2005)
Frisch, K.R.: The logarithmic potential method for convex programming. (Manuscript. Institute of Economics,
University of Oslo, Oslo, May) (1955)
Fukushima, M.: A successive quadratic programming algorithm with global and superlinear convergence properties.
Math. Program. 35, 253–264 (1986)
Gabay, D.: Reduced quasi-Newton methods with feasibility improvement for nonlinearly constrained optimization.
Math. Program. Study. 16, 18–44 (1982)
Gabriele, G.A., Ragsdell, K.M.: Large scale nonlinear programming using the generalized reduced gradient method.
Trans. ASME J. Mech. Des. 102, 566–573 (1980)
Garner, J., Benedek, R.: Solution of Ginzburg-Landau equations for inhomogeneous superconductors by nonlinear
optimization. Phys. Rev. B. 42, 376–385 (1990)
Gay, D.M.: Computing optimal locally constrained steps. SIAM J. Sci. Stat. Comput. 2, 186–197 (1981)
Gay, D.M.: Electronic mail distribution of linear programming test problems. Math. Program. Soc. COAL Newsl. 13,
10–12 (1985)
Gay, D.M., Overton, M.L., Wright, M.H.: A primal-dual interior method for nonconvex nonlinear programming.
(Technical Report 97-4-08, Bell Lab. Murray Hill, July 29) (1997)
Ge, R.-P., Powell, M.J.D.: The convergence of variable metric matrices in unconstrained optimization. Math. Program.
27, 123–143 (1983)
George, A., Liu, J.W.-H.: The evolution of the minimum degree ordering. SIAM Rev. 31, 1–19 (1989)
Gilbert, J.C., Lemaréchal, C.: Some numerical experiments with variable-storage quasi-Newton algorithms. Math.
Program. Ser. B. 45, 407–435 (1989)
Gilbert, J.C.: SQPlab – A MATLAB software for solving nonlinear optimization problems and optimal control
problems. (Technical Report, INRIA-Rocquencourt, BP 105, F-78153 Le Chesnay Cedex, France) (2009)
Gilbert, J.C., Nocedal, J.: Global convergence properties of conjugate gradient methods for optimization. SIAM
J. Optim. 2, 21–42 (1992)
Gill, P.E., Kungurtsev, V., Robinson, D.P.: A stabilized SQP method: Global convergence. IMA J. Numer. Anal. 37,
407–443 (2017a)
Gill, P.E., Kungurtsev, V., Robinson, D.P.: A stabilized SQP method: Superlinear convergence. Math. Program. 163,
369–410 (2017b)
Gill, P.E., Leonard, M.W.: Limited-memory reduced-Hessian methods for unconstrained optimization. SIAM J. Optim.
14, 380–401 (2003)
Gill, P.E., Murray, W.: Quasi-Newton methods for unconstrained optimization. J. Inst. Math. Appl. 9, 91–108 (1972)
Gill, P.E., Murray, W.: Safeguarding steplength algorithms for optimization using descent methods. (Technical Report
NAC 37, National Physical Laboratory, Teddington, UK) (1974a)
References 779

Gill, P.E., Murray, W.: Methods for large-scale linearly constrained problems. In: Gill, P.E., Murray, W. (eds.)
Numerical Methods for Constrained Optimization, pp. 93–147. Academic Press, London, New York, San Francisco
(1974b)
Gill, P.E., Murray, W.: Numerically stable methods for quadratic programming. Math. Program. 14, 349–372 (1978)
Gill, Ph.E., Murray, W.: Conjugate gradient methods for large-scale nonlinear optimization. (Report SOL 79-15.
Department of Operations Research, Stanford University, Stanford) (1979)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: Sparse matrix methods in optimization. (Technical Report
SOL82-17, Department of Operations Research, Stanford University, Stanford, California) (1982)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: User’s guide for SOL/QPSOL. (Technical Report SOL84-6,
Department of Operations Research, Stanford University, Stanford, California) (1984)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: User’s guide for NPSOL (version 4.0): A Fortran package for
nonlinear programming. (Report SOL 86-2, Department of Operations Research, Stanford University, Stanford,
CA) (1986)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: Maintaining LU factors of a general sparse matrix. Linear
Algebra Appl. 88(89), 239–270 (1987)
Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: Some theoretical properties of an augmented Lagrangean merit
function. In: Pardalos, P.M. (ed.) Advances in Optimization and Parallel Computing, pp. 101–128. North-Holland,
Amsterdam (1992)
Gill, P.E., Murray, W., Wright, M.H.: Practical Optimization. Academic Press, London (1981)
Gill, P.E., Murray, W., Saunders, M.A.: User’s guide for SNOPT (version 5.3): A Fortran package for large-scale
nonlinear programming. (Technical Report NA 97-4, Department of Mathematics, University of California, San
Diego) (1997)
Gill, P.E., Murray, W., Saunders, M.A.: SNOPT: A SQP algorithm for large-scale constrained optimization. SIAM
J. Optim. 12, 979–1006 (2002)
Gill, P.E., Murray, W., Saunders, M.A.: SNOPT: An SQP algorithm for large-scale constrained optimization. SIAM
Rev. 47, 99–131 (2005)
Gill, P.E., Murray, W., Saunders, M.A.: User’s guide for SQOPT Version 7: Software for large-scale nonlinear
programming. (Report, Department of Mathematics, University of California, San Diego) (2006)
Gill, P.E., Golub, G.H., Murray, W., Saunders, M.A.: Methods for modifying matrix factorizations. Math. Comput. 28,
505–535 (1974)
Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, Berlin (1984)
€
G€odel, K.: Uber formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I. Monatshefte für
Mathematik und Physik. 38, 173–199 (1931)
Goldfarb, D.: A family of variable metric method derived by variation mean. Math. Comput. 23, 23–26 (1970)
Goldfarb, D.: Matrix factorizations in optimization of nonlinear functions subject to linear constraints. Math. Program.
10, 1–31 (1975)
Goldfarb, D.: Using the steepest-edge simplex algorithm to solve sparse linear programs. In: Bunch, J.R., Rose,
D. (eds.) Sparse Matrix Computations, pp. 227–240. Academic Press, New York (1976)
Goldfarb, D.: Curvilinear path steplength algorithms for minimization which use directions of negative curvature.
Math. Program. 18, 31–40 (1980)
Goldfarb, D., Liu, S., Wang, S.: A logarithmic barrier function algorithm for quadratically constrained convex quadratic
programming. SIAM J. Optim. 1(2), 252–267 (1991)
Goldfarb, D., Mu, C., Wright, J., Zhou, C.: Using negative curvature in solving nonlinear programs, arXiv preprint
arXiv:1706.00896 (2017)
Goldfarb, D., Idnani, A.: A numerically stable dual method for solving strictly convex quadratic programs. Math.
Program. 27, 1–33 (1983)
Goldfarb, D., Reid, J.K.: A practicable steepest-edge simplex algorithm. Math. Program. 12, 361–371 (1977)
Goldfeld, S.M., Quandt, R.E., Trotter, H.F.: Maximization by quadratic hill-climbing. Econometrica. 34, 541–551
(1966)
Goldstein, A.A.: On steepest descent. SIAM J. Control. 3, 147–151 (1965)
Goldstein, A.A., Price, J.: An effective algorithm for minimization. Numer. Math. 10, 184–189 (1967)
Golub, G.H., Van Loan, C.F.: Matrix Computation, 3rd edn. The Johns Hopkins University Press, Baltimore (1996)
Goncalves, A.S.: A primal-dual method for quadratic programming with bounded variables. In: Lootsma, F.A. (ed.)
Numerical Methods for Nonlinear Optimization, pp. 255–263. Academic Press, London (1972)
Goodman, J., Kohn, R., Reyna, L.: Numerical study of a relaxed variational problem from optimal design. Comput.
Methods Appl. Mech. Eng. 57, 107–127 (1986)
Gould, N.I.M.: On the accurate determination of search directions for simple differentiable penalty functions. IMA
J. Numer. Anal. 6, 357–372 (1986)
Gould, N.I.M.: An algorithm for large scale quadratic programming. IMA J. Numer. Anal. 11, 299–324 (1991)
780 References

Gould, N.I.M.: An Introduction to Algorithms for Continuous Optimization. Oxford University Computing Laboratory
and Rutherford Appleton Laboratory, UK (2006)
Gould, N.I.M., Hribar, M.E., Nocedal, J.: On the solution of equality constrained quadratic problems arising in
optimization. SIAM J. Sci. Comput. 23, 1375–1394 (2001)
Gould, N.I.M., Orban, D., Toint, P.L.: GALAHAD, a library of thread-safe Fortran 90 package for large-scale nonlinear
optimization. ACM Trans. Math. Softw. 29(4), 353–372 (2004)
Gould, N.I.M., Orban, D., Toint, P.L.: Numerical methods for large-scale nonlinear optimization. Acta Numer. 14,
299–361 (2005a)
Gould, N.I.M., Robinson, D.P.: A second derivative SQP methods: Local convergence. (Numerical Analysis Report
08/21, Oxford University Computing Laboratory) (2008)
Gould, N.I.M., Robinson, D.P.: A second derivative SQP methods: Global convergence. SIAM J. Optim. 20,
2023–2048 (2010)
Gould, N.I.M., Sainvitu, C., Toint, P.L.: A filter-trust-region method for unconstrained optimization. SIAM J. Optim.
16, 341–357 (2005b)
Gould, N.I.M., Toint, P.L.: Numerical methods for large-scale non-convex quadratic programming. In: Siddiqi, A.H.,
Kocvara, M. (eds.) Trends in Industrial and Applied Mathematics, pp. 149–179. Kluwer Academic Publishers,
Dordrecht (2002a)
Gould, N.I.M., Toint, P.L.: An iterative working-set method for large-scale non-convex quadratic programming. Appl.
Numer. Math. 43, 09–128 (2002b)
Gould, N.I.M., Toint, P.L.: Preprocessing for quadratic programming. Math. Program. 100(1), 95–132 (2004)
Gould, N.I.M., Toint, Ph.L.: A Quadratic Programming Bibliography. (RAL Numerical Analysis Group Internal Report
2000–1, March 28) (2012)
Greenbaum, A.: Iterative Methods for Solving Linear Systems. Frontiers in Applied Mathematics. SIAM, Philadelphia
(1997)
Greenbaum, A., Strakoš, Z.: Predicting the behavior of finite precision Lanczos and conjugate gradient computations.
SIAM J. Matrix Anal. Appl. 13, 121–137 (1992)
Griewank, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM, Philadelphia
(2000)
Griewank, A., Toint, P.L.: Partitioned variable metric updates for large structured optimization problems. Numer. Math.
39, 119–137 (1982a)
Griewank, A., Toint, P.L.: Local convergence analysis of partitioned quasi-Newton updates. Numer. Math. 39, 429–448
(1982b)
Griffith, R.E., Stewart, R.A.: A nonlinear programming technique for the optimization of continuous processing
systems. Manag. Sci. 7(4), 379–392 (1961)
Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer.
Anal. 23, 707–716 (1986)
Grippo, L., Lampariello, F., Lucidi, S.: A truncated Newton method with nonmonotone line search for unconstrained
optimization. J. Optim. Theory Appl. 60, 401–419 (1989)
Grippo, L., Lampariello, F., Lucidi, S.: A class of nonmonotone stabilization methods in unconstrained optimization.
Numer. Math. 59, 779–805 (1991)
Grippo, L., Sciandrone, M.: Nonmonotone globalization techniques for the Barzilai-Borwein gradient method. Comput.
Optim. Appl. 23, 143–169 (2002)
Griva, I., Nash, S.G., Sofer, A.: Linear and Nonlinear Optimization, 2nd edn. SIAM, Philadelphia (2009)
Gr€
otschel, M.: Optimization Stories. Documenta Mathematica, Journal der Deutschen Mathematiker-Vereinigung,
Berlin (2012)
Gu, N.Z., Mo, J.T.: Incorporating nonmonotone strategies into the trust region method for unconstrained optimization.
Comput. Math. Appl. 55, 2158–2172 (2008)
Guéret, C., Prins, C., Sevaux, M.: Applications of Optimization with Xpress-MP. Dash Optimization (2002)
Guo, Q., Liu, J.G., Wang, D.H.: A modified BFGS method and its superlinear convergence in nonconvex minimization
with general line search rule. J. Appl. Math. Comput. 28(1–2), 435–446 (2008)
Hager, W.W., Zhang, H.: A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM
J. Optim. 16(2005), 170–192 (2005)
Hager, W.W., Zhang, H.: Algorithm 851: CG-Descent, a conjugate gradient method with guaranteed descent. ACM
Trans. Math. Softw. 32(1), 113–137 (2006a)
Hager, W.W., Zhang, H.: A survey of nonlinear conjugate gradient methods. Pac. J. Optim. 2(1), 35–58 (2006b)
Han, S.P.: Superlinearly convergent variable metric algorithms for general nonlinear programming problems. Math.
Program. 11, 263–282 (1976)
Han, S.P.: A globally convergent method for nonlinear programming. J. Optim. Theory Appl. 22, 297–309 (1977)
References 781

Han, J.Y., Liu, G.H., Yin, H.X.: Convergence of Perry and Shanno’s memoryless quasi-Newton method for nonconvex
optimization problems. OR Trans. 1, 22–28 (1997)
Han, S.P., Mangasarian, O.L.: Exact penalty functions in nonlinear programming. Math. Program. 17, 251–269 (1979)
Han, X., Zhang, J., Chen, J.: A new hybrid conjugate gradient algorithm for unconstrained optimization. Bull. Iranian
Math. Soc. 43(6), 2067–2084 (2017)
Heinkenschlos, M., Ulbrich, M., Ulbrich, S.: Superlinear and quadratic convergence of affine-scaling interior-point
Newton methods for problems with simple bounds without strict complementarity assumption. Math. Program. 86,
615–635 (1999)
Hellerman, E., Rarick, D.: Reinversion with the preassigned pivot procedure. Math. Program. 1, 195–216 (1971)
Hellerman, E., Rarick, D.: The partitioned preassigned pivot procedure (P4). In: Rose, D.J., Willoughby, R.A. (eds.)
Sparse Matrices and Their Applications, pp. 67–76. Plenum Press, New York (1972)
Herskovits, J.: A view on nonlinear optimization. In: Herskovits, J. (ed.) Advances in Structural Optimization. Kluwer
Academic Publishers, Dordrecht (1995)
Hestenes, M.R.: Iterative methods for solving linear equations. J. Optim. Theory Appl. 11, 323–334 (1951)
Hestenes, M.R.: Iterative computational methods. Commun. Pure Appl. Math. 8, 85–96 (1955)
Hestenes, M.R.: The conjugate–gradient method for solving linear systems. In: Proceedings of the Sixth Symposium in
Applied Mathematics 1953, pp. 83–102. McGraw-Hill, New York (1956a)
Hestenes, M.R.: Hilbert space methods in variational theory and numerical analysis. In: Proceedings of the International
Congress of Mathematicians 1954, pp. 229–236, North-Holland, Amsterdam (1956b)
Hestens, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)
Hestenes, M.R.: Conjugate-Gradient Methods in Optimization. Springer, Berlin (1980)
Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 49,
409–436 (1952)
Himmelblau, D.M.: Applied Nonlinear Programming. McGraw-Hill, New York (1972)
Hock, W., Schittkowski, K.: Test Examples for Nonlinear Programming Codes Lecture Notes in Economics and
Mathematical Systems, vol. 187. Springer, Berlin (1981)
Hooke, R., Jeeves, T.A.: Direct search solution of numerical and statistical problems. J. Assoc. Comput. Mach. 8,
212–229 (1961)
Horst, R., Pardalos, P.M., Thoai, N.V.: Introduction to Global Optimization, 2nd edn. Kluwer Academic Publishers,
Dordrecht (2000)
Hough, P., Kolda, T., Torczon, V.: Asynchronous parallel pattern search for nonlinear optimization. SIAM J. Optim. 23,
134–156 (2001)
Hu, Y.F., Storey, C.: Global convergence result for conjugate gradient methods. J. Optim. Theory Appl. 71, 399–405
(1991)
Huang, S., Wan, Z., Chen, X.: A new nonmonotone line search technique for unconstrained optimization. Numer.
Algorithms. 68, 671–689 (2014)
Huang, H., Wei, Z., Yao, S.: The proof of the sufficient descent condition of the Wei-Yao-Liu conjugate gradient
method under the strong Wolfe-Powell line search. Appl. Math. Comput. 189, 1241–1245 (2007)
Hürlimann, T.: Mathematical Modeling and Optimization: An Essay for the Design of Computer-Based Modeling
Tools. Springer, Berlin (1999)
ILOG CPLEX 8.0: User’s Manual, ILOG SA Gentilly, France (2002)
Irizarry, R.: A generalized framework for solving dynamic optimization problems using the artificial chemical process
paradigm: Applications to particulate processes and discrete dynamic systems. Chem. Eng. Sci. 60, 5663–5681
(2005)
Jensen, D.L., Polyak, R.: The convergence of a modified barrier method for convex programming. (Research Report RC
18570, IBM Research Division, T.J. Watson Research Center, Yorktown Heights, New York) (1992)
Jiao, B.C., Chen, L.P., Pan, C.Y.: Convergence properties of a hybrid conjugate gradient method with Goldstein line
search. Math. Numer. Sin. 29(2), 137–146 (2007)
Jian, J., Han, L., Jiang, X.: A hybrid conjugate gradient method with descent property for unconstrained optimization.
Appl. Math. Comput. 39(3–4), 1281–1290 (2015)
Jittorntrum, K., Osborne, M.: A modified barrier function method with improved rate of convergence for degenerate
problems. J. Aust. Math. Soc. Ser. B. 21, 305–329 (1980)
John, F.: Extremum problems with inequalities as subsidiary conditions. In: Studies and Essays, Presented to R. Courant
on his 60th birthday, January 8, 1948, pp. 187–204. Interscience, New York (1948)
Jones, D., Perttunen, C., Stuckman, B.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory
Appl. 79, 157–181 (1993)
Kalan, J.E.: Aspects of large-scale in-core linear programming. In: Proceedings of the ACM, pp. 304–313 (1971)
Kall, P., Wallace, S.W.: Stochastic Programming. Wiley, New York (1994)
782 References

Kallrath, J., Wilson, J.M.: Business Optimization Using Mathematical Programming. Macmillan (Palgrave),
Basingstoke (1997)
Karmarkar, N.: A new polynomial time algorithm for linear programming. Combinatorica. 4, 373–395 (1984)
Karush, W.: Minima of Functions of Several Variables with Inequalities as Side Constraints. M.Sc. Dissertation,
Department of Mathematics, University of Chicago, Chicago, Illinois (1939)
Kearfott, R., Novoa, M.: INTBIS, a portable interval Newton bisection package. ACM Trans. Math. Softw. 16, 152–157
(1990)
Keller, C., Gould, N.I.M., Wathen, A.J.: Constraint preconditioning for indefinite linear systems. SIAM J. Matrix Anal.
Appl. 21, 1300–1317 (2000)
Kelley, C.T.: Iterative Methods for Linear and Nonlinear Equations. Frontiers in Applied Mathematics. SIAM,
Philadelphia (1995)
Kelley, C.T.: Iterative Methods for Optimization, No 18 in Frontiers in Applied Mathematics. SIAM Publications,
Philadelphia (1999)
Kelley, C.T., Sachs, E.W.: Local convergence of the symmetric rank one iteration. Comput. Optim. Appl. 9, 43–63
(1998)
Khalfan, H.F., Byrd, R.H., Schnabel, R.B.: A theoretical and experimental study of the symmetric rank-one update.
SIAM J. Optim. 3(1), 1–24 (1993)
Kim, K.: The effective integration of simplex and interior point thechniques. Part I: Decomposition. Part II: Null-space
affine scaling. Ph.D. Thesis, Department of Pure and Applied Mathematics, Washington State University, Pullman,
Washington (1991)
Kim, K., Nazareth, J.L.: Implementation of a primal null-space affine scaling method and its extensions. (Technical
Report 92-1, Washington State University, Pullman, Washington) (1992)
Kim, K., Nazareth, J.L.: A primal null-space affine-scaling method. ACM Trans. Math. Softw. 20(3), 373–392 (1994)
Kocvara, M., Stingl, M.: PENNON – A code for convex nonlinear and semidefinite programming. Optim. Methods
Softw. 18(3), 317–333 (2003)
Kolda, T.G., Lewis, R.M., Troczon, V.: Optimization by direct search: New perspectives on some classical and modern
methods. SIAM Rev. 45, 385–482 (2003)
Kollerstrom, N.: Thomas Simpson and “Newton’s method of approximation”: An enduring myth. Br. J. Hist. Sci. 25(3),
347–354 (1992)
Kortanek, K.O., Potra, F.A., Ye, Y.: On some efficient interior point methods for nonlinear convex programming.
Linear Algebra Appl. 152, 169–189 (1991)
Kou, C.X., Dai, Y.H.: A modified self-scaling memoryless Broyden-Fletcher-Goldfarb-Shanno method for uncon-
strained optimization. J. Optim. Theory Appl. 165, 209–224 (2015)
Kuhn, H.W., Tucker, A.W.: Nonlinear programming. In: Proceedings of 2nd Berkeley Symposium, pp. 481–492.
University of California Press, Berkeley (1951)
Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E.: Convergence properties of the Nelder-Mead simplex algorithm
in low dimensions. SIAM J. Optim. 9, 112–147 (1998)
Lalee, M., Nocedal, J., Plantenga, T.: On the implementation of an algorithm for large-scale equality constrained
optimization. SIAM J. Optim. 8, 682–706 (1998)
Lanczos, C.: An iteration method for the solution of the eigenvalue problem of linear differential and integral
operators. J. Res. Natl. Bur. Stand. 45, 252–282 (1950)
Lanczos, C.: Solution of systems of linear equations by minimized iterations. J. Res. Natl. Bur. Stand. 49, 33–53 (1952)
Lapidus, L., Luus, R.: The control of nonlinear systems: Part II: Convergence by combined first and second variations.
AIChE J. 13, 108–113 (1967)
Larrosa, J.A.E.: New heuristics for global optimization of complex bioprocesses. Ph.D. Thesis, Universidade de Vigo,
Departamento de Enxeñerı́a Quı́mica, Vigo (2008)
Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. (Mathematics and Computer Science
Division, Argonne National Laboratory, Lemont, IL 60439, USA) (2019)
Lasdon, L.S., Fox, R.L., Ratner, M.W.: Nonlinear optimization using the generalized reduced gradient method. RAIRO.
3, 73–104 (1974)
Lasdon, L.S., Waren, A.D.: Generalized reduced gradient software for linearly and nonlinearly constrained
problems. In: Greenberg, H.J. (ed.) Design and Implementation of Optimization Software, pp. 335–362. Sijthoff
and Noordhoff, Holland (1978)
Lasdon, L.S., Waren, A.D.: A survey of nonlinear programming applications. Oper. Res. 28, 34–50 (1980)
Lasdon, L.S., Waren, A.D.: GRG2 User’s Guide. (Department of General Business, School of Business Administration,
University of Texas, Austin, May 1982) (1982)
Lasdon, L.S., Waren, A.D., Jain, A., Ratner, M.: Design and testing of a generalized reduced gradient code for nonlinear
programming. ACM Trans. Math. Softw. 4, 34–50 (1978)
References 783

Lemaréchal, C.: A view of line search. In: Auslander, A., Oettli, W., Stoer, J. (eds.) Optimization and Optimal Control,
pp. 59–78. Springer, Berlin (1981)
Lemaréchal, C., Nemirovskii, A., Nesterov, Y.: New variants of bundle methods. Math. Program. 69, 111–147 (1995)
Lemke, C.E.: A method of solution for quadratic programs. Manag. Sci. 8, 442–453 (1962)
Lenstra, J.K., Rinnooy Kan, A.H.G., Schrijver, A. (eds.): History of Mathematical Programming. A Collection of
Personal Reminiscences. CWI Amsterdam and North-Holland, Amsterdam (1991)
Lescrenier, M.: Convergence of trust region algorithms for optimization with bounds when strict complementarity does
not hold. SIAM J. Numer. Anal. 28, 476–495 (1991)
Levenberg, K.: A method for the solution of certain problems in least squares. Q. Appl. Math. 2, 164–168 (1944)
Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966)
Lewis, R.M., Torczon, V., Trosset, M.W.: Direct search methods: then and now. J. Comput. Appl. Math. 124(1–2),
191–207 (2000)
Leyffer, S., Mahajan, A.: Foundations of Constrained Optimization. (Preprint ANL/MCS-P1767-0610. Argonne
National Laboratory, Mathematics and Computer Science Division, June 17) (2010)
Li, D.H., Fukushima, M.: A modified BFGS method and its global convergence in nonconvex minimization. J. Comput.
Appl. Math. 129(1–2), 15–35 (2001a)
Li, D.H., Fukushima, M.: On the global convergence of the BFGS method for nonconvex unconstrained optimization
problems. SIAM J. Optim. 11(4), 1054–1064 (2001b)
Liao, A.: Modifying BFGS method. Oper. Res. Lett. 20, 171–177 (1997)
Lin, C.-J., Moré, J.J.: Newton’s method for large bound-constrained optimization problems. SIAM J. Optim. 9,
1100–1127 (1999)
Lin, Y., Cryer, C.W.: An alternating direction implicit algorithm for the solution of linear complementarity problems
arising from free boundary problems. Appl. Math. Optim. 13, 1–7 (1985)
Liu, J.K., Li, S.J.: New hybrid conjugate gradient method for unconstrained optimization. Appl. Math. Comput. 245,
36–43 (2014)
Liu, H.W., Liu, Z.X.: An efficient Barzilai-Borwein conjugate gradient method for unconstrained
optimization. J. Optim. Theory Appl. 180(3), 879–906 (2019)
Liu, D.C., Nocedal, J.: On the limited-memory BFGS method for large optimization. Math. Program. 45, 503–528
(1989)
Liu, Y., Storey, C.: Efficient generalized conjugate gradient algorithms. Part 1: Theory. J. Optim. Theory Appl. 69,
129–137 (1991)
Liuzzi, G., Lucidi, S., Sciandrone, M.: Sequential penalty derivative-free methods for nonlinear constrained optimiza-
tion. SIAM J. Optim. 20, 2614–2635 (2010)
Livieris, I.E., Tampakas, V., Pintelas, P.: A descent hybrid conjugate gradient method based on the memoryless BFGS
update. Numer. Algorithms. 79(4), 1169–1185 (2018)
Lucia, A.: An explicit quasi-Newton update for sparse optimization calculations. Math. Comput. 40(161), 317–322
(1983)
Luenberger, D.G.: Introduction to Linear and Nonlinear Programming. Addison-Wesley Publishing Company, Reading
(1973)
Luenberger, D.G.: Introduction to Linear and Nonlinear Programming, 2nd edn. Addison-Wesley Publishing Company,
Reading (1984)
Luenberger, D.G., Ye, Y.: Linear and Nonlinear Programming. International Series in Operations Research &
Management Science 228, 4th edn. Springer, New York (2016)
Lustig, I.J., Marsten, R.E., Shanno, D.F.: The primal-dual interior point method on the Cray supercomputer. In:
Coleman, T.F., Li, Y. (eds.) Large-Scale Numerical Optimization, pp. 70–80. SIAM, Philadelphia (1990)
Lustig, I.J., Marsten, R.E., Shanno, D.F.: Computational experience with a primal-dual interior point method for linear
programming. Linear Algebra Appl. 152, 191–222 (1991)
Lustig, I.J., Marsten, R.E., Shanno, D.F.: On implementing Mehrotra’s predictor-corrector interior-point method for
linear programming. SIAM J. Optim. 2(3), 435–449 (1992)
Lustig, I.J., Marsten, R.E., Shanno, D.F.: Computational experience with a globally convergent primal-dual predictor-
corrector algorithm for linear programming. Math. Program. 66, 123–135 (1994)
Luus, R.: Picewise linear continuous optimal control by iterative dynamic programming. Ind. Eng. Chem. Res. 32,
859–865 (1993)
Mangasarian, O.L.: Nonlinear Programming. McGraw-Hill, New York. [Reprinted by SIAM Publications, 1995]
(1969)
Mangasarian, O.: Nonlinear Programming. McGraw-Hill, New York. [Reprinted by SIAM Publications, Philadelphia,
USA, 1969] (1995)
Manne, A.S.: ETA-MACRO: A model of energy-economy interactions. In: Hitch, C.J. (ed.) Modeling Energy-
Economy Interactions: Five Approaches. Resources for the Future, Washington, DC (1977)
784 References

Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. SIAM. 11(2), 431–441 (1963)
Matthews, A., Davies, D.: A comparison of modified Newton methods for unconstrained optimization. Comput. J. 14,
293–294 (1971)
Mayne, D.Q., Polak, E.: A superlinearly convergent algorithm for constrained optimization problems. Math. Program.
Stud. 16, 45–61 (1982)
Maratos, N.: Exact penalty function algorithms for finite-dimensional and control optimization problems. Ph. D. Thesis,
University of London (1978)
Marazzi, M., Nocedal, J.: Wedge trust region methods for derivative free optimization. Math. Program. Ser. A. 91,
289–305 (2002)
Markowitz, H.M.: Portofolio selection. J. Financ. 8, 77–91 (1952)
Markowitz, H.M.: The elimination form of the inverse and its application to linear programming. Manag. Sci. 3,
255–269 (1957)
McCormick, P., Ritter, K.: Alternative proofs of the convergence properties of the conjugate gradient method. J. Optim.
Theory Appl. 13(5), 497–518 (1974)
McKinney, D.C., Savitsky, A.G.: Basic optimization models for water and energy management. June 1999 (revision
6, February 2003) (2003)
Megiddo, N.: Chapter 8: Pathways to the optimal set in linear programming. In: Megiddo, N. (ed.) Progress in
Mathematical Programming: Interior-Point and Related Methods, pp. 131–158. Springer, New York (1989)
Mehrotra, S.: On the implementation of a (primal-dual) interior point method. (Technical Report 90-03, Department of
Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois 60208) (1989)
Mehrotra, S.: On the implementation of a primal-dual interior point method. SIAM J. Optim. 2, 575–601 (1992)
Meintjes, K., Morgan, A.P.: Chemical-equilibrium systems as numerical test problems. ACM Trans. Math. Softw. 16,
143–151 (1990)
Mészáros, C.: The inexact minimum local fill-in ordering algorithm. Working Paper 95-7, Computer and Automation
Institute, Hungarian Academy of Sciences, Budapest, Hungary (1995)
Meyer, C.D.: Matrix Analysis and Applied Linear Algebra. SIAM, Philadelphia (2000)
Mizuno, S., Todd, M.J., Ye, Y.: On adaptive-step primal-dual interior-point algorithms for linear programming. Math.
Oper. Res. 18, 964–981 (1993)
Moré, J.J.: Recent developments in algorithms and software for trust-region methods. In: Bachem, A., Gr€ otschel, M.,
Korte, B. (eds.) Mathematical Programming: State of the Art, pp. 258–287. Springer, Berlin (1983)
Moré, J.J., Sorensen, D.C.: On the use of direction of negative curvature in modified Newton method. Math. Program.
16, 1–20 (1979)
Moré, J.J., Sorensen, D.C.: On the use of directions of negative curvature in a modified Newton method. Math. Program.
16, 1–20 (1983)
Moré, J.J., Sorensen, D.C.: Newton’s method. In: Studies in Numerical Analysis, vol. 24 of MAA Studies in
Mathematics, pp. 29–82. The Mathematical Association of America (1984)
Moré, J.J., Thuente, D.J.: On the line search algorithms with guaranteed sufficient decrease. (Mathematics and
Computer Science Division Preprint MCS-P153-0590, Argonne National Laboratory, Argonne. IL) (1990)
Moré, J.J., Thuente, D.J.: Line search algorithms with guaranteed sufficient decrease. (Mathematics and Computer
Science Division, Preprint MCS-P330-1092, Argonne National Laboratory, October) (1992)
Moré, J.J., Thuente, D.J.: Line search algorithms with guaranteed sufficient decrease. ACM Trans. Math. Softw. 20,
286–307 (1994)
Moré, J.J., Toraldo, G.: Algorithms for bound constrained quadratic programming problems. Numer. Math. 55,
377–400 (1989)
Moré, J.J., Toraldo, G.: On the solution of large quadratic programming problems with bound constraints. SIAM
J. Optim. 1, 93–113 (1991)
Morgan, A.P.: Solving Polynomial Systems Using Continuation for Scientific and Engineering Problems. Prentice Hall,
Englewood Cliff (1987)
Murtagh, B.A., Saunders, M.A.: Large-scale linearly constrained optimization. Math. Program. 14, 41–72 (1978)
Murtagh, B.A., Saunders, M.A.: MINOS/AUGMENTED User’s Manual. (Technical Report SOL 80-14, Systems
Optimization Laboratory, Department of Operations Research, Stanford University, Stanford, California, CA
94305) (1980)
Murtagh, B.A., Saunders, M.A.: A projected Lagrangian algorithm and its implementation for sparse nonlinear
constraints. Math. Program. Study. 16, 84–117 (1982)
Murtagh, B.A., Saunders, M.A.: MINOS 5.0. User’s Guide. (Technical Report SOL 83-20, Department of Operations
Research, Stanford University, 1983. [Revised as MINOS 5.1 User’s Guide, Report SOL 83-20R, 1987]) (1987)
Murtagh, B.A., Saunders, M.A.: MINOS 5.4 User’s Guide. (Technical Report SOL 83-20R, Systems Optimization
Laboratory, Department of Operations Research, Stanford University, Stanford, California, CA 94305, February)
(1995)
References 785

Murray, W., Overton, M.L.: Steplength algorithms for minimizing a class of nondifferentiable functions. Computing.
23, 309–331 (1979)
Murty, K.G., Kabadi, S.N.: Some NP-complete problems in quadratic and nonlinear programming. Math. Program. 19,
200–212 (1987)
Naiman, A.E., Babuska, I.M., Elman, H.C.: A note on conjugate gradient convergence. Numer. Math. 76, 209–230
(1997)
NAG – The Numerical Algorithms Group Ltd: Mathematical optimization software. https://www.nag.com/
Narushima, Y., Wakamatsu, T., Yabe, H.: Extended Barzilai-Borwein method for unconstrained optimization
problems. Pac. J. Optim. 6(3), 591–614 (2008)
Narushima, Y., Yabe, H., Ford, J.A.: A three-term conjugate gradient method with sufficient descent property for
unconstrained optimization. SIAM J. Optim. 21, 212–230 (2011)
Nash, S.G.: Newton-type minimization via the Lanczos method. SIAM J. Numer. Anal. 21, 770–788 (1984a)
Nash, S.G.: User’s guide for TN/TNBC: Fortran routines for nonlinear optimization. (Report 397, Baltimore, MD:
Mathematical Sciences Department, The John Hopking University) (1984b)
Nash, S.G.: Preconditioning of truncated-Newton methods. SIAM J. Sci. Stat. Comput. 6, 599–616 (1985)
Nash, S.G.: SUMT (revisited). Oper. Res. 46, 763–775 (1998)
Nash, S.G.: A survey of truncated-Newton methods. J. Comput. Appl. Math. 124, 45–59 (2000)
Nash, S.G., Nocedal, J.: A numerical study of the limited memory BFGS method and the truncated-Newton method for
large-scale optimization. SIAM J. Optim. 1, 358–372 (1991)
Nash, S.G., Polyak, R., Sofer, A.: A numerical comparison of barrier and modified-barrier methods for large-scale
bound-constrained optimization. In: Hager, W.W., Hearn, D.W., Pardlos, P.M. (eds.) Large Scale Optimization:
State of the Art, pp. 319–338. Kluwer Academic Publishers, Dordrecht, Boston, London (1994)
Navon, I.M., Legler, D.M.: Conjugate gradient methods for large-scale minimization in meteorology. Mon. Weather
Rev. 115, 1479–1502 (1987)
Nazareth, J.L.: A conjugate direction algorithm without line search. J. Optim. Theory Appl. 23, 373–387 (1977)
Nazareth, J.L.: Conjugate gradient methods less dependent on conjugacy. SIAM Rev. 28(4), 501–511 (1986)
Nazareth, J.L.: Computer Solution of Linear Programs. Oxford University Press, New York (1987)
Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)
Nelson, C.V., Hodgkin, B.C.: Determination of magnitudes, directions and locations of two independent dipoles in a
circular conducting region from boundary potential measurements. IEEE Trans. Biomed. Eng. 28, 817–823 (1981)
Nemirovskii, A., Todd, M.: Interior point methods for optimization. Acta Numer. 17, 181–234 (2008)
Nemirovsky, A.S., Yudin, D.B.: Problem Complexity and Method Efficiency in Optimization. Wiley-Interscience
Series in Discrete Mathematics, New York (1983)
Nemhauser, G.L., Wolsey, L.A.: Integer and Combinatorial Optimization. Wiley, New York (1988)
Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia
(1994)
Ng, E., Peyton, B.W.: Block sparse Cholesky algorithms on advanced uniprocessor computers. SIAM J. Sci. Comput.
14, 1034–1056 (1993)
Ni, Q., Yuan, Y.: A subspace limited memory quasi-Newton algorithm for large-scale nonlinear bound constrained
optimization. Math. Comput. 66, 1509–1520 (1997)
Nishida, N., Liu, Y.A., Lapidus, L., Hiratsuka, S.: An effective computational algorithm for suboptimal singular and/or
bang-bang control. AICHE J. 22(3), 505–513 (1976)
Nitsche, J.C.C.: Lectures on Minimal Surfaces, vol. 1. Cambridge University Press, Cambridge, UK (1989)
Nocedal, J.: Updating quasi-Newton matrices with limited storage. Math. Comput. 35, 773–782 (1980)
Nocedal, J.: Theory of algorithms for unconstrained optimization. Acta Numer. 1, 199–242 (1992)
Nocedal, J.: Conjugate gradient methods and nonlinear optimization. In: Adams, L., Nazareth, J.L. (eds.) Linear and
Nonlinear Conjugate Gradient-Related Methods, pp. 9–23. SIAM, Philadelphia (1996)
Nocedal, J., Wächter, A., Waltz, R.A.: Adaptive barrier strategies for nonlinear interior methods. (Technical Report RC
23563, IBM Watson Research Center, Yorktown Heights, NY, USA) (2005)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research, 2nd edn. Springer Science +
Business Media, New York (2006)
Nocedal, J., Yuan, Y.X.: Analysis of self-scaling quasi-Newton method. Math. Program. 61, 19–37 (1993)
Omojokun, E.: Trust region algorithms for optimization with nonlinear equality and inequality constraints. Ph.D.
Dissertation, Department of Computer Science, University of Colorado (1989)
O’Neill, R.: Algorithm AS 47: Function minimization using a simplex procedure. Appl. Stat. 20(3), 338–345 (1971)
Oren, S.S. Self-scaling variable metric algorithms for unconstrained optimization. Ph.D. Thesis, Department of
Engineering-Economic Systems, Stanford University, Stanford (1972)
Oren, S.S.: Self-scaling variable metric algorithm. Part II. Manag. Sci. 20, 863–874 (1974)
786 References

Oren, S.S., Luenberger, D.G.: Self-scaling variable metric (SSVM) algorithms, part I: criteria and sufficient conditions
for scaling a class of algorithms. Manag. Sci. 20, 845–862 (1974)
Oren, S.S., Spedicato, E.: Optimal conditioning of self-scaling variable metric algorithm. Math. Program. 10, 70–90
(1976)
Ortega, J.M., Rheinboldt, W.C.: Iterative Solutions of Nonlinear Equations in Several Variables. SIAM Classics in
Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (2000)
Ou, Y., Liu, Y.: A memory gradient method based on the nonmonotone technique. J. Ind. Manag. Optim. 13(2),
857–872 (2017)
Ou, Y., Lin, H.: A class of accelerated conjugate-gradient-like methods based on a modified secant equation. J. Ind.
Manag. Optim. 16(3), 1503–1518 (2020)
Panier, E.R., Tits, A.L.: Avoiding the Maratos effect by means of a nonmonotone line search. I: General constrained
problema. SIAM J. Numer. Anal. 28, 1183–1195 (1991)
Pant, M., Thangaraj, R., Singh, V.P.: Particle swarm optimization with crossover operator and its engineering
applications. IAENG Int. J. Comput. Sci. 36, 112–121 (2009)
Papadimitriou, C.H., Steiglitz, K.: Combinatorial Optimization: Algorithms and Complexity. Prentice Hall, Englewood
Cliffs (1982)
Pardalos, P.M., Rosen, J.B.: Constrained Global Optimization: Algorithms and Applications. Springer, Berlin (1987)
Peressini, A.L., Sullivan, F.E., Uhl, J.J.: The Mathematics of Nonlinear Programming. Springer, New York Inc (1988)
Perry, A.: A modified conjugate gradient algorithm. (Discussion Paper No. 229, Center for Mathematical Studies in
Economics and Management Science, Northwestern University) (1976)
Perry, A.: A class of conjugate gradient algorithms with two step variable metric memory. (Discussion paper
269, Center for Mathematical Studies in Economics and Management Science. Northwestern University, IL,
USA) (1977)
Peyret, R., Taylor, T.D.: Computational Methods for Fluid Flow. Springer, New York (1985)
Polak, E., Ribiére, G.: Note sur la convergence de méthods de direction conjugées. Revue Francaise d’Informatique et
de Recherche Opérationnelle. 16, 35–43 (1969)
Polyak, B.T.: The conjugate gradient method in extremal problems. USSR Comput. Math. Math. Phys. 9, 94–112
(1969)
Polyak, R.: Modified barrier functions (theory and methods). Math. Program. 54, 177–222 (1992)
Potra, F.A., Rheinboldt, W.C.: On the monotone convergence of Newton’s method. Computing. 36(1), 81–90 (1986)
Potra, F.A., Shi, Y.: Efficient line search algorithm for unconstrained optimization. J. Optim. Theory Appl. 85, 677–704
(1995)
Powell, M.J.D.: An efficient method for finding the minimum of a function of several variables without calculating
derivatives. Comput. J. 17, 155–162 (1964)
Powell, M.J.D.: A method for nonlinear constraints in optimization problems. In: Fletcher, R. (ed.) Optimization,
pp. 283–297. Academic Press, New York (1969)
Powell, M.J.D.: Convergence properties of a class of minimization algorithms. In: Mangasarian, O., Meyer, R.,
Robinson, S. (eds.) Nonlinear Programming 2, pp. 1–27. Academic Press, New York (1975)
Powell, M.J.D.: Some global convergence properties of a variable-metric algorithm for minimization without exact line
searches. In: Cottle, R.W., Lemke, C.E. (eds.) Nonlinear Programming, SIAM-AMS Proceedings, vol. 9, pp. 53–72,
Philadelphia, PA, USA (1976)
Powell, M.J.D.: Restart procedures of the conjugate gradient method. Math. Program. 2, 241–254 (1977)
Powell, M.J.D.: A fast algorithm for nonlinearly constrained optimization calculations. In: Watson, G.A. (ed.) Numeri-
cal Analysis, Dundee 1977, Lecture Notes in Mathematics, vol. 630, pp. 144–157. Springer, Berlin (1978a)
Powell, M.J.D.: The convergence of variable metric methods of nonlinearly constrained optimization calculations. In:
Mangasarian, O.L., Meyer, R.R., Robinson, S.M. (eds.) Nonlinear Programming 3, pp. 27–63. Academic Press,
New York (1978b)
Powell, M.J.D.: Algorithms for nonlinear constraints that use Lagrangian functions. Math. Program. 14, 224–248
(1978c)
Powell, M.J.D.: ZQPCVX: A Fortran subroutine for convex quadratic programming. (Technical Report, Department of
Applied Mathematics and Theoretical Physics, Cambridge University) (1983)
Powell, M.J.D.: Nonconvex minimization calculations and the conjugate gradient method. In: Griffiths, D.F. (ed.)
Numerical Analysis (Dundee, 1983), Lecture Notes in Mathematics, vol. 1066, pp. 122–141 (1984)
Powell, M.J.D.: How bad are the BFGS and DFP methods when the objective function is quadratic? Math. Program. 34,
34–47 (1986a)
Powell, M.J.D.: Convergence properties of algorithms for nonlinear optimization. SIAM Rev. 28(4), 487–500 (1986b)
Powell, M.J.D.: Updating conjugate directions by the BFGS formula. Math. Program. 38, 693–726 (1987)
References 787

Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear
interpolation. (Department of Applied Mathematics and Theoretical Physics, University of Cambridge (revised
August 1993)) (1993)
Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear
interpolation. In: Gomez, S., Hennart, J.P. (eds.) Advances in Optimization and Numerical Analysis, pp. 51–67.
Kluwer Academic Publishers (1994)
Powell, M.J.D.: Direct search algorithms for optimization calculations. Acta Numer. 7, 287–336 (1998)
Powell, M.J.D.: UOBYQA: Unconstrained optimization by quadratic approximation. Math. Program. 92, 555–582
(2002)
Powell, M.J.D.: On trust region methods for unconstrained minimization without derivatives. Math. Program. 97,
605–623 (2003)
Powell, M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. (Department of Applied
Mathematics and Theoretical Physics, Centre for Mathematical Sciences, Cambridge) (2004)
Powell, M.J.D.: The NEWUOA software for unconstrained minimization without derivatives. In: Di Pillo, G., Roma,
M. (eds.) Large-Scale Nonlinear Optimization, pp. 255–297. Springer (2006)
Powell, M.J.D.: The BOBYQA algorithm for bound constrained optimization without derivatives. (Technical Report,
Department of Applied Mathematics and Theoretical Physics, Cambridge, England) (2009)
Powell, M.J.D.: On the convergence of trust region algorithms for unconstrained minimization without derivatives.
(Technical Report, Department of Applied Mathematics and Theoretical Physics, Cambridge, England) (2011)
Powell, M.J.D., Toint, P.L.: On the estimation of sparse Hessian matrices. SIAM J. Numer. Anal. 16, 1060–1074 (1979)
Pulkkinen, S.: A review of methods for unconstrained optimization: Theory, implementation and testing. Master’s
Thesis, University of Helsinki, Department of Mathematics and Statistics (2008)
Price, W.L.: Global optimization by controlled random search. J. Optim. Theory Appl. 55(1983), 333–348 (1983)
Price, C.J., Coope, D., Byatt, D.: A convergent variant of the Nelder-Mead algorithm. J. Optim. Theory Appl. 113, 5–19
(2002)
Ratschek, H., Rokne, J.: A circuit design problem. J. Global Optim. 3, 501 (1993)
Raydan, M.: On the Barzilai and Borwein choice of steplength for the gradient method. IMA J. Numer. Anal. 13,
321–326 (1993)
Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM
J. Optim. 7, 26–33 (1997)
Raydan, M., Svaiter, B.F.: Relaxed steepest descent and Cauchy-Barzilai-Borwein method. Comput. Optim. Appl. 21,
155–167 (2002)
Reid, J.K.: Sparse in core linear programming. In: Watson, G. (ed.) Numerical Analysis Dundee, 1975, Lecture Notes in
Mathematics, 506, pp. 176–189 (1975)
Reid, J.K.: A sparsity-exploiting variant of the Bartels-Golub decomposition for linear programming bases. Math.
Program. 24, 55–69 (1982)
Rijckaert, M.J.: Engineering applications of geometric programming. In: Avriel, M., Rijckaert, M.J., Wilde, M. (eds.)
Optimization in Design. Prentice-Hall, Englewood Cliffs (1973)
Rios, L.M., Shainidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software
implementations. J. Glob. Optim. 56, 1247–1293 (2013)
Robinson, S.M.: A quadratically convergent algorithm for general nonlinear programming problems. Math. Program. 3,
145–156 (1972)
Robinson, S.M.: Perturbed Kuhn-Tucker points and rates of convergence for a class of nonlinear programming
algorithms. Math. Program. 7, 1–16 (1974)
Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
Rose, D.J., Tarjan, R.E.: Algorithmic aspects of vertex elimination on graphs. In: Proceedings of the 7th Annual
Symposium on the Theory of Computing, pp. 245–254. Association for Computing Machinery, New York (1975)
Rosen, J.B., Kreuser, J.: A gradient projection algorithm for nonlinear constraints. In: Lootsma, F. (ed.) Numerical
Methods for Nonlinear Optimization, pp. 297–300. Academic Press, London (1972)
Rosen, J.B.: The gradient projection method for nonlinear programming. Part I – linear constraints. J. SIAM Appl.
Math. 8(1), 181–217 (1960)
Rosen, J.B.: The gradient projection method for nonlinear programming. Part II – nonlinear constraints. J. SIAM Appl.
Math. 9(4), 414–432 (1961)
Rosenbrock, H.H.: An automatic method for finding the greatest or least value of a function. Comput. J. 3, 175–184
(1960)
Rosser, J.B.: Rapidly converging iterative methods for solving linear equations. In: Paige, L.J., Taussky, O. (eds.)
Simultaneous Linear Equations and the Determination of Eigenvalues. Applied Mathematics Series 29, pp. 59–64.
National Bureau of Standards, U.S. Government Printing Office, Washington, DC (1953)
788 References

Rothberg, E., Hendrickson, B.: Sparse matrix ordering methods for interior-point linear programming. (Technical
Report SAND96-0475J, Sandia National Laboratory) (1996)
Sainvitu, C., Toint, Ph.L.: A filter-trust-region method for simple-bound constrained optimization. (Technical Report,
Department of Mathematics, University of Namur, Belgium) (2006)
Sandgren, E.: Nonlinear integer and discrete programming in mechanical design. In: Proceedings of the ASME Design
Technology Conference, pp. 95–105, Kissimme (1988)
Sandgren, E., Ragsdell, K.M.: The utility of nonlinear programming algorithms: A comparative study – Part I. J. Mech.
Des. 102(3), 540–546 (1980)
Sargent, E., Murtagh, B.A.: Projection methods for nonlinear programming. Math. Program. 4, 245–268 (1973)
Saunders, M.: Augmented Lagrangian methods. (Notes 9. Stanford University, Management Science & Engineering.
Spring 2015) (2015a)
Saunders, M.: NPSOL and SNOPT – SQP Methods. (Notes 11. Stanford University, Management Science & Engineer-
ing. Spring 2015) (2015b)
Schlick, T.: Modified Cholesky factorizations for sparse preconditioners. SIAM J. Sci. Comput. 14, 424–445 (1993)
Schlick, T., Fogelson, A.: TNPACK – A truncated Newton minimization package for large-scale problems: I Algorithm
and usage. ACM Trans. Math. Softw. 18, 46–70 (1992a)
Schlick, T., Fogelson, A.: TNPACK – A truncated Newton minimization package for large-scale problems: II
implementation examples. ACM Trans. Math. Softw. 18, 71–111 (1992b)
Schittkowski, K.: The nonlinear programming method of Wilson, Han and Powell with an augmented Lagrangean type
line search function. Part I: Convergence analysis. Numer. Math. 38, 83–114 (1981)
Schittkowski, K.: On the convergence of a sequential quadratic programming method with an augmented Lagrange line
search function. Math. Operationsforschung Statistik, Ser. Optim. 14(2), 197–216 (1983)
Schittkowski, K.: NLPQL: A Fortran subroutine for solving constrained nonlinear programming problems. Ann. Oper.
Res. 5(2), 485–500 (1985)
Schittkowski, K.: NLPQL: A Fortran subroutine solving constrained nonlinear programming problems. Ann. Oper. Res.
5, 485–500 (1986)
Schittkowski, K.: More Test Examples for Nonlinear Programming Codes. Springer, Berlin (1987)
Schittkowski, K.: NLPQLP: A Fortran implementation of a sequential quadratic programming algorithm. User’s guide.
(Report, Department of Mathematics, University of Bayreuth) (2002)
Schittkowski, K.: QL: A Fortran code for convex quadratic programming. User’s guide, Version 2.11. (Technical
Report, Department of Mathematics, University of Bayreuth, July) (2005)
Schittkowski, K.: NLPQO: A Fortran implementation of a sequential quadratic programming algorithm with distributed
and non-monotone line search. User’s guide, version 3.0. (Technical Report, Department of Computer Science,
University of Bayreuth) (2009)
Schittkowski, K.: A robust implementation of a sequential quadratic programming algorithm with successive error
restoration. (Technical Report, Department of Computer Science, University of Bayreuth) (2010)
Schnabel, R.B., Chow, T.T.: Tensor methods for unconstrained optimization using second derivatives. SIAM J. Optim.
1, 293–315 (1991)
Schnabel, R.B., Frank, P.D.: Tensor methods for nonlinear equations. SIAM J. Numer. Anal. 21, 815–843 (1984)
Schreiber, R., Keller, H.: Driven cavity flows by efficient numerical techniques. J. Comput. Phys. 49, 310–333 (1983)
Schultz, G.A., Schnabel, R.B., Byrd, R.H.: A family of trust-region-based algorithms for unconstrained minimization
with strong global convergence properties. SIAM J. Numer. Anal. 22, 47–67 (1985)
Shacham, M.: Numerical solution of constrained nonlinear algebraic equations. Int. J. Numer. Methods Eng. 23,
1455–1481 (1986)
Shanno, D.F.: Conditioning of quasi-Newton methods for function minimization. Math. Comput. 24, 647–656 (1970)
Shanno, D.F.: Conjugate gradient methods with inexact searches. Math. Oper. Res. 3, 244–256 (1978a)
Shanno, D.F.: On the convergence of a new conjugate gradient algorithm. SIAM J. Numer. Anal. 15, 1247–1257
(1978b)
Shanno, D.F.: CONMIN – A Fortran subroutine for minimizing an unconstrained nonlinear scalar valued function of a
vector variable x either by the BFGS variable metric algorithm or by a Beale restarted conjugate gradient algorithm.
Private communication, October 17, 1983
Shanno, D.F.: Globally convergent conjugate gradient algorithms. Math. Program. 33, 61–67 (1985)
Shanno, D.F.: Who invented the interior-point method? In: Gr€ otschel, M. (ed.) Optimization stories. Documenta
Mathematica, Journal der Deutschen Methematiker-Vereinigung, Extra volume, 21st International Symposium on
Mathematical Programming, pp. 55–64, Berlin, August 19–24, 2012
Shanno, D.F., Breitfeld, M.G., Simantiraki, E.M.: Implementing barrier methods for nonlinear programming. In:
Terlaky, T. (ed.) Interior Point Methods of Mathematical Programming, pp. 399–414. Kluwer Academic Publishers
(1996)
References 789

Shanno, D.F., Phua, K.H.: Algorithm 500. Minimization of unconstrained multivariable functions. ACM Trans. Math.
Softw. 2, 87–94 (1976)
Shanno, D.F., Phua, K.H.: Matrix conditioning and nonlinear optimization. Math. Program. 14, 149–160 (1978)
Shanno, D.F., Phua, K.H.: Remark on algorithm 500. ACM Trans. Math. Softw. 6, 618–622 (1980)
Shanno, D.F., Simantiraki, E.M.: Interior-point methods for linear and nonlinear programming. In: Duff, I.S., Watson,
G.A. (eds.) The State of the Art in Numerical Analysis, pp. 339–362. Oxford University Press, New York (1997)
Shi, Z.-J.: Convergence of line search methods for unconstrained optimization. Appl. Math. Comput. 157, 393–405
(2004)
Shi, Z.-J., Shen, J.: Step-size estimation for unconstrained optimization methods. Comput. Appl. Math. 24(3), 399–416
(2015)
Sorensen, D.C.: Newton’s method with a model trust region modification. SIAM J. Numer. Anal. 19, 409–426 (1982)
Spall, J.C.: Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control. Wiley Series in
Discrete Mathematics and Optimization. Wiley, New York (2003)
Spellucci, P.: An SQP method for general nonlinear programs using only equality constrained subproblems. Math.
Program. 3, 413–448 (1998)
Spendley, W., Hext, G.R., Himsworth, F.R.: Sequential applications of simplex designs in optimization and evolution-
ary operation. Technometrics. 4, 441–461 (1962)
Spedicato, E., Zhao, J.: Explicit general solution of the Quasi-Newton equation with sparsity and symmetry. Optim.
Methods Softw. 2(3–4), 311–319 (1993)
Stewart, G.W.: A modification of Davidon’s method to accept difference approximation of derivatives. ACM. 14,
72–83 (1967)
Steihaug, T.: The conjugate gradient method and trust-regions in large-scale optimization. SIAM J. Numer. Anal. 20,
626–637 (1983)
Stiefel, E.L.: Kernel polynomials in linear algebra and their numerical applications. In: Further Contributions to the
Determination of Eigenvalues. National Bureau of Standards, Applied Mathematical Series, 49, pp. 1–22 (1958)
Stoer, J., Yuan, Y.X.: A subspace study on conjugate gradient algorithms. ZAMM – J. Appl. Math. Mech. 75, 69–77
(1995)
Strogatz, S.H.: Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering.
Perseus Books, Cambridge, MA (1994)
Suhl, U.H., Suhl, L.M.: Computing sparse LU factorizations for large-scale linear programming bases. ORSA
J. Comput. 2, 325–335 (1990)
Suhl, L.M., Suhl, U.H.: A fast LU-update for linear programming. (Arbeitspapier des Instituts fur Wirtschaftsin-
formatik, Freie Universitaet – Berlin, August) (1991)
Sun, W., Liu, H., Liu, Z.: A class of accelerated subspace minimization conjugate gradient methods. J. Optim. Theory
Appl. 190, 811–840 (2021)
Sun, W., Yuan, Y.X.: Optimization Theory and Methods. Nonlinear Programming. Springer Science + Business Media,
New York (2006)
Surry, P.D., Radcliffe, N.J., Boyd, I.D.: A multi-objective approach to constrained optimization of gas supply networks:
The COMOGA method. In: Fogarty, T.C. (ed.) Evolutionary Computing: AISB Workshop, Number 993. Lecture
Notes in Computer Science, pp. 166–180. Springer, Berlin (1995)
Tapia, R.A.: A stable approach to Newton’s method for general mathematical programming problems in ℝn. J. Optim.
Theory Appl. 14, 453–476 (1974)
Tits, A.L., Wächter, A., Bakhtiari, S., Urban, T.J., Lawrence, C.T.: A primal-dual interior-point method for nonlinear
programming with strong global and local convergence properties. SIAM J. Optim. 14, 173–199 (2003)
Toint, P.L.: On sparse and symmetric matrix updating subject to a linear equation. Math. Comput. 31, 954–961 (1977)
Toint, P.L.: Towards an efficient sparsity exploiting Newton method for minimization. In: Sparse Matrices and Their
Uses, pp. 57–87. Academic Press, New York (1981)
Toint, P.L.: An assessment of nonmonotone line search techniques for unconstrained optimization. SIAM J. Sci.
Comput. 17, 725–739 (1996)
Toint, P.L.: A nonmonotone trust-region algorithm for nonlinear optimization subject to convex constraints. Math.
Program. 77, 69–94 (1997)
Topkis, D.M., Veinott, A.F.: On the convergence of some feasible direction algorithms for nonlinear programming.
SIAM J. Control. 5, 268–279 (1967)
Touati-Ahmed, D., Storey, C.: Efficient hybrid conjugate gradient techniques. J. Optim. Theory Appl. 64, 379–397
(1990)
Traub, J.: Iterative Methods for the Solution of Equations. Prentice-Hall, Englewood Cliffs (1964)
Trefethen, L.N., Bau III, D.: Numerical Linear Algebra. SIAM, Philadelphia (1997)
Trefethen, L.N., Schreiber, R.S.: Average-case stability of Gaussian elimination. SIAM J. Matrix Anal. Appl. 11(3),
335–360 (1990)
790 References

Tr€oltzsch, A.: Benchmarking of bound-constrained optimization software. (CERFACS working note: WN/PA/07/143,
pp. 1–39) (2007)
Ulbrich, S.: On the superlinear local convergence of a filter-SQP method. Math. Program. 100(1), 217–245 (2004)
Ulbrich, M., Ulbrich, S., Heinkenschlos, M.: Global convergence of affine-scaling interior-point Newton methods for
infinite-dimensional nonlinear problems with pointwise bounds. SIAM J. Control. Optim. 37, 731–764 (1999)
Ulbrich, M., Ulbrich, S., Vicente, L.N.: A globally convergent primal-dual interior-point filter method for nonconvex
nonlinear programming. Math. Program. 100, 379–410 (2004)
Vanderbei, R.J.: ALPO: Another linear program optimizer. (Technical Report AT&T Bell Laboratories) (1990)
Vanderbei, R.J.: An interior point code for quadratic programming. (Technical Report SOR 94-15, Princeton Univer-
sity) (1994)
Vanderbei, R.J.: LOQO: An interior point code for quadratic programming. (Technical Report SOR 94-15, Princeton
University) (1995)
Vanderbei, R.: Linear Programming Foundations and Extensions. Kluwer Academic Publishers, Boston, London,
Dordrecht (1996)
Vanderbei, R.J.: Linear Programming: Foundations and Extensions, 2nd edn. Springer, New York (2001)
Vanderbei, R.J., Shanno, D.F.: An interior-point algorithm for nonconvex nonlinear programming. (Technical Report
SOR 97-21, Princeton University) (1997)
Vanderbei, R.J., Shanno, D.F.: An interior point algorithm for nonconvex nonlinear programming. Comput. Optim.
Appl. 13, 231–252 (1999)
Vanderplaats, G.N.: DOT Users Manual. Version 4.20, Vanderplaats Research & Development, Inc, Colorado Springs
(1995)
Van der Pol, B.: Forced oscillations in a circuit with nonlinear resistance (receptance with reactive triode). Lond. Edinb.
Dublin Philos. Mag. 3, 65–80 (1927)
Van der Vorst, H.A.: Lecture Notes on Iterative Methods. (Report Mathematical Institute, University of Utrecht) (1993)
Vardi, A.: A trust region algorithm for equality constrained minimization: convergence properties and implementation.
SIAM J. Numer. Anal. 22, 575–591 (1985)
Vassiliadis, V.S., Floudas, C.A.: The modified barrier function approach for large-scale optimization. Comp. Chem.
Engng. 21(8), 855–874 (1997)
Vavasis, S.A.: Quadratic programming is NP. Inf. Process. Lett. 36, 73–77 (1990)
Vavasis, S.A.: Nonlinear Optimization. Oxford University Press, New York and Oxford (1991)
Verschelde, J., Verlinden, P., Cools, R.: Homotopies exploiting Newton polytopes for solving sparse polynomial
systems. SIAM J. Numer. Anal. 31, 915–930 (1994)
Von Stryk, O.: User’s guide for DIRCOL (Version 2.1): A direct collocation method for the numerical solution of
optimal control problems. (Technical Report, Technische Universität München) (1999)
Wächter, A.: An interior point algorithm for large scale nonlinear optimization with applications in process engineering.
Ph.D. Thesis, Carnegie Mellon University, Pittsburg, PA, January 2002
Wächter, A., Biegler, L.T.: On the implementation of an interior-point filter line-search algorithm for large-scale
nonlinear programming. Math. Program. 106, 25–57 (2000)
Wächter, A., Biegler, L.T.: Line search filter methods for nonlinear programming: Motivation and global convergence.
(Technical Report RC 23036, Yorktown Heights, NY: IBM T.J. Watson Research Center (revised 2004)) (2001)
Wächter, A., Biegler, L.T.: Line search filter methods for nonlinear programming: Motivation and global convergence.
SIAM J. Optim. 16, 1–31 (2005a)
Wächter, A., Biegler, L.T.: Line search filter methods for nonlinear programming: Local convergence. SIAM J. Optim.
16, 32–48 (2005b)
Wächter, A., Biegler, L.T.: On the implementation of an interior-point filter line-search algorithm for large-scale
nonlinear programming. Math. Program. 106, 25–57 (2006)
Walsh, G.R.: Methods of Optimization. Wiley, London, New York, Sydney, Toronto (1975)
Waltz, R.A.: KNITRO 4.0 User’s Manual. (Technical Report, Ziena Optimization Inc., Evanston, IL, October) (2004)
Waltz, R.A., Morales, J.L., Nocedal, J., Orban, D.: An interior algorithm for nonlinear optimization that combines line
search and trust region step. (Technical Report 2003-6, Optimization Technology Center, Northwestern University,
Evanston, IL, USA, June). [Math. Program. 107, 2006, 391–408] (2003)
Wan, Z., Huang, S., Zheng, X.D.: New cautious BFGS algorithm based on modified Armijo-type line search. J. Inequal.
Appl. 241, 1–10 (2012)
Wan, Z., Teo, K.L., Shen, X.L., Hu, C.M.: New BFGS method for unconstrained optimization problem based on
modified Armijo line search. Optimization. 63(2), 285–304 (2014)
Wang, H.J., Yuan, Y.X.: A quadratic convergence method for one-dimensional optimization. Chin. J. Oper. Res. 11,
1–10 (1992)
Wei, Z., Li, G., Qi, L.: New nonlinear conjugate gradient formulas for large-scale unconstrained optimization problems.
Appl. Math. Comput. 179, 407–430 (2006a)
References 791

Wei, Z., Yao, S., Liu, L.: The convergence properties of some new conjugate gradient methods. Appl. Math. Comput.
183, 1341–1350 (2006b)
Wei, Z., Yu, G., Yuan, G., Lian, Z.: The superlinear converence of a modified BFGS-type method for unconstrained
optimization. Comput. Optim. Appl. 29, 315–332 (2004)
Wilkinson, J.H.: The Algebraic Eigenvalue Problem. Oxford University Press, London (1965)
Wilson, R.B.: A simplicial algorithm for concave programming. Ph.D Thesis, Harvard University (1963)
Wolfe, P.: A duality theorem for nonlinear programming. Q. Appl. Math. 19, 239–244 (1961)
Wolfe, P.: Methods of nonlinear programming. In: Abadie, J. (ed.) Nonlinear Programming, pp. 97–131. North-
Holland, Amsterdam (1967)
Wolfe, P.: Convergence conditions for ascent methods. SIAM Rev. 11, 226–235 (1969)
Wolfe, P.: Convergence conditions for ascent methods. II: Some corrections. SIAM Rev. 13, 185–188 (1971)
Wolsey, L.A.: Integer Programming. Wiley-Interscience Series in Discrete Mathematics and Optimization. Wiley,
New York (1998)
Wright, M.H.: Numerical methods for nonlinearly constrained optimization. (SLAC Report No.193, 1976, Stanford
University, California. (Ph.D. Dissertation)) (1976)
Wright, M.H.: Direct search methods: Once scorned, now respectable. In: Numerical Analysis 1996 (Proceedings of the
1995 Dundee Bienal Conference in Numerical Analysis), pp. 191–208. Addison Wesley Longman (1996)
Wright, S.J.: Implementing proximal point methods for linear programming. J. Optim. Theory Appl. 65, 531–554
(1990)
Wright, M.H.: Interior methods for constrained optimization. Acta Numer. 1, 341–407 (1991)
Wright, S.J.: Primal-Dual Interior-Point Methods. SIAM, Philadelphia (1997)
Wu, G., Liang, H.: A modified BFGS method and its convergence. Comput. Model. New Technol. 18(11), 43–47 (2014)
Xavier, A.E.: Hyperbolic penalty: A new method for nonlinear programming with inequalities. Int. Trans. Oper. Res. 8,
659–671 (2001)
Yabe, H., Martı́nez, H.J., Tapia, R.A.: On sizing and shifting the BFGS update within the sized Broyden family of
secant updates. SIAM J. Optim. 15(1), 139–160 (2004)
Yabe, H., Ogasawara, H., Yoshino, M.: Local and superlinear convergence of quasi-Newton methods based on modified
secant conditions. J. Comput. Appl. Math. 205, 717–632 (2007)
Yamashita, H.: A globally convergent primal-dual interior-point method for constrained optimization. Optim. Methods
Softw. 10, 443–469 (1998)
Yamashita, N.: Sparse Quasi-Newton Updates with Positive Definite Matrix Completion. Department of Applied
Mathematics of Physics, Graduate School of Informatics, Kyoto University (2005)
Yao, S., Wei, Z., Huang, H.: A note about WYL’s conjugate gradient method and its application. Appl. Math. Comput.
191, 381–388 (2007)
Yang, X., Luo, Z., Dai, X.: A global convergence of LS-CD hybrid conjugate gradient method. Adv. Numer. Anal.
2013, 517452 (2013). https://doi.org/10.1155/2013/517452
Yang, E.K., Tolle, J.W.: A class of methods for solving large convex quadratic programs subject to box constraints.
Math. Program. 51, 223–228 (1991)
Ye, Y.: Interior Point Algorithms: Theory and Analysis. Wiley, New York (1997)
Ye, Y., Todd, M.J., Mizuno, S.: An O((nl)1/2)-iterations homogeneous and self-dual linear programming algorithm.
Math. Oper. Res. 19, 53–67 (1994)
Ypma, T.J.: Historical development of the Newton-Raphson method. SIAM Rev. 37, 531–551 (1995)
Yuan, Y.X.: A modified BFGS algorithm for unconstrained optimization. IMA J. Numer. Anal. 11, 325–332 (1991)
Yuan, Y.X.: Analysis on the conjugate gradient method. Optim. Methods Softw. 2, 19–29 (1993)
Yuan, Y.: A new stepsize for the steepest descent method. J. Comput. Math. 24, 149–156 (2006)
Yuan, Y.: Advances in trust region algorithms. Math. Program. 151, 249–281 (2015)
Yuan, Y.X., Byrd, R.: Non-quasi-Newton updates for unconstrained optimization. J. Comput. Math. 13(2), 95–107
(1995)
Yuan, G., Sheng, Z., Wang, B., Hu, W., Li, C.: The global convergence of a modified BFGS method for nonconvex
functions. J. Comput. Appl. Math. 327, 274–294 (2018)
Yuan, G., Wei, Z.: Convergence analysis of a modified BFGS method on convex minimizations. Comput. Optim. Appl.
47, 237–255 (2010)
Yuan, G., Wei, Z., Lu, X.: Global convergence of BFGS and PRP methods under a modified weak Wolfe–Powell line
search. Appl. Math. Model. 47, 811–825 (2017)
Zangwill, W.I.: The convex simplex method. Manag. Sci. 14(3), 221–238 (1967)
Zhang, Y.: Interior-point gradient methods with diagonal-scaling for simple-bound constrained optimization. (Techni-
cal Report TR04-06, Department of Computational and Applied Mathematics, Rice University, Houston, Texas)
(2004)
Zhang, L.: Two modified Dai-Yuan nonlinear conjugate gradient methods. Numer. Algorithms. 50(1), 1–16 (2009a)
792 References

Zhang, L.: New versions of the Hestenes-Stiefel nonlinear conjugate gradient method based on the secant condition for
optimization. Comput. Appl. Math. 28(1), 1–23 (2009b)
Zhang, J., Deng, N.Y., Chen, L.H.: New quasi-Newton equation and related methods for unconstrained optimization. J
Optim. Theory Appl. 102, 147–167 (1999)
Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization.
SIAM J. Optim. 14, 1043–1056 (2004)
Zhang, Y., Tapia, R.A., Dennis, J.E.: On the superlinear and quadratic convergence of primal-dual interior-point linear
programming algorithms. SIAM J. Optim. 2, 304–324 (1992)
Zhang, J., Xu, C.: Properties and numerical performance of quasi-Newton methods with modified quasi-Newton
equations. J. Comput. Appl. Math. 137, 269–278 (2001)
Zhang, L., Zhou, W.: Two descent hybrid conjugate gradient methods for optimization. J. Comput. Appl. Math. 216,
251–264 (2008)
Zhang, L., Zhou, W., Li, H.: Some descent three-term conjugate gradient methods and their global convergence. Optim.
Methods Softw. 22(4), 697–711 (2007)
Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B, FORTRAN subroutines for large scale bound
constrained optimization. ACM Trans. Math. Softw. 23, 550–560 (1997)
Zhu, M., Nazareth, J.L., Wolkowicz, H.: The quasi-Cauchy relation and diagonal updating. SIAM J. Optim. 9(4),
1192–1204 (1999)
Zhu, H., Wen, S.: A class of generalized quasi-Newton algorithms with superlinear convergence. Int. J. Nonlinear Sci.
2(3), 140–146 (2006)
Zou, X., Navon, I.M., Berger, M., Phua, K.H., Schlick, T., Le Dimet, F.X.: Numerical experience with limited-memory
quasi-Newton and truncated Newton methods. SIAM J. Optim. 3(3), 582–608 (1993)
Zoutendijk, G.: Methods of Feasible Directions. Elsevier, Amsterdan (1960)
Zoutendijk, G.: Nonlinear programming, computational methods. In: Abadie, J. (ed.) Integer and Nonlinear Program-
ming, pp. 38–86. North-Holland, Amsterdam (1970)
Zwart, P.B.: Nonlinear Programming: A Quadratic Analysis of Ridge Paralysis. Washington University, Report
COO-1493-21. St. Louis (1969)
 Author Index

A Beale, E.M.L., 246, 260, 450

Abadie, J., 15, 569, 579, 597 Bebernes, J., 762
Adjiman, C.S., 722, 723 Benedek, R., 764
Al-Baali, M., 204, 206, 208, 253, 260, 278, 280, 287 Benson, H.Y., 302, 659, 661
Alguacil, N., 8 Ben-Tal, A., 11, 490, 492, 499, 519
Alonso, A.A., 749 Berger, M., 314
Andersen, E.D., 474, 642 Bergman, L., 557
Andersen, K.D., 474, 642 Bertsekas, D.P., 9, 19, 22, 382, 383, 409, 411, 415–417,
Andreani, R., 518 419, 484, 485, 490, 516
Andrei, N., 1, 3, 6, 9, 13, 15–17, 19, 22, 68, 75, 77, 78, 80, Betts, J., 642
88, 92, 95, 97, 98, 100, 101, 104, 106, 107, 133, Bicanic, N., 168
135, 146, 157, 163, 165, 169, 181, 183, 184, 213, Biegler, L.T., 18, 299, 314, 437, 544, 567, 606, 633, 642,
214, 217–220, 222, 225, 229, 231, 235–238, 241, 645, 661, 663–667, 669, 671–673, 678, 749
245, 250, 253, 256, 259, 260, 267, 272, 278, 280– Biggs, M.C., 278–280, 282, 544
283, 288–290, 302, 304, 313, 314, 355, 359, 365, Birge, J.R., 11
366, 369, 370, 378, 380, 382, 383, 420, 430, 452, Birgin, E., 16, 378, 411, 416, 417, 419, 437, 494, 518
461, 473, 475, 488, 494, 501, 502, 506, 510, 516, Bisschop, J., 6, 8
519, 567, 576, 577, 579, 593, 597, 611, 615, 621, Boggs, P.T., 313, 375, 566
624, 625, 644, 679, 721–729, 731, 732, 735, 741, Bondarenko, A.S., 737
756, 764, 765 Bongartz, I., 5
Antoniou, A., 80 Bonnans, J., 556, 567
Aris, R., 120, 762 Bortz, D.M., 737
Armijo, L., 38, 68, 380 Borwein, J.M., 53, 280, 416, 417
Arrow, K.J., 511 Boyd, I.D., 659
Arzam, M.R., 278 Boyd, S., 19, 382, 383
Averick, B.M., 13, 120, 121, 430, 723, 724, 757, 759, 761, Breitfeld, M.G., 488, 490, 493, 495, 497, 499, 614
764 Brent, R.P., 369
Axelsson, O., 181 Brooke, A., 6, 145
Brown, A.A., 15, 501
B Broyden, C.G., 264, 273
Babaie-Kafaki, S., 107, 220, 278, 285, 286 Buckley, A.G., 314
Babuska, I.M., 184 Bulirsch, R., 69
Bakhtiari, S., 661 Bunch, J.R., 160, 162, 450, 604
Balakrishnan, V., 382 Byatt, D., 357
Ballard, D.H., 729 Byrd, R.H., 16–18, 269–271, 278, 279, 282, 285, 287,
Banga, J.R., 749 297, 299, 300, 302, 309, 312–314, 330, 340, 353,
Bartels, R.H., 543 374, 378, 411, 416, 417, 421–424, 426, 427, 437,
Bartholomew-Biggs, M.C., 9, 15, 19, 22, 80, 382, 444, 494, 542, 557, 558, 561, 563, 566, 631, 633–635,
454, 455, 501, 719, 731 637, 639, 642, 643, 661–663
Bartlett, R., 567
Barzilai, J., 53, 280, 416, 417 C
Bau III, D., 719 Carolan, W.J., 510
Bazaraa, M.S., 9, 19, 22, 382, 383, 409, 716, 719 Cauchy, A., 9, 58
Bărbulescu, M., 473 Carpentier, J., 569, 579, 597

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 793
Switzerland AG 2022
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
794 Author Index

Carter, R.G., 13, 120, 121, 430, 723, 724, 757, 759, 761, Drud, A., 17, 567, 590, 591, 593, 594, 597
764, 765 Duran, M., 651
Castillo, E., 8
Cesari, L., 736 E
Chachuat, B.C., 383, 409 Eberly, D., 762
Chan, S.C., 302 Ecker, J.G., 15
Chen, J., 220 Eisenstat, S.C., 315, 322, 330
Chen, H., 260, 278, 285, 302 El-Bakry, A.S., 611, 614, 622, 661
Chen, L.P., 219, 220 Eldersveld, S.K., 544, 546, 642
Chen, X., 50 El Ghaoui, L., 11, 382
Cheng, S.H., 604 Elman, H.C., 184
Cheng, W.Y., 280, 282 Esposito, W.R., 722, 723
Chin, C.M., 374, 648
Choi, T.D., 369 F
Chow, T.T., 15 Facchinei, F., 141, 411, 416, 437
Cimatti, G., 759 Feron, E., 382
Cohen, A., 202 Fiacco, A.V., 141, 157, 159, 300, 376, 490, 599, 601, 645,
Coleman, T.F., 415, 416, 536, 566 685
Colombani, Y., 6 Finkel, D.E., 370
Concus, P., 327 Fletcher, R., 12, 18, 68, 80, 141, 157, 160, 171, 201, 208,
Conejo, A.J., 8 249, 264, 272, 309, 313, 314, 353, 374, 375, 379,
Conn, A.R., 5, 9, 17, 299, 300, 302, 311, 314, 330, 353, 381, 450, 453, 454, 474, 482, 526, 536, 544, 566,
369, 370, 378, 379, 382, 411, 416, 421, 437, 499, 567, 599, 610, 647, 648, 650, 651, 653–659, 661,
500, 518, 519, 536, 543, 566, 643, 644 678
Contreras, M., 278 Floudas, C.A., 10, 519, 722, 723
Cools, R., 724 Fogelson, A., 9, 326, 330, 416
Coope, D., 357 Ford, J.A., 260, 286
Cottle, R.W., 409 Forsgren, A., 141, 376, 603, 644
Courant, R., 516 Forsythe, G.E., 169
Crowder, H.P., 202 Fourer, R., 6, 8, 146
Cryer, C.W., 759 Fox, R.L., 579
Curtis, A., 712 Frank, M., 15, 570
Curtis, F.E., 330 Frank, P.D., 15, 642
Cuthrell, J.E., 749 Freeman, T.L., 141, 157, 160
Friedlander, A., 416
D Friedlander, M.P., 378
Dai, X., 220 Frisch, K.R., 490, 599, 645
Dai, Y.H., 15, 46, 49, 54, 172, 188, 192, 195, 196, 198, Fukushima, M., 285, 287, 536
199, 201, 202, 208, 210, 218–220, 232, 245,
253–255, 259, 260, 287, 304 G
Dai, Z., 219, 220 Gabay, D., 566
Daniel, J.W., 201, 459 Gabriele, G.A., 579
Dantzig, G.B., 1, 9, 504 Garcı́a-Bertrand, R., 8
Davidon, W.C., 14, 80, 264, 313 Garner, J., 764
Davies, D., 141 Gay, M., 6, 146, 352, 510, 611
Debreu, G., 487 Ge, R.-P., 313
Dehmiry, A.H., 278, 287 George, A., 605
Dembo, R.S., 315, 322, 415, 428, 728, 731, 732 Ghanbari, R., 220, 278, 286
Demmel, J.W., 704, 719 Gilbert, J., 201, 213, 219–221, 254, 288, 292, 312, 314,
Denchfield, A., 106 567, 631, 634, 661
Dener, A., 106 Gill, P.E., 9, 12, 17, 22, 80, 141, 143, 144, 157, 159, 278,
Deng, N.Y., 260, 278, 285, 556 288, 309, 314, 327, 375, 376, 382, 383, 429, 453,
Dennis, J.E., 9, 68, 75, 80, 165, 273, 288, 313, 353, 382, 459, 474, 507, 519, 542, 544, 546, 549–551, 566,
416, 417, 615, 622, 712, 719 567, 603, 643, 644, 699, 700, 712
Deuflhard, P., 315 Glowinski, R., 757
Dikin, I.I., 9, 644 G€
odel, K., 3
Dixon, L.C.W., 330 Goldfarb, D., 141, 264, 450, 456–459, 593, 611
Dolan, E.D., 13, 78, 420, 502, 735–737 Goldfeld, S.M., 141
Author Index 795

Goldstein, A.A., 40, 68, 141 Hulbert, L.A., 415

Golub, G., 186, 327, 459, 467, 562, 704, 719 Hürlimann, T., 6, 8
Goncalves, A.S., 450
Goodman, J., 759, 760 I
Gould, N.I.M., 5, 9, 17, 80, 299, 300, 302, 311, 314, 330, Idnani, A., 450, 456–459
353, 374, 378, 379, 382, 411, 416, 417, 421, 437, Irizarry, R., 754
450, 474, 487, 499, 500, 518, 519, 542, 557, 558,
561–563, 566, 643, 644, 659 J
Graggs, W.B., 459 Jeeves, T.A., 369
Grandinetti, L., 278, 287 Jelinek, C.A., 729
Greenbaum, A., 181, 184 Jensen, D.L., 499
Griewank, A., 12, 314 Jain, A., 579
Griffith, R.E., 15, 373 Jian, J., 219, 220
Grippo, L., 48, 54, 416, 417, 556 Jiang, X., 219, 220
Griva, I., 9, 22, 54, 80, 333, 409, 719 Jiao, B.C., 219, 220
Grothey, A., 314 Jittorntrum, K., 490
Grossmann, I.E., 651 John, F., 409
Gr€
otschel, M., 19 Jones, D., 370
Gu, N.Z., 50 Johnson, K., 168
Guéret, C., 474 Júdice, J., 416
Guerrero, G., 15
Guigou, J., 579 K
Gümüs, Z.H., 722, 723 Kabadi, S.N., 474
Guo, Q., 286 Kalan, J.E., 591
Kall, P., 11
H Kallrath, J., 6
Hager, W.W., 15, 44, 46, 49, 50, 54, 68, 78, 208, 231, 233, Karmarkar, N., 9, 382, 644
234, 238, 259, 260, 411, 415, 417, 437 Karush, W., 409
Haggag, A., 579 Kaufman, L., 162, 450, 459, 604
Han, J.Y., 210, 220, 234 Kearfott, R., 722
Han, L., 219, 220 Keller, C., 562
Han, S.P., 479, 480, 524, 528, 544, 548, 566 Keller, H., 122
Han, X., 220 Kelley, C.T., 80, 155, 168, 181, 300, 312, 369, 382, 411,
Harding, S.T., 722, 723 413, 414, 437, 721
Heinkenschlos, M., 416 Kendrick, D., 6, 145
Heipcke, S., 6 Kennington, J.L., 510
Hellerman, E., 509, 591 Kernighan, B.W., 6, 146
Hendrickson, B., 605 Khalfan, H.F., 299, 300, 302
Heng-Bin, A., 330 Kim, K., 9
Herskovits, J., 566 Klepeis, J.L., 722, 723
Hestenes, M.R., 9, 169, 171, 172, 201, 280, 482, 511, 517 Kocvara, M., 519
Hext, G.R., 369 Kohn, R., 759, 760
Higham, N.J., 604 Kolda, T., 355, 370, 686
Hill, J.E., 510 Kollerstrom, N., 168
Himmelblau, D.M., 723 Kortanek, K.O., 611
Himsworth, F.R., 369 Kou, C.X., 15, 46, 245, 253–255, 259, 260
Hiratsuka, S., 754 Kreuser, J., 503, 518
Hock, W., 624, 727–729, 731 Kristjansson, B., 6
Hodgkin, B.C., 724 Kuhn, H.W., 409
Hooke, R., 369 Kungurtsev, V., 141
Horst, R., 10 Kupferschmid, M., 15
Hough, P., 370
Hribar, M.E., 18, 299, 314, 353, 562, 563, 631, 634, 637, L
642, 661 Lagarias, J.C., 357
Hu, C.M., 278, 286 Lalee, M., 539, 544
Hu, W., 278, 287 Lam, W.H., 302
Hu, Y.F., 218, 219, 221, 222 Lampariello, F., 48, 416, 417, 556
Huang, H., 219, 220 Lanczos, C., 169
Huang, S., 50, 278, 286 Lapidus, L., 738, 754
796 Author Index

Larrosa, J.A.E., 750 M

Larson, J., 22, 355, 689 Mahajan, A., 371, 375, 380, 382
Lasdon, L.S., 579 Mangasarian, O.L., 9, 409, 479, 480
Lawrence, C.T., 661 Manne, A.S., 557
Le Dimet, F.X., 314 Maratos, N., 379, 534
Legler, D.M., 259 Marazzi, M., 370, 633
Lemaréchal, C., 68, 80, 288, 292, 312, 314, 567, 659 Markowitz, H.M., 474, 519, 591
Lemke, C.E., 450 Marsten, R.E., 9, 613
Lenir, A., 314 Marquardt, D.W., 141, 352
Lenstra, J.K., 18 Martı́nez, H.J., 16, 278, 378, 411, 416, 417, 419, 437, 494,
Leonard, M.W., 278, 314 518
Lescrenier, M., 416 Matthews, A., 141
Levenberg, K., 141, 352 Mayne, D.Q., 536, 566
Levitin, E.S., 415 McCormick, G.P., 141, 157, 159, 202, 300, 376, 490, 599,
Lewis, J.G., 642 601, 645, 685
Lewis, R.M., 355, 370, 686 McKinney, D.C., 740, 744, 745, 748
Leyffer, S., 18, 314, 353, 371, 375, 379–382, 482, 544, Mead, R., 9, 16, 355, 369, 679
566, 567, 599, 610, 647, 648, 650, 651, 654–659, Meeraus, A., 6, 8, 145
661, 678 Megiddo, N., 558, 639
Li, C., 278, 287 Mehrotra, S., 165, 465
Li, D.H., 202, 280, 282, 285, 287 Meintjes, K., 723
Li, G., 219, 220, 283, 285 Menchi, O., 759
Li, H., 220 Menickelly, M., 22, 355, 689
Li, S.J., 227 Mészáros, C., 605
Li, Y., 416 Meyer, C.A., 722, 723
Liao, A., 278, 281, 282 Meyer, C.D., 719
Liao, L.Z., 54, 172, 202, 232, 304 Minguez, R., 8
Lian, Z., 278, 283 Mizuno, S., 9
Liang, H., 278 Moghrabi, I.A., 286
Lin, C.-J., 16, 107, 351, 352, 411, 416, 437, 759 Mo, J.T., 50
Lindskog, G., 181 Morales, J.L., 18, 631, 633, 642
Liu, D.C., 9, 279, 292, 294, 312, 314, 327, 556, 608 Moré, J.J., 9, 13, 16, 68, 75, 78, 120, 121, 141, 273, 292,
Liu, G., 662, 663 313, 327, 333, 344, 351–353, 378, 411, 415, 416,
Liu, G.H., 210, 220, 234 420, 421, 430, 437, 502, 699, 723, 724, 735–737,
Liu, H.W., 54, 107 757, 759, 761, 764, 765
Liu, J.G., 286 Morgan, A.P., 723, 724
Liu, J.K., 227 Mu, C., 141
Liu, J.W.-H., 605 Munson, T., 106, 502, 735–737
Liu, L., 219, 220 Murray, W., 9, 12, 17, 22, 80, 141, 143, 144, 157, 159,
Liu, S., 611 288, 309, 314, 327, 375, 383, 429, 453, 459, 474,
Liu, Y., 51, 201, 214 507, 519, 542, 544, 546, 549–551, 566, 567, 643,
Liu, Y.A., 754 699, 700, 712
Liu, Z., 107 Murtagh, B.A., 17, 378, 475, 488, 503–505, 509, 511,
Liu, Z.X., 54 512, 514, 515, 518, 519, 544, 557, 643
Liuzzi, G., 18, 680, 684–688 Murty, K.G., 474
Livieris, I.E., 253
Louveaux, F., 11 N
Lu, P., 16, 299, 314, 378, 411, 416, 417, 421, 424, 426, Naiman, A.E., 184
427, 437, 494 Narushima, Y., 54, 260, 286
Lu, X., 278, 287 Nash, S.G., 9, 15, 16, 22, 54, 80, 255, 315, 326, 327, 329,
Lucia, A., 309 330, 333, 378, 409, 416, 417, 428–430, 437, 490,
Lucidi, S., 18, 48, 141, 411, 416, 417, 437, 556, 680, 684– 494, 645, 719
688 Navon, I.M., 259, 314
Luenberger, D.G., 9, 19, 22, 253, 278, 280, 281, 382, 383, Nazareth, J.L., 9, 260, 288
394, 409, 576, 577 Nelder, J.A., 9, 16, 355, 369, 679
Luo, Z., 220 Nelson, C.V., 724
Lustig, I.J., 9, 613 Nemirovskii, A., 11, 88, 376, 377, 659
Luus, R., 738, 749 Nemhauser, G.L., 10
Author Index 797

Nesterov, Y., 377, 659 Q

Ng, E., 605 Qi, L., 219, 220, 283, 285
Niemi, S., 510 Quandt, R.E., 141
Nishida, N., 754
Ni, Q., 219, 416 R
Nitsche, J.C.C., 764 Radcliffe, N.J., 659
Nocedal, J., 9, 16–19, 22, 75, 80, 195, 201, 204, 213, 219– Ragsdell, K.M., 579, 724
221, 253–255, 259, 269–271, 278–282, 287, 291, Raman, R., 6, 145
292, 294, 297, 299, 309, 312–314, 323, 325, 327, Rarick, D., 509, 591
330, 331, 343, 344, 346, 349, 351–353, 370, 374, Ratner, M.W., 579
378, 381–383, 409, 411, 416, 417, 421–424, 426, Ratschek, H., 725
427, 437, 448, 449, 452, 463, 465, 474, 477, 479, Raydan, M., 16, 49, 54, 93, 95, 378, 411, 416, 417, 419,
482, 483, 494, 516, 534, 539, 542, 544, 556–558, 437, 494, 556
561–563, 566, 599, 603, 605, 606, 608, 627, 629, Reeds, J.A., 357
631–635, 637, 639, 642–644, 661–663, 685, 719 Reeves, C.M., 171, 201
Novoa, M., 722 Reid, J.K., 591, 593, 712
Reyna, L., 759, 760
O Rezaee, S., 107
Ogasawara, H., 278 Rheinboldt, W.C., 164, 168
O’Leary, D.P., 327 Ribiére, G., 171, 201, 210
Omojokun, E., 381, 539 Rijckaert, M.J., 728
O’Neill, R., 366, 367, 370 Rinnooy Kan, A.H.G., 18
Orban, D., 18, 450, 518, 566, 631, 633, 642, 644 Rios, L.M., 22, 355, 370, 689
Oren, S.S., 253, 278, 280, 281, 327 Ritter, K., 202
Ortega, J.M., 164 Robinson, D.P., 141, 374
Osborne, M., 490 Robinson, S.M., 503, 511, 518, 525
Ou, Y., 51, 107 Rockafellar, R.T., 11
Overton, M.L., 80, 611 Roelofs, M., 6
Rokne, J., 725
P Rose, D.J., 605
Palagi, L., 416, 437 Rosen, J.B., 10, 15, 415, 503, 518, 569, 597
Pan, C.Y., 219, 220 Rosenbrock, H.H., 369
Panier, E.R., 556 Rosenthal, R.E., 6, 145
Pant, M., 462 Rosser, J.B., 169
Papadimitriou, C.H., 10 Rothberg, E., 605
Pardalos, P.M., 10, 722, 723
Parlett, B.N., 160, 604 S
Pedregal, P., 8 Sachs, E.W., 300
Peressini, A.L., 382, 383, 719 Sagastizábal, C., 567
Perry, A., 233, 244, 245, 248, 252, 259 Sainvitu, C., 417
Perttunen, C., 370 Sainz de la Maza, E., 374, 566
Peyret, R., 748 Sandgren, E., 724, 726
Peyton, B.W., 605 Santos, S.A., 416
Phua, K.H., 75, 234, 259, 278, 314 Sargent, E., 511
Pintelas, P., 253 Sartenaer, A., 518
Plantenga, T., 539, 544 Saunders, M.A., 17, 309, 375, 378, 459, 474, 475, 488,
Polak, E., 171, 201, 210, 536, 566 503–505, 509, 511, 512, 514–516, 518, 519, 542,
Polyak, R., 171, 201, 210, 414, 415, 490, 499 544, 546, 549–551, 557, 566, 567, 643
Potra, F.A., 68, 168, 611 Savitsky, A.G., 740, 744, 745, 748
Powell, M.J.D., 16, 18, 68, 80, 201, 202, 204, 210, 213, Scheinberg, K., 369, 370
232, 234, 247, 259, 260, 264, 270, 279, 313, 352, Schinzinger, R., 729
353, 355, 358, 369, 370, 417, 461, 482, 500, 511, Schittkowski, K., 17, 54, 461, 500, 533, 544, 553–555,
517, 524, 528, 534, 544, 547, 555, 566, 679, 680, 566, 624, 724, 728–731
684, 712 Schlick, T., 9, 314, 326, 330, 416
Price, C.J., 141, 357 Schmid, C., 544
Price, R.C., 330 Schnabel, R.B., 9, 15, 68, 75, 80, 297, 299, 300, 302, 309,
Price, W.L., 729 312, 313, 340, 353, 382, 417, 421–423, 566, 615,
Prins, C., 474 622, 712, 719
Pulkkinen, S., 80 Schrage, L., 6
798 Author Index

Schreiber, R., 122, 704, 719 Thangaraj, R., 462

Schrijver, A., 18 Thoai, N.V., 10
Schultz, G.A., 340, 353 Thuente, D.J., 68, 75, 292, 327
Schuverdt, M., 518 Tits, A.L., 556, 661
Schweiger, C.A., 722, 723 Todd, M.J., 9, 376
Sciandrone, M., 18, 54, 680, 684–688 Toint, Ph.L., 5, 9, 17, 299, 300, 302, 309, 311, 314, 330,
Sevaux, M., 474 353, 370, 378, 379, 382, 411, 416, 417, 421, 437,
Shacham, M., 722 450, 474, 500, 518, 519, 556, 566, 643, 644, 648,
Shainidis, N.V., 22, 355, 370, 698 651, 659, 712
Shanno, D.F., 9, 68, 75, 233, 234, 244–247, 249, 250, 252, Tolle, J.W., 313, 375, 415, 566
259, 260, 264, 277, 278, 302, 307, 488, 490, 493, Topkis, D.M., 571
495, 497, 499, 611, 613, 614, 632, 633, 642, 645, Toraldo, G., 378, 415, 421, 759
659, 661 Torczon, V., 355, 370, 686
Shen, C., 659 Touati-Ahmed, D., 218, 219, 222
Shen, J., 80 Tsuchiya, T., 611, 614, 622, 661
Shen, X.L., 278, 286 Traub, J., 164
Sheng, Z., 278, 287 Trefethen, L.N., 704, 719
Shi, Y., 68 Tr€
oltzsch, A., 437
Shi, Z.-J., 63, 80 Trosset, M.W., 370
Sherali, H.D., 9, 19, 22, 382, 383, 409, 716, 719 Trotter, H.F., 141
Shetty, C.M., 9, 19, 22, 382, 383, 409, 716, 719 Tucker, A.W., 409
Shultz, G.A., 566 Tulowitzki, U., 415
Simantiraki, E.M., 614
Singh, R.P., 749 U
Singh, V.P., 462 Uhl, J.J., 382, 383, 719
Soares, J., 416 Ulbrich, M., 416, 644, 645, 659, 661
Sofer, A., 9, 22, 54, 80, 333, 409, 490, 719 Ulbrich, S., 416, 644, 645, 659, 661
Sollow, R.M., 511 Urban, T.J., 661
Sorensen, D.C., 141, 344, 352, 699
Spall, J.C., 11 V
Spedicato, E., 278, 309, 327 Vandenberghe, L., 19, 383
Spellucci, P., 474, 544 Vanderbei, R.J., 1, 9, 474, 504, 611, 613, 614, 632, 633,
Spendley, W., 369 642, 645, 659, 661
Steiglitz, K., 10 Vanderplaats, G.N., 437
Steihaug, T., 315, 322, 340, 429 Van der Pol, B., 755
Stewart, G.W., 154, 459 Van der Vorst, H.A., 181
Stewart, R.A., 15, 373 Van Loan, C.F., 186, 467, 562, 704, 719
Stiefel, E., 169, 171, 172, 201, 280 Vardi, A., 375
Stingl, M., 519 Vassiliadis, V.S., 519
Stoer, J., 69, 238 Vavasis, S.A., 9, 474
Storey, C., 201, 214, 218, 219, 221, 222 Veinott, A.F., 571
Strakoš, Z., 184 Verlinden, P., 724
Strogatz, S.H., 755 Verschelde, J., 724
Stuckman, B., 370 Vicente, L.N., 416, 644, 645, 659, 661
Suhl, L.M., 591 Von Stryk, O., 737
Suhl, U.H., 591
Sullivan, F.E., 382, 383, 719 W
Sun, D.F., 210, 220 Wächter, A., 18, 299, 314, 437, 606, 632, 633, 642, 644,
Sun, W., 9, 19, 22, 80, 107, 141, 157, 168, 187, 382, 383, 645, 659, 661, 663–667, 669, 671–673, 678
409, 411, 474, 603 Wakamatsu, T., 54
Surry, P.D., 659 Walker, H.F., 330
Svaiter, B.F., 93, 95 Wallace, S.W., 11
Walsh, G.R., 80
T Waltz, R.A., 17, 18, 374, 542, 557, 558, 561, 563, 566,
Tampakas, V., 253 631–633, 635, 638, 639, 642, 643
Tapia, R.A., 165, 278, 548, 566, 611, 614, 622, 661 Wan, Z., 50, 278, 286
Tarjan, R.E., 605 Wang, B., 278, 287
Taylor, T.D., 740 Wang, D.H., 286
Teo, K.L., 278, 286 Wang, H.J., 279
Author Index 799

Wang, S., 611 Yang, E.K., 415

Waren, A.D., 579 Yang, X., 220
Wathen, A.J., 562 Yao, S., 219, 220
Wei, Z., 219, 220, 278, 283, 285, 287 Ye, Y., 9, 19, 382, 409, 576, 577, 611, 614
Wen, F., 219, 220 Yin, H.X., 210, 220, 234
Wen, S., 278 Yoshino, M., 278
Wild, S.M., 22, 355, 689 Ypma, T.J., 168
Wilkinson, J.H., 278 Yu, G., 278, 283, 287
Wilson, J.M., 8, 566 Yuan, G., 278, 283, 287
Wilson, R.B., 524 Yuan, Y.X., 9, 19, 22, 80, 141, 157, 168, 187, 188, 192,
Wochmann, S.J., 510 198, 201, 202, 208, 210, 218–220, 238, 253, 260,
Wolfe, P., 15, 41, 68, 189, 202, 407, 504, 569, 570, 576, 269–271, 278–282, 285, 313, 352, 382, 383, 409,
597 411, 416, 474, 603
Wolkowicz, H., 288 Yudin, D.B., 88
Wolsey, L.A., 10 Yuzefovich, I., 490, 492
Wright, J., 141
Wright, M.H., 9, 12, 22, 80, 309, 357, 369, 376, 377, 382, Z
383, 474, 514, 515, 519, 533, 544, 546, 551, 567, Ze-Yao, M., 330
603, 611, 614, 644, 699, 700, 712 Zhang, H., 15, 44, 46, 49, 50, 54, 68, 78, 208, 231, 233,
Wright, P.E., 357 234, 238, 259, 260, 411, 415, 417, 437
Wright, S.J., 9, 19, 22, 75, 80, 204, 291, 309, 323, 325, Zhang, J., 220, 260, 278, 280, 285
331, 343, 344, 346, 349, 351, 352, 381–383, 409, Zhang, L., 220
411, 415, 448, 449, 452, 463, 465, 474, 477, 479, Zhang, Y., 165, 416, 566, 611, 614, 622, 661
482, 483, 516, 534, 539, 566, 599, 603, 605, 606, Zangwill, W.I., 15, 569, 578, 597
608, 617, 627, 629, 644, 685, 719 Zhao, J., 309
Wu, G., 278 Zheng, X.D., 278, 286
Wu-Sheng, L., 80 Zhou, C., 141
Zhou, F.J., 556
X Zhou, J.L., 556
Xavier, A.E., 519 Zhou, W., 220
Xiao, Y., 556 Zhu, C., 16, 299, 314, 378, 411, 416, 417, 421, 424, 426,
Xing-Ping, L., 330 427, 437
Xu, C., 278, 280 Zhu, H., 278
Xue, G.L., 13, 120, 121, 430, 723, 724, 757, 759, 761, Zhu, M., 288
764, 765 Zibulevsky, M., 490, 492, 499, 519
Zou, X., 314
Y Zoutendijk, G., 15, 57, 58, 189, 203, 569, 570, 597
Yabe, H, 54, 260, 278, 286 Zwart, P.B., 141
Yamashita, H., 309, 661
 Subject Index

A Bratu’s problem, 120

Active Broyden class, 267
constraint, 390 Broyden-Fletcher-Goldfarb-Shanno (BFGS), 261, 264,
set, 390 265
methods, 571 memoryless, 277
Accelerated steepest descent algorithm, 98 with modified line-search, 286
Acceptable line-search, 30 with modified secant equation, 283
Accumulation point, 707
Accuracy of algorithms, 12 C
A-conjugate directions, 170 Cauchy point, 337, 469
Advantages of the Newton method, 140 quadratic programming simple bounds, 460
Algebraic characterization of a sequence, 707
descent direction, 387 trust-region, 337
feasible direction, 390 Cauchy-Schwarz inequality, 692
tangent space, 395 Cautious BFGS, 285
Algebraic modeling languages, 6 Cayley-Hamilton theorem, 705
ALGENCAN, 518 Central finite-difference formula, 147
Angle between two nonzero vectors, 692 Central path, 464
Angle condition, 48, 54 Centroid, 356
A-orthogonal directions, 170 CD-method (conjugate gradient) (Fletcher), 208
Approximate Wolfe line-search, 44 CG-DESCENT (conjugate gradient), 233, 252
Armijo line-search, 38 Chance-constrained optimization, 11
Augmented Lagrange function, 482 Chebyshev polynomials, 179
Augmented Lagrangian methods, 377, 482 Circuit design, 120, 150, 166, 725
Automatic differentiation, 712 Circularity danger, 3
Cholesky factorization, 143, 698
B modified, 699
Backtracking, 46, 47, 68 Closed set, 715
Balance between function and constraints, 394 COBYLA (direct search), 680
Banach lemma, 695 Combinatorial difficulty, 392
Barzilai-Borwein line-search, 53 Compact representation of the L-BFGS updating, 296,
Barrier method, 600 422
Basic assumptions (conjugate gradient), 189 Compactifying matrix, 310
Basic matrix (GRG), 579 Compact set, 715
Basis, 691 Comparison of TN versus
Binding conjugate gradient algorithms, 325
constraint, 404 L-BFGS, 327
simple bound constraint, 404 Complementarity
Biobjective optimization problem, 648 measure, 464
Bolzano-Weierstrass theorem, 707 slackness, 391
Bound-constrained Lagrangian methods, 378 Complexity analysis, 12
Bounded deterioration property, 273 Composite Newton method
Bounded set, 715 of order 1, 164
Bracketing phase in line-search, 68 of order m, 164

# The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature 801
Switzerland AG 2022
N. Andrei, Modern Numerical Nonlinear Optimization, Springer Optimization and Its Applications 195,
https://doi.org/10.1007/978-3-031-08720-2
802 Subject Index

Computational sciences, 4 Determinant of different BFGS updatings, 706

Computer science, 4 Deterministic optimization, 11
Conditioning and stability, 704 Diagonal updating of the Hessian, 287
Condition number, 704 Differentiable manifold, 394
Conjugate directions, 170 Dilemma, 5
Conjugate gradient methods memoryless BFGS Direction of negative curvature, 157
preconditioned, 244 Direct search method
CONMIN (conjugate gradient), 245, 249 constrained optimization, 679
CONOPT, 567, 587, 589 unconstrained optimization, 355
linear mode, 593 Disadvantages of the Newton method, 140
nonlinear mode, 593 Discarding the inactive constraints, 402
Consistency of quasi-Newton method, 273 Discrete optimization, 9
Constrained Dogleg method, 338
optimization, 9 Dual
qualification, 392 algorithm for quadratic programming, 457
Constraint, 1 feasibility, 391
normal, 397 problem, 406
Continuous optimization, 9 Duality, 406
Convergence from remote starting points, 377 Dynamic restart strategy, 255
Convergence of the MM-SR1gen method, 305 DYNAMIC (quadratic programming application), 462
Convex
function, 10, 716 E
nonconvex, 439 Efficiency of algorithms, 12, 14, 78
program, 386 Eigenvalues, 697
programming, 11 Eigenvalues in tangent space, 399
quadratic program, 439 Elastic-plastic torsion application, 294, 430, 757
equality constraints, 467 Elastic programming, 543
set, 10, 716 Element functions, 310
strictly convex, 439 Elements of
Convex simplex method (Zangwill), 578 analysis, 707
Criticism of the penalty and augmented Lagrangian applied numerical linear algebra, 691
methods, 486 convexity, 716
Crossover technique, 638 topology in the Euclidian space, 715
Curvature condition (line-search), 42, 262 Ellipsoid norm, 693
Errors in functions, gradients and hessians, 154
D Euclidean norm, 692
Dai and Kou line-search, 46 ETA-MACRO application (NLPQLP), 557
Dai-Liao conjugate gradient method, 232 Exact line-search, 30, 37
Damped Newton method, 129 Expansion (Nelder-Mead), 356
Day-Yuan method (conjugate gradient), 208
Debreu theorem, 488 F
DEEPS algorithm (direct search), 359, 361 Factorization of the full KKT system (quadratic
reduction phase, 363 programming), 440
stalling phase, 363 Farkas theorem, 718
DFG (direct search), 684 Feasible
Degeneracy, 404 direction, 389
Dennis-Moré condition, 274 method (Zoutendijk), 569
Departure from linearity, 503 domain, 1, 383
Derivative-free interior-point methods, 609
constrained optimization, 679 solution, 383
unconstrained optimization, 355 Filter
DFP (Davidon-Fletcher-Powell), 261, 263, 264 algorithm
DK/CGOPT (conjugate gradient), 251, 253, 254 sequential linear programming, 649, 650
Descent sequential quadratic programming, 658
condition, 29 envelope, 648
direction, 22, 188, 387 methods, 379, 647
DESCON (conjugate gradient), 236 with line search, 665
accelerated, 240 Finite-difference approximation, 146, 710
Determinant of a matrix, 704 Finite termination property, 175
Subject Index 803

First-order Grippo-Lampariello-Lucidi line-search, 48

necessary conditions, 396 Gu-Mo line-search, 50
optimality conditions, 2, 25, 387
sufficient conditions, 388 H
First-and second-order necessary conditions, 403 Hager-Zhang
Fletcher-Reeves (conjugate gradient), 171, 202 conjugate gradient, 78
Flow in a drive cavity, 121 line-search, 44
Forcing sequence, 315, 316, 322 Hessian, 22, 125, 708
Forward finite-difference formula, 146 Hestenes-Stiefel method (conjugate gradient), 75, 171, 213
Fraction to the boundary rule, 602, 665 Hypersurface, 394
Frank-Wolfe Method, 570 H€
older
Frobenius norm, 693, 700 continuity, 27, 117
Function inequality, 692
bounded, 707 Homotopy methods, 600
continuous, 707 Huang-Wan-Chen line-search, 50
first derivative, 708 Hybrid conjugate gradient methods, 219
uniformly continuous, 707 convex combination, 218, 225
second derivative, 708 projection concept, 218
Fundamental Hybrid convex combination of LS and DY, 227
ratio, 88
theorem of calculus, 23 I
theorem of linear algebra, 694 Inactive constraint, 402
Indefinite quadratic programming, 450
G Implicit function theorem, 709
GALAHAD, 518 Improved Wolfe line-search, 46, 255
Gaussian Elimination (LU Factorization), 697 Incomplete Cholesky factorization, 186
with complete pivoting, 698 Inertia and singularity, 605
with partial pivoting, 698 Inertia correction and regularization, 606, 671
Gauss-Newton method, 124 Inexact
General barrier, 489 line-search, 30, 38
General nonlinear Newton method, 315, 316
conjugate gradient algorithm, 202 Infeasible
memoryless BFGS preconditioned, 244 constraints, 543
optimization, 1 point, 372
General penalty-barrier function (SPENBAR), 490 Infimum, 383
Generalized Inhomogeneous Superconductors 1-D Ginzburg-Landau
Cauchy point, 352 application, 296, 436, 764
computation, 426 Inverse of a matrix, 694
reduced gradient method, 579, 580, 586 Inverse of some BFGS updatings, 695
secant equation, 302 IPOPT, 662
Wolfe line-search, 43 feasibility restoration phase, 672
Geometric necessary condition, 389, 395 primal-dual barrier approach, 662
Gill and Murray method, 143 Inside contraction (Nelder-Mead), 356
Global Integer programming, 10
convergence analysis, 12 Interior point algorithm
convergence of BFGS, 270 line-search, 610
maximum, 385 prototype, 602
minimization, 22, 385 Interior point methods, 375, 376, 599
solution, 10
Globalization strategy, 381 J
Goldfeld, Quandt and Trotter method, 141 Jacobian matrix, 27, 109, 124
Goldstein and Price method, 142 Jamming, 204
Goldstein line-search, 40
Gordan theorem, 718 K
Gradient, 22, 708 Kantorovich inequality, 84
projected, 415 KKT matrix (quadratic programming), 440
for simple bounds, 417 KKT point, 391
method (Rosen), 573 KKT necessary conditions, 391
Greatest lower bound, 384 KKT sufficient conditions, 393, 405
804 Subject Index

KNITRO M
ACTIVE (Active-Set Sequential Linear-Quadratic Mathematical
Programming), 557, 559, 560 programming, 18
crossover algorithm, 639 model, 3, 7
INTERIOR-CG (Interior-Point Sequential Linear- Maratos effect, 534
Quadratic), 631, 636 Powell example, 534
INTERIOR-DIRECT (Interior-Point Sequential Matrices, 692
Linear-Quadratic), 631, 632, 634 elementary, 701
determinant, 704
L identity, 692
l1 penalty method, 481 inverse, 694
LACOP collection, 13, 727 lower triangular, 693
comparison: MINOS, KNITRO/ACTIVE, SNOPT norms, 693
and CONOPT, 597 pentadiagonal, 693
solution with COBYLA, 684 symmetric, 692
solution with CONOPT, 594 upper triangular, 693
solution with DFL, 688 trace, 706
solution with filterSD, 652 transpose, 692
solution with IPOPT, 675 tridiagonal, 693
solution with KNITRO/ACTIVE, 564 Matthews and Davies method, 142
solution with KNITRO/INTERIOR, 640 Maximum, 384
solution with MINOS, 516, 517 Mean value theorem, 708
solution with NLPQLP, 557 Memoryless
solution with PDIP, 626 BFGS method, 277
solution with SNOPT, 551 SR1 method with generalized secant
solution with SPENBAR, 502 equation, 302
Lagrange duality, 408 Mental models, 2
Lagrange multipliers, 391, 397 Merit function, 532, 616, 637
interpretation, 400, 401, 405, 406 MFCO constraint qualification, 393
LANCELOT, 518 Minimal surfaces with Enneper boundary conditions
Large-scale optimization, 11 application, 296, 762
L-BFGS, 292 Minimum, 384
L-BFGS-B (simple bounds), 421, 427 MINOS algorithm, 503
L-BFGS-B versus SPG, 427 linear constraints, 504, 508
Least upper bound, 384 nonlinear constraints, 510, 514
Levenberg-Marquardt method, 141 MINPACK-2 collection, 757
LICO constrained qualification, 393, 397 solution with CG-DESCENT, 237
Limited-memory quasi-Newton methods, 290 solution with CONMIN, 251
Line-search solution with DESCON, 244
algorithm, 172 solution with DK+w, DK+aw, DK+iw, 257
method, 380 solution with hybrid conjugate gradient, 224
strategy, 22 solution with L-BFGS, 294
Linear combination, 691 solution with L-BFGS-B, SPG, TNBC, 430
Linear conjugate gradient algorithm, 177 solution with MM-SR1gen and MM-BFGS, 308
Linear constraint qualification, 403 solution with standard conjugate gradient, 217
Linearly constrained Lagrangian methods, 377 solution with TRON, 351
Linear programming, 1, 9, 382 Mixed integer programming, 10
Linguistic models, 2 Modern modeling scheme, 7
Lipschitz continuity, 26, 111, 715 Modified
Liu-Storey Method (conjugate gradient), 213 augmented Lagrangian (MINOS), 503
Local Cholesky factorization, 145, 167
affine model, 109 Lagrangian (MINOS), 503
convergence analysis, 12 log-barrier function, 490
minimizer, 21 secant equation, 285
models (constrained optimization), 373, 375 Wolfe line search, 239
solution, 10 Modified augmented Lagrangian (MINOS), 503, 511
Lower bound, 2, 383, 411 Moore-Penrose pseudo-inverse,701
Subject Index 805

N profile, 13, 14
Necessary optimality conditions, 2, 24 Perpendicular, 691
Nonlinear Perry/Shanno search direction, 248, 252
algebraic systems, 27 Perturbed KKT (quadratic programming), 464
optimization, 9 Polak-Ribière-Polyak conjugate gradient, 171, 210
programming, 9, 383 Positive
NELMED algorithm, 356, 357 definite matrices, 697
Newton semidefinite matrices, 697
decrement, 133 Preconditioned conjugate gradient algorithm, 186
method, 125, 140 Predictor-corrector algorithm for quadratic Programming,
with central-difference, 152 466
with finite differences, 145 Pressure distribution in a journal bearing application, 294,
with line-search, 129, 134 432, 759
step, 133 Primal
system, 110, 126, 145, 613 feasibility, 391
Newton-CG method, 177, 324, 340 problem, 406
NEWUOA algorithm (direct method), 357, 358 Primal-dual
NLPQLP (successive error restoration), 553, 556 active set method (quadratic programming), 456
linear search in, 555 central path, 600, 613
Nocedal condition, 195 interior-point algorithm, 620
Nonlinear elasting mode (SNOPT), 543 system, 602, 603
Nonconvex quadratic programming, 450 Projected conjugate gradient method (quadratic
Nondegenerate stationary point, 414 programming), 448
Nonsmooth augmented system approach, 449
exact penalty functions, 479 normal equations approach, 449
penalty method, 479 Propan combustion in air, 119, 150, 166, 723
NPSOL, 567 Prototype of the interior-point algorithm, 602
Null-space
of a matrix, 694 Q
method (quadratic programming), 442 Quadratic
model of function f, 125, 262, 331
O penalty
Objective function, 1, 21, 383 function, 475
Open questions (conjugate gradient), 260 method, 475, 476
Open programming, 1, 9
ball, 715 active-set method, 451, 452
set, 715 convex, 439
Optimal design with composite materials application, elimination of variables, 471
295, 433, 759 equality constrained, 439
Optimality conditions, 2 indefinite, 450
Optimization based on simulation, 679 inequality constrained, 449
Order notation, 714 interior point method, 463
Orthogonality, 696 nonconvex, 439, 450
Outside contraction (Nelder-Mead), 356 primal-dual active-set method, 456
Ou-Liu line-search, 51 strictly convex, 439
with simple bounds, 468
P Quadratic programming solver SQOPT, 549
Path-following primal-dual methods (quadratic QR decomposition, 700
programming), 464 Quasi-Newton methods assumptions, 271
Parameter, 1
Penalty R
barrier method, 491 Range of a
parameter, 476 function, 693
Penalty and merit function methods, 379 matrix, 693
Penalty-barrier method, 491 Rate of convergence, 34, 710
PENNON, 519 Reduced gradient, 445, 545, 577, 588
Perfect line-search, 30 inequality constraints (quadratic programming), 454
Performance linear equality constraints (quadratic programming),
ratio, 14 445
806 Subject Index

Reduced gradient (cont.) Shrink operation (Nelder-Mead), 356

method (Wolfe), 576 Simple bounds, 1, 411
simple bounds (quadratic programming), 455 optimization, 9, 411
Reduced Hessian, 442, 545 necessary conditions, 412
quadratic programming, 531 sufficient conditions, 414
Reduced Newton system, 614 Simplified Newton method, 163
Reduced system (quadratic programming), 442 Singular value decomposition, 700
conjugate gradient applied, 446 Slater condition, 403
preconditioned conjugate gradient, 447 Small-scale optimization, 11
Refining mechanism, 381 SMUNO collection, 12, 117, 721
Reflection (Nelder-Mead), 356 solution with DEEPS, 368, 369
Regular point solution with NELMED, 367
equality constraints, 394 solution with NEWUOA, 367, 368
general case, 402 Smooth manifold, 394
inequality constraints, 391 SNOPT (sequential quadratic programming), 542, 551
Regularity assumption, 397 Solid fuel ignition, 120
Regularization techniques, 381 Span of a set of vectors, 691
Relative nonlinearity, 116 Sparse quasi-Newton updates, 309
Relaxed steepest descent algorithm, 92 Spectral projected gradient (SPG) algorithm, 417, 418
Residual cubic interpolation, 420
conjugate gradient, 173 quadratic interpolation, 420
truncated Newton, 315, 316 SPENBAR algorithm, 488, 495
Restart criterion (Powell), 202 SQOPT (quadratic programming solver), 549
Restart vectors of Beale, 247 SQPlab, 567
Restricted Broyden class, 269 Spectral decomposition, 701
Robot kinematics problem, 118, 149, 166, 722 SR1 method, 299
Robust optimization, 11 with generalized secant equation, 302
Robustness of algorithms, 12, 14, 78 Stable Cholesky factorization, 143
Stable Newton method, 157
S Stable Newton negative curvature method
Scalar product, 691 Fiacco and McCormick, 159
Scaling of BFGS method, 278 Fletcher and Freeman, 160
Schur complement, 442 Gill and Murray, 158
method, 441 Standard conjugate gradient methods, 201
Secant equation, 262, 264 Stationary point, 387, 487
Second-order Stationary solution of a chemical reactor application, 117,
necessary conditions, 387, 398 149, 165, 722
sufficient conditions, 388, 399, 405 Steady-State combustion application, 295, 434, 761
Second-order conditions, 2 Steepest descent algorithm, 75, 81, 82
Second-order correction (Maratos), 534, 536 Stochastic optimization, 11
Selection phase in line-search, 68 Strict complementarity, 404
Self-correcting property, 273 Strict global
Sensitivity analysis, 2 maximum, 385
Separable unconstrained minimization, 310 minimum, 385
Sequential Strictly convex function, 11
linear programming, 373 Strong convex function, 62, 65, 717
linear-quadratic programming, 374, 541 Strongly active (binding) constraint, 404
quadratic programming, 373 Strongly active simple bound constraints, 404
equality constrained, 521, 523 Sufficient reduction (line search), 42
inequality constrained, 524, 525, 528 Subroutine
simple approach, 526 back, 69
with approximate Hessian, 529 LSbis, 71
with line search, 537 WolfeLS, 72
with successive error restoration, 553 Subspaces, 693
with trust-region, 538, 539 Sufficient
Sequential l1 quadratic programming for the equality descent condition, 22, 63, 189, 270
and inequality constraints, 540 optimality conditions, 2, 25
Sherman-Morrison formula, 695 Superbasic variables, 505
Sherman-Morrison-Woodbury, 695 Supremum, 384
Subject Index 807

T solution with TN, 325

Tangent solution with TNBC, 429
set, 394 Upper bound, 2, 384, 411
space, 394
Taylor’s theorem, 24, 708 V
TNBC (TN with simple bounds), 428 Variables, 1
TNBC versus LBFGS-B and versus SPG, 429, 430 artificial (CONOPT), 588
Transversality, 391 basic (GRG), 579
Trace of BFGS, 707 nonbasic (GRG), 579
TRON, 351 slack (CONOPT), 588
Truncated Newton (TN) method, 315, 316 Vectors, 691
Trust-region length, 692
algorithm, 333 linearly independent, 691
for barrier problems, 628 magnitude, 692
interior-point algorithm, 627, 630 orthogonal, 691
method, 380 Von Neumann lemma, 695
radius, 23, 332
sequential quadratic programming algorithm, 539 W
strategy, 22 Warm start, 525
Weakly active constraints, 404
U Weierstrass theorem, 386
Unconstrained optimization, 9 Wolfe duality, 408
Uniformly convex function, 62 Wolfe line-search (strong), 42, 188, 263
UOP collection, 12 Wolfe line-search (weak), 41, 188, 262
solution with ASDB, 104 Wolfe line-search with
solution with BFGS, 267 safeguarding and cubic interpolation, 68
solution with CG-DESCENT, 235 simple bisection, 52, 68
solution with CONMIN, 250 Word rectification, 3
solution with DESCON, 243 Working set, 392, 450
solution with DK/CGOPT, 255 Working matrix (SNOPT), 545
solution with hybrid conjugate gradient methods, 222 Working surface, 572
solution with L-BFGS-B, 427
solution with MM-BFGS, 307 Z
solution with MM-SR1gen, 307 Zhang-Hager line search, 49
solution with SDB, SDW, 88 Zoutendijk
solution with RSDB, 97 condition, 58, 63, 190
solution with SPG, 420 method (constrained optimization), 57
standard conjugate gradient, 214